分析kdump(vmcore)

1.环境

[root@localhost ~]# cat /etc/redhat-release 
CentOS Linux release 7.9.2009 (Core)
[root@localhost ~]# uname -a
Linux localhost.localdomain 3.10.0-1160.62.1.el7.x86_64 #1 SMP Tue Apr 5 16:57:59 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

2.分下kdump文件需要安装crash与kernel-debuginfo

yum -y install crash
yum --enablerepo=base-debuginfo install -y kernel-debuginfo-$(uname -r)

3.启动

crash /opt/127.0.0.1-2022-07-24-13_22_33/vmcore /usr/lib/debug/lib/modules/3.10.0-1160.62.1.el7.x86_64/vmlinux

启动失败，由于调试机器的内核版本与产生coredump的机器的内核版本不一致，导致crash启动失败，根据vmcore-dmesg.txt文件可知内核版本2.6.32-431.el6.x86_64，根据https://en.wikipedia.org/wiki/CentOS找到centos6.5对应的内核版本就是2.6.32-431.el6.x86_64

4.安装虚拟机

通过虚拟机装centos6.5 下载镜像https://vault.centos.org/6.5/isos/x86_64/CentOS-6.5-x86_64-bin-DVD1.iso 安装完虚拟机之后，更新yum报错

[root@localhost ]# yum update
 
...repomd.xml: [Errno 14] problem making ssl connection
Essai d'un autre miroir.
Erreur : Cannot retrieve repository metadata (repomd.xml) for repository: base. Please verify its path and try again"

以上错误 

Fixing yum repos on CentOS 6 now it’s EOL – Mark Gilbert's Tech Blog

解决方案更新/etc/yum.repos.d/CentOS-Base.repo内容

[C6.10-base]
name=CentOS-6.10 - Base
baseurl=http://linuxsoft.cern.ch/centos-vault/6.10/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
enabled=1
metadata_expire=never
 
[C6.10-updates]
name=CentOS-6.10 - Updates
baseurl=http://linuxsoft.cern.ch/centos-vault/6.10/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
enabled=1
metadata_expire=never
 
[C6.10-extras]
name=CentOS-6.10 - Extras
baseurl=http://linuxsoft.cern.ch/centos-vault/6.10/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
enabled=1
metadata_expire=never
 
[C6.10-contrib]
name=CentOS-6.10 - Contrib
baseurl=http://linuxsoft.cern.ch/centos-vault/6.10/contrib/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
enabled=0
metadata_expire=never
 
[C6.10-centosplus]
name=CentOS-6.10 - CentOSPlus
baseurl=http://linuxsoft.cern.ch/centos-vault/6.10/centosplus/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
enabled=0
metadata_expire=never

5.centos6.5安装kernel-debuginfo

下载kernel-debuginfo

wget http://debuginfo.centos.org/6/x86_64/kernel-debuginfo-2.6.32-431.el6.x86_64.rpm
wget http://debuginfo.centos.org/6/x86_64/kernel-debuginfo-common-x86_64-2.6.32-431.el6.x86_64.rpm

安装kernel-debuginfo

#先安装
rpm -ivh kernel-debuginfo-common-x86_64-2.6.32-431.el6.x86_64.rpm
 
#再安装
rpm -ivh kernel-debuginfo-2.6.32-431.el6.x86_64.rpm

6.启动crash

[root@localhost debuginfo]# crash ./vmcore /usr/lib/debug/lib/modules/2.6.32-431.el6.x86_64/vmlinux 
 
crash 7.1.0-8.el6
Copyright (C) 2002-2014  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
 
please wait... (gathering task table data)      
crash: invalid task address: ffff880631fdb500
 
crash: invalid task address: ffff88022fcaaae0
 
crash: invalid task address: ffff8803d2f0d540
 
crash: invalid task address: ffff8804ddefe040
      KERNEL: /usr/lib/debug/lib/modules/2.6.32-431.el6.x86_64/vmlinux
    DUMPFILE: ./vmcore  [PARTIAL DUMP]
        CPUS: 8
        DATE: Sat Jul 23 22:22:31 2022
      UPTIME: 442 days, 20:23:48
LOAD AVERAGE: 1.12, 1.16, 1.11
       TASKS: 1241
    NODENAME: dzswj-hlw-fpzx-ap08
     RELEASE: 2.6.32-431.el6.x86_64
     VERSION: #1 SMP Fri Nov 22 03:15:09 UTC 2013
     MACHINE: x86_64  (2100 Mhz)
      MEMORY: 28 GB
       PANIC: "general protection fault: 0000 [#1] SMP "
         PID: 9415
     COMMAND: "tomcatName.sh"
        TASK: ffff8802d075caa0  [THREAD_INFO: ffff880514d0e000]
         CPU: 6
       STATE: TASK_RUNNING (PANIC)
 
crash> 

备注

调试机器的内核版本要与产生coredump的机器的内核版本一致。 

参考

Analyzing the Core Dump - CentOS - Linux

CentOS 7 - How to install kernel-debuginfo · GitHub

How to use kdump for Linux Kernel Crash Analysis

Bug hunting — The Linux Kernel documentation