如何定位线上CPU占用过高的问题

服务器线上问题开发系列

---

---

前言

---

一、定位问题流程

项目上线，CPU飙高不下，触发报警，如何定位排查问题。有两种办法1、通过堆栈 2、通过火焰图（本文略）
1、top查看进程占用率最高的进程
2、ps -mp pid定位到进程中cpu占用最高的线程
ps -mp 1153 -o THREAD,tid,pid
3、pstack 打印堆栈调用

二、使用实例流程

1.代码

#include 
#include 

void test1()
{
    while(1){
        sleep(0.1);
    }
}
void test2()
{
    while(1){
        sleep(10);
    }
}

void dbdemo()
{
    std::thread(test1).detach();    
    std::thread(test2).detach(); 
}

int main()
{
	dbdemo();
	return 0;
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

2.定位问题实践过程

1、top查出占用率高的进程DBclient的Pid 5896

2、ps -mp 5869 -o THREAD,tid,pid 查出DBclient进程中占用率过高的Tid，5900,转为16进制0x170C

3、pstack 5896，找到对应5900的线程，即可发现调用栈
发现线程4调用关系，test1中sleep函数导致cpu占用过高，
#0 0x00007fa889545626 in sleep () from /lib64/libc.so.6
#1 0x000000000042dd35 in test1() ()

[root@localhost ~]# pstack 5896
Thread 6 (Thread 0x7fa88616e700 (LWP 5897)):
#0  0x00007fa88954585d in nanosleep () from /lib64/libc.so.6
#1  0x00007fa889576134 in usleep () from /lib64/libc.so.6
#2  0x00007fa88ab51422 in bvar::detail::SamplerCollector::run (this=0x16d9b40) at /opt/data/code/vplatform_thirdparty/BRPC-1.0.0-rc02/src/bvar/detail/sampler.cpp:180
#3  0x00007fa88ab51f49 in bvar::detail::SamplerCollector::sampling_thread (arg=<optimized out>) at /opt/data/code/vplatform_thirdparty/BRPC-1.0.0-rc02/src/bvar/detail/sampler.cpp:110
#4  0x00007fa88a075ea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa88957e96d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fa88596d700 (LWP 5898)):
#0  0x00007fa889573c3d in poll () from /lib64/libc.so.6
#1  0x00007fa887a33948 in mongoc_socket_poll () from /home/vagrant/Mycode/DBClient/../libs/gcc_lib/libmongoc-1.0.so.0
#2  0x00007fa887a36627 in _mongoc_stream_socket_poll () from /home/vagrant/Mycode/DBClient/../libs/gcc_lib/libmongoc-1.0.so.0
#3  0x00007fa887a35815 in mongoc_stream_poll () from /home/vagrant/Mycode/DBClient/../libs/gcc_lib/libmongoc-1.0.so.0
#4  0x00007fa887a31cbb in _server_monitor_awaitable_ismaster_recv () from /home/vagrant/Mycode/DBClient/../libs/gcc_lib/libmongoc-1.0.so.0
#5  0x00007fa887a32208 in mongoc_server_monitor_check_server () from /home/vagrant/Mycode/DBClient/../libs/gcc_lib/libmongoc-1.0.so.0
#6  0x00007fa887a32c12 in _server_monitor_thread () from /home/vagrant/Mycode/DBClient/../libs/gcc_lib/libmongoc-1.0.so.0
#7  0x00007fa88a075ea5 in start_thread () from /lib64/libpthread.so.0
#8  0x00007fa88957e96d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fa88516c700 (LWP 5900)):
#0  0x00007fa889545626 in sleep () from /lib64/libc.so.6
#1  0x000000000042dd35 in test1() ()
#2  0x0000000000433e73 in void std::_Bind_simple<void (*())()>::_M_invoke<>(std::_Index_tuple<>) ()
#3  0x0000000000433dcd in std::_Bind_simple<void (*())()>::operator()() ()
#4  0x0000000000433d66 in std::thread::_Impl<std::_Bind_simple<void (*())()> >::_M_run() ()
#5  0x00007fa889e1b330 in ?? () from /lib64/libstdc++.so.6
#6  0x00007fa88a075ea5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007fa88957e96d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fa88496b700 (LWP 5901)):
#0  0x00007fa88954585d in nanosleep () from /lib64/libc.so.6
#1  0x00007fa8895456f4 in sleep () from /lib64/libc.so.6
#2  0x000000000042dd45 in test2() ()
#3  0x0000000000433e73 in void std::_Bind_simple<void (*())()>::_M_invoke<>(std::_Index_tuple<>) ()
#4  0x0000000000433dcd in std::_Bind_simple<void (*())()>::operator()() ()
#5  0x0000000000433d66 in std::thread::_Impl<std::_Bind_simple<void (*())()> >::_M_run() ()
#6  0x00007fa889e1b330 in ?? () from /lib64/libstdc++.so.6
#7  0x00007fa88a075ea5 in start_thread () from /lib64/libpthread.so.0
#8  0x00007fa88957e96d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fa87ffff700 (LWP 5902)):
#0  0x00007fa88a079de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fa887a32b69 in mongoc_server_monitor_wait () from /home/vagrant/Mycode/DBClient/../libs/gcc_lib/libmongoc-1.0.so.0
#2  0x00007fa887a32dc2 in _server_monitor_rtt_thread () from /home/vagrant/Mycode/DBClient/../libs/gcc_lib/libmongoc-1.0.so.0
#3  0x00007fa88a075ea5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007fa88957e96d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fa88bd6ab80 (LWP 5896)):
#0  0x00007fa88954585d in nanosleep () from /lib64/libc.so.6
#1  0x00007fa8895456f4 in sleep () from /lib64/libc.so.6
#2  0x000000000042deba in main ()
[root@localhost ~]# 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

---

总结

如果你对线上进程cpu飙高不知道如何定位，可以通过本文尝试一下，希望对你有所帮助。

老铁，如果觉得不错，请点赞收藏。