如何快速定位到报错日志中的关键信息，一招学会，赶快GET吧

一般的服务器日志一个可能大的有几十上百m，小的也得几百k，里面内容是比较多的，如拿到日志没思路去看的话，下面一些办法可以让你快速定位到日志中的异常错误信息

下面的这块日志可以作为练习参考

g_2.11-3.9.0.jar:/export/server/kafka/bin/../libs/scala-reflect-2.11.12.jar:/export/server/kafka/bin/../libs/slf4j-api-1.7.25.jar:/export/server/kafka/bin/../libs/slf4j-log4j12-1.7.25.jar:/export/server/kafka/bin/../libs/snappy-java-1.1.7.2.jar:/export/server/kafka/bin/../libs/validation-api-1.1.0.Final.jar:/export/server/kafka/bin/../libs/zkclient-0.10.jar:/export/server/kafka/bin/../libs/zookeeper-3.4.13.jar:/export/server/kafka/bin/../libs/zstd-jni-1.3.5-4.jar (org.apache.zookeeper.ZooKeeper)
[2022-11-15 20:46:32,836] INFO Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
[2022-11-15 20:46:32,836] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2022-11-15 20:46:32,836] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2022-11-15 20:46:32,836] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2022-11-15 20:46:32,836] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2022-11-15 20:46:32,836] INFO Client environment:os.version=3.10.0-1160.76.1.el7.x86_64 (org.apache.zookeeper.ZooKeeper)
[2022-11-15 20:46:32,837] INFO Client environment:user.name=root (org.apache.zookeeper.ZooKeeper)
[2022-11-15 20:46:32,837] INFO Client environment:user.home=/root (org.apache.zookeeper.ZooKeeper)
[2022-11-15 20:46:32,837] INFO Client environment:user.dir=/root (org.apache.zookeeper.ZooKeeper)
[2022-11-15 20:46:32,839] INFO Initiating client connection, connectString=master:2181,slave1:2181,slave2:2181 sessionTimeout=6000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@77fbd92c (org.apache.zookeeper.ZooKeeper)
[2022-11-15 20:46:32,946] INFO Opening socket connection to server slave2/192.168.241.23:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2022-11-15 20:46:32,949] INFO [ZooKeeperClient] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
[2022-11-15 20:46:32,970] INFO Socket error occurred: slave2/192.168.241.23:2181: 拒绝连接 (org.apache.zookeeper.ClientCnxn)
[2022-11-15 20:46:33,073] INFO Opening socket connection to server slave1/192.168.241.22:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2022-11-15 20:46:33,074] INFO Socket error occurred: slave1/192.168.241.22:2181: 拒绝连接 (org.apache.zookeeper.ClientCnxn)
[2022-11-15 20:46:33,174] INFO Opening socket connection to server master/192.168.241.21:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2022-11-15 20:46:33,175] INFO Socket connection established to master/192.168.241.21:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2022-11-15 20:46:33,182] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2022-11-15 20:46:34,747] INFO Opening socket connection to server slave2/192.168.241.23:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2022-11-15 20:46:34,748] INFO Socket error occurred: slave2/192.168.241.23:2181: 拒绝连接 (org.apache.zookeeper.ClientCnxn)
[2022-11-15 20:46:35,820] INFO Opening socket connection to server slave1/192.168.241.22:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2022-11-15 20:46:35,821] INFO Socket error occurred: slave1/192.168.241.22:2181: 拒绝连接 (org.apache.zookeeper.ClientCnxn)
[2022-11-15 20:46:36,209] INFO Opening socket connection to server master/192.168.241.21:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2022-11-15 20:46:36,209] INFO Socket connection established to master/192.168.241.21:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2022-11-15 20:46:36,210] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2022-11-15 20:46:37,505] INFO Opening socket connection to server slave2/192.168.241.23:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2022-11-15 20:46:37,510] INFO Socket error occurred: slave2/192.168.241.23:2181: 拒绝连接 (org.apache.zookeeper.ClientCnxn)
[2022-11-15 20:46:38,515] INFO Opening socket connection to server slave1/192.168.241.22:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2022-11-15 20:46:38,515] INFO Socket error occurred: slave1/192.168.241.22:2181: 拒绝连接 (org.apache.zookeeper.ClientCnxn)
[2022-11-15 20:46:38,951] INFO [ZooKeeperClient] Closing. (kafka.zookeeper.ZooKeeperClient)
[2022-11-15 20:46:39,623] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
[2022-11-15 20:46:39,624] INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn)
[2022-11-15 20:46:39,627] INFO [ZooKeeperClient] Closed. (kafka.zookeeper.ZooKeeperClient)
[2022-11-15 20:46:39,629] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
        at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply$mcV$sp(ZooKeeperClient.scala:268)
        at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply(ZooKeeperClient.scala:264)
        at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply(ZooKeeperClient.scala:264)
        at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
        at kafka.zookeeper.ZooKeeperClient.kafka$zookeeper$ZooKeeperClient$$waitUntilConnected(ZooKeeperClient.scala:264)
        at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:97)
        at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1693)
        at kafka.server.KafkaServer.kafka$server$KafkaServer$$createZkClient$1(KafkaServer.scala:348)
        at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:372)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:202)
        at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
        at kafka.Kafka$.main(Kafka.scala:75)
        at kafka.Kafka.main(Kafka.scala)
[2022-11-15 20:46:39,633] INFO shutting down (kafka.server.KafkaServer)
[2022-11-15 20:46:39,636] INFO shut down completed (kafka.server.KafkaServer)
[2022-11-15 20:46:39,638] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)
[2022-11-15 20:46:39,647] INFO shutting down (kafka.server.KafkaServer)
(END)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

步骤1:定位到错误信息再那个日志中(grep)

当天日志可能比较多，被切分成很多个小日志
如下图

如果出现这种情况，我们可以使用这种办法来定位错误信息再那个日志中
比如我们有相关的关键字可以直接用，用关键字匹配日志
grep -ri "关键字" 所以查询的日志(可以使用*进行匹配)

[root@master logs]# grep -ri "ERROR Fatal" server.log.2022-11-14-21*
server.log.2022-11-14-21:[2022-11-15 20:46:39,629] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
server.log.2022-11-14-21-1:[2022-11-15 20:46:39,629] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
1
2
3

从上面我们可以直接定位到错误信息在server.log.2022-11-14-21和server.log.2022-11-14-21-1都有

步骤2:查看日志上下文信息

查看上下多少x行信息(cat指令)

查看指定文件中关键词上下5行的信息cat xxx |grep -C 5 "关键信息"

[root@master logs]# cat server.log.2022-11-14-21 |grep -C 5 "ERROR Fatal"
[2022-11-15 20:46:38,515] INFO Socket error occurred: slave1/192.168.241.22:2181: 拒绝连接 (org.apache.zookeeper.ClientCnxn)
[2022-11-15 20:46:38,951] INFO [ZooKeeperClient] Closing. (kafka.zookeeper.ZooKeeperClient)
[2022-11-15 20:46:39,623] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
[2022-11-15 20:46:39,624] INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn)
[2022-11-15 20:46:39,627] INFO [ZooKeeperClient] Closed. (kafka.zookeeper.ZooKeeperClient)
[2022-11-15 20:46:39,629] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
        at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply$mcV$sp(ZooKeeperClient.scala:268)
        at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply(ZooKeeperClient.scala:264)
        at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply(ZooKeeperClient.scala:264)
        at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
[root@master logs]#


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

如图示效果

查看关键字下多少x行信息(cat指令)

查看指定文件中关键词往下10行的信息cat xxx |grep -A 10 "关键信息"

[root@master logs]# cat server.log.2022-11-14-21 |grep -A 10  "ERROR Fatal"
[2022-11-15 20:46:39,629] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
        at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply$mcV$sp(ZooKeeperClient.scala:268)
        at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply(ZooKeeperClient.scala:264)
        at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply(ZooKeeperClient.scala:264)
        at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
        at kafka.zookeeper.ZooKeeperClient.kafka$zookeeper$ZooKeeperClient$$waitUntilConnected(ZooKeeperClient.scala:264)
        at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:97)
        at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1693)
        at kafka.server.KafkaServer.kafka$server$KafkaServer$$createZkClient$1(KafkaServer.scala:348)
        at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:372)
[root@master logs]#

1
2
3
4
5
6
7
8
9
10
11
12
13
14

效果如图所示

直接进入日志用关键字定位(less指令)

less xxx.log

[root@master logs]# less server.log.2022-11-14-21
1

进入日志后敲一下/ + 关键字，之后按一下回车
具体看图操作

回车之后就定位到关键词的地方了，如果还有其他相同的关键字日志继续/ + 方向键↑就能快速把刚才的关键字打出来了，完了在敲回车

直接进入日志用关键字定位(vim指令-但是不太推荐)

同less指令的后续操作
直接vim进去或者vi进去，不太建议使用是有时候shell连接在你正在vi或者vim的时候断掉之后会产生一部分快照文件，如果下一次连到服务器的时候需要把快照文件删除掉才能继续编辑

特殊情况(日志被打成tar包了。。。)

如果日志比较大了，而且是前n天的，比如说是10天前，有可能会被压缩tar包(具体看你们公司相关配置)，这个时候使用可以解压文件之后再查看，但是如果权限不够，不允许解压，就需要另求他法了
查看tar包中的日志内容:zcat xxx.tar.gz | egrep -a "关键字"

[root@master logs]# zcat test.tar.gz |egrep -a "ERROR Fatal"
[2022-11-15 20:46:39,629] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)

1
2
3
4