内核systemtap工具安装

  • 启用yum debug 仓库。 Alibaba Cloud Linux3 默认debug 仓库是禁用状态,可以通过-enablerepo或编辑文件的方式启用下。

  • 执行yum --enablerepo=alinux3-plus-debug install -y systemtap kernel-debuginfo-$(uname -r) kernel-devel-$(uname -r) 安装systemtap

  • 执行stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}' 检查systemtap是否安装成功。

    1
    2
    3
    4
    5
    6
    7
    8
    
    [root@waf]# stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}'
    Pass 1: parsed user script and 485 library scripts using 340260virt/95012res/17136shr/79768data kb, in 150usr/30sys/190real ms.
    Pass 2: analyzed script: 1 probe, 1 function, 5 embeds, 0 globals using 537540virt/293776res/18764shr/277048data kb, in 2140usr/210sys/2360real ms.
    Pass 3: using cached /root/.systemtap/cache/7b/stap_7b2bb1f97f025d77d0ae5c702821ef63_2589.c
    Pass 4: using cached /root/.systemtap/cache/7b/stap_7b2bb1f97f025d77d0ae5c702821ef63_2589.ko
    Pass 5: starting run.
    read performed
    Pass 5: run completed in 10usr/20sys/341real ms.
    
  • debuginfo-install openresty 安装openresty-debuginfo

准备收集lua运行信息及生成火焰图工具包

  • stapxx,收集lua运行信息。

    1
    
    git clone https://github.com/philipp-classen/stapxx
    

    官方的https://github.com/openresty/stapxx不支持gcc 64位模式

  • openresty官方提供的systemtap工具包。

    1
    
    git clone https://github.com/openresty/openresty-systemtap-toolkit.git
    
  • 火焰图生成工具。

    1
    
    git clone https://github.com/brendangregg/FlameGraph.git
    

收集lua运行信息并生成火焰图

  1. 采样:

    1
    2
    3
    
    cd stapxx
    export PATH=$PWD:$PATH
    ./samples/lj-lua-stacks.sxx --arg time=10 --skip-badvars -x {WORK_PID} > tmp.bt
    

    其中-x指定nginx worker的PID。

  2. 使用openresty-system-toolkit工具,调整一下tmp.bt的采样结果:

    1
    2
    3
    
    cd openresty-systemtap-toolkit/
    ./fix-lua-bt ../stapxx/tmp.bt > flame.bt
    mv flame.bt /codes/FlameGraph/
    
  3. 切换到火焰图生成工具目录,开始生成火焰图

    1
    2
    
    cd  FlameGraph
    ./stackcollapse-stap.pl flame.bt |./flamegraph.pl > flame.svg
    

脚本

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
PID=$1
#rm -rf tmp.bt
#rm -rf flame.bt
#rm -rf flame.svg

cd stapxx
export PATH=$PWD:$PATH
cd ..
./stapxx/samples/lj-lua-stacks.sxx --arg time=30 --skip-badvars -x $PID > tmp.bt
./openresty-systemtap-toolkit/fix-lua-bt tmp.bt > flame.bt
./FlameGraph/stackcollapse-stap.pl flame.bt |./FlameGraph/flamegraph.pl > flame.svg

flame

参考

https://juejin.cn/post/7157751082960027679