43. Debugging & Profiling

43.1. Debugging with GDB

You can use a launcher script generated through the build process. You may run the game server with GDB attached, you may attach a GDB to a running game server process.

For example, you may run like the following. (It assumes that the project is hello and the flavor is game.)

$ ./hello.game-local --gdb

If there is a running game server process, it will automatically attaches to the process. If not, it will run a new game server with attached GDB. For the latter case, you can start your debugging session by typing run command.

 Please type `run` to start debugging.
 If you want to check the arguments, please type `show args`.


 GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
 Copyright (C) 2016 Free Software Foundation, Inc.
 ... (skipped)
 Reading symbols from /usr/bin/funapi_runner...(no debugging symbols found)...done.
 (gdb) run

Tip

Due to the security enforcment on linux operating system, you may not allowed to attach a debugger to the process ran by the same user.

You can use the sudo command as a workaround. Or you may change the system configuration by running the following command. It is worth noting that you must not run the following command on a production system.

$ echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope

After rebooting the machine, the configuration will be reset to the default state.

If there are multiple game-server instances of the same flavor, you can choose the process by specifying the process id on the command line.

# Please substitute the {PID} with the process id.
$ ./hello.game-local --gdb --pid {PID}

43.2. Debugging dump files

iFun Engine uses Google Breakpad to generate minidump files (.dmp) in the dump directory when the game server crashes. Minidump files are much smaller than core dump files. You can use these files to easily check the thread stacks that crashed.

43.2.1. Minidump file location

These files are saved in the logs/dumps/ directory in the build directory during development. If the game server is run using a daemon on a live server, dump files are saved in the /var/crash/funapi/{{project-name}}/ directory.

43.2.2. Viewing minidump data

You can see stacks in the minidump files by using the following commands:

Note

The following explanation assumes use of a daemon on a live server. If you are in the development phase, you can change /var/crash/funapi/{{project-name}}/ to logs/dumps/.

$ funapi_stackwalk /var/crash/funapi/{{project-name}}/{{dump-uuid}}.dmp \
    /usr/share/{{project-name}}/symbols

The following is an example of operation.

 $ funapi_stackwalk /var/crash/funapi/example/08835639-ff4e-e778-6c14c453-5147e752.dmp /usr/share/example/symbols
 OS Linux 0.0.0 Linux 3.10.0-123.20.1.el7.x86_64 #1 SMP Thu Jan 29 18:05:33 UTC 2015 x86_64
 CPU amd64 family 6 model 42 stepping 7 8
 Crash SIGSEGV 0x0 31

 0 libexample.so: /build/example-source/src/event_handlers.cc:155 (+0x4) - example::OnTick
 1 libexample.so: /usr/include/boost/function/function_template.hpp:112 (+0x14) - ...
 2 funapi_runner: ...
 ...
 7 libboost_thread-mt.so.1.53.0 (+0xd24a)
 8 libpthread-2.17.so (+0x7df5)
 9 libc-2.17.so (+0xf61ad)

43.2.3. Debugging minidumps with GDB

Note

The following explanation assumes use of a daemon on a live server. If you are in the development phase, you can change /var/crash/funapi/{{project-name}}/ to logs/dumps/.

Use the following commands to debug minidump files with GDB.

$ funapi_core_analyze /usr/lib/lib{{project-name}}.so \
    /var/crash/funapi/{{project-name}}/{{dump-uuid}}.dmp
 $ funapi_core_analyze /usr/lib/libhello.so /var/crash/funapi/hello/3acd4cbf-5654-695b-0dd043c9-3c70e30d.dmp
 Finding game server .so file
 Converting dumps/3acd4cbf-5654-695b-0dd043c9-3c70e30d.dmp into coredump
 Launching gdb to inspect generated coredump (hello.23945.core).
 Please press 'y' to load required symbols

 ...

 Failed to read a valid object file image from memory.
 Core was generated by `/usr/bin/funapi_runner --main_program_name=hello_server.game --app_flavor=game --'.
 #0  0x00007fc18d76c2de in ?? ()
 add symbol table from file "src/libhello.so" at
   .text_addr = 0x7fc18d0ceaf0
 Reading symbols from src/libhello.so...done.
 (gdb) bt
 #0  0x00007fc18d76c2de in hello::TestFilters () at ../src/filter.cc:367
 #1  0x00007fc18d43094f in (anonymous namespace)::HelloServer::Install (arguments=...) at ../src/hello_server.cc:113
 #2  0x00007fc18d430f18 in HelloServer_install (arguments=...) at ../src/hello_server.cc:182
 #3  0x00000000007328d3 in ?? ()
 #4  0x0000000000601b5c in ?? ()
 #5  0x00007fc1962d2ec5 in ?? ()
 #6  0x0000000000000000 in ?? ()
 (gdb) info locals
 x = 0x0
 (gdb)

43.2.4. Generating core dumps instead of minidumps

In order to minimize disk usage, you can only see stack values and directly referenced values in minidumps. If you need more data for debugging, you can save core dumps instead of minidumps and debug with those.

Warning

Since the entire memory footprint of the core dump process is saved on the disk, disk usage may increase when this process uses a lot of memory.

Important

You need to check OS settings first to see whether core dumps are generated.

# 아래와 같이 0이 나오는 경우 코어덤프가 디스크에 남지 않습니다.
$ ulimit -c
0

# 남길 수 있는 코어덤프 용량 제한을 해제
$ ulimit -c unlimited

Run the launcher with the following settings.

$ ./hello-local --enable_breakpad=false

If OS settings are not changed, core files are generated in the current directory after crashes. You can debug applicable files as follows:

$ gdb /usr/bin/funapi_runner core
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1
...
Core was generated by `/usr/bin/funapi_runner --main_program_name=hello_server.default --app_flavor=de'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f28715bad44 in (anonymous namespace)::HelloServer::Start () at ../src/hello_server.cc:71
71      *x = 0;
[Current thread is 1 (Thread 0x7f287e5ec800 (LWP 21649))]
(gdb) bt
#0  0x00007f28715bad44 in (anonymous namespace)::HelloServer::Start () at ../src/hello_server.cc:71
#1  0x00007f28715bad8a in HelloServer_start () at ../src/hello_server.cc:83
#2  0x0000000000845c03 in fun::StartComponents() ()
#3  0x000000000065eabe in main ()
(gdb)

43.3. CPU profiling

Note

This section deals with general CPU profiling. Please see Event profiling and debugging to learn more about profiling specific to iFun Engine.

Note

C# version profiling will be supported later.

You can find CPU bottlenecks using the following process even without rebuilding the game server.

43.3.1. Installing required programs

43.3.1.1. perf utility

Ubuntu

$ sudo apt-get install linux-tools-generic

Important

You need the linux-tools package when using non-generic kernels. i.e. linux-tools-<kernel name>.

CentOS

$ sudo yum install perf

43.3.1.2. FlameGraph

$ sudo wget -O /usr/local/bin/flamegraph.pl \
    https://raw.githubusercontent.com/brendangregg/FlameGraph/master/flamegraph.pl

$ sudo wget -O /usr/local/bin/stackcollapse-perf.pl \
    https://raw.githubusercontent.com/brendangregg/FlameGraph/master/stackcollapse-perf.pl

$ sudo chmod a+x /usr/local/bin/flamegraph.pl /usr/local/bin/stackcollapse-perf.pl

43.3.2. Outputting profiling results

Step 1. Run the game server. (Skip if already running)

$ ./hello-local
  1. Run the profiling command as follows:

    $ sudo /usr/bin/funapi_profile {{ProjectName}}
    

    Tip

    만약 Flavors: Identifying servers according to their role 를 쓰는 경우 아래처럼 flavor 를 지정합니다.

    $ sudo funapi_profile {{ProjectName}} {{FlavorName}}
    

    Example

    $ sudo /usr/bin/funapi_profile hello
    Profiling server for activity (for 60 seconds)
    [sudo] password for user:
    [ perf record: Woken up 7 times to write data ]
    [ perf record: Captured and wrote 1.626 MB perf.23428.data (~71057 samples) ]
    Generating flamegraph
    [kernel.kallsyms] with build id d7fccc433a6b698638325b755df2772c15636d04 not found, continuing without symbols
    Generated: perf.hello_server.default.23428.svg
    
  2. Open the SVG file on the last output line in a browser. This is perf.hello_server.default.23428.svg in the example above.

In this example, when the hello::OnTick() function is the bottleneck, it is displayed horizontally in the flamegraph. You can find and optimize this function.

Tip

Click this function to expand the SVG file to show more details.

Tip

Hover over a function in the SVG file to show the CPU usage of that function and its sub-functions at the lower left.

Tip

Use the Search function in the upper right of the SVG file to easily see which parts of the project code use the CPU the most.

Assuming the project is hello, enter ^hello:: into Search to show only the project functions in purple.