43. Dump file debugging & CPU profiling

43.1. Debugging dump files

iFun Engine uses Google Breakpad to generate minidump files (.dmp) in the dump directory when the game server crashes. Minidump files are much smaller than core dump files. You can use these files to easily check the thread stacks that crashed.

43.1.1. Minidump file location

These files are saved in the logs/dumps/ directory in the build directory during development. If the game server is run using a daemon on a live server, dump files are saved in the /var/crash/funapi/{{project-name}}/ directory.

43.1.2. Viewing minidump data

You can see stacks in the minidump files by using the following commands:


The following explanation assumes use of a daemon on a live server. If you are in the development phase, you can change /var/crash/funapi/{{project-name}}/ to logs/dumps/.

$ funapi_stackwalk /var/crash/funapi/{{project-name}}/{{dump-uuid}}.dmp \

The following is an example of operation.

 $ funapi_stackwalk /var/crash/funapi/example/08835639-ff4e-e778-6c14c453-5147e752.dmp /usr/share/example/symbols
 OS Linux 0.0.0 Linux 3.10.0-123.20.1.el7.x86_64 #1 SMP Thu Jan 29 18:05:33 UTC 2015 x86_64
 CPU amd64 family 6 model 42 stepping 7 8
 Crash SIGSEGV 0x0 31

 0 libexample.so: /build/example-source/src/event_handlers.cc:155 (+0x4) - example::OnTick
 1 libexample.so: /usr/include/boost/function/function_template.hpp:112 (+0x14) - ...
 2 funapi_runner: ...
 7 libboost_thread-mt.so.1.53.0 (+0xd24a)
 8 libpthread-2.17.so (+0x7df5)
 9 libc-2.17.so (+0xf61ad)

43.1.3. Debugging minidumps with GDB


The following explanation assumes use of a daemon on a live server. If you are in the development phase, you can change /var/crash/funapi/{{project-name}}/ to logs/dumps/.

Use the following commands to debug minidump files with GDB.

$ funapi_core_analyze /usr/lib/lib{{project-name}}.so \
 $ funapi_core_analyze /usr/lib/libhello.so /var/crash/funapi/hello/3acd4cbf-5654-695b-0dd043c9-3c70e30d.dmp
 Finding game server .so file
 Converting dumps/3acd4cbf-5654-695b-0dd043c9-3c70e30d.dmp into coredump
 Launching gdb to inspect generated coredump (hello.23945.core).
 Please press 'y' to load required symbols


 Failed to read a valid object file image from memory.
 Core was generated by `/usr/bin/funapi_runner --main_program_name=hello_server.game --app_flavor=game --'.
 #0  0x00007fc18d76c2de in ?? ()
 add symbol table from file "src/libhello.so" at
   .text_addr = 0x7fc18d0ceaf0
 Reading symbols from src/libhello.so...done.
 (gdb) bt
 #0  0x00007fc18d76c2de in hello::TestFilters () at ../src/filter.cc:367
 #1  0x00007fc18d43094f in (anonymous namespace)::HelloServer::Install (arguments=...) at ../src/hello_server.cc:113
 #2  0x00007fc18d430f18 in HelloServer_install (arguments=...) at ../src/hello_server.cc:182
 #3  0x00000000007328d3 in ?? ()
 #4  0x0000000000601b5c in ?? ()
 #5  0x00007fc1962d2ec5 in ?? ()
 #6  0x0000000000000000 in ?? ()
 (gdb) info locals
 x = 0x0

43.1.4. Generating core dumps instead of minidumps

In order to minimize disk usage, you can only see stack values and directly referenced values in minidumps. If you need more data for debugging, you can save core dumps instead of minidumps and debug with those.


Since the entire memory footprint of the core dump process is saved on the disk, disk usage may increase when this process uses a lot of memory.


You need to check OS settings first to see whether core dumps are generated.

# 아래와 같이 0이 나오는 경우 코어덤프가 디스크에 남지 않습니다.
$ ulimit -c

# 남길 수 있는 코어덤프 용량 제한을 해제
$ ulimit -c unlimited

Run the launcher with the following settings.

$ ./hello-local --enable_breakpad=false

If OS settings are not changed, core files are generated in the current directory after crashes. You can debug applicable files as follows:

$ gdb /usr/bin/funapi_runner core
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1
Core was generated by `/usr/bin/funapi_runner --main_program_name=hello_server.default --app_flavor=de'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f28715bad44 in (anonymous namespace)::HelloServer::Start () at ../src/hello_server.cc:71
71      *x = 0;
[Current thread is 1 (Thread 0x7f287e5ec800 (LWP 21649))]
(gdb) bt
#0  0x00007f28715bad44 in (anonymous namespace)::HelloServer::Start () at ../src/hello_server.cc:71
#1  0x00007f28715bad8a in HelloServer_start () at ../src/hello_server.cc:83
#2  0x0000000000845c03 in fun::StartComponents() ()
#3  0x000000000065eabe in main ()

43.2. CPU profiling


This section deals with general CPU profiling. Please see Event profiling and debugging to learn more about profiling specific to iFun Engine.


C# version profiling will be supported later.

You can find CPU bottlenecks using the following process even without rebuilding the game server.

43.2.1. Installing required programs perf utility


$ sudo apt-get install linux-tools-generic


You need the linux-tools package when using non-generic kernels. i.e. linux-tools-<kernel name>.


$ sudo yum install perf FlameGraph

$ sudo wget -O /usr/local/bin/flamegraph.pl \

$ sudo wget -O /usr/local/bin/stackcollapse-perf.pl \

$ sudo chmod a+x /usr/local/bin/flamegraph.pl /usr/local/bin/stackcollapse-perf.pl

43.2.2. Outputting profiling results

Step 1. Run the game server. (Skip if already running)

$ ./hello-local
  1. Run the profiling command as follows:

    $ sudo /usr/bin/funapi_profile {{ProjectName}}


    만약 Flavors: Identifying servers according to their role 를 쓰는 경우 아래처럼 flavor 를 지정합니다.

    $ sudo funapi_profile {{ProjectName}} {{FlavorName}}


    $ sudo /usr/bin/funapi_profile hello
    Profiling server for activity (for 60 seconds)
    [sudo] password for user:
    [ perf record: Woken up 7 times to write data ]
    [ perf record: Captured and wrote 1.626 MB perf.23428.data (~71057 samples) ]
    Generating flamegraph
    [kernel.kallsyms] with build id d7fccc433a6b698638325b755df2772c15636d04 not found, continuing without symbols
    Generated: perf.hello_server.default.23428.svg
  2. Open the SVG file on the last output line in a browser. This is perf.hello_server.default.23428.svg in the example above.

In this example, when the hello::OnTick() function is the bottleneck, it is displayed horizontally in the flamegraph. You can find and optimize this function.


Click this function to expand the SVG file to show more details.


Hover over a function in the SVG file to show the CPU usage of that function and its sub-functions at the lower left.


Use the Search function in the upper right of the SVG file to easily see which parts of the project code use the CPU the most.

Assuming the project is hello, enter ^hello:: into Search to show only the project functions in purple.