Automatic core dump when Golang program crashes

/ Laixintao | SG

Original link: https://www.kawabangga.com/posts/5175

Some time ago, I encountered a problem. The program crashed inexplicably, and the stack didn’t see any clues. Today I changed a parameter to make the golang program core dump when it crashes.

In fact, the core is to add an environment variable, GOTRACEBACK=1. But there are some other system-related problems, which are briefly recorded in this article.

After Golang 1.6, the optional values ​​of variables in this environment have changed, and the new values ​​are as follows:

  • GOTRACEBACK=none will suppress all tracebacks, you only get the panic message.
  • GOTRACEBACK=single is the new default behavior that prints only the goroutine believed to have caused the panic.
  • GOTRACEBACK=all causes stack traces for all goroutines to be shown, but stack frames related to the runtime are suppressed.
  • GOTRACEBACK=system is the same as the previous value, but frames related to the runtime are also shown, this will reveal goroutines started by the runtime itself.
  • GOTRACEBACK=crash is unchanged from Go 1.5.

Some points to note:

First of all, in addition to the GOTRACEBACK parameter, there are other useful environment variables that can control the runtime of golang. This article summarizes it very well.

Then this parameter is invalid on macOS, so don’t waste effort on MAC.

Linux is also limited by ulimit. You can use ulimit -c to view the size limit for Core dump. If it is 0 , the dump will not come out, and it is not recommended to set it to ulimited . I changed it to 50G. If the program is started by systemd, you can set LimitCORE= parameter in the service unit file, the effect is the same.

Where is the generated core dump stored?

It can be viewed here:

 cat /proc/sys/kernel/core_pattern

This defines the naming method of the core dump file.

But in ubuntu, you will see this output:

 |/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E

It means that the pipe is directed to apport. apport is the core dump management service chosen by the ubuntu distribution.

By default, user programs will not have a core dump. Then we have two solutions:

  1. Close apport, use the system’s core dump to write directly to the disk
  2. Configure apport to also write the user’s core dump file

The first method is relatively simple, just modify the /proc/sys/kernel/core_pattern file above:

 echo "kernel.core_pattern=/tmp/%e.%t.%p.%s.core" > /etc/sysctl.d/60-core-pattern.conf 
  
sysctl --system

Then you can disable apport:

 systemctl stop apport 
  
systemctl disable apport

The second way, first make sure apport is running. It can be viewed through systemctl status apport . You can also look at the apport log:

 tail -f /var/log/apport.log

Trigger a core dump and you will see:

 executable does not belong to a package, ignoring

It means that the core dump is not from the software packaged by ubuntu, ignore it.

The configuration method is to modify the ~/.config/apport/settings (if not, manually create) file and write:

 [main] 
  
unpackaged=true

Trigger the core dump again, and this time there will be information written in the log:

 ERROR: apport (pid 3709) Thu Jun 22 07:00:58 2023: called for pid 3704, signal 6, core limit 0, dump mode 1 
  
ERROR: apport (pid 3709) Thu Jun 22 07:00:58 2023: executable: /tmp/go-build2218735647/b001/exe/crash (command line "/tmp/go-build2218735647/b001/exe/crash") 
  
ERROR: apport (pid 3709) Thu Jun 22 07:00:59 2023: wrote report /var/crash/_tmp_go-build2218735647_b001_exe_crash.0.crash

Also note that this file is not a core dump file, but a debug file packaged by apport, which can be unpacked using apport-unpack:

 apport-unpack _tmp_go-build2218735647_b001_exe_crash.0.crash crash_dump.core

The unpacked CoreDump can be analyzed with gdb. Other files record some system-related information. (It feels like Ubuntu is used to let users report bugs)

Finally, if the core dump is not generated under the workdir of the process, you can check whether it is in /var/lib/systemd/coredump/. It is said on the Internet that the system using systemd will be stored here, but I have not encountered it.

This article is transferred from: https://www.kawabangga.com/posts/5175
This site is only for collection, and the copyright belongs to the original author.