How to collect dumps for Firebird on Linux and generate stack traces

Introduction

In case of serious problem with Firebird, like hanging or abnormal termination, it is necessary to configure dump collection and then create stack traces. With dumps and stack traces, Firebird core developers of IBSurgeon can analyze the detailed picture of the hanging/failure and give the solution.


	Dump and stack traces analysis is available as a part of advanced Firebird technical support service from IBSurgeon. Our plans start from USD199/server/month, and include support through chat (WhatsApp, Telegram, etc), so in case of a serious problem you will have professional help almost immediately. For details, visit: /en/enterprise-subscription-plans-for-firebird/


	All actions below require root or sudoer.

1. Prerequisites

1.1 Delete or Stop Interfering Packages

Ubuntu/Debian

systemd-coredump

Code

apt-get purge --auto-remove systemd-coredump

To verify deletion:

Code

cat /proc/sys/kernel/core_pattern

There should be no output like:

Code

/lib/systemd/systemd-coredump%P %u %g %s %t %e

apport

Code

# Search
service --status-all | grep -m1 -i apport

# Mock deletion
apt-get -s remove apport

# Delete
sudo apt-get remove apport

# Clean remaining files
sudo apt-get purge apport

CentOS/RedHat

abrtd

To check:

Code

cat /proc/sys/kernel/core_pattern

If you see output like:

Code

/usr/libexec/abrt-hook-ccpp %s %c %p %u %g%t e %P %I

Stop and delete abrtd:

For CentOS 6:

Code

service abrtd stop

For CentOS 7:

Code

systemctl disable abrtd.service

Remove:

Code

yum -y remove abrt*

# Clean remaining files
rm -r /var/cache/abrt/

# Verify cleanup
cat /var/log/yum.log | grep abrt

1.2 Install GDB


	GDB is required to create stack traces from collected dumps. GDB installation does not affect performance or put Firebird in “debug” mode.

For CentOS:

Code

yum install gdb

For Ubuntu/Debian:

Code

apt-get install gdb

2. Remove Dump Size Limitations

Check current dump size limit:

Code

ulimit -c

If you see 0, dump creation is prohibited.

For systems with systemd, modify the service file:

CentOS/RedHat

Code

/usr/lib/systemd/system/firebird-superserver.service

Ubuntu/Debian

Code

/lib/systemd/system/firebird-superserver.service

Add to [Service] section:

Code

LimitCORE=infinity

Restart services:

Code

systemctl daemon-reload
systemctl restart firebird-superserver.service

Verify unlimited dump size:

Code

strings /proc/$(pgrep firebird)/limits

3. Configure Dump File Location and Naming


	This step is optional but highly recommended to specify meaningful dump file names and ensure adequate storage space.

Temporary Configuration

Code

sysctl -w kernel.core_pattern=/path/to/dumps/core.%e.%t.%p

Permanent Configuration

Add to /etc/sysctl.conf:

Code

kernel.core_pattern=/path/to/dumps/core.%e.%t.%p

Set Ownership

Code

chown firebird /path/to/dumps -R

4. Test Dump Collection

Prerequisites

Restart SSH
Verify ulimit -c returns “unlimited”

Test Crash Commands

Firebird 3+ (SuperServer or SuperClassic)

Code

echo "shell kill -6 $(pgrep firebird);" | /opt/firebird/bin/isql -q localhost:employee -user sysdba -pas masterkey

Firebird 2.5 (SuperClassic)

Code

echo "shell kill -6 $(pgrep fb_smp_server);" | /opt/firebird/bin/isql -q localhost:employee -user sysdba -pas masterkey

Classic Mode

Code

select mon$server_pid from mon$attachments where mon$attachment_id = current_connection;
shell kill -6 <found_PID>;

5. Install Debug Files


	- Debug files must match your Firebird build version - Installing debug files does not affect performance - Update debug files when updating Firebird

Expected debug file locations:

Code

/opt/firebird/bin/.debug/firebird.debug
/opt/firebird/bin/.debug/gbak.debug
/opt/firebird/bin/.debug/isql.debug
/opt/firebird/lib/.debug/libfbclient.so.3.0.11.debug
/opt/firebird/lib/.debug/libib_util.so.debug
/opt/firebird/plugins/.debug/libfbtrace.so.debug
/opt/firebird/plugins/.debug/libEngine12.so.debug


	HQbird installations include correct debug files and handle updates automatically.

6. Create Stack Traces from Running Processes

Create gdb-commands.txt:

Code

thread apply all bt
quit
yes

Generate stack trace:

Code

gdb -q -x ./gdb-commands.txt /opt/firebird/bin/firebird $(pgrep firebird) 1>./stacktrace.txt 2>&1

7. Create Dump from Running Process

Code

gcore -o /path/to/dumps/firebirdprefix $(pgrep firebird)


	The tool will append the process ID to the prefix.

8. Create Stack Trace from Existing Dump

Code

gdb -q -x ./gdb-commands.txt /opt/firebird/bin/firebird /tmp/core.firebird.23041 1>./stacktrace1.txt 2>&1

9. Deep Dump Analysis

To analyze shared libraries:

Code

echo -e "info sharedlibrary\nquit\n" | gdb