Mariadb stuck during startup

[anshulkothari810]

I have been running the firefly_iii app on a proxmox lxc for sometime. The app makes use of mariadb. The setup was working fine but recently it has stopped working as mariadb is not starting up.

I think the issue has been happening since 06/30. The mariadb container is using 100% of the CPU but the container logs only show the following three log lines so its not clear to me what's going on:

2025-07-04 20:04:59+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.8.2+maria~ubu2404 started.
2025-07-04 20:05:07+00:00 [Warn] [Entrypoint]: /sys/fs/cgroup///memory.pressure not writable, functionality unavailable to MariaDB
2025-07-04 20:05:07+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'

My docker config to start mariadb is:

db:
image: mariadb:lts
hostname: fireflydb
container_name: firefly_iii_db
restart: always
env_file: .db.env
volumes:
- firefly_iii_db:/var/lib/mysql
networks:
- firefly_iii

I have tried doing a clean installation as well as downgrading to 11.4 but nothing helps as I observe the same behavior.

I have tried attaching to the docker container and looking at /var/logs/mysql to see if I can see any more logs but it is empty.

Any idea on how to debug this?

[grooverdan]

Can you obtain the processlist of the db container? Which proxmox version?

MariaDB logs in the container go straight to stdout/stderr so you're not missing anything with filesystem based logs.

To see if this is a bug that we have already fixes the development container should be equalivant:
quay.io/mariadb-foundation/mariadb-devel:11.4.

In this repo there is a Containerfile.debug that will install gdb and the debuginfo symbols. Build with:

$ buildah bud --tag mariadb_debug:11.4 --build-arg BASE=mariadb:11.4 -f Containerfile.debug .

Or the equivalent compose/docker build.

Add to the compose for db:

    cap_add:
      - SYS_PTRACE

To capture what MariaDB is doing assuming its running and has made it to the pid1 after switching user (which I'm not 100% sure it has by the logs).

docker exec db gdb --batch --eval-command="set print frame-arguments all"  --eval-command="thread apply all bt full" /usr/sbin/mariadbd 1

[anshulkothari810]

Given mariadb was working for me before and I have an initialized db, I was able to solve this problem by starting the db docker using mysql user instead of root, which I believe avoided calling the gosu command.

Offcourse, this didn't work for a clean installation which I was playing with. From the clean installation, when I switch back to using root as the user, I see following processes:

root@fireflydb:/# ps -aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 93.1 0.0 704000 384 ? Rsl 07:11 4:01 gosu mysql /usr/local/bin/docker-entrypoint.sh mariadbd
root 47 0.0 0.0 5016 3968 pts/0 Ss 07:13 0:00 /bin/bash
root 57 0.0 0.0 8280 4224 pts/0 R+ 07:15 0:00 ps -aux

I am not sure what is the correct way to use containerfile.debug so I just updated my existing compose file to use following image: quay.io/mariadb-foundation/mariadb-debug:11.8". I added sys_ptrace capability also. After running the docker exec command you suggested, I only see gosu related threads. For example,

root@firefly2:/Shared/Scripts/compose-files/firefly_debug# docker exec firefly_iii_db gdb --batch --eval-command="set print frame-arguments all" --eval-command="thread apply all bt full" /usr/sbin/mariadbd 1
[New LWP 33]
[New LWP 32]
[New LWP 31]
[New LWP 30]
0x0000000000461b87 in ?? ()

Thread 5 (LWP 30 "gosu"):
#0 0x0000000000461a23 in ?? ()
No symbol table info available.
#1 0x000000000042f24f in ?? ()
No symbol table info available.
#2 0x00000000006145b8 in ?? ()
No symbol table info available.
#3 0x0000000000000080 in ?? ()
No symbol table info available.
#4 0x000000c000045eb8 in ?? ()
No symbol table info available.
#5 0x0000000000000000 in ?? ()
No symbol table info available.

Thread 4 (LWP 31 "gosu"):
#0 0x0000000000461a23 in ?? ()
No symbol table info available.
#1 0x000000000042f1d6 in ?? ()
No symbol table info available.
#2 0x000000c000036548 in ?? ()
No symbol table info available.
#3 0x0000000000000080 in ?? ()
No symbol table info available.
#4 0x0000000000000000 in ?? ()
No symbol table info available.

Thread 3 (LWP 32 "gosu"):
#0 0x0000000000461a23 in ?? ()
No symbol table info available.
#1 0x000000000042f1d6 in ?? ()
No symbol table info available.
#2 0x000000c000036948 in ?? ()
No symbol table info available.
#3 0x0000000000000080 in ?? ()
No symbol table info available.
#4 0x0000000000000000 in ?? ()
No symbol table info available.

Thread 2 (LWP 33 "gosu"):
#0 0x0000000000461a23 in ?? ()
No symbol table info available.
#1 0x000000000042f1d6 in ?? ()
No symbol table info available.
#2 0x0000000000642958 in ?? ()
No symbol table info available.
#3 0x0000000000000080 in ?? ()
No symbol table info available.
#4 0x0000000000000000 in ?? ()
No symbol table info available.

Thread 1 (LWP 1 "gosu"):
#0 0x0000000000461b87 in ?? ()
No symbol table info available.
#1 0x000000000045ba54 in ?? ()
No symbol table info available.
#2 0x000000c000036000 in ?? ()
No symbol table info available.

[grooverdan]

so it appears that gosu is stuck. Ideas @tianon ?

[tianon]

Definitely weird -- the only time I've seen something similar has been QEMU user-mode emulation bugs (32bit arm on 64bit x86, to be specific). 🤔