Gradual CPU leak: 5% to 345% over 13 days with auditd-logs parser (v1.7.8)

[Zendorea]

Describe the bug
CrowdSec v1.7.8 gradually accumulates CPU usage over days/weeks until it consumes 345%+ CPU. The process starts at ~5% after restart and slowly climbs without bound. After 13 days of uptime, systemd reported the process consumed 3 weeks 2 days 14 hours of CPU time.

The log volume being processed is minimal (~8.5k lines from auditd, ~47 from auth.log), so this is not caused by high throughput.

To Reproduce

1. Install CrowdSec v1.7.8 on Ubuntu 24.04
2. Configure acquisition for /var/log/audit/audit.log (auditd) and /var/log/syslog
3. Let it run for 7-14 days
4. Observe CPU usage climbing from ~5% to 100%+ over time

Expected behavior
CPU usage should remain stable at ~5-10% regardless of uptime duration.

Evidence

Systemd stop output after 13 days:

May 20 01:49:00 Media-Server systemd[1]: crowdsec.service: Consumed 3w 2d 14h 58min 23.887s CPU time, 573.9M memory peak, 66.0M memory swap peak.

Process state at discovery:

root  3363534  345%  1.0  3110892 159888 ?  Rsl  May13 34007:09 /usr/bin/crowdsec -c /etc/crowdsec/config.yaml

After restart:

root  2781659  5.8%  1.2  2385428 205484 ?  Ssl  01:49  0:32 /usr/bin/crowdsec -c /etc/crowdsec/config.yaml

Acquisition metrics (showing minimal volume):

| Source                         | Lines read | Lines parsed | Lines unparsed |
| file:/var/log/audit/audit.log  | 8.59k      | 8.46k        | 128            |
| file:/var/log/auth.log         | 47         | 17           | 30             |

Acquisition config

filenames:
  - /var/log/audit/*.log
labels:
  type: auditd
source: file
---
filenames:
  - /var/log/messages
  - /var/log/syslog
  - /var/log/kern.log
labels:
  type: syslog
source: file
---
filenames:
  - /var/log/nginx/*.log
labels:
  type: nginx
source: file

Parsers installed

• crowdsecurity/auditd-logs
• crowdsecurity/sshd-logs
• crowdsecurity/nginx-logs
• crowdsecurity/http-logs

Environment

• CrowdSec version: v1.7.8-debian-pragmatic-amd64-63227459
• Go version: 1.26.2
• OS: Ubuntu 24.04 LTS (kernel 6.8.0-111-generic)
• Platform: linux/amd64 (Hetzner CPX42)
• RAM: 16GB
• Installed via: apt (debian package)

Workaround
Weekly or watchdog-based systemctl restart crowdsec prevents the leak from becoming critical. A watchdog script checking CPU > 50% for 30 minutes and restarting is our current mitigation.

Additional context

• The leak appears to be in the Go runtime or parser goroutines, not in log processing volume
• Memory peaks at ~574MB which is high for processing <10k lines
• The auditd-logs parser chain (child-child-child-crowdsecurity/auditd-logs) may be involved
• No panics or errors in journalctl logs — it just silently consumes more CPU over time

[github-actions]

@Zendorea: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

1. Check Crowdsec Documentation to see if your issue can be self resolved.
2. You can also join our Discord.
3. Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[github-actions]

@Zendorea: There are no 'kind' label on this issue. You need a 'kind' label to start the triage process.

• /kind feature
• /kind enhancement
• /kind refactoring
• /kind bug
• /kind packaging

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[Zendorea]

/kind bug

[blotus]

Hello,

Could you provide a pprof profile when the CPU usage is high ?

Assuming you are using the default 6060 port for the metrics:
curl http://localhost:6060/debug/pprof/profile -o profile.gz
The command will take 30s to run.

Could you also provide crowdsec log file ?

[Zendorea]

Thanks for the quick response. Unfortunately we already restarted the service to restore normal operation before filing this issue, so CPU is back to ~5% now.

We have a watchdog script that monitors CrowdSec CPU usage. I'll update it to automatically capture a pprof profile before restarting next time the leak manifests. Based on our observation, it takes 7-14 days for the CPU to climb above the threshold.

Will provide both the pprof profile and log file once we have the data from the next occurrence.

[Zendorea]

Got the pprof profile captured! The CPU leak recurred today (May 23, 2026).

Timeline

• May 20 01:49 UTC: Restarted CrowdSec (previous leak)
• May 23 08:10 UTC: Watchdog detected 51% CPU sustained for 30 minutes
• May 23 08:10 UTC: Captured pprof profile + heap dump, then restarted

So it took ~2.5 days to reach 51% this time (previously it was 13 days to reach 345% — the difference is likely due to higher audit log volume this week).

Files

Profile and heap dump (base64-encoded gzip): https://gist.github.com/Zendorea/b7883e205b8b15228953116157dbf128

To decode and analyze:

Environment

• CrowdSec v1.7.8 (alphaga), installed via apt on Ubuntu 24.04
• Kernel 6.8.0-111-generic, Hetzner CPX42 (8 vCPU, 16GB RAM)
• Parsers: , sshd-logs, nginx-logs
• Scenarios: auditd-base64-exec-behavior, auditd-postexploit-pkill, auditd-postexploit-rm, auditd-suid-crash, auditd-sus-exec
• Custom whitelist parser () filters ~99.7% of audit events
• Audit log volume: ~120,000 events/hour (Docker host with heavy cron/exec activity)
• CrowdSec metrics show 7.11M audit events processed in the 2.5 days before the leak

Observations

• The leak correlates with auditd-logs parser volume. This server generates ~3M audit events/day from Docker exec and cron activity.
• The custom whitelist parser short-circuits most events early, but the leak still occurs.
• Removing scenario (which was triggering on our own ffmpeg/Docker processes) did not prevent the leak.
• Memory also grows: 317MB at start, peaked at 1.5GB before restart.

Let me know if you need anything else or if the profile format is correct.

[blotus]

Looking at the profile, I think it's something similar to #4464 : almost all the time seems to be spent in sqlite calls, but it's weird for the CPU usage to increase slowly over time instead of instantly.

Do you have any custom profile/scenarios that would make DB calls (eg, GetDecisionsCount, GetDecisionsSinceCount, ... helpers) ?

We don't have a package yet for you to try the fix, but you could either try with the dev docker image, or manually building crowdsec from the master branch then replacing the binary on your server.

Another solution would be to try to migrate to PG (or mysql), but it's a lot more involved.

[Zendorea]

No custom profiles or scenarios that make direct DB calls. Here's what we have:

Custom scenario:

• custom/wordpress-scanner — standard leaky bucket, no DB helper calls

Custom parsers:

• custom/whitelist-admin-audit — whitelists root uid audit events (expression-based, no DB calls)
• custom/whitelists — standard IP/CIDR whitelist

DB info:

• SQLite, 31MB
• No custom profiles overriding the default

Installed scenarios (all from hub):

• 5x auditd (base64-exec, pkill, rm, suid-crash, sus-exec)
• ssh-bf, ssh-slow-bf
• nginx http-* scenarios
• crowdsecurity/http-* scenarios

The high audit log volume (~120K events/hour, ~3M/day) is likely what's stressing the SQLite path. Most events are whitelisted early by the parser, but the ones that pass through to scenarios still hit the DB for decision checks.

Happy to test the dev docker image or a master branch build if you can point me to the specific commit/PR that addresses #4464. Our watchdog will capture another profile if the fix doesn't resolve it.

For now we'll keep the watchdog running with the current stable release. Let me know if you need any other info.

[Zendorea]

Analyzed the pprof profile with go tool pprof — can confirm your SQLite hypothesis is correct.

CPU Profile Analysis

Duration: 30s, Total samples = 17.17s (57.23%)
Top functions:
      flat  flat%   cum   cum%
    16.12s 93.88% 16.12s 93.88%  runtime.cgocall
         0     0% 16.13s 93.94%  github.com/mattn/go-sqlite3.(*SQLiteRows).Next.func1
         0     0% 16.12s 93.88%  github.com/mattn/go-sqlite3.(*SQLiteRows).nextSyncLocked
         0     0% 16.07s 93.59%  github.com/mattn/go-sqlite3._Cfunc__sqlite3_step_internal

93.88% of all CPU time is spent in go-sqlite3 → _sqlite3_step_internal. Only 1.92% is in parser logic and 0.76% in bucket processing. The application itself is essentially idle — SQLite is consuming all available CPU.

Heap Profile

Heap shows only 12MB in use — this is not a memory leak. The 317MB→1.5GB growth reported by systemd is likely SQLite's internal page cache or OS-level RSS growth from the CGo bridge under sustained load.

Correlation with PR #4473

Our build is v1.7.8 (built 2026-05-11), which ships with the go-sqlite3 version that introduced the regression. PR #4473 (merged 2026-05-19, 8 days after our build) downgrades to v1.14.32 which should resolve this.

This matches the behavior perfectly:

• v1.7.7 (go-sqlite3 v1.14.23) → no issue
• v1.7.8 (go-sqlite3 v1.14.44) → CPU leak over time
• Fix: downgrade to v1.14.32

Our Path Forward

We can downgrade to v1.7.7 via apt as a workaround, but we'd prefer to stay on v1.7.8 with the watchdog restarting every 2-3 days until a v1.7.9 (or patch release) ships with the fix from #4473 included. Is there an ETA on the next release?

In the meantime, our watchdog auto-restarts CrowdSec when CPU exceeds 50% for 30 minutes, which keeps the impact manageable.