"Too many open files" error when benchmarking

[zizhengtai]

I wanted to benchmark the example in the readme but got this Error: Os { code: 24, kind: Other, message: "Too many open files" } error.

• macOS 10.15.5
• 2.3 GHz 8-Core Intel Core i9 (16 logical cores)
• rustc 1.44.0 (49cae5576 2020-06-01)

$ ulimit --all
Maximum size of core files created                           (kB, -c) 0
Maximum size of a process’s data segment                     (kB, -d) unlimited
Maximum size of files created by the shell                   (kB, -f) unlimited
Maximum size that may be locked into memory                  (kB, -l) unlimited
Maximum resident set size                                    (kB, -m) unlimited
Maximum number of open file descriptors                          (-n) 256
Maximum stack size                                           (kB, -s) 8192
Maximum amount of cpu time in seconds                   (seconds, -t) unlimited
Maximum number of processes available to a single user           (-u) 5568
Maximum amount of virtual memory available to the shell      (kB, -v) unlimited

Code:

use std::io;

#[async_std::main]
async fn main() -> io::Result<()> {
    let mut app = tide::new();
    app.at("/hello").get(|_| async { Ok("Hello, world!") });
    app.listen("127.0.0.1:8000").await?;
    Ok(())
}

Dependencies:

tide = "0.10.0"
async-std = { version = "1.6.0", features = ["attributes"] }

Tide command and output (same for both "/" and "/hello"):

$ cargo run --release
    Finished release [optimized] target(s) in 0.13s
     Running `target/release/test_tide`
Error: Os { code: 24, kind: Other, message: "Too many open files" }

wrk command and output for "/":

$ wrk -t10 -c1000 -d10s --latency http://127.0.0.1:8000
Running 10s test @ http://127.0.0.1:8000
  10 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.05ms    1.29ms   5.36ms   63.78%
    Req/Sec   380.00    229.91     0.85k    70.00%
  Latency Distribution
     50%    2.19ms
     75%    4.45ms
     90%    4.80ms
     99%    5.13ms
  392 requests in 10.07s, 31.39KB read
  Socket errors: connect 757, read 1702, write 8, timeout 0
  Non-2xx or 3xx responses: 392
Requests/sec:     38.91
Transfer/sec:      3.12KB

wrk command and output for "/hello":

$ wrk -t10 -c1000 -d10s --latency http://127.0.0.1:8000/hello
Running 10s test @ http://127.0.0.1:8000/hello
  10 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.19ms    1.53ms   7.90ms   86.61%
    Req/Sec   419.80    216.12   817.00     60.00%
  Latency Distribution
     50%    2.53ms
     75%    4.49ms
     90%    4.80ms
     99%    7.84ms
  433 requests in 10.06s, 54.55KB read
  Socket errors: connect 757, read 1521, write 22, timeout 0
Requests/sec:     43.05
Transfer/sec:      5.42KB

[jbr]

Some questions:

• Is it intentional that the route is at /hello but the wrk command is at /?
• What operating system are you on?
• How many cores (virtual or real) does your test machine have?
• What's an example app with a different rust server that you're using to compare?
• What version of rust?
• What's your output for ulimit --all

Here's my setup for an identical app:

• MacOS 10.15.5
• 2.4 GHz Quad-Core Intel Core i5 (8 virtual)
• rustc 1.46.0-nightly (feb3536eb 2020-06-09)

$ ulimit --all
Maximum size of core files created                           (kB, -c) 0
Maximum size of a process’s data segment                     (kB, -d) unlimited
Maximum size of files created by the shell                   (kB, -f) unlimited
Maximum size that may be locked into memory                  (kB, -l) unlimited
Maximum resident set size                                    (kB, -m) unlimited
Maximum number of open file descriptors                          (-n) 10496
Maximum stack size                                           (kB, -s) 8192
Maximum amount of cpu time in seconds                   (seconds, -t) unlimited
Maximum number of processes available to a single user           (-u) 2784
Maximum amount of virtual memory available to the shell      (kB, -v) unlimited

here is my output for /:

$ wrk -t10 -c1000 -d10s --latency http://127.0.0.1:8000
Running 10s test @ http://127.0.0.1:8000
  10 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     5.26ms    2.89ms  36.98ms   66.10%
    Req/Sec    10.88k     2.10k   17.79k    76.50%
  Latency Distribution
     50%    4.94ms
     75%    7.22ms
     90%    9.06ms
     99%   13.11ms
  1085635 requests in 10.05s, 84.90MB read
  Socket errors: connect 0, read 951, write 0, timeout 0
  Non-2xx or 3xx responses: 1085635
Requests/sec: 108017.03
Transfer/sec:      8.45MB

and for /hello

$ wrk -t10 -c1000 -d10s --latency http://127.0.0.1:8000/hello
Running 10s test @ http://127.0.0.1:8000/hello
  10 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     5.33ms    3.27ms  58.77ms   72.07%
    Req/Sec    10.44k     2.43k   16.05k    75.00%
  Latency Distribution
     50%    4.69ms
     75%    7.12ms
     90%    9.86ms
     99%   15.20ms
  1042624 requests in 10.05s, 128.27MB read
  Socket errors: connect 0, read 918, write 0, timeout 0
Requests/sec: 103769.75
Transfer/sec:     12.77MB

[zizhengtai]

@jbr I updated the issue with more information. Seems my Maximum number of open file descriptors is much smaller than yours.

And here is the result for actix-web:

Code:

use actix_web::{get, App, HttpServer};

#[get("/hello")]
async fn hello() -> &'static str {
    "Hello, world!"
}

#[actix_rt::main]
async fn main() -> std::io::Result<()> {
    HttpServer::new(|| App::new().service(hello))
        .bind("127.0.0.1:8000")?
        .run()
        .await
}

Dependencies:

actix-rt = "1"
actix-web = "2"

wrk for "/":

$ wrk -t10 -c1000 -d10s --latency http://127.0.0.1:8000
Running 10s test @ http://127.0.0.1:8000
  10 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.57ms   68.55us   5.41ms   94.10%
    Req/Sec    12.25k     5.71k   22.45k    50.45%
  Latency Distribution
     50%    1.56ms
     75%    1.58ms
     90%    1.60ms
     99%    1.86ms
  1229966 requests in 10.10s, 96.18MB read
  Socket errors: connect 757, read 113, write 0, timeout 0
  Non-2xx or 3xx responses: 1229966
Requests/sec: 121754.41
Transfer/sec:      9.52MB

wrk for "/hello":

$ wrk -t10 -c1000 -d10s --latency http://127.0.0.1:8000/hello
Running 10s test @ http://127.0.0.1:8000/hello
  10 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.54ms   68.17us   4.55ms   93.35%
    Req/Sec    12.48k     6.56k   23.68k    70.17%
  Latency Distribution
     50%    1.54ms
     75%    1.56ms
     90%    1.58ms
     99%    1.83ms
  1252976 requests in 10.10s, 155.34MB read
  Socket errors: connect 757, read 113, write 0, timeout 0
Requests/sec: 124024.71
Transfer/sec:     15.38MB

actix-web didn't crash or report any errors.

[jbr]

I'm not sure why actix-web doesn't crash or report errors in this case, but it seems like sometimes it does, check out actix-web issue 1268. If you google "too many open files," there are results for many different languages and servers, and the recommended fix for all of them is to configure your ulimits appropriately.

I've added #586 as a possible solution for systems that cannot be configured at the OS level, but for benchmarking or production servers, the correct solution is to increase ulimits