Benchmark tracking

[porcuquine]

Issue for tracking benchmarks over time.

[porcuquine]

3e92fa9f1be80575f1e5a0b1617a75b04805f152 on my development machine

(base) ➜  neptune git:(master) cargo bench
    …
    Finished bench [optimized] target(s) in 28.58s
     Running target/release/deps/neptune-6aa7ba28772916f6

running 3 tests
test poseidon::tests::hash_det ... ignored
test poseidon::tests::reset ... ignored
test tests::constants_consistency ... ignored

test result: ok. 0 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out

     Running target/release/deps/hash-6e8d97b43ee19a2b
Gnuplot not found or not usable, disabling plotting
`gnuplot --version` failed with error message:
dyld: Library not loaded: /usr/local/opt/readline/lib/libreadline.7.dylib
  Referenced from: /usr/local/bin/gnuplot
  Reason: image not found

hash/Sha2 256/Generated scalars
                        time:   [2.7975 us 2.8167 us 2.8364 us]
                        change: [-2.2134% -1.5500% -0.9210%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 6 outliers among 100 measurements (6.00%)
  6 (6.00%) high mild
hash/Sha2 512/Generated scalars
                        time:   [2.6885 us 2.7036 us 2.7203 us]
                        change: [+1.8016% +2.7368% +3.8340%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 11 outliers among 100 measurements (11.00%)
  5 (5.00%) high mild
  6 (6.00%) high severe
hash/Poseidon hash/Generated scalars
                        time:   [63.715 us 64.102 us 64.499 us]
                        change: [-1.6427% -0.9227% -0.2130%] (p = 0.01 < 0.05)
                        Change within noise threshold.
Found 4 outliers among 100 measurements (4.00%)
  4 (4.00%) high mild

Gnuplot not found or not usable, disabling plotting
`gnuplot --version` failed with error message:
dyld: Library not loaded: /usr/local/opt/readline/lib/libreadline.7.dylib
  Referenced from: /usr/local/bin/gnuplot
  Reason: image not found

[porcuquine]

https://github.com/filecoin-project/neptune/commit/615ec19c3570a4c4e268273115bc8535db2b20ca

Binary hash (width 3):

hash-64/Sha2 256/Generated scalars
                        time:   [1.4690 us 1.4713 us 1.4738 us]
                        change: [-1.7410% -1.3525% -0.8979%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 8 outliers among 100 measurements (8.00%)
  2 (2.00%) high mild
  6 (6.00%) high severe
hash-64/Sha2 512/Generated scalars
                        time:   [1.2767 us 1.2783 us 1.2800 us]
                        change: [-1.4560% -1.0486% -0.6472%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 10 outliers among 100 measurements (10.00%)
  4 (4.00%) high mild
  6 (6.00%) high severe
hash-64/Poseidon hash/Generated scalars
                        time:   [29.887 us 29.937 us 29.991 us]
                        change: [-1.8042% -1.3567% -0.9145%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 11 outliers among 100 measurements (11.00%)
  7 (7.00%) high mild
  4 (4.00%) high severe

Circuit constraints (ARITY = 2): https://github.com/filecoin-project/neptune/blob/615ec19c3570a4c4e268273115bc8535db2b20ca/src/circuit.rs#L282

[porcuquine]

7acbdcf

https://github.com/filecoin-project/neptune/blob/7acbdcfa70d1f3f11d47e2c38ed79ff7da84559b/src/circuit.rs#L416

[porcuquine]

e483a51

https://github.com/filecoin-project/neptune/blob/e483a514666ab828ab38051413e132fc49dbe867/src/circuit.rs#L445

Perf regression might be related to benchmark. We're now computing (rather than just loading) constants on first access, and benchmark may be including this time.

[UPDATE: benchmark was actually run with arity 2, although the commit shown is arity 8. That setting change for comparison wasn't committed.]

hash-64/Sha2 256/Generated scalars
                        time:   [1.4922 us 1.4979 us 1.5042 us]
                        change: [-0.6302% -0.2113% +0.2502%] (p = 0.36 > 0.05)
                        No change in performance detected.
Found 8 outliers among 100 measurements (8.00%)
  5 (5.00%) high mild
  3 (3.00%) high severe
hash-64/Sha2 512/Generated scalars
                        time:   [1.3212 us 1.3259 us 1.3303 us]
                        change: [-0.4008% +0.1080% +0.6053%] (p = 0.68 > 0.05)
                        No change in performance detected.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild
hash-64/Poseidon hash/Generated scalars
                        time:   [32.169 us 32.284 us 32.406 us]
                        change: [+2.5868% +3.2309% +3.8328%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 8 outliers among 100 measurements (8.00%)
  1 (1.00%) low mild
  7 (7.00%) high mild

[porcuquine]

8bc8794

(base) ➜  neptune git:(master) git rev-parse HEAD
8bc879426d11212797ada4edb9c3010fd92618e2

(base) ➜  neptune git:(master) cargo bench
    Finished bench [optimized] target(s) in 0.13s
     Running target/release/deps/neptune-678d721d93608f16

hash-64/Sha2 256/Generated scalars
                        time:   [1.6942 us 1.6991 us 1.7046 us]
                        change: [-1.7510% -1.1445% -0.5614%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 7 outliers among 100 measurements (7.00%)
  7 (7.00%) high mild
hash-64/Sha2 512/Generated scalars
                        time:   [1.5103 us 1.5197 us 1.5322 us]
                        change: [-0.8553% -0.3516% +0.2200%] (p = 0.21 > 0.05)
                        No change in performance detected.
Found 7 outliers among 100 measurements (7.00%)
  2 (2.00%) high mild
  5 (5.00%) high severe
hash-64/Poseidon hash/Generated scalars
                        time:   [23.866 us 23.994 us 24.123 us]
                        change: [-2.4276% -1.8550% -1.3182%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild
hash-64/Poseidon hash optimized/Generated scalars
                        time:   [17.558 us 17.621 us 17.704 us]
                        change: [-4.1792% -3.7571% -3.3465%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
  2 (2.00%) high mild
  2 (2.00%) high severe

hash-128/Sha2 256/Generated scalars
                        time:   [3.1106 us 3.1175 us 3.1244 us]
                        change: [-1.8971% -1.5546% -1.1891%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
  1 (1.00%) low mild
  5 (5.00%) high mild
  2 (2.00%) high severe
hash-128/Sha2 512/Generated scalars
                        time:   [2.9772 us 2.9877 us 2.9996 us]
                        change: [-1.7984% -1.2300% -0.6251%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 3 outliers among 100 measurements (3.00%)
  1 (1.00%) high mild
  2 (2.00%) high severe
hash-128/Poseidon hash/Generated scalars
                        time:   [58.596 us 58.887 us 59.241 us]
                        change: [+1.2014% +1.6733% +2.1535%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 5 outliers among 100 measurements (5.00%)
  1 (1.00%) low mild
  4 (4.00%) high mild
hash-128/Poseidon hash optimized/Generated scalars
                        time:   [31.912 us 32.007 us 32.105 us]
                        change: [-0.8927% -0.4021% +0.0908%] (p = 0.13 > 0.05)
                        No change in performance detected.
Found 3 outliers among 100 measurements (3.00%)
  1 (1.00%) low mild
  2 (2.00%) high mild

hash-256/Sha2 256/Generated scalars
                        time:   [6.0731 us 6.0949 us 6.1204 us]
                        change: [-1.2142% -0.7038% -0.1072%] (p = 0.01 < 0.05)
                        Change within noise threshold.
Found 8 outliers among 100 measurements (8.00%)
  3 (3.00%) high mild
  5 (5.00%) high severe
hash-256/Sha2 512/Generated scalars
                        time:   [5.6254 us 5.6399 us 5.6546 us]
                        change: [-1.9873% -1.5588% -1.1264%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) high mild
hash-256/Poseidon hash/Generated scalars
                        time:   [169.73 us 170.14 us 170.58 us]
                        change: [-0.3360% -0.0222% +0.3106%] (p = 0.90 > 0.05)
                        No change in performance detected.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild
hash-256/Poseidon hash optimized/Generated scalars
                        time:   [63.139 us 63.286 us 63.442 us]
                        change: [-1.1950% -0.7944% -0.3897%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 6 outliers among 100 measurements (6.00%)
  1 (1.00%) low mild
  4 (4.00%) high mild
  1 (1.00%) high severe

hash-352/Sha2 256/Generated scalars
                        time:   [7.9285 us 7.9429 us 7.9579 us]
                        change: [-2.4148% -1.9924% -1.5447%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
  6 (6.00%) high mild
  4 (4.00%) high severe
hash-352/Sha2 512/Generated scalars
                        time:   [7.3065 us 7.3227 us 7.3410 us]
                        change: [-2.9885% -2.5386% -2.1119%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
  1 (1.00%) low mild
  7 (7.00%) high mild
  1 (1.00%) high severe
hash-352/Poseidon hash/Generated scalars
                        time:   [287.88 us 288.84 us 289.99 us]
                        change: [-2.3777% -1.8167% -1.3036%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
  1 (1.00%) low mild
  4 (4.00%) high mild
  4 (4.00%) high severe
hash-352/Poseidon hash optimized/Generated scalars
                        time:   [91.337 us 91.649 us 91.969 us]
                        change: [-1.9678% -1.5258% -1.0567%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe

GPU:

➜  gbench git:(master) RUST_LOG=info cargo run --release
    Finished release [optimized] target(s) in 0.05s
     Running `target/release/gbench`
[2020-05-02T01:27:06Z INFO  gbench] KiB: 4194304
[2020-05-02T01:27:06Z INFO  gbench] leaves: 134217728
[2020-05-02T01:27:06Z INFO  gbench] max column batch size: 400000
[2020-05-02T01:27:06Z INFO  gbench] max tree batch size: 700000
[2020-05-02T01:27:06Z INFO  gbench] Creating ColumnTreeBuilder
[2020-05-02T01:27:10Z INFO  gbench] ColumnTreeBuilder created
[2020-05-02T01:27:10Z INFO  gbench] Using effective batch size 400000 to build columns
[2020-05-02T01:27:10Z INFO  gbench] adding column batches
[2020-05-02T01:27:10Z INFO  gbench] start commitment
...............................................................................................................................................................................................................................................................................................................................................
[2020-05-02T01:29:23Z INFO  gbench] adding final column batch and building tree
[2020-05-02T01:29:39Z INFO  gbench] end commitment
[2020-05-02T01:29:39Z INFO  gbench] commitment time: 149.518970373s
[2020-05-02T01:29:39Z INFO  gbench] end

[porcuquine]

Latest number with pure OpenCL implementation are reported here: https://github.com/filecoin-project/neptune/pull/78#issue-556044272