perf: improve performance of timestamp truncate for some formats by andygrove · Pull Request #2996 · apache/datafusion-comet

andygrove · 2025-12-27T15:42:37Z

Which issue does this PR close?

Part of #2995

Rationale for this change

Sub-day truncations (MICROSECOND, MILLISECOND, SECOND, MINUTE, HOUR) are timezone-independent - truncating to the minute boundary is the same regardless of timezone display. This allows simple integer arithmetic instead of expensive DateTime conversions.

Criterion Benchmarks

Format	Before	After	Speedup
HOUR	957 µs	32 µs	~30x faster (96.6% improvement)
MINUTE	640 µs	32 µs	~20x faster (95% improvement)
SECOND	405 µs	32 µs	~12x faster (92% improvement)
MILLISECOND	430 µs	32 µs	~13x faster (92.6% improvement)
MICROSECOND	435 µs	27 µs	~16x faster (93.7% improvement)

Microbenchmarks

OpenJDK 64-Bit Server VM 17.0.17+10-Ubuntu-122.04 on Linux 6.8.0-90-generic
AMD Ryzen 9 7950X3D 16-Core Processor
Timestamp Truncate - HOUR:                Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                93             95           1         11.3          88.2       1.0X
Comet (Scan)                                         90             92           1         11.6          86.0       1.0X
Comet (Scan + Exec)                                  38             40           1         27.4          36.6       2.4X

OpenJDK 64-Bit Server VM 17.0.17+10-Ubuntu-122.04 on Linux 6.8.0-90-generic
AMD Ryzen 9 7950X3D 16-Core Processor
Timestamp Truncate - MINUTE:              Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                94             96           2         11.2          89.3       1.0X
Comet (Scan)                                         90             92           2         11.7          85.4       1.0X
Comet (Scan + Exec)                                  38             40           1         27.4          36.5       2.5X

OpenJDK 64-Bit Server VM 17.0.17+10-Ubuntu-122.04 on Linux 6.8.0-90-generic
AMD Ryzen 9 7950X3D 16-Core Processor
Timestamp Truncate - SECOND:              Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                33             35           1         31.6          31.6       1.0X
Comet (Scan)                                         31             33           2         34.3          29.2       1.1X
Comet (Scan + Exec)                                  38             40           1         27.4          36.5       0.9X

OpenJDK 64-Bit Server VM 17.0.17+10-Ubuntu-122.04 on Linux 6.8.0-90-generic
AMD Ryzen 9 7950X3D 16-Core Processor
Timestamp Truncate - MILLISECOND:         Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                33             35           1         31.7          31.6       1.0X
Comet (Scan)                                         30             31           1         35.4          28.2       1.1X
Comet (Scan + Exec)                                  38             40           1         27.6          36.2       0.9X

OpenJDK 64-Bit Server VM 17.0.17+10-Ubuntu-122.04 on Linux 6.8.0-90-generic
AMD Ryzen 9 7950X3D 16-Core Processor
Timestamp Truncate - MICROSECOND:         Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                29             31           2         36.3          27.6       1.0X
Comet (Scan)                                         24             25           1         43.5          23.0       1.2X
Comet (Scan + Exec)                                  33             34           1         31.9          31.3       0.9X

What changes are included in this PR?

This PR does not improve performance for YEAR, MONTH, DAY, WEEK, or QUARTER, which all remain very slow.

How are these changes tested?

codecov-commenter · 2025-12-27T16:16:08Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 59.58%. Comparing base (f09f8af) to head (6c98518).
⚠️ Report is 829 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2996      +/-   ##
============================================
+ Coverage     56.12%   59.58%   +3.45%     
- Complexity      976     1368     +392     
============================================
  Files           119      167      +48     
  Lines         11743    15497    +3754     
  Branches       2251     2573     +322     
============================================
+ Hits           6591     9234    +2643     
- Misses         4012     4966     +954     
- Partials       1140     1297     +157

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

sqlbenchmark · 2026-01-02T19:00:57Z

Comet Microbenchmark Results: CometDatetimeExpressionBenchmark

Commit: 75a53cb - clippy

Benchmark Results

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Date Truncate  - YEAR:                    Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                38             41           2         27.4          36.5       1.0X
Comet (Scan)                                         37             40           2         28.3          35.3       1.0X
Comet (Scan + Exec)                                  41             44           1         25.7          38.9       0.9X

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Date Truncate  - YYYY:                    Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                33             35           1         31.6          31.7       1.0X
Comet (Scan)                                         34             36           1         30.7          32.6       1.0X
Comet (Scan + Exec)                                  39             41           1         27.0          37.0       0.9X

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Date Truncate  - YY:                      Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                32             34           1         32.6          30.7       1.0X
Comet (Scan)                                         33             35           1         32.0          31.2       1.0X
Comet (Scan + Exec)                                  39             40           1         27.0          37.1       0.8X

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Date Truncate  - MON:                     Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                34             35           2         31.2          32.0       1.0X
Comet (Scan)                                         34             35           1         30.6          32.7       1.0X
Comet (Scan + Exec)                                  34             35           1         30.8          32.5       1.0X

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Date Truncate  - MONTH:                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                34             35           1         31.0          32.3       1.0X
Comet (Scan)                                         33             34           1         31.8          31.4       1.0X
Comet (Scan + Exec)                                  34             35           1         30.6          32.7       1.0X

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Date Truncate  - MM:                      Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                34             35           1         31.1          32.2       1.0X
Comet (Scan)                                         33             34           1         31.7          31.6       1.0X
Comet (Scan + Exec)                                  34             35           1         30.8          32.4       1.0X

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Date Truncate (Dictionary) - YEAR:        Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                31             32           1         34.1          29.3       1.0X
Comet (Scan)                                         31             33           1         33.4          29.9       1.0X
Comet (Scan + Exec)                                  37             38           1         28.5          35.1       0.8X

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Date Truncate (Dictionary) - YYYY:        Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                31             32           1         34.3          29.1       1.0X
Comet (Scan)                                         31             33           1         33.5          29.8       1.0X
Comet (Scan + Exec)                                  37             38           1         28.2          35.5       0.8X

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Date Truncate (Dictionary) - YY:          Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                30             31           1         34.9          28.6       1.0X
Comet (Scan)                                         31             32           1         33.7          29.7       1.0X
Comet (Scan + Exec)                                  37             38           1         28.1          35.6       0.8X

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Date Truncate (Dictionary) - MON:         Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                33             34           1         32.1          31.1       1.0X
Comet (Scan)                                         32             33           1         32.4          30.9       1.0X
Comet (Scan + Exec)                                  32             34           1         32.5          30.8       1.0X

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Date Truncate (Dictionary) - MONTH:       Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                32             33           1         32.5          30.8       1.0X
Comet (Scan)                                         32             34           2         32.3          31.0       1.0X
Comet (Scan + Exec)                                  33             35           1         31.9          31.3       1.0X

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Date Truncate (Dictionary) - MM:          Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                34             34           1         31.3          32.0       1.0X
Comet (Scan)                                         33             34           1         31.7          31.6       1.0X
Comet (Scan + Exec)                                  34             35           1         31.2          32.1       1.0X

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Timestamp Truncate  - YEAR:               Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                               106            108           1          9.9         100.9       1.0X
Comet (Scan)                                        105            106           1         10.0         100.3       1.0X
Comet (Scan + Exec)                                 336            337           1          3.1         320.4       0.3X

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Timestamp Truncate  - YYYY:               Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                               108            109           1          9.7         102.6       1.0X
Comet (Scan)                                        105            106           1         10.0         100.0       1.0X
Comet (Scan + Exec)                                 336            337           1          3.1         320.8       0.3X

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Timestamp Truncate  - YY:                 Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                               107            108           1          9.8         102.2       1.0X
Comet (Scan)                                        104            106           1         10.0          99.5       1.0X
Comet (Scan + Exec)                                 336            338           1          3.1         320.6       0.3X

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Timestamp Truncate  - MON:                Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                               108            109           1          9.7         103.1       1.0X
Comet (Scan)                                        105            107           1         10.0         100.5       1.0X
Comet (Scan + Exec)                                 283            284           1          3.7         270.0       0.4X

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Timestamp Truncate  - MONTH:              Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                               108            109           2          9.7         103.1       1.0X
Comet (Scan)                                        106            107           3          9.9         100.7       1.0X
Comet (Scan + Exec)                                 284            284           1          3.7         270.6       0.4X

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Timestamp Truncate  - MM:                 Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                               108            109           1          9.7         103.3       1.0X
Comet (Scan)                                        106            107           1          9.9         100.8       1.0X
Comet (Scan + Exec)                                 277            281           3          3.8         263.7       0.4X

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Timestamp Truncate  - DAY:                Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                90             91           1         11.6          86.0       1.0X
Comet (Scan)                                         88             89           1         12.0          83.7       1.0X
Comet (Scan + Exec)                                 237            238           1          4.4         226.0       0.4X

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Timestamp Truncate  - DD:                 Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                91             92           1         11.5          86.6       1.0X
Comet (Scan)                                         88             89           1         12.0          83.5       1.0X
Comet (Scan + Exec)                                 238            239           1          4.4         226.8       0.4X

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Timestamp Truncate  - HOUR:               Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                97             98           1         10.8          92.9       1.0X
Comet (Scan)                                         95             96           1         11.1          90.5       1.0X
Comet (Scan + Exec)                                  36             37           1         29.4          34.0       2.7X

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Timestamp Truncate  - MINUTE:             Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                92             94           3         11.4          87.6       1.0X
Comet (Scan)                                         88             89           1         11.9          84.0       1.0X
Comet (Scan + Exec)                                  36             37           1         29.4          34.1       2.6X

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Timestamp Truncate  - SECOND:             Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                31             32           1         33.3          30.0       1.0X
Comet (Scan)                                         28             29           1         36.9          27.1       1.1X
Comet (Scan + Exec)                                  34             35           1         30.7          32.5       0.9X

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Timestamp Truncate  - WEEK:               Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                               102            103           0         10.3          97.1       1.0X
Comet (Scan)                                        100            101           2         10.5          94.9       1.0X
Comet (Scan + Exec)                                 276            277           1          3.8         263.4       0.4X

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Timestamp Truncate  - QUARTER:            Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                               126            127           1          8.4         119.8       1.0X
Comet (Scan)                                        123            124           1          8.5         117.7       1.0X
Comet (Scan + Exec)                                 340            340           1          3.1         324.1       0.4X

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Timestamp Truncate (Dictionary) - YEAR:   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                97             99           2         10.8          92.8       1.0X
Comet (Scan)                                         94             95           1         11.1          89.9       1.0X
Comet (Scan + Exec)                                 326            328           2          3.2         310.5       0.3X

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Timestamp Truncate (Dictionary) - YYYY:   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                97            100           7         10.8          92.7       1.0X
Comet (Scan)                                         94             95           1         11.1          89.7       1.0X
Comet (Scan + Exec)                                 331            333           2          3.2         315.9       0.3X

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Timestamp Truncate (Dictionary) - YY:     Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                99            101           5         10.6          94.5       1.0X
Comet (Scan)                                         94             95           1         11.2          89.2       1.1X
Comet (Scan + Exec)                                 327            328           1          3.2         311.7       0.3X

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Timestamp Truncate (Dictionary) - MON:    Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                96             98           1         10.9          91.9       1.0X
Comet (Scan)                                         94             95           1         11.2          89.7       1.0X
Comet (Scan + Exec)                                 279            280           1          3.8         265.9       0.3X

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Timestamp Truncate (Dictionary) - MONTH:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                96             96           1         11.0          91.2       1.0X
Comet (Scan)                                         93             94           1         11.3          88.3       1.0X
Comet (Scan + Exec)                                 273            275           2          3.8         260.8       0.3X

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Timestamp Truncate (Dictionary) - MM:     Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                96             97           1         10.9          91.4       1.0X
Comet (Scan)                                         94             95           1         11.1          89.9       1.0X
Comet (Scan + Exec)                                 279            279           1          3.8         265.7       0.3X

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Timestamp Truncate (Dictionary) - DAY:    Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                81             82           1         12.9          77.6       1.0X
Comet (Scan)                                         78             79           1         13.4          74.4       1.0X
Comet (Scan + Exec)                                 234            234           1          4.5         222.9       0.3X

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Timestamp Truncate (Dictionary) - DD:     Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                83             84           1         12.7          78.9       1.0X
Comet (Scan)                                         80             83           3         13.1          76.2       1.0X
Comet (Scan + Exec)                                 238            240           2          4.4         227.0       0.3X

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Timestamp Truncate (Dictionary) - HOUR:   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                82             83           1         12.8          78.3       1.0X
Comet (Scan)                                         79             80           1         13.2          75.8       1.0X
Comet (Scan + Exec)                                  30             31           1         35.0          28.6       2.7X

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Timestamp Truncate (Dictionary) - MINUTE:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                81             83           3         12.9          77.6       1.0X
Comet (Scan)                                         79             80           1         13.3          74.9       1.0X
Comet (Scan + Exec)                                  30             31           1         35.2          28.4       2.7X

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Timestamp Truncate (Dictionary) - SECOND:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                27             27           1         39.4          25.3       1.0X
Comet (Scan)                                         23             24           1         45.6          21.9       1.2X
Comet (Scan + Exec)                                  29             30           1         35.9          27.9       0.9X

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Timestamp Truncate (Dictionary) - WEEK:   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                                93             94           1         11.3          88.5       1.0X
Comet (Scan)                                         90             91           1         11.7          85.8       1.0X
Comet (Scan + Exec)                                 278            280           2          3.8         265.0       0.3X

OpenJDK 64-Bit Server VM 17.0.17+10 on Linux 6.8.0-57-generic
AMD Ryzen 9 7950X 16-Core Processor
Timestamp Truncate (Dictionary) - QUARTER:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
-------------------------------------------------------------------------------------------------------------------------
Spark                                                119            120           1          8.8         113.3       1.0X
Comet (Scan)                                         116            117           0          9.0         110.7       1.0X
Comet (Scan + Exec)                                  340            341           1          3.1         324.3       0.3X

Automated benchmark run by dfbench

parthchandra · 2026-01-08T18:21:02Z

Sub-day truncations (MICROSECOND, MILLISECOND, SECOND, MINUTE, HOUR) are timezone-independent

The timezone Asia/Kathmandu is UTC+5:45; the above statement might not be strictly accurate.

…cate # Conflicts: # spark/src/test/scala/org/apache/spark/sql/benchmark/CometDatetimeExpressionBenchmark.scala

…cate

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

andygrove · 2026-02-10T19:27:43Z

Sub-day truncations (MICROSECOND, MILLISECOND, SECOND, MINUTE, HOUR) are timezone-independent

The timezone Asia/Kathmandu is UTC+5:45; the above statement might not be strictly accurate.

Hmm..good point, thank you. I think I will close this PR for now and revisit another time.

andygrove · 2026-02-10T19:29:28Z

Sub-day truncations (MICROSECOND, MILLISECOND, SECOND, MINUTE, HOUR) are timezone-independent

The timezone Asia/Kathmandu is UTC+5:45; the above statement might not be strictly accurate.

Hmm..good point, thank you. I think I will close this PR for now and revisit another time.

I filed #3477

andygrove added 2 commits December 27, 2025 08:40

perf: improve performance of timestamp truncate

d9abf64

clippy

75a53cb

andygrove marked this pull request as draft January 2, 2026 19:03

upmerge

8075a6c

andygrove changed the title ~~perf: improve performance of timestamp truncate~~ perf: improve performance of timestamp truncate for some formats Jan 7, 2026

improve benchmark

6c98518

andygrove marked this pull request as ready for review January 7, 2026 15:52

andygrove requested review from comphead, mbutrovich and parthchandra January 7, 2026 15:54

Merge remote-tracking branch 'origin/main' into faster-timestamp-trun…

3bdd0f3

…cate # Conflicts: # spark/src/test/scala/org/apache/spark/sql/benchmark/CometDatetimeExpressionBenchmark.scala

andygrove marked this pull request as draft January 30, 2026 01:48

andygrove and others added 2 commits February 10, 2026 12:25

Merge remote-tracking branch 'apache/main' into faster-timestamp-trun…

ee87199

…cate

fix: add missing closing brace in unixTimestampFromDateBenchmark

2653613

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

andygrove closed this Feb 10, 2026

andygrove mentioned this pull request Feb 10, 2026

perf: optimize timestamp truncation for sub-day formats #3477

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

perf: improve performance of timestamp truncate for some formats#2996

perf: improve performance of timestamp truncate for some formats#2996
andygrove wants to merge 7 commits intoapache:mainfrom
andygrove:faster-timestamp-truncate

andygrove commented Dec 27, 2025 •

edited

Loading

Uh oh!

codecov-commenter commented Dec 27, 2025 •

edited

Loading

Uh oh!

sqlbenchmark commented Jan 2, 2026

Uh oh!

parthchandra commented Jan 8, 2026

Uh oh!

andygrove commented Feb 10, 2026

Uh oh!

andygrove commented Feb 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

andygrove commented Dec 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

Criterion Benchmarks

Microbenchmarks

What changes are included in this PR?

How are these changes tested?

Uh oh!

codecov-commenter commented Dec 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sqlbenchmark commented Jan 2, 2026

Comet Microbenchmark Results: CometDatetimeExpressionBenchmark

Benchmark Results

Uh oh!

parthchandra commented Jan 8, 2026

Uh oh!

andygrove commented Feb 10, 2026

Uh oh!

andygrove commented Feb 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

andygrove commented Dec 27, 2025 •

edited

Loading

codecov-commenter commented Dec 27, 2025 •

edited

Loading