Format benchmarks

[sarahchen6]

What Does This Do

WIP

Motivation

Additional Notes

Contributor Checklist

• Format the title according to the contribution guidelines
• Assign the type: and (comp: or inst:) labels in addition to any other useful labels
• Avoid using close, fix, or any linking keywords when referencing an issue
  Use solves instead, and assign the PR milestone to the issue
• Update the CODEOWNERS file on source file addition, migration, or deletion
• Update public documentation with any new configuration flags or behaviors

Jira ticket: [PROJ-IDENT]

Note: Once your PR is ready to merge, add it to the merge queue by commenting /merge. /merge -c cancels the queue request. /merge -f --reason "reason" skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.

[sarahchen6]

should not need to append reports. each comparison-baseline-vs-candidate.md is just in its respective benchmark directory

[faydef]

To follow up on our discussion from yesterday
The steps I see to make it cleaner are the following:

• Convert the results using bp-analyzer since now we have the conversion scripts in place
• Merge the runs that need to be merged (like the 10 runs from decapo into one) like done in here https://github.com/DataDog/dd-trace-java/pull/10550/changes#:~:text=benchmark_analyzer%20merge%20%5C,%24%7Bk6_files%5B%40%5D%7D%22
• compare the results to output in md format using bp_analyzer
  example:

  benchmark_analyzer compare pairwise \
    --baseline='{"baseline_or_candidate":"baseline"}' \
    --candidate='{"baseline_or_candidate":"candidate"}' \
    --format='$BP_FORMAT' \
    --outpath="$ARTIFACTS_DIR/comparison-baseline-vs-candidate.md" \
    "$ARTIFACTS_DIR"/baseline*.converted.json "$ARTIFACTS_DIR"/candidate*.converted.json

Adapt the blob pattern as needed

• you can skip the setup for pr commenting using the bp-tools image and piping the md output to the pr-commenter like
  cat "$ARTIFACTS_DIR/comparison-baseline-vs-candidate.md" | pr-commenter --for-repo="$CI_PROJECT_NAME" --for-pr="$CI_COMMIT_REF_NAME" --header='$BP_HEADER' --on-duplicate=replace
  (if you want to append multiple files as comment you can use --on-duplicate=minimize)
  Let me know if anything is still unclear 🙏