Add End-to-End tests for the Miden Node

[Dominik1999]

We need to define and run integration tests for the Miden Node.

[phklive]

Linked to:

• Add full docker-compose #93
• Expose metrics for each service #144
• Add healthchecking #145

[hackaugusto]

I think this issue is more about an end-to-end test (E2E) instead of the Rust's integration tests. Here are a few notes that I believe are important for these types of tests:

1. CI:
   1. The E2E only run after successful unit tests
   2. Logs should be aggressively collected and persisted by the CI
2. Tests:
   1. Have overridable timeouts for environment setup/teardown and test execution. default timeout for tests of around 30secs
   2. Can skip the teardown on test failure
   3. Must have automatic retries with a resetable test timeout configurable by test and disabled by default
3. Environment:
   1. Must be containerized

The rationale of the above is:

1. i. E2E tests are super slow, if the unit tests fail it doesn't make sense to waste CI time on the E2E tests
   ii. Debugging E2E tests from CI can be challenging, and the logs are the only tool we have in some cases (specially when tests fail because the CI machines are not that powerful).
2. i. When debugging/running a single test, the setup/teardown time will dominate the test runtime, this means these settings should be different. The test timeout is important to detect issues that are outside of the test runner, and to incentivize good test writting.
   ii. When debugging a test, it is useful to keep the environment around to inspect further.
   iii. Flakiness is a given, retries on CI will be needed. Retries should be disabled by default because otherwise super badly written tests can get in, this setting is just a last resort kinda of tool.
3. i. it is impossible to do environment cleanup properly without it (e.g. double fork creates a daemon and there is no reliable way of caughting it without a cgroup). Using containers also allows for parallel test execution (useful when working on different branches).

Ideally we would also have:

• The ability to run a new E2E test multiple times when it is introduced, say 5 times. To detect flakiness before merging a PR.
• Collect some metrics about the tests, to know what is slow/flaky and needs an eye to improve
• Test should be written with no assumption they have a clean slate. This would require a setup/teardown per test, which is super slow.

With the above said. I think the best approach here is to have a hand written test runner, that will setup the container environment first, and then call the tests (which can be a rust binary).

[phklive]

#232 (comment)

[phklive]

@bobbinth would be glad to have your view and input on end-to-end tests that we could perform on the miden node.

Thanks

[Dominik1999]

Is there something you need to close this issue? Are you actively working on it?

[phklive]

To be able to close this PR I need this PR to be merged, it has been approved it's a matter of hours before I can continue working on the end-to-end testing: #257