Proposal: keploy validate command for pre-test validation (before keploy test)

[Syedowais312]

Related issue: #3340

Hi maintainers 👋

I opened this discussion to get early feedback on a proposed CLI command:

[Syedowais312]

The goal is to allow users to validate test cases and mock files before running keploy test, especially for CI/CD pipelines and large test suites.

---

❓ Problem

Currently, issues such as:

• malformed YAML
• missing or incorrect schema fields
• invalid HTTP status codes
• missing or orphaned mocks

are only discovered after running the full test suite, which:

• wastes CI time
• makes debugging harder
• leads to confusing or late failures

---

💡 Proposed Solution

Introduce a new CLI command:

keploy validate

Example usage

keploy validate
keploy validate --test-set test-1
keploy validate --format json
keploy validate --strict

---

What it would validate

• YAML: syntax, indentation, duplicate keys
• Schema: required fields (version, kind, name, spec), correct types
• HTTP tests: method, URL format, status codes (100–599)
• gRPC tests: valid method and path format
• Mocks: referenced mock files exist and are valid
• Cross-references: duplicate test names, orphaned mocks
• Best practices: warnings for large response bodies, hardcoded timestamps

---

Sample Output (example)

✖ Validation Failed

Errors:
test-set-2/tests/test-3.yaml:15
  ERROR: Invalid HTTP status code '6xx' – must be between 100–599

Warnings:
test-set-3/tests/test-2.yaml
  WARNING: Response body exceeds 1MB – consider using body matching

Summary: 25 test cases validated, 2 errors, 1 warning
Exit code: 1

---

Use Cases

• CI/CD: fail fast before expensive test runs
• Test maintenance: validate after manual edits or migrations
• Onboarding: help new users verify their setup quickly
• Version upgrades: ensure test suites remain compatible with new Keploy versions

---

Feedback Requested

Before implementing this, I’d love guidance on:

• Does this align with Keploy’s CLI roadmap?
• Should this be:
  • a standalone command (keploy validate), or
  • part of keploy test (e.g. --validate-only)?
• Is the proposed scope reasonable for an initial version?

Happy to iterate based on feedback. Thanks!

[Akshat005Chaudhary]

Contributor Feedback

Just to clarify my position: I’m a beginner contributor and still getting familiar with Keploy’s internals.

From my understanding so far, the proposal aligns well with Keploy’s focus on developer productivity and test reliability. Failing fast on malformed or incompatible test assets seems particularly valuable, especially in CI/CD contexts. A standalone keploy validate command also feels cleaner and avoids overloading keploy test.

For an initial version, it might make sense to prioritize:

• YAML syntax and schema validation
• Missing or orphaned mocks
• HTTP/gRPC method, URL/path, and status code checks

Best-practice checks (e.g., large response bodies, hardcoded timestamps) could remain non-blocking and be gated behind --strict or introduced incrementally.

As a convenience feature (possibly later), --validate could exist as a flag on keploy test, acting as a thin wrapper that internally invokes keploy validate.

Overall

This seems like it could meaningfully improve the developer experience, particularly for onboarding and automated pipelines. I’d appreciate guidance on whether this direction makes sense and what a reasonable beginner-friendly scope would be.

[officialasishkumar]

Hi @Syedowais312 ,

thanks for the proposal. We currently don't need this.

[Syedowais312]

Thank you for taking the time to review this.

I understand, could you please let me know when or under what circumstances this might be needed in the future? I'm interested in working on it when it becomes relevant.