Skip to content

test(smoke): skip recurring external-credential flakes instead of failing CI - #1792

Open
orhanrauf wants to merge 3 commits into
mainfrom
fix/skip-external-credential-flakes
Open

test(smoke): skip recurring external-credential flakes instead of failing CI#1792
orhanrauf wants to merge 3 commits into
mainfrom
fix/skip-external-credential-flakes

Conversation

@orhanrauf

@orhanrauf orhanrauf commented May 21, 2026

Copy link
Copy Markdown
Member

Summary

Three smoke tests have been failing on every PR (and on main) for days because their external dependencies are unavailable — not because of any product bug:

  • test_oauth_token_injection_notion / test_oauth_token_defaults_sync_immediately_true in test_source_connections_oauth.py: the Notion authorization behind the Composio test account has been revoked, so POST /source-connections returns 400 Bad Request on the connect attempt.
  • 8 tests in test_source_connections_token_injection.py: same revoked Notion token, same backend 400.
  • test_composio_auth_provider_sync_immediately in test_source_connections_auth_provider.py: the Composio→Todoist account returns 401 "credentials invalid or revoked", so the sync job ends with error_category='auth_provider_credentials_invalid'.

What this PR does

  • Both fresh_notion_token fixtures now probe api.notion.com/v1/users/me before yielding. If the token isn't actually live, dependent tests pytest.skip (mirroring the established pattern in test_cleanup.py:124). Missing config also becomes skip instead of fail.
  • _wait_for_job_status (and the immediate-after-POST check) skips specifically when error_category == \"auth_provider_credentials_invalid\". Every other failure category still fails the test.

These changes only convert external-state flakes to skips; product regressions surfaced through the same code paths will still fail loudly.

Why these flakes weren't caught by the existing retry logic

The CI script retries failed tests up to 2 times, but only when fewer than 4 tests fail. Shard 0 hits exactly 10 failures (all from the revoked Notion token), so it never retries — and even if it did, the upstream creds wouldn't recover between attempts.

Test plan

  • Changed-lines mypy gate: 100% on all three files (verified locally with `diff-quality --violations=mypy --fail-under=100`).
  • Changed-lines ruff gate: 100% on all three files.
  • Pre-commit hooks pass (ruff, import-linter, unit tests).
  • Verify CI passes — the Public API Test job should go green on this PR (the previously-failing tests will report as "skipped" rather than "failed").

Summary by cubic

Skips recurring smoke test failures caused by revoked external credentials (Notion via Composio and Composio→Todoist). Keeps CI green on upstream outages while still failing on real product issues.

  • Bug Fixes
    • Notion token fixtures probe /v1/users/me before yielding. Invalid token or missing config → pytest.skip.
    • Auth-provider sync skips when error_category="auth_provider_credentials_invalid" (both on initial read and during polling). Other categories still fail.
    • For CodeQL, keep token validation inside the try and pre-assign token/token_valid defaults so the analyzer sees them initialized; behavior unchanged.

Written for commit fa12a40. Summary will update on new commits. Review in cubic

…ling CI

Three smoke tests have been failing on every PR (and on main) for days
because their external dependencies are unavailable:

- test_oauth_token_injection_notion / test_oauth_token_defaults_sync_immediately_true:
  the Notion authorization behind the Composio test account has been revoked,
  so /source-connections returns 400 on the connect attempt.
- test_token_injection_* (8 tests in test_source_connections_token_injection.py):
  same revoked Notion token, same backend 400.
- test_composio_auth_provider_sync_immediately: the Composio→Todoist account
  is returning 401 ("credentials invalid or revoked"), so the sync job fails
  with error_category='auth_provider_credentials_invalid'.

Make the fresh_notion_token fixtures probe Notion's /users/me before yielding,
and skip dependent tests if the token isn't actually live. Mirror the existing
pattern in test_cleanup.py. For the auth-provider sync test, treat
'auth_provider_credentials_invalid' specifically as a skip — every other
failure category still fails the test.

These changes only convert *external-state* flakes to skips; product
regressions surfaced through the same code paths will still fail loudly.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 3 files

Re-trigger cubic

Comment thread backend/tests/e2e/smoke/test_source_connections_oauth.py Fixed
Comment thread backend/tests/e2e/smoke/test_source_connections_token_injection.py Fixed
… var

CodeQL flagged 'token may be used before initialized' on the post-except
_validate_notion_token call. pytest.skip raises, but the static analyzer
doesn't model that. Moving the validation call inside the try block makes
the data flow explicit and silences the alert without changing behavior.
Comment thread backend/tests/e2e/smoke/test_source_connections_oauth.py Fixed
Comment thread backend/tests/e2e/smoke/test_source_connections_token_injection.py Fixed
…alysis

CodeQL doesn't model pytest.skip as no-return, so it still flagged token_valid
as possibly-unset after the previous fix. Pre-assign both variables before the
try block so the analyzer sees them as definitely-initialized on every path.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants