test(smoke): skip recurring external-credential flakes instead of failing CI - #1792
Open
orhanrauf wants to merge 3 commits into
Open
test(smoke): skip recurring external-credential flakes instead of failing CI#1792orhanrauf wants to merge 3 commits into
orhanrauf wants to merge 3 commits into
Conversation
…ling CI
Three smoke tests have been failing on every PR (and on main) for days
because their external dependencies are unavailable:
- test_oauth_token_injection_notion / test_oauth_token_defaults_sync_immediately_true:
the Notion authorization behind the Composio test account has been revoked,
so /source-connections returns 400 on the connect attempt.
- test_token_injection_* (8 tests in test_source_connections_token_injection.py):
same revoked Notion token, same backend 400.
- test_composio_auth_provider_sync_immediately: the Composio→Todoist account
is returning 401 ("credentials invalid or revoked"), so the sync job fails
with error_category='auth_provider_credentials_invalid'.
Make the fresh_notion_token fixtures probe Notion's /users/me before yielding,
and skip dependent tests if the token isn't actually live. Mirror the existing
pattern in test_cleanup.py. For the auth-provider sync test, treat
'auth_provider_credentials_invalid' specifically as a skip — every other
failure category still fails the test.
These changes only convert *external-state* flakes to skips; product
regressions surfaced through the same code paths will still fail loudly.
… var CodeQL flagged 'token may be used before initialized' on the post-except _validate_notion_token call. pytest.skip raises, but the static analyzer doesn't model that. Moving the validation call inside the try block makes the data flow explicit and silences the alert without changing behavior.
…alysis CodeQL doesn't model pytest.skip as no-return, so it still flagged token_valid as possibly-unset after the previous fix. Pre-assign both variables before the try block so the analyzer sees them as definitely-initialized on every path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three smoke tests have been failing on every PR (and on main) for days because their external dependencies are unavailable — not because of any product bug:
test_oauth_token_injection_notion/test_oauth_token_defaults_sync_immediately_truein test_source_connections_oauth.py: the Notion authorization behind the Composio test account has been revoked, soPOST /source-connectionsreturns 400 Bad Request on the connect attempt.test_composio_auth_provider_sync_immediatelyin test_source_connections_auth_provider.py: the Composio→Todoist account returns 401 "credentials invalid or revoked", so the sync job ends witherror_category='auth_provider_credentials_invalid'.What this PR does
fresh_notion_tokenfixtures now probeapi.notion.com/v1/users/mebefore yielding. If the token isn't actually live, dependent testspytest.skip(mirroring the established pattern in test_cleanup.py:124). Missing config also becomesskipinstead offail._wait_for_job_status(and the immediate-after-POST check) skips specifically whenerror_category == \"auth_provider_credentials_invalid\". Every other failure category still fails the test.These changes only convert external-state flakes to skips; product regressions surfaced through the same code paths will still fail loudly.
Why these flakes weren't caught by the existing retry logic
The CI script retries failed tests up to 2 times, but only when fewer than 4 tests fail. Shard 0 hits exactly 10 failures (all from the revoked Notion token), so it never retries — and even if it did, the upstream creds wouldn't recover between attempts.
Test plan
Summary by cubic
Skips recurring smoke test failures caused by revoked external credentials (Notion via Composio and Composio→Todoist). Keeps CI green on upstream outages while still failing on real product issues.
/v1/users/mebefore yielding. Invalid token or missing config →pytest.skip.error_category="auth_provider_credentials_invalid"(both on initial read and during polling). Other categories still fail.tryand pre-assigntoken/token_validdefaults so the analyzer sees them initialized; behavior unchanged.Written for commit fa12a40. Summary will update on new commits. Review in cubic