Added LLM-based website docs validation section to CONTRIBUTING.md #194
Added LLM-based website docs validation section to CONTRIBUTING.md #194
Conversation
Adds a new section in CONTRIBUTING.md with an LLM prompt to validate Kotlin/Android code snippets in the web documentation against the SDK source code. This validation step is now integrated into the release process (Step 5) to ensure documentation accuracy before each release. Also renumbers subsequent release steps and clarifies that SDK version PRs are required even without API changes.
sacOO7
changed the title
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughRewrote CONTRIBUTING.md to center on publishing to Maven Central: added OSSRH/GPG prerequisites, a publishing command with CI in-memory signing notes, reorganized the Release Process and PR/docs workflow, and added a "Validate website doc snippets" verification procedure. (49 words) Changes
Sequence Diagram(s)sequenceDiagram
participant Dev as Developer
participant CI as CI Pipeline
participant Site as Website/Docs
participant Maven as Maven Central (OSSRH)
participant SDK as SDK (validation runtime)
Dev->>Site: Update docs/snippets
Dev->>Repo: Open PR (includes doc + code changes)
CI->>Repo: Run tests & build
CI->>SDK: Run "Validate website doc snippets" (execute Kotlin/Android snippets)
SDK-->>CI: Validation report (pass/fail, details)
CI->>Maven: Execute `publishAndReleaseToMavenCentral` with in-memory signing
Maven-->>CI: Publish result (success/failure)
CI-->>Dev: Report results (validation + publish status)
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
sacOO7
changed the title
Code Coverage
|
|
@coderabbitai review |
✅ Actions performedReview triggered.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
CONTRIBUTING.md (1)
239-240: Consider making the validation step unconditional to align with the mandatory docs PR in Step 6.Step 5 is guarded by a condition ("If you've deprecated…or are uncertain"), but Step 6 mandates a docs PR for every release regardless of API changes. A developer who is confident their internal-only changes have no public API impact could reasonably skip Step 5 yet must still open a Step 6 PR. Running the validation unconditionally before every release would ensure the docs PR in Step 6 always starts from a verified baseline.
✏️ Proposed fix
-5. If you've deprecated any public methods or properties, made changes to public interfaces, or are uncertain about the impact of your updates, run the [Validate website docs snippet changes](`#validate-website-docs-snippet-changes`) task locally. This will verify that the `kotlin` or `android` code snippets in the web documentation (https://github.com/ably/docs) are accurate and aligned with the current SDK source. Review the generated report and address any issues it identifies. +5. Run the [Validate website docs snippet changes](`#validate-website-docs-snippet-changes`) task locally to verify that the `kotlin` and `android` code snippets in the web documentation (https://github.com/ably/docs) are accurate and aligned with the current SDK source. Review the generated report and address any issues it identifies.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@CONTRIBUTING.md` around lines 239 - 240, Update CONTRIBUTING.md so Step 5 is unconditional: remove the conditional phrasing ("If you’ve deprecated…or are uncertain") and state that the "Validate website docs snippet changes" task must be run for every release before creating the docs PR in Step 6; ensure the text explains that this validation verifies kotlin/android snippets in the web docs and must be run even if there are no public API changes so the docs PR in Step 6 always starts from a verified baseline.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@CONTRIBUTING.md`:
- Around line 196-197: The two bullets reference non-Kotlin examples; replace
the "Missing keywords (e.g., `new` for constructors)" hint with Kotlin-relevant
cases (e.g., omitted `val`/`var` for declarations, missing `suspend` on
coroutine functions, incorrect companion object accessors) and replace the
"PascalCase for C# properties, camelCase for JavaScript" line with Kotlin naming
guidance (classes/objects use UpperCamelCase; functions, properties and local
variables use lowerCamelCase), ensuring the wording and examples match Kotlin
idioms.
- Line 181: The opening fenced code block that contains the prose prompt
starting with "Verify all `kotlin` and `android` annotated code snippets ..." is
missing a language identifier which triggers MD040; update that fence to include
a language (e.g., add "text" or "markdown" after the opening ```), so the block
becomes a fenced code block with a language specifier to satisfy markdownlint.
---
Nitpick comments:
In `@CONTRIBUTING.md`:
- Around line 239-240: Update CONTRIBUTING.md so Step 5 is unconditional: remove
the conditional phrasing ("If you’ve deprecated…or are uncertain") and state
that the "Validate website docs snippet changes" task must be run for every
release before creating the docs PR in Step 6; ensure the text explains that
this validation verifies kotlin/android snippets in the web docs and must be run
even if there are no public API changes so the docs PR in Step 6 always starts
from a verified baseline.
f1307ef to
6e5f72e
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
CONTRIBUTING.md (2)
215-215: Specify an output path for the report file.
chat_kotlin_api_verification_report.mdhas no path qualifier, so the LLM will create it in whatever directory happens to be its working context. Anchoring it to{SDK_PATH}(or{DOCS_PATH}) makes the output deterministic and easy to find.✏️ Proposed fix
-Create/update a markdown report file `chat_kotlin_api_verification_report.md` with all findings. +Create/update a markdown report file at `{SDK_PATH}/chat_kotlin_api_verification_report.md` with all findings.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@CONTRIBUTING.md` at line 215, The directive to "Create/update a markdown report file `chat_kotlin_api_verification_report.md`" lacks a deterministic path; update the CONTRIBUTING.md line to specify an anchored output path (e.g., prefix the filename with {SDK_PATH}/ or {DOCS_PATH}/) so the generated report is always created in the expected location; reference the exact filename `chat_kotlin_api_verification_report.md` in your change and include the chosen placeholder ({SDK_PATH} or {DOCS_PATH}) to make the output path explicit.
177-177: Outdated LLM model references."Claude Opus" and "GPT-4" are now several generations behind. Current top-tier models include Claude Opus 4.5 and GPT-5.2, with Gemini 3 Pro also ranking among the most capable. Consider updating the examples.
✏️ Suggested update
-> This prompt should be run with the most powerful LLM available to you (e.g. Claude Opus, GPT-4, etc.) for the best results. +> This prompt should be run with the most powerful LLM available to you (e.g. Claude Opus 4.5, GPT-5, Gemini 3 Pro, etc.) for the best results.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@CONTRIBUTING.md` at line 177, Update the sentence "This prompt should be run with the most powerful LLM available to you (e.g. Claude Opus, GPT-4, etc.) for the best results." in CONTRIBUTING.md to cite current top-tier models instead of outdated ones; replace the example list with modern models such as "Claude Opus 4.5, GPT-5.2, Gemini 3 Pro" (or similar current equivalents) so the guidance stays accurate, ensuring the edited line still reads clearly and preserves the original intent.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@CONTRIBUTING.md`:
- Line 186: The sentence that reads "code blocks with the `kotlin` and `android`
annotation in `.mdx` files" is ambiguous; change it to use "or" so it clearly
matches code fences annotated with either `kotlin` or `android` (e.g., replace
"`kotlin` and `android`" with "`kotlin` or `android`" in the line describing
search of `.mdx` code blocks) to ensure the filter selects blocks annotated with
either language.
---
Nitpick comments:
In `@CONTRIBUTING.md`:
- Line 215: The directive to "Create/update a markdown report file
`chat_kotlin_api_verification_report.md`" lacks a deterministic path; update the
CONTRIBUTING.md line to specify an anchored output path (e.g., prefix the
filename with {SDK_PATH}/ or {DOCS_PATH}/) so the generated report is always
created in the expected location; reference the exact filename
`chat_kotlin_api_verification_report.md` in your change and include the chosen
placeholder ({SDK_PATH} or {DOCS_PATH}) to make the output path explicit.
- Line 177: Update the sentence "This prompt should be run with the most
powerful LLM available to you (e.g. Claude Opus, GPT-4, etc.) for the best
results." in CONTRIBUTING.md to cite current top-tier models instead of outdated
ones; replace the example list with modern models such as "Claude Opus 4.5,
GPT-5.2, Gemini 3 Pro" (or similar current equivalents) so the guidance stays
accurate, ensuring the edited line still reads clearly and preserves the
original intent.
6e5f72e to
84d7427
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
CONTRIBUTING.md (1)
182-182:⚠️ Potential issue | 🟡 MinorUnresolved:
andshould beorin the annotation filter (both occurrences).A single code fence carries exactly one annotation, so the conjunction
andwould logically require a block to simultaneously carry bothkotlinandandroid— which is impossible — and would match nothing. The intended union filter needsor.✏️ Proposed fix
-Verify all `kotlin` and `android` annotated code snippets in `.mdx` files located at `{DOCS_PATH}` against the `ably-chat-kotlin` source code repository at `{SDK_PATH}`. +Verify all `kotlin` or `android` annotated code snippets in `.mdx` files located at `{DOCS_PATH}` against the `ably-chat-kotlin` source code repository at `{SDK_PATH}`.-1. **Find all code snippets**: Search for all code blocks with the `kotlin` and `android` annotation in `.mdx` files. +1. **Find all code snippets**: Search for all code blocks with the `kotlin` or `android` annotation in `.mdx` files.Also applies to: 186-186
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@CONTRIBUTING.md` at line 182, Replace the incorrect conjunction "and" with "or" in the annotation filter text that reads "Verify all `kotlin` and `android` annotated code snippets..." (both occurrences) so the sentence correctly describes a union filter (i.e., `kotlin` OR `android`) for code fences; update both identical lines to use "or" to ensure the documentation matches the intended annotation logic.
ℹ️ Review info
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Disabled knowledge base sources:
- Jira integration is disabled
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (1)
CONTRIBUTING.md
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@CONTRIBUTING.md`:
- Line 215: The report filename chat_kotlin_api_verification_report.md is
ambiguous and may be emitted into {SDK_PATH} and accidentally committed; update
the instruction to write the report to an explicit path (for example
{SDK_PATH}/chat_kotlin_api_verification_report.md or
{DOCS_PATH}/chat_kotlin_api_verification_report.md) or add
chat_kotlin_api_verification_report.md to .gitignore as a fallback so the file
cannot be staged unintentionally.
---
Duplicate comments:
In `@CONTRIBUTING.md`:
- Line 182: Replace the incorrect conjunction "and" with "or" in the annotation
filter text that reads "Verify all `kotlin` and `android` annotated code
snippets..." (both occurrences) so the sentence correctly describes a union
filter (i.e., `kotlin` OR `android`) for code fences; update both identical
lines to use "or" to ensure the documentation matches the intended annotation
logic.
…dated release process steps
sacOO7
force-pushed
the
feature/doc-update-release-process
branch
from
84d7427 to
46a297d
Compare
There was a problem hiding this comment.
♻️ Duplicate comments (2)
CONTRIBUTING.md (2)
182-186:⚠️ Potential issue | 🟡 MinorUse
orinstead ofandfor snippet language filters.Line 182 and Line 186 still read as if a fence must be both
kotlinandandroid. That can be interpreted too literally and miss snippets; this should be a union filter.Suggested wording fix
-Verify all `kotlin` and `android` annotated code snippets in `.mdx` files located at `{DOCS_PATH}/src/pages/docs/chat` against the `ably-chat-kotlin` source code repository at `{SDK_PATH}`. +Verify all `kotlin` or `android` annotated code snippets in `.mdx` files located at `{DOCS_PATH}/src/pages/docs/chat` against the `ably-chat-kotlin` source code repository at `{SDK_PATH}`. -1. **Find all code snippets**: Search for all code blocks with the `kotlin` and `android` annotation in `.mdx` files. +1. **Find all code snippets**: Search for all code blocks with the `kotlin` or `android` annotation in `.mdx` files.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@CONTRIBUTING.md` around lines 182 - 186, Update the verification wording that currently requires code fences to be annotated with both `kotlin` and `android` to instead use a union filter; replace instances of "kotlin and android" with "kotlin or android" (e.g., the verification step that says "Find all code snippets: Search for all code blocks with the `kotlin` and `android` annotation") so the search captures snippets annotated with either `kotlin` or `android`.
215-215:⚠️ Potential issue | 🟡 MinorAnchor report output to an explicit path.
Line 215 still leaves the report location ambiguous. Please write to a concrete path to avoid accidental generation/staging in the SDK repo root.
Suggested wording fix
-Create/update a markdown report file `chat_kotlin_api_verification_report.md` with all findings. +Create/update a markdown report file at `{DOCS_PATH}/chat_kotlin_api_verification_report.md` with all findings.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@CONTRIBUTING.md` at line 215, Update the ambiguous instruction that creates/updates "chat_kotlin_api_verification_report.md" to write to a concrete path (e.g., tools/reports/chat_kotlin_api_verification_report.md or docs/reports/chat_kotlin_api_verification_report.md) so the report is never generated/staged in the repo root; change the line that mentions the report file name to explicitly include the chosen directory (reference: the string "chat_kotlin_api_verification_report.md") and ensure the wording says "Create/update the markdown report at <explicit/path>/chat_kotlin_api_verification_report.md with all findings."
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Duplicate comments:
In `@CONTRIBUTING.md`:
- Around line 182-186: Update the verification wording that currently requires
code fences to be annotated with both `kotlin` and `android` to instead use a
union filter; replace instances of "kotlin and android" with "kotlin or android"
(e.g., the verification step that says "Find all code snippets: Search for all
code blocks with the `kotlin` and `android` annotation") so the search captures
snippets annotated with either `kotlin` or `android`.
- Line 215: Update the ambiguous instruction that creates/updates
"chat_kotlin_api_verification_report.md" to write to a concrete path (e.g.,
tools/reports/chat_kotlin_api_verification_report.md or
docs/reports/chat_kotlin_api_verification_report.md) so the report is never
generated/staged in the repo root; change the line that mentions the report file
name to explicitly include the chosen directory (reference: the string
"chat_kotlin_api_verification_report.md") and ensure the wording says
"Create/update the markdown report at
<explicit/path>/chat_kotlin_api_verification_report.md with all findings."
ℹ️ Review info
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Disabled knowledge base sources:
- Jira integration is disabled
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (1)
CONTRIBUTING.md
There was a problem hiding this comment.
Like idea very much! May I suggest a bit different implementation, we created https://github.com/ably/sdk-workspace repo recently. It early draft. But the main idea that in this repo we keep common scripts for sdk development. We require some predefine structure:
sdk-workspace-
- ably-java
- ably-chat-kotlin
- docs
- ...
because we know the structure we can predefine prompts, and use them as claude code skills. Take a look at the existing skills. I think this way it will be easier to run this skills we can also share context.
|
|
||
| 1. **Find all code snippets**: Search for all code blocks with the `kotlin` and `android` annotation in `.mdx` files. | ||
|
|
||
| 2. **Understand SDK structure**: Analyze the SDK source code to understand: |
There was a problem hiding this comment.
Maybe it will be easier for LLM to understand context from dokka-generated docs? I believe context will be much smaller and it definitely will only have public methods, maybe we should instruct to run doc generation task and check docs (in xml or md format to avoid additional html noise)
There was a problem hiding this comment.
Dokka documentation would only supply context about method signatures, and navigating XML/HTML files would likely take a significant amount of time for LLMs. This approach hasn’t been tested yet like the current prompt, but we could consider adding it to Claude skills as part of https://ably.atlassian.net/browse/DX-837
This was already suggested by |
sacOO7
force-pushed
the
feature/doc-update-release-process
branch
from
046d43f to
049941c
Compare
There was a problem hiding this comment.
♻️ Duplicate comments (2)
CONTRIBUTING.md (2)
182-186:⚠️ Potential issue | 🟡 MinorUse
orinstead ofandfor snippet language filters.On Line 182 and Line 186, "
kotlinandandroid" can be interpreted as requiring both annotations at once. That risks missing valid snippets. Use "kotlinorandroid" in both places for unambiguous union matching.✏️ Proposed fix
-Verify all `kotlin` and `android` annotated code snippets in `.mdx` files located at `{DOCS_PATH}/src/pages/docs/chat` against the `ably-chat-kotlin` source code repository at `{SDK_PATH}`. +Verify all `kotlin` or `android` annotated code snippets in `.mdx` files located at `{DOCS_PATH}/src/pages/docs/chat` against the `ably-chat-kotlin` source code repository at `{SDK_PATH}`. ... -1. **Find all code snippets**: Search for all code blocks with the `kotlin` and `android` annotation in `.mdx` files. +1. **Find all code snippets**: Search for all code blocks with the `kotlin` or `android` annotation in `.mdx` files.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@CONTRIBUTING.md` around lines 182 - 186, Replace the phrase "kotlin and android" with "kotlin or android" in the verification steps so the language filter matches either annotation instead of requiring both; locate the two occurrences of the exact string "kotlin and android" in the CONTRIBUTING.md verification paragraph and update them to "kotlin or android" to ensure union matching for code snippet detection.
215-215:⚠️ Potential issue | 🟡 MinorMake the report output path explicit.
On Line 215,
chat_kotlin_api_verification_report.mdhas no directory prefix, so output location depends on the LLM/tool working directory. Please anchor it to{DOCS_PATH}(or{SDK_PATH}) to avoid accidental placement/commit noise.✏️ Proposed fix
-Create/update a markdown report file `chat_kotlin_api_verification_report.md` with all findings. +Create/update a markdown report file `{DOCS_PATH}/chat_kotlin_api_verification_report.md` with all findings.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@CONTRIBUTING.md` at line 215, Update the report creation instruction so the report path is anchored to a known directory: replace the bare filename chat_kotlin_api_verification_report.md with an explicit path such as {DOCS_PATH}/chat_kotlin_api_verification_report.md (or {SDK_PATH}/chat_kotlin_api_verification_report.md) in CONTRIBUTING.md; ensure any commands or examples that reference chat_kotlin_api_verification_report.md are updated to use the chosen anchored variable so output is deterministic and not dependent on the current working directory.
🧹 Nitpick comments (1)
CONTRIBUTING.md (1)
170-170: Tighten wording for readability.Line 170 reads a bit awkwardly with “in the setup/install guide, at minimum.” Consider simplifying for clarity in contributor instructions.
✏️ Suggested rewrite
-Any releases must be accompanied by a PR to bump the library install version in the setup/install guide, at minimum. +Every release must include a docs PR that bumps the library version in the setup/install guide.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@CONTRIBUTING.md` at line 170, The sentence "Any releases must be accompanied by a PR to bump the library install version in the setup/install guide, at minimum." is awkward; rephrase it for clarity by simplifying and removing "at minimum." Replace it with a concise instruction such as "Each release must include a PR that updates the library version in the setup/install guide," ensuring the contributor requirement remains clear and direct.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Duplicate comments:
In `@CONTRIBUTING.md`:
- Around line 182-186: Replace the phrase "kotlin and android" with "kotlin or
android" in the verification steps so the language filter matches either
annotation instead of requiring both; locate the two occurrences of the exact
string "kotlin and android" in the CONTRIBUTING.md verification paragraph and
update them to "kotlin or android" to ensure union matching for code snippet
detection.
- Line 215: Update the report creation instruction so the report path is
anchored to a known directory: replace the bare filename
chat_kotlin_api_verification_report.md with an explicit path such as
{DOCS_PATH}/chat_kotlin_api_verification_report.md (or
{SDK_PATH}/chat_kotlin_api_verification_report.md) in CONTRIBUTING.md; ensure
any commands or examples that reference chat_kotlin_api_verification_report.md
are updated to use the chosen anchored variable so output is deterministic and
not dependent on the current working directory.
---
Nitpick comments:
In `@CONTRIBUTING.md`:
- Line 170: The sentence "Any releases must be accompanied by a PR to bump the
library install version in the setup/install guide, at minimum." is awkward;
rephrase it for clarity by simplifying and removing "at minimum." Replace it
with a concise instruction such as "Each release must include a PR that updates
the library version in the setup/install guide," ensuring the contributor
requirement remains clear and direct.
ℹ️ Review info
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Disabled knowledge base sources:
- Jira integration is disabled
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (1)
CONTRIBUTING.md
sacOO7
force-pushed
the
feature/doc-update-release-process
branch
from
049941c to
a41bf81
Compare
3 participants
Summary by CodeRabbit