Skip to content

experimental: import ai-dev-kit skills into experimental/ directory#73

Open
jamesbroadhead wants to merge 16 commits into
mainfrom
experimental-aidevkit
Open

experimental: import ai-dev-kit skills into experimental/ directory#73
jamesbroadhead wants to merge 16 commits into
mainfrom
experimental-aidevkit

Conversation

@jamesbroadhead
Copy link
Copy Markdown

@jamesbroadhead jamesbroadhead commented May 12, 2026

Summary

Adds an experimental/ directory containing 23 agent skills from databricks-solutions/ai-dev-kit databricks-skills/, imported as a snapshot on a best-effort basis. Excluded: databricks-model-serving (TODO #1b — different surface than stable, heavy MCP coupling) and databricks-spark-declarative-pipelines (TODO #5 — different surface than stable databricks-pipelines). databricks-lakebase-provisioned is not in the upstream experimental branch either, so absent here.

The manifest now exposes them under a new top-level experimental_skills map alongside the existing skills map so consumers can skip them by default.

Paired with databricks/cli#5243 which teaches databricks experimental aitools skills install to:

  • skip experimental_skills by default,
  • install all of them with --experimental,
  • install one by name (with --experimental required and -experimental suffix on the install dir).

Source

Import is from 9c7a5b3
on the appkit-on-experimental branch of jamesbroadhead/ai-dev-kit — the head of a-d-k PR #533,
which targets a-d-k's experimental branch (not main). 1 commit ahead of origin/experimental at import time.

Landing dependency: a-d-k PR #533 should merge before this PR so the first periodic sync from a-d-k doesn't conflict. The rename (databricks-app-pythondatabricks-apps-python) is load-bearing — it's what prevents a 3rd skill-name collision with d-a-s's stable databricks-apps.

Direction caveat — please read

In the Apr 28 thread (Slack link), Dustin's stated plan was to move databricks-agent-skills skills into ai-dev-kit's experimental branch as defaults. This PR goes the other direction (a-d-k content → d-a-s/experimental). I don't see any d-a-s commits from Dustin yet, and the timing has slipped. Opening this so we have something concrete to iterate on — happy to drop it if the original direction is still preferred.

TODOs / caveats for iteration

  1. Name collisions. Resolved in this PR:

    • 1a. databricks-jobs — merged into stable. Imported the comprehensive reference content from a-d-k's databricks-jobs skill into skills/databricks-jobs/, bumping version to 0.2.0. The merged skill keeps stable's scaffolding workflow + parent: databricks-core hierarchy + Codex agents/openai.yaml + compatibility note, and adds the experimental's full task-types reference (9 types), trigger types (6), notifications/health/retries/queues, and 7 worked end-to-end examples. Layered structure: SKILL.md as overview + four reference files (task-types.md, triggers-schedules.md, notifications-monitoring.md, examples.md). Cleanups during merge: dropped trigger-spam description, normalized /Workspace/Users/user@example.com/... paths to /Workspace/Shared/.... The experimental copy is removed; the install-time -experimental suffix collision-handling on the cli side becomes unnecessary for jobs.
    • 1b. databricks-model-serving — dropped from this PR. After a deep compare, the two skills cover almost entirely different surfaces: stable is ops-focused (manage existing endpoints via CLI: serving-endpoints create/get/query/update-config/build-logs/put-ai-gateway/get-permissions/..., AI Gateway, traffic config, app integration via databricks-apps skill); experimental is dev-focused (build & ship MLflow models / GenAI agents: autolog → mlflow.pyfunc.log_modeldatabricks.agents.deploy() → query, with full Classical ML / Custom PyFunc / ResponsesAgent + LangGraph / UCFunctionToolkit / VectorSearchRetrieverTool coverage). Near-zero content overlap. Experimental version also has heavy MCP-tool dependency (60+ refs to ai-dev-kit's manage_serving_endpoint, manage_workspace_files, manage_jobs, manage_job_runs, execute_code that don't exist in the d-a-s/databricks experimental aitools flow). Removed experimental/databricks-model-serving/ from this PR; manifest regenerated. Follow-up: port the high-value dev-side content into the stable skill — classical-ml autolog patterns (mlflow.{sklearn,xgboost,lightgbm,pytorch,tensorflow,spark}.autolog()), Custom PyFunc signatures, ResponsesAgent pattern with the create_text_output_item helper-method gotcha, UCFunctionToolkit + VectorSearchRetrieverTool with resource passthrough for auth, the Foundation Model API endpoint table. Strip MCP refs; replace with CLI/SDK equivalents. Owners: @databricks/eng-apps-devex (per CODEOWNERS).
  2. CODEOWNERS for experimental/ — open. Discussing access model offline.

  3. No sync mechanism with upstream a-d-k. Resolved with a paired RFC. Two-part plan:

    • Pre-lock (this PR): periodic manual re-syncs from upstream ai-dev-kit into experimental/. Documented in experimental/README.md.
    • Post-lock (follow-up): invert the direction. a-d-k becomes the consumer; databricks-skills/imported/ in a-d-k is a git subtree of this repo's experimental/. RFC PR opened against a-d-k: RFC: subtree-sync skills from databricks-agent-skills/experimental databricks-solutions/ai-dev-kit#530 (draft). To make subtree work, d-a-s needs to publish an experimental-only branch via git subtree split --prefix=experimental after every push to main — that's a small workflow to add here in a follow-up PR. A one-shot preview branch experimental-only-preview was pushed to this repo to enable the RFC demo and should be deleted once the auto-publish workflow lands.
  4. No agent metadata. Resolved. Imported skills install fine on Codex CLI — the missing agents/openai.yaml was a cosmetic gap, not a functional blocker (skill files still get copied; only the marketplace UI metadata is absent). scripts/skills.py now auto-generates agents/openai.yaml + copies shared assets for each experimental skill on generate, using SKILL.md frontmatter as the source. Stubs are only written when missing, so upstream a-d-k can override by shipping its own files in the skill. The auto-generated names are titlecased from the skill key — most look good (Databricks Iceberg, Databricks Genie); a few degrade gracefully (Databricks Aibi Dashboards). Refining those is a follow-up.

  5. databricks-pipelines was deliberately excluded. Resolved. a-d-k doesn't ship a databricks-pipelines skill under that name, but it does ship databricks-spark-declarative-pipelines covering the same product. After a deep compare, that experimental version covers a different surface than stable: scaffolding (databricks pipelines init + bundle/MCP workflow A/B/C), DLT migration guide, language-selection rules, per-language performance reference. The stable skill covers feature reference (decision tree, common traps, format options, fine-grained per-feature × per-language refs). Partial overlap; experimental's DAB-coupled workflow is the exact concern Dustin flagged in the Apr 28 Slack thread for demo-generator flows. Removed experimental/databricks-spark-declarative-pipelines/ from this PR (24 experimental skills now). Follow-up TODO (post-merge): port the high-value pieces into stable skills/databricks-pipelines/ — DLT migration guide, workflow A/B/C decision matrix, per-language performance reference, language-selection rules. Strip MCP-tool refs. Owners: @lennartkats-db / @camielstee-db (per CODEOWNERS).

  6. spark-python-data-source naming exception. Kept as-is. The skill is about the OSS Apache Spark 4+ PySpark DataSource API (building custom connector libraries), not a Databricks product — only lightly flavored with Databricks idioms. The convention break is acceptable given the content.

  7. Versioning. Resolved. Bumped the extract_version_from_skill fallback in scripts/skills.py from 0.0.00.0.1 so the manifest never reports 0.0.0 (which some tools treat as "unset"). Applies to all 24 experimental skills + the stable databricks-dabs skill, which currently have no explicit version: in their SKILL.md frontmatter. Skills with an explicit version are unchanged. The change is sync-safe: when upstream a-d-k eventually adds version fields, those win; until then, the manifest reports the floor.

  8. installed_dir for experimental skills. All experimental skills install under a -experimental suffix. Every experimental skill installs to ~/.claude/skills/<name>-experimental/ regardless of whether there's a stable skill with the colliding base name. Implemented in databricks/cli#5243 via a new SourceName field on SkillMeta: the manifest key (and install dir) carry the -experimental suffix; SourceName preserves the unsuffixed name for fetching from experimental/<name>/ in this repo. Users see at a glance which installed skills are experimental.

  9. Excluded a-d-k content. Confirmed scope. Excluded: TEMPLATE/ (template, not a skill), install_skills.sh + install_genie_code_skills.py (a-d-k's installers — we use the cli installer instead), databricks-builder-app/ (a Python app for a-d-k's builder UI), databricks-mcp-server/ (the a-d-k MCP server — separate concern from skills), databricks-tools-core/ (Python lib used by a-d-k tooling — no experimental skill references it), hooks/hooks.json (a-d-k plugin lifecycle hooks tied to ${CLAUDE_PLUGIN_ROOT}/.claude-plugin/setup.sh/check_update.sh — plugin-specific, not skill content), plus top-level repo metadata (.github/, LICENSE.md, README.md, VERSION, install.{sh,ps1}, etc.). Verified no experimental skill cross-references any excluded path.

  10. README placement. Verified. experimental/README.md retains the adapted a-d-k skill list with a top warning block; the root README.md has an "Experimental Skills" section with an install-by-name example. Three concrete fixes applied during the verification pass: (a) dropped the stale databricks-model-serving collision example since that skill was removed from the PR, (b) install commands updated to include the -experimental suffix + flag per TODO Add CODEOWNERS #8's resolution, (c) added a short note in experimental/README.md explaining why the in-repo dir names don't carry the suffix (it's added at install time).

Test plan

  • python3 scripts/skills.py generate regenerates the manifest cleanly.
  • python3 scripts/skills.py validate passes.
  • CI green on this branch.
  • Manual: databricks experimental aitools skills install (no flag) installs only stable skills.
  • Manual: databricks experimental aitools skills install --experimental installs both.
  • Manual: databricks experimental aitools skills install databricks-iceberg errors because it's experimental.
  • Manual: databricks experimental aitools skills install databricks-iceberg --experimental installs that one skill.

This pull request and its description were written by Claude.

@jamesbroadhead jamesbroadhead force-pushed the experimental-aidevkit branch from 2035bab to 2a45b83 Compare May 12, 2026 16:06
@jamesbroadhead jamesbroadhead requested a review from fjakobs as a code owner May 12, 2026 16:09
@jamesbroadhead jamesbroadhead force-pushed the experimental-aidevkit branch from 8387654 to a0fbb21 Compare May 12, 2026 16:14
jamesbroadhead added a commit to databricks/cli that referenced this pull request May 12, 2026
The default DATABRICKS_SKILLS_REF pin is a release tag that pre-dates
the experimental_skills manifest section (see
databricks/databricks-agent-skills#73). Users who pass --experimental
against that ref today silently get no experimental skills installed.

Log a Warnf at install time pointing them at the env var override
(=main, or a future release that includes the section).

Helper: manifestHasExperimental(), unit-tested in source_test.go.

Co-authored-by: Isaac
@jamesbroadhead jamesbroadhead changed the title experimental: import ai-dev-kit skills as best-effort skills experimental: import ai-dev-kit skills into experimental/ director May 12, 2026
@jamesbroadhead jamesbroadhead changed the title experimental: import ai-dev-kit skills into experimental/ director experimental: import ai-dev-kit skills into experimental/ directory May 12, 2026
jamesbroadhead added a commit that referenced this pull request May 12, 2026
Replaces the previous import (a-d-k commit 2228c3e on add_appkit) with the
head of a-d-k PR #533 (commit 9c7a5b3 on appkit-on-experimental), which
targets a-d-k's experimental branch.

Changes:
- Refresh 23 experimental skill directories from the new source.
- Drop databricks-lakebase-provisioned — removed on a-d-k experimental.
- databricks-apps-python: rename + SKILL.md now leads with AppKit
  (TypeScript + React SDK) and demotes Python frameworks to alternatives;
  6-mcp-approach.md replaced with 6-cli-approach.md.
- databricks-lakebase-autoscale/references/connection-patterns.md: change
  placeholder `user:password` to `<user>:<password>` so the secret scanner
  doesn't flag the doc-only example. Cosmetic only.
- Continue to exclude databricks-model-serving and
  databricks-spark-declarative-pipelines (PR #73 TODOs #1b and #5).
- Regenerate manifest.json and agents/openai.yaml stubs via
  scripts/skills.py generate.
- Update experimental/README.md provenance section with the new SHA,
  branch, and divergence notes.

Co-authored-by: Isaac
jamesbroadhead added a commit that referenced this pull request May 12, 2026
Merges the comprehensive jobs reference content from
experimental/databricks-jobs/ into skills/databricks-jobs/ and removes
the experimental copy.

What's new in stable databricks-jobs (v0.2.0):
- Full task-types reference (9 types: notebook, spark_python,
  python_wheel, sql, dbt, pipeline, spark_jar, run_job, for_each)
- All 6 trigger types with examples (cron, periodic, file_arrival,
  table_update, continuous, manual) + combining + pause/resume
- Notifications + health rules + retries + timeouts + queues
- 7 end-to-end worked examples (ETL, warehouse refresh, event-driven,
  ML training, multi-env, streaming, cross-job orchestration)
- run_if conditions, environments (serverless deps), permissions

What's retained from the prior stable skill:
- parent: databricks-core hierarchy
- Compatibility note + version metadata (bumped 0.1.0 → 0.2.0)
- Scaffolding workflow (databricks bundle init + CLAUDE.md/AGENTS.md
  template + project structure)
- Unit testing + development workflow sections
- agents/openai.yaml + assets/

Cleanups during the merge:
- Replaced the trigger-spam description with a terse one
- Normalized hard-coded /Workspace/Users/user@example.com/ paths in
  the imported reference files to /Workspace/Shared/

scripts/skills.py: updated SKILL_METADATA description for jobs to
reflect the broader scope. Manifest regenerated; experimental count
drops from 23 to 22.

Resolves PR #73 TODO #1a.

Co-authored-by: Isaac
Adds an experimental/ directory containing the 26 agent skills from
databricks-solutions/ai-dev-kit. These are imported as a snapshot on a
best-effort basis — they are not officially supported skills and follow a
looser contract than skills/ (no agents/openai.yaml, no shared-asset sync,
no SKILL_METADATA gate).

The manifest now exposes them under a new top-level experimental_skills
map so consumers can distinguish them from stable skills and skip them by
default. scripts/skills.py handles the new directory; the existing
generate / validate flow is unchanged for stable skills.

Co-authored-by: Isaac
Owners are the top contributors to databricks-solutions/ai-dev-kit
(>=10 commits at import time). The cross-org team
@databricks-solutions/ai-dev-kit-maintainers is the canonical owner;
this line can be replaced with that team handle if the team gets
write access to this repo.

Co-authored-by: Isaac
Adds a Provenance & sync model section to experimental/README.md:
- Transition phase: source of truth is upstream ai-dev-kit; this dir
  gets periodic manual re-syncs.
- Post-lock: source of truth is this repo; ai-dev-kit's
  databricks-skills/ becomes read-only.

Co-authored-by: Isaac
Adds two helpers to scripts/skills.py that run as part of `generate`:

- `ensure_experimental_codex_metadata` copies the shared
  assets/databricks.{svg,png} into each experimental skill (mirroring
  what stable skills get via sync_assets).
- `synthesize_openai_yaml` writes agents/openai.yaml from each
  experimental skill's SKILL.md frontmatter (display_name from the
  skill name, short_description from the first sentence of the
  frontmatter description, brand_color and icon paths fixed).

Both run only when the destination file is absent, so upstream
ai-dev-kit can override by shipping its own openai.yaml or assets.

This closes the cosmetic gap that experimental skills installed into
Codex CLI would render without an icon or marketplace metadata.

Co-authored-by: Isaac
Same pattern as databricks-model-serving: the experimental version
covers a different surface than the stable databricks-pipelines skill
(workflow / scaffolding / DLT-migration / per-language performance vs
feature reference / decision tree / common traps / format options).
The DAB-coupled scaffolding workflow is also the specific concern
Dustin flagged in the Apr 28 Slack thread for demo-generator flows.

Removed experimental/databricks-spark-declarative-pipelines/;
manifest regenerated (24 experimental skills). Follow-up TODO: port
the high-value pieces (DLT migration guide, workflow A/B/C decision
matrix, per-language performance reference, language-selection rules)
into skills/databricks-pipelines/, stripping MCP-tool refs.

Co-authored-by: Isaac
Adds the actual a-d-k commit hash (2228c3e on the add_appkit branch,
5 commits ahead of origin/main at import time) along with a note
about the local deltas vs public main. Surfaces the key one: the
databricks-app-python -> databricks-apps-python rename hadn't merged
upstream, and pulling from the renamed version is what avoids a 3rd
skill-name collision with d-a-s's stable databricks-apps.

Co-authored-by: Isaac
extract_version_from_skill() falls back to a synthetic version when a
skill's SKILL.md frontmatter has no version: field. The previous
fallback was 0.0.0, which several install tools treat as "unset"
rather than "first release".

Bump to 0.0.1. Affects all 24 experimental skills (imported from
ai-dev-kit without versions) plus the stable databricks-dabs skill.
Skills with an explicit version are unchanged.

Co-authored-by: Isaac
- experimental/README.md: install examples now use the -experimental
  suffix on the skill name + the --experimental flag (matching the
  install-path behaviour landed in databricks/cli#5243). Adds a short
  note explaining why the in-repo dir name and the install dir name
  differ.
- experimental/README.md: drop databricks-model-serving from the
  collision example (it was removed from this PR earlier).
- experimental/README.md: update the (also available as stable skill)
  note for databricks-jobs to point at the open TODO #1a.
- Root README: clarify the suffixed install name in the by-name install
  example.

Co-authored-by: Isaac
Replaces the previous import (a-d-k commit 2228c3e on add_appkit) with the
head of a-d-k PR #533 (commit 9c7a5b3 on appkit-on-experimental), which
targets a-d-k's experimental branch.

Changes:
- Refresh 23 experimental skill directories from the new source.
- Drop databricks-lakebase-provisioned — removed on a-d-k experimental.
- databricks-apps-python: rename + SKILL.md now leads with AppKit
  (TypeScript + React SDK) and demotes Python frameworks to alternatives;
  6-mcp-approach.md replaced with 6-cli-approach.md.
- databricks-lakebase-autoscale/references/connection-patterns.md: change
  placeholder `user:password` to `<user>:<password>` so the secret scanner
  doesn't flag the doc-only example. Cosmetic only.
- Continue to exclude databricks-model-serving and
  databricks-spark-declarative-pipelines (PR #73 TODOs #1b and #5).
- Regenerate manifest.json and agents/openai.yaml stubs via
  scripts/skills.py generate.
- Update experimental/README.md provenance section with the new SHA,
  branch, and divergence notes.

Co-authored-by: Isaac
The previous regex-only parser in extract_description_from_skill()
captured the YAML block-scalar indicator (`>-`) verbatim, so any SKILL.md
that wrote `description: >-\n  multi-line content` produced a manifest
entry of `">-"`. The new ai-dev-kit import (PR #533) brought two such
files — databricks-dbsql and databricks-execution-compute — which
landed corrupted descriptions in manifest.json and corrupted
short_description / default_prompt in agents/openai.yaml.

Walk the frontmatter line by line: if the value is a block-scalar
indicator (|, |-, |+, >, >-, >+), aggregate the indented continuation
lines (folded with spaces for `>`-style, newlines for `|`-style).

Regenerate manifest.json and the two affected agents/openai.yaml stubs.

Co-authored-by: Isaac
Replaces the hand-rolled block-scalar walker (added one commit ago) with
PyYAML's safe_load. PyYAML's default SafeLoader is pure-Python — no C
extension required — and handles every YAML edge case for free instead
of reimplementing them.

Side-benefit: also fixes a second latent bug. The regex parser stripped
the outer YAML quotes but left inner `\"` escapes intact as literal
backslash-quote characters, so descriptions like
`"... mentions \"switch workspace\"..."` ended up in manifest.json
with the backslashes preserved. yaml.safe_load resolves these
correctly. Regenerated manifest reflects the fix for databricks-config.

Co-authored-by: Isaac
Merges the comprehensive jobs reference content from
experimental/databricks-jobs/ into skills/databricks-jobs/ and removes
the experimental copy.

What's new in stable databricks-jobs (v0.2.0):
- Full task-types reference (9 types: notebook, spark_python,
  python_wheel, sql, dbt, pipeline, spark_jar, run_job, for_each)
- All 6 trigger types with examples (cron, periodic, file_arrival,
  table_update, continuous, manual) + combining + pause/resume
- Notifications + health rules + retries + timeouts + queues
- 7 end-to-end worked examples (ETL, warehouse refresh, event-driven,
  ML training, multi-env, streaming, cross-job orchestration)
- run_if conditions, environments (serverless deps), permissions

What's retained from the prior stable skill:
- parent: databricks-core hierarchy
- Compatibility note + version metadata (bumped 0.1.0 → 0.2.0)
- Scaffolding workflow (databricks bundle init + CLAUDE.md/AGENTS.md
  template + project structure)
- Unit testing + development workflow sections
- agents/openai.yaml + assets/

Cleanups during the merge:
- Replaced the trigger-spam description with a terse one
- Normalized hard-coded /Workspace/Users/user@example.com/ paths in
  the imported reference files to /Workspace/Shared/

scripts/skills.py: updated SKILL_METADATA description for jobs to
reflect the broader scope. Manifest regenerated; experimental count
drops from 23 to 22.

Resolves PR #73 TODO #1a.

Co-authored-by: Isaac
PR #533 has merged into upstream a-d-k experimental. The
databricks-skills/ tree is byte-identical between the previous
import SHA (9c7a5b3) and the merge commit (7b07f18) — only
install.{sh,ps1} changed, which we don't import. README updated
to point at the now-authoritative branch + SHA and drop the "one
commit ahead of origin/experimental" caveat.

Co-authored-by: Isaac
@jamesbroadhead jamesbroadhead force-pushed the experimental-aidevkit branch from 50467c3 to 5719156 Compare May 15, 2026 09:45
@jamesbroadhead
Copy link
Copy Markdown
Author

Stable ↔ experimental skill overlap

Surfacing the overlap explicitly per Quentin's question. There are 6 stable/experimental pairs with non-trivial topical overlap; the remaining 16 experimental skills cover surfaces with no stable equivalent.

# Stable Experimental (this PR) Topic Overlap Stable strengths (kept) Experimental strengths (potential merge candidates) Direction
1 databricks-apps (196L SKILL + 4 refs) databricks-apps-python (259L SKILL + 6 refs + 4 examples) Building Databricks Apps Both scaffold apps on the Apps platform; both cover Lakebase integration, model serving, deployment AppKit-first (TypeScript/React) — SQL queries, tRPC, smoke tests, Playwright selector rules, proto-first contracts, appkit lint cast rules, AppKit version pinning, Genie agent workflow Python framework menu (Streamlit/Dash/Gradio/Flask/FastAPI/Reflex), explicit OAuth authorization patterns, Foundation Model API examples (fm-minimal-chat, fm-parallel-calls, fm-structured-outputs), app-resources schema Keep both. Disjoint primary audience (TS vs Python). Port FM-API examples + Python-framework matrix into stable's references/other-frameworks.md post-merge.
2 databricks-dabs (39L SKILL + 5 refs, 450L total) databricks-bundles (324L SKILL + SDP/alerts files, 497L total) Declarative Automation Bundles Both cover databricks.yml, resources, targets, deploy/validate, SDP pipelines, SQL alerts Layered references (bundle-structure / deploy-and-run / resource-permissions / alerts / sdp-pipelines); naming convention <name>.<type>.yml; --strict validate guidance; permissions matrix Self-contained single-file primer with full databricks.yml skeleton (variables, dev/prod targets); similar SDP + alerts content but flatter Stable supersedes. Drop the experimental copy, or trim it to a thin pointer. The duplication is the cleanest example of what Quentin flagged.
3 databricks-lakebase (300L SKILL + 3 refs, 871L total) databricks-lakebase-autoscale (232L SKILL + 5 refs, 1146L total) Lakebase Postgres (Autoscaling) Both cover projects/branches/endpoints, connectivity, OAuth token refresh, reverse-ETL via synced tables, compute sizing Resource-hierarchy diagram, capacity planning + sizing details, Data API / PostgREST, app-integration via databricks-apps, compliance matrix (HIPAA/C5/TISAX) Per-area refs (projects / branches / computes / connection-patterns / reverse-etl); 354L connection-patterns deep-dive; field-mask update-* CLI examples; region matrix (AWS + Azure-beta) Stable supersedes on hierarchy/synced-tables. Port the deeper connection-patterns content + field-mask CLI usage into stable's references/connectivity.md.
4 databricks-core (CLI/auth/profile entrypoint) databricks-config Workspace/profile management Both cover databricks auth, ~/.databrickscfg, profile switching, OAuth + PAT login "NEVER auto-select a profile" rule, CLI version-check + install flow, REST-API fallback for sandboxed envs, Claude Code separate-shell guidance, parent-skill index Plain cheatsheet of auth describe / config get / auth login / configure Stable supersedes. Experimental is a strict subset — drop.
5 databricks-core (CLI-first) databricks-python-sdk SDK + Connect quickstart Both touch Databricks Connect, CLI version, profile selection Profile-selection rules, CLI install, REST fallback databricks-sdk Python usage, WorkspaceClient REST patterns, Connect quickstart, doc-index reference Mostly disjoint. Keep as SDK-focused complement, or fold the SDK section into a sibling stable skill — does not collide with databricks-core's CLI scope.
6 databricks-serverless-migration databricks-execution-compute Compute selection / execution Both mention serverless vs classic / Databricks Connect Migration lifecycle (Ingest/Analyze/Test/Validate); blocker detection (RDDs, DBFS, HMS, streaming triggers, init scripts); Spark Connect fixes 3-mode decision matrix (Connect / Serverless Job / Interactive Cluster); compute + warehouse CRUD cheatsheet; cold-start expectations Keep both. Different angles — migration vs day-to-day execution choice. Cross-link from stable.

No stable equivalent (16 skills — no overlap)

databricks-agent-bricks, databricks-ai-functions, databricks-aibi-dashboards, databricks-dbsql, databricks-docs, databricks-genie, databricks-iceberg, databricks-metric-views, databricks-mlflow-evaluation, databricks-spark-structured-streaming, databricks-synthetic-data-gen, databricks-unity-catalog, databricks-unstructured-pdf-generation, databricks-vector-search, databricks-zerobus-ingest, spark-python-data-source.

TL;DR

Three pairs are real duplicates that should converge over time: dabs/bundles (drop experimental), lakebase/lakebase-autoscale (drop experimental, port connection-patterns content first), core/config (drop experimental).

Two pairs are complementary and worth keeping side-by-side: apps/apps-python (TS vs Python audiences), serverless-migration/execution-compute (migration vs runtime selection).

One pair is mostly disjoint: core/python-sdk (CLI vs SDK).

Intentional for this PR per the rationale in the description — port adk mostly intact, then resolve the duplicates as a series of single-skill PRs against stable, with the experimental copies removed in the same change. Happy to start that series if folks agree on the deltas above.

This comment was written by Claude.

Removes three experimental skills that duplicate stable equivalents
without adding net new content:

- databricks-bundles → use stable `databricks-dabs`
- databricks-lakebase-autoscale → use stable `databricks-lakebase`
- databricks-config → use stable `databricks-core`

Cross-references in the surviving experimental skills (apps-python,
python-sdk, zerobus-ingest, synthetic-data-gen, 4-deployment.md) now
point at the stable names by bare name — matching the convention
already used in stable skills' "Related Skills" / "Product Skills"
sections.

`experimental/README.md` records the removals and points readers at
the stable replacements. Manifest regenerated — experimental skill
count drops from 22 to 19.

Constraint per James: root must not depend on experimental, but
experimental may depend on root. Honored — all new references go in
that direction, and no stable skill content was changed by this
commit.

Co-authored-by: Isaac
User-walkthrough testing of experimental/README.md surfaced 13
broken cross-reference links in 9 SKILL.md files, all in
"Related Skills" sections. Each pointed at
../<name>/SKILL.md for a skill that doesn't exist in
experimental/ — they reference earlier-resolved name collisions
(databricks-jobs, databricks-model-serving) or
experimental-only content that was excluded from this import
(databricks-spark-declarative-pipelines).

Replacement strategy mirrors c4daa14: use the bare skill name
without a link, matching the convention stable skills use in
their own cross-references. Targets:

- databricks-spark-declarative-pipelines (excluded per TODO #5)
  → databricks-pipelines (stable analogue)
- databricks-model-serving (dropped per TODO #1b) →
  databricks-model-serving (bare; stable exists at same name)
- databricks-jobs (merged per TODO #1a) → databricks-jobs
  (bare; stable exists at same name)

Also rewrites the stale "collision handling for jobs and
model-serving" sentence in experimental/README.md — those
collisions were resolved via merge + drop respectively in this
PR, not via the suffix mechanism the sentence implied.

Co-authored-by: Isaac
…ix subskill example

User-walkthrough testing of the root README surfaced three doc
issues:

1. "Available Skills" listed only databricks-apps. The repo ships
   8 stable skills — added all of them with one-line descriptions.

2. Install section mixed two valid command forms without
   explanation: `databricks experimental aitools install`
   (agent-detection wrapper, used in the Claude Code section) and
   `databricks experimental aitools skills install` (skills-only
   subcommand, used in the Experimental section). Settled on the
   wrapper as the primary form and explained the subcommand as
   the power-user path with --experimental support.

3. The "Adding New Skills" example used a name (ai-databricks-apps)
   and a prose mechanism ("First, load the base ... skill") that
   do not match the actual subskill pattern stable skills use.
   Rewritten to demonstrate the real mechanism: parent:
   <skill-name> in frontmatter, matching the CLAUDE.md convention
   and what every stable subskill does today.

Co-authored-by: Isaac
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant