Repository navigation
Add acceptance tests for aitools list - #7161
Merged
Merged
Conversation
Contributor
|
Commit: Regression test reportSelected 4 changed tests (limit=20, 0 not selected) 4 tests: regression=2 unreleased=0 coverage=2 failing=0
TestAccept/cmd/aitools/list/latest: branch ✅ | main (66fce77) ❌ | latest (v1.22.0) ❌main (66fce77): latest (v1.22.0): TestAccept/cmd/aitools/list/latest-lookup-fails: branch ✅ | main (66fce77) ❌ | latest (v1.22.0) ❌main (66fce77): latest (v1.22.0): |
No acceptance test ran `aitools list`, and the aitools tests set DATABRICKS_SKILLS_REF, so the compat manifest and the latest skills release lookup were never covered. Add DATABRICKS_SKILLS_API_URL so the test server can replace api.github.com, and cover an explicit ref, a pinned compat entry, latest with a successful lookup, and latest with a 403 from the lookup. Co-authored-by: Isaac <no-reply@databricks.com>
- Reuse DATABRICKS_CLI_GITHUB_API_URL (from libs/versioncheck) for the
latest skills release lookup instead of a new env var, and drop the
unit test that only covered the new var.
- Run acceptance/cmd/aitools in the test-exp-aitools CI target.
- Commit the compat manifests, move shared setup and the jq allow-list
(now with summary) to script.prepare, and serve the skills manifest
from one {ref} stub.
- Assert the lookup failure with musterr, and pin the compat cache
settings so the shell environment does not change the result.
Co-authored-by: Isaac <no-reply@databricks.com>
- Export versioncheck.GitHubAPIURL and use it for the skills release lookup, so DATABRICKS_CLI_GITHUB_API_URL behaves the same for both. - Use top-level alternation in the test-exp-aitools pattern; the parenthesized form selected only TestAccept/apps. - Clear DATABRICKS_SKILLS_REF in the list tests, drop the unused ~/.claude setup, and explain why the lookup failure is expected. Co-authored-by: Isaac <no-reply@databricks.com>
- GitHubAPIURL removes a trailing slash, and its doc comment names the env var; keep the original gitHubAPIURLEnv comment. - Keep acceptance/cmd/aitools out of test-exp-aitools sources, so a change there still runs task test on Windows. - Drop the text-output step in explicit-ref (agent config env vars can change it) and the setup_list_test wrapper, and fix comments. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
rugpanov
force-pushed
the
aitools-list-acceptance
branch
from
October 22, 2026 12:08
4aebbe8 to
a504050
Compare
Collaborator
Integration test reportCommit: a504050
Top 3 slowest tests (at least 2 minutes):
|
simonfaltum
approved these changes
Oct 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
databricks aitools listinacceptance/cmd/aitools/list/:explicit-ref:DATABRICKS_SKILLS_REFis set. The CLI fetches only the skills manifest.pinned: the compat entry pins0.2.32. The CLI does not call the latest-release API.latest: the compat entry islatest, and the API returnsv0.2.33. The CLI lists the skills from that release.latest-lookup-fails: the compat entry islatest, and the API returns 403. The command fails withresolve latest skills release: HTTP 403 fetching latest skills release.DATABRICKS_CLI_GITHUB_API_URL, the existing override fromlibs/versioncheck.versioncheck.GitHubAPIURLis now exported, and both lookups use it.versioncheckalready owns this variable, so the helper stays there. The helper now also removes a trailing/.-runpattern oftest-exp-aitoolsnow uses top-level alternation and includescmd/aitools.TestAccept/(apps|experimental/aitools)selected onlyapps, becausego testdoes not split on a/inside parentheses. So a change inlibs/aitoolsnow runs the new tests, and the 10experimental/aitoolsacceptance tests also run in this target now. Thesourcesdo not change: a change inacceptance/cmd/aitoolsstill runstask test, which includes Windows.Why
No acceptance test ran
aitools list. The aitools acceptance tests setDATABRICKS_SKILLS_REF, so they never used the compat manifest or the latest-release lookup. Because of this, the CLI tests did not find the 1.22.0 regression from #7069. The VS Code unit tests found it in databricks/databricks-vscode#2269, and #7159 mitigates it.The
latest-lookup-failsoutput records the current behavior. Whenaitools listgets an embedded fallback, the test must change, and the diff shows the new behavior.Each test copies a committed
compat-manifest.jsonintoDATABRICKS_CACHE_DIR. So the result does not depend on the embedded or the remote compat manifest.test.tomlalso setsDATABRICKS_SKILLS_REF,DATABRICKS_FORCE_EMBEDDED_COMPAT, andDATABRICKS_CACHE_ENABLED, so values from the shell do not change the result.Tests
go test ./acceptance -run 'TestAccept/apps|TestAccept/cmd/aitools|TestAccept/experimental/aitools': all pass, also withDATABRICKS_SKILLS_REF=mainandCLAUDE_CONFIG_DIRset in the shell.go test ./libs/versioncheck/... ./libs/aitools/... ./cmd/aitools/... ./cmd/version/... ./libs/clicompat/..., with the newTestGitHubAPIURL(default, override, and trailing/).golangci-linton the changed packages: 0 issues.Known limitations
aitools list. Plugin installs use the same lookup and have no acceptance test.release,skills, andsummary, because agent detection depends on the binaries onPATH.test-exp-aitoolshas no Windows runner. So a PR that changes onlylibs/aitoolsorcmd/aitoolsruns the new tests on Linux and macOS only.DATABRICKS_CLI_GITHUB_API_URLnow also redirects the skills lookup. A mirror that has onlydatabricks/clireleases makesaitools listfail.This pull request and its description were written by Isaac.