Skip to content

air: show workspace-supported accelerators in compute help - #6993

Open
caroline-db wants to merge 3 commits into
mainfrom
air/list-compute-options
Open

caroline-db wants to merge 3 commits into
mainfrom
air/list-compute-options

Conversation

@caroline-db

@caroline-db caroline-db commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Changes

  • Fetch workspace compute options from the AI Training endpoint for databricks air run -h config.compute.
  • Display the options returned by the backend, including per-node accelerator counts and preview stages. The CLI does not apply its own accelerator filter.
  • Show an explicit message when the backend returns no options.
  • Bound the lookup to three seconds and fall back to the built-in list when the request is unavailable or fails. Unexpected failures also produce a warning.

Why

Static help can advertise accelerators that cannot be used in the selected workspace. The AI Training backend determines which options are supported by both the workspace and the Jobs AI Runtime task API, keeping that compatibility logic out of the CLI.

The fallback preserves useful help while the endpoint is unavailable or still being rolled out.

Tests

  • ./task test
  • ./task lint
  • ./task checks
  • Verified successful, empty, unavailable, and failed endpoint responses.
  • Verified backend filtering end to end.

@github-actions github-actions Bot added the AIR Databricks AI Runtime CLI label Oct 8, 2026
@caroline-db
caroline-db force-pushed the air/list-compute-options branch 2 times, most recently from 4464f3f to 6b2cfe5 Compare October 8, 2026 18:37
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: efefea9

Run: 39650353516

Env 🔄​flaky ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 281 16 4:13
🔄​ aws-windows-latest-4core-16gb 2 281 14 3:51
✅​ azure linux-2core-8gb 280 16 4:06
✅​ azure-windows-latest-4core-16gb 282 14 4:48
✅​ gcp linux-2core-8gb 281 16 4:19
🔄​ gcp-windows-latest-4core-16gb 2 281 14 3:52
Test Name aws-windows-latest-4core-16gb gcp-windows-latest-4core-16gb
🔄​ TestFsCpDir ✅​p 🔄​f
🔄​ TestFsCpDir/dbfs_to_uc-volumes ✅​p 🔄​f
🔄​ TestFilerRecursiveDelete 🔄​f ✅​p
🔄​ TestFilerRecursiveDelete/files 🔄​f ✅​p
Top 3 slowest tests (at least 2 minutes):
duration env testname
4:45 azure-windows-latest-4core-16gb TestAccept
3:31 aws-windows-latest-4core-16gb TestAccept
3:27 gcp-windows-latest-4core-16gb TestAccept

Comment thread cmd/air/runconfig.go Outdated
Comment thread cmd/air/run.go
defer cancel()
cmd.SetContext(root.SkipLoadBundle(root.SkipPrompt(ctx)))

if !cmdctx.HasWorkspaceClient(cmd.Context()) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make sure if any of these are RPC calls that they have retries

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yup, this uses the databricks sdk client.Do which retries SDK-classified transient failures, and there is a 3-second context timeout here

Comment thread cmd/air/run.go
}
}

options, err := listWorkspaceComputeOptions(cmd.Context(), cmdctx.WorkspaceClient(cmd.Context()))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

@caroline-db
caroline-db force-pushed the air/list-compute-options branch from c35e61a to 4af9e9a Compare October 15, 2026 23:04
@caroline-db
caroline-db force-pushed the air/list-compute-options branch from 4af9e9a to efefea9 Compare October 21, 2026 18:08
@github-actions

Copy link
Copy Markdown
Contributor

Commit: efefea990 | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

1 tests: regression=1 unreleased=0 coverage=0 failing=0

test branch main (bef6321) latest (v1.21.0)
TestAccept/air/config-help ✅ ❌ ❌
TestAccept/air/config-help: branch ✅ | main (bef6321) ❌ | latest (v1.21.0) ❌

main (bef6321):

acceptance_test.go:1307: Diff:
        --- air/config-help/output.txt
        +++ /tmp/TestAcceptairconfig-help1628838081/002/output.txt
        @@ -74,10 +74,6 @@
         config.compute
           Which accelerators to run on and how many.
         
        -  Supported accelerator types:
        -    GPU_1xH100 - 1 accelerator per node [Private Preview]
        -    GPU_8xH100 - 8 accelerators per node
        -
           Fields:
             num_accelerators  Total number of GPUs to allocate.
             accelerator_type  Which accelerator to run on, e.g. GPU_1xA10.
        
FAIL

latest (v1.21.0): same output as main (bef6321)

@caroline-db
caroline-db marked this pull request as ready for review October 21, 2026 19:21
@caroline-db
caroline-db requested review from a team as code owners October 21, 2026 19:21
@caroline-db
caroline-db requested a review from rugpanov October 21, 2026 19:21

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AIR Databricks AI Runtime CLI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants