[denik]

Stack, 2 of 4. Based on #7130; next: #7143, then #7098.

Changes

Five mutators now read and edit the typed config directly instead of going through b.Config.Mutate:

• PrependWorkspacePrefix: the workspace.*_path fields.
• ApplyBundlePermissions: the permissions field of each supported resource.
• ExpandPipelineGlobPaths: pipeline libraries.
• ProcessStaticResources: lists resource keys from the typed Resources.
• ProcessRootIncludes: writes the expanded include list after the included files are merged, because merging reloads the typed config from the dynamic tree.

Split out of #7098 (#7098).

Why

This is a step toward removing dyn from the mutator pipeline. Mutators that edit typed config are simpler and don't need to convert between the two forms.

Intentional behavior notes

• (a) Invalid-permission-level warnings now show their source location (in databricks.yml:L:C). Before, ApplyBundlePermissions rebuilt the permissions without locations. One golden changed because of this.
• (b) ApplyBundlePermissions now walks resource names in sorted order. This only changes the order of overlap warnings.
• (c) Expanded include items, and pipeline libraries after glob expansion, now get the location of the original entry at the same index. Before, they had no location.
• (d) In typed config, a library with both notebook and file and an explicit notebook.path: "" looks the same as one with no notebook path.

Tests

• New unit tests for the edge cases: empty and unset permissions/libraries, library order, includes of included files, and resource keys for every resource type. They pass on both the old and the new code.
• The full TestAccept/bundle suite ran locally. The only golden diff is the one in validate/secret_scope_invalid_permission_level.

This pull request and its description were written by Isaac.

[github-actions]

Commit: 26b7cc42d | Workflow run

Regression test report

Selected 6 changed tests (limit=20, 0 not selected)

12 tests: regression=2 unreleased=0 coverage=10 failing=0

test	branch	main (16d722d)	latest (v1.22.0)
TestAccept/bundle/override/cluster_id_clears_new_cluster_reference/DMS=	✅	✅	✅
TestAccept/bundle/override/cluster_id_clears_new_cluster_reference/DMS=true	✅	✅	✅
TestAccept/bundle/override/preset_clears_forced_schedule_reference/DMS=	✅	✅	✅
TestAccept/bundle/override/preset_clears_forced_schedule_reference/DMS=true	✅	✅	✅
TestAccept/bundle/resource_deps/whole_map_reference_hides_entries/DMS=	✅	✅	✅
TestAccept/bundle/resource_deps/whole_map_reference_hides_entries/DMS=true	✅	✅	✅
TestAccept/bundle/validate/nulls/target_adds_null_keeps_it/DMS=	✅	✅	✅
TestAccept/bundle/validate/nulls/target_adds_null_keeps_it/DMS=true	✅	✅	✅
TestAccept/bundle/validate/required_struct_reference/DMS=	✅	✅	✅
TestAccept/bundle/validate/required_struct_reference/DMS=true	✅	✅	✅
TestAccept/bundle/validate/secret_scope_invalid_permission_level/DMS=	✅	❌	❌
TestAccept/bundle/validate/secret_scope_invalid_permission_level/DMS=true	✅	❌	❌

TestAccept/bundle/validate/secret_scope_invalid_permission_level/DMS=: branch ✅ | main (16d722d) ❌ | latest (v1.22.0) ❌

main (16d722d):

acceptance_test.go:1307: Diff:
        --- bundle/validate/secret_scope_invalid_permission_level/output.txt
        +++ /tmp/TestAcceptbundlevalidatesecret_scope_invalid_permission_level3148319721/002/output.txt
        @@ -2,11 +2,9 @@
         >>> [CLI] bundle validate
         Warning: invalid value "12" for enum field. Valid values are [READ WRITE MANAGE]
           at resources.secret_scopes.bad_levels.permissions[0].level
        -  in databricks.yml:10:18
         
         Warning: invalid value "fuzz_badlevel" for enum field. Valid values are [READ WRITE MANAGE]
           at resources.secret_scopes.bad_levels.permissions[1].level
        -  in databricks.yml:12:18
         
         Name: test-bundle
         Target: default
        @@ -19,11 +14,9 @@
         >>> [CLI] bundle validate --strict
         Warning: invalid value "12" for enum field. Valid values are [READ WRITE MANAGE]
           at resources.secret_scopes.bad_levels.permissions[0].level
        -  in databricks.yml:10:18
         
         Warning: invalid value "fuzz_badlevel" for enum field. Valid values are [READ WRITE MANAGE]
           at resources.secret_scopes.bad_levels.permissions[1].level
        -  in databricks.yml:12:18
         
         Name: test-bundle
         Target: default

latest (v1.22.0): same output as main (16d722d)

TestAccept/bundle/validate/secret_scope_invalid_permission_level/DMS=true: branch ✅ | main (16d722d) ❌ | latest (v1.22.0) ❌

main (16d722d):

acceptance_test.go:1307: Diff:
        --- bundle/validate/secret_scope_invalid_permission_level/output.txt
        +++ /tmp/TestAcceptbundlevalidatesecret_scope_invalid_permission_level1171386184/002/output.txt
        @@ -2,11 +2,9 @@
         >>> [CLI] bundle validate
         Warning: invalid value "12" for enum field. Valid values are [READ WRITE MANAGE]
           at resources.secret_scopes.bad_levels.permissions[0].level
        -  in databricks.yml:10:18
         
         Warning: invalid value "fuzz_badlevel" for enum field. Valid values are [READ WRITE MANAGE]
           at resources.secret_scopes.bad_levels.permissions[1].level
        -  in databricks.yml:12:18
         
         Name: test-bundle
         Target: default
        @@ -19,11 +14,9 @@
         >>> [CLI] bundle validate --strict
         Warning: invalid value "12" for enum field. Valid values are [READ WRITE MANAGE]
           at resources.secret_scopes.bad_levels.permissions[0].level
        -  in databricks.yml:10:18
         
         Warning: invalid value "fuzz_badlevel" for enum field. Valid values are [READ WRITE MANAGE]
           at resources.secret_scopes.bad_levels.permissions[1].level
        -  in databricks.yml:12:18
         
         Name: test-bundle
         Target: default

latest (v1.22.0):

acceptance_test.go:1307: Diff:
        --- bundle/validate/secret_scope_invalid_permission_level/output.txt
        +++ /tmp/TestAcceptbundlevalidatesecret_scope_invalid_permission_level3554799691/002/output.txt
        @@ -2,11 +2,9 @@
         >>> [CLI] bundle validate
         Warning: invalid value "12" for enum field. Valid values are [READ WRITE MANAGE]
           at resources.secret_scopes.bad_levels.permissions[0].level
        -  in databricks.yml:10:18
         
         Warning: invalid value "fuzz_badlevel" for enum field. Valid values are [READ WRITE MANAGE]
           at resources.secret_scopes.bad_levels.permissions[1].level
        -  in databricks.yml:12:18
         
         Name: test-bundle
         Target: default
        @@ -19,11 +14,9 @@
         >>> [CLI] bundle validate --strict
         Warning: invalid value "12" for enum field. Valid values are [READ WRITE MANAGE]
           at resources.secret_scopes.bad_levels.permissions[0].level
        -  in databricks.yml:10:18
         
         Warning: invalid value "fuzz_badlevel" for enum field. Valid values are [READ WRITE MANAGE]
           at resources.secret_scopes.bad_levels.permissions[1].level
        -  in databricks.yml:12:18
         
         Name: test-bundle
         Target: default
        
FAIL

[eng-dev-ecosystem-bot]

Integration test report

Commit: 26b7cc4

Run: 39759257887

	Env	✅​pass	🙈​skip	Time
✅​	aws linux-2core-8gb	281	21	4:39
✅​	aws-windows-latest-4core-16gb	283	19	3:26
✅​	azure linux-2core-8gb	280	21	4:26
✅​	azure-windows-latest-4core-16gb	282	19	3:20
✅​	gcp linux-2core-8gb	281	21	4:50
✅​	gcp-windows-latest-4core-16gb	283	19	3:21

Top 3 slowest tests (at least 2 minutes):

duration	env	testname
3:20	aws-windows-latest-4core-16gb	TestAccept
3:03	gcp-windows-latest-4core-16gb	TestAccept
2:46	azure-windows-latest-4core-16gb	TestAccept