Skip to content

Support dependencies for encrypted private ingredients - #3857

Merged
icanhasmath merged 2 commits into
masterfrom
marcg/private-ingredient-dependencies
Oct 21, 2026
Merged

icanhasmath merged 2 commits into
masterfrom
marcg/private-ingredient-dependencies

Conversation

@icanhasmath

Copy link
Copy Markdown
Contributor

state publish --build (encrypted private ingredients) had no practical way to set dependencies: --meta was read only after the wheel was built, and --edit overwrote whatever the meta file supplied. This adds dependency support from pyproject.toml, fixes --meta/--edit, and adds a command to export an ingredient's meta file for editing.

Changes

Dependencies from pyproject.toml (pyproject.go)

  • [project].dependencies become runtime dependencies on language/python packages. Names are normalized in one inventory API call (new model.FetchNormalizedNames).
  • Version ranges carry over; ~=1.16 → >=1.16,<2, ==4.2.* → >=4.2,<4.3.
  • OS and Python version markers become conditions, matching how the platform encodes PyPI ingredients: sys_platform/platform_system/os_name == "nt" → kernel/Windows|Linux|Darwin; python_version/python_full_version → language/python. python_version is major.minor only, so > "3.9" becomes >=3.10 rather than >3.9 (which would match 3.9.18).
  • Unsupported entries (or, !=, extra, platform_machine, URL references) are skipped with a notice pointing to --depend.
  • The wheel's METADATA records them as Requires-Dist.
  • Precedence: --depend* flags > meta file > pyproject.toml.

--meta with --build: the meta file is now read before the wheel is built, so its name/version/description are used (flags > meta file > pyproject.toml). Previously a meta file without a matching pyproject.toml failed with "missing metadata".

--edit fixes (publish.go)

  • Dependencies and authors from the meta file are kept instead of being overwritten by the previous revision's.
  • Inherited dependencies keep their type and conditions (both were dropped, so python's build/runtime/test entries collapsed into duplicates and Windows-only dependencies became unconditional).
  • builder/private-builder is added only if not already inherited.
  • Author/description changes on an existing version are rejected up front; the API error ("Cannot change authors or description without a new version") previously surfaced as Error received: <no value>.

New command: state export ingredient-meta <namespace>/<name>[@version] writes an existing ingredient's meta file (name, namespace, version, full dependency list with conditions) in the format --meta reads, to edit and pass back with state publish --edit --meta. Authors and description are omitted since they can't change on an existing version.

Crash fix: model.InventoryRequirementsToString dereferenced requirements without a version (present on some public ingredients' conditions, e.g. click). It now skips them.

Docs: internal/runners/publish/readme.md walks through publishing an encrypted ingredient with dependencies and editing them.

Testing

  • Unit tests for requirement/marker translation, edit inheritance (dependencies, types, conditions), private-builder dedupe, and pyproject.toml parsing.
  • TestPublishBuildEncrypted now includes a pyproject.toml dependency (six) that the package imports, so the test fails unless the dependency is published and installed. Not yet run under the integration-test account.
  • Manually against the platform (private/ActiveStateBE/python/test-00N-greeting, with an org-key contract via env):
    • publish with pyproject.toml dependencies, then install and import: dependency installed, sentinel printed;
    • conditional dependencies on Linux / Python 3.11: Linux-only and python_version > "3.9" dependencies installed, Windows-only and < "3.11" not;
    • --edit with a meta file, --edit inheriting dependencies, and the export → edit → --edit --meta → export round trip;
    • a new version without --edit publishes; the same version without --edit is refused.

Not addressed

  • On --edit, a meta file or pyproject.toml replaces the whole dependency list; there is no merge.
  • Other publish API errors still render as Error received: <no value>; the real message is only in the log.

🤖 Generated with Claude Code

`state publish --build` can now declare and edit the dependencies of an
encrypted private ingredient:

- Read [project].dependencies from pyproject.toml and publish them as
  runtime dependencies on language/python packages, with names normalized
  by the inventory API. Version ranges carry over (~= and ==X.* are
  expanded). OS and Python version markers become conditions
  (kernel/Windows|Linux|Darwin, language/python); unsupported markers are
  skipped with a notice. The wheel's METADATA records them as
  Requires-Dist.
- Read the --meta file before building, so a meta file's name, version
  and description feed the wheel (flags > meta file > pyproject.toml).
- Add `state export ingredient-meta <namespace>/<name>[@Version]`, which
  writes an existing ingredient's meta file for use with --edit --meta.

Fix --edit along the way: keep the meta file's dependencies and authors
instead of overwriting them with the previous revision's, keep inherited
dependency types and conditions, add the private-builder dependency only
once, and reject author/description changes on an existing version up
front instead of failing in the API. InventoryRequirementsToString no
longer panics on requirements without a version.

Add a readme walking through the publish and edit workflow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@icanhasmath
icanhasmath requested a review from MDrakos October 21, 2026 19:57

@MDrakos MDrakos left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes because --edit can now silently rewrite the wrong dependency list: one path drops the inherited deps, and the export round trip can land on a different version than the one exported.

Another note on tests: the new state export ingredient-meta command (FetchMeta), the up-front author/description rejection, and the nil-version crash fix in InventoryRequirementsToString have no tests, and the only pyproject-deps coverage is TestPublishBuildEncrypted, which per the description hasn't run yet. I think a unit test for the nil-requirement case and one for prepareEditRequest with a condition that carries requirements would be cheap and pin the crash fix.

2 inline comments below.

Comment thread internal/runners/publish/publish.go Outdated
if err != nil {
return errs.Wrap(err, "Could not read pyproject.toml dependencies")
}
reqVars.Dependencies = deps

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If every pyproject.toml entry gets skipped (say the only dep has a platform_machine marker), pyprojectDependencies returns nil, and on --edit this line wipes the deps prepareEditRequest just inherited. The publish then goes out with only builder/private-builder. That contradicts the readme ("If neither the meta file nor pyproject.toml lists any, the previous revision's dependencies are kept") and it's a regression from master, where --edit --build kept them.

Suggested change
reqVars.Dependencies = deps
if len(deps) > 0 {
reqVars.Dependencies = deps
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, fixed in 5dfae2a as suggested. The step now lives in applyPyprojectDependencies, which only replaces the dependencies when at least one survives translation. TestApplyPyprojectDependencies covers both no entries and all entries skipped, and fails with the guard removed.

Checked against the platform too: --edit --build on a version whose pyproject.toml only had a platform_machine dependency kept all of the previous revision's dependencies.

return captain.NewCommand(
"ingredient-meta",
locale.Tl("export_ingredient_meta_title", "Exporting Ingredient Meta File"),
locale.Tl("export_ingredient_meta_description", "Export the meta file of a published ingredient, to modify and pass to '[ACTIONABLE]state publish --edit --meta[/RESET]'"),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This points users at state publish --edit --meta without --build, but on that path the meta file's version is never used: Run only looks up params.Version, so it falls through to the latest version and prepareEditRequest overwrites reqVars.Version with it. So exporting mypkg@1.0.0 while 1.1.0 is latest and publishing it back rewrites 1.1.0's dependencies with 1.0.0's list, no error. imo the fix is in Run: look up cmp.Or(params.Version, reqVars.Version) instead of params.Version, so the exported version is what gets edited (the --build path already does this through generateEncryptedArtifact).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, fixed in 5dfae2a. Right after the meta file is read, params.Version = cmp.Or(params.Version, reqVars.Version), so the meta file's version drives the lookup, the edit and the published version, with --version still taking precedence. Doing it on params rather than only in the lookup keeps everything downstream in agreement, the same way --build already worked. As a side effect, publishing a new version from a meta file without --build no longer fails with "already exists".

Checked your scenario on the platform: with 0.0.1 and 0.0.2 published, exporting @0.0.1, changing a dependency and running state publish --edit --meta (no --build) created revision 2 of 0.0.1. 0.0.2 was unchanged.

- When every pyproject.toml dependency is skipped, keep the dependencies
  already in place (on --edit, those inherited from the previous
  revision) instead of replacing them with an empty list.
- Let a meta file's version select the version to publish or edit, like
  --version (which still takes precedence). An exported meta file passed
  back with `state publish --edit --meta` without --build was editing
  the latest version instead of the exported one.

Add tests for both, for the up-front author/description rejection on
--edit, for InventoryRequirementsToString with requirements missing a
version or comparator, and for inherited conditions that carry
requirements.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@icanhasmath

Copy link
Copy Markdown
Contributor Author

@MDrakos thanks, both inline issues are fixed in 5dfae2a (details in the threads).

On tests:

  • InventoryRequirementsToString nil case: TestInventoryRequirementsToString covers requirements missing a version or a comparator. It panics against master's implementation, as exporting click did.
  • prepareEditRequest with a condition carrying requirements: TestPrepareEditRequestDependencies now inherits a language/python <3.11 condition and checks the requirement string survives.
  • Author/description rejection: moved into checkEditKeepsIdentity, with TestCheckEditKeepsIdentity covering revisions, a new version via --edit, and a new ingredient.
  • Inherited deps when all pyproject entries are skipped: TestApplyPyprojectDependencies (your first comment).

Still not covered by unit tests:

  • FetchMeta / state export ingredient-meta only wraps platform calls. It's exercised by the platform runs above and in the description, but there's no mocked test.
  • TestPublishBuildEncrypted still hasn't run under the integration-test account, so CI with the publish tag will be its first run.

@icanhasmath
icanhasmath requested a review from MDrakos October 21, 2026 20:33
@icanhasmath
icanhasmath merged commit 44e750f into master Oct 21, 2026
8 of 12 checks passed
@icanhasmath
icanhasmath deleted the marcg/private-ingredient-dependencies branch October 21, 2026 20:56
@icanhasmath icanhasmath mentioned this pull request Oct 21, 2026
icanhasmath added a commit that referenced this pull request Oct 21, 2026
* Support dependencies for encrypted private ingredients

`state publish --build` can now declare and edit the dependencies of an
encrypted private ingredient:

- Read [project].dependencies from pyproject.toml and publish them as
  runtime dependencies on language/python packages, with names normalized
  by the inventory API. Version ranges carry over (~= and ==X.* are
  expanded). OS and Python version markers become conditions
  (kernel/Windows|Linux|Darwin, language/python); unsupported markers are
  skipped with a notice. The wheel's METADATA records them as
  Requires-Dist.
- Read the --meta file before building, so a meta file's name, version
  and description feed the wheel (flags > meta file > pyproject.toml).
- Add `state export ingredient-meta <namespace>/<name>[@Version]`, which
  writes an existing ingredient's meta file for use with --edit --meta.

Fix --edit along the way: keep the meta file's dependencies and authors
instead of overwriting them with the previous revision's, keep inherited
dependency types and conditions, add the private-builder dependency only
once, and reject author/description changes on an existing version up
front instead of failing in the API. InventoryRequirementsToString no
longer panics on requirements without a version.

Add a readme walking through the publish and edit workflow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Address review: keep inherited deps, edit the meta file's version

- When every pyproject.toml dependency is skipped, keep the dependencies
  already in place (on --edit, those inherited from the previous
  revision) instead of replacing them with an empty list.
- Let a meta file's version select the version to publish or edit, like
  --version (which still takes precedence). An exported meta file passed
  back with `state publish --edit --meta` without --build was editing
  the latest version instead of the exported one.

Add tests for both, for the up-front author/description rejection on
--edit, for InventoryRequirementsToString with requirements missing a
version or comparator, and for inherited conditions that carry
requirements.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
icanhasmath added a commit that referenced this pull request Oct 21, 2026
* Support dependencies for encrypted private ingredients

`state publish --build` can now declare and edit the dependencies of an
encrypted private ingredient:

- Read [project].dependencies from pyproject.toml and publish them as
  runtime dependencies on language/python packages, with names normalized
  by the inventory API. Version ranges carry over (~= and ==X.* are
  expanded). OS and Python version markers become conditions
  (kernel/Windows|Linux|Darwin, language/python); unsupported markers are
  skipped with a notice. The wheel's METADATA records them as
  Requires-Dist.
- Read the --meta file before building, so a meta file's name, version
  and description feed the wheel (flags > meta file > pyproject.toml).
- Add `state export ingredient-meta <namespace>/<name>[@Version]`, which
  writes an existing ingredient's meta file for use with --edit --meta.

Fix --edit along the way: keep the meta file's dependencies and authors
instead of overwriting them with the previous revision's, keep inherited
dependency types and conditions, add the private-builder dependency only
once, and reject author/description changes on an existing version up
front instead of failing in the API. InventoryRequirementsToString no
longer panics on requirements without a version.

Add a readme walking through the publish and edit workflow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Address review: keep inherited deps, edit the meta file's version

- When every pyproject.toml dependency is skipped, keep the dependencies
  already in place (on --edit, those inherited from the previous
  revision) instead of replacing them with an empty list.
- Let a meta file's version select the version to publish or edit, like
  --version (which still takes precedence). An exported meta file passed
  back with `state publish --edit --meta` without --build was editing
  the latest version instead of the exported one.

Add tests for both, for the up-front author/description rejection on
--edit, for InventoryRequirementsToString with requirements missing a
version or comparator, and for inherited conditions that carry
requirements.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants