Skip to content

fix(idempotency): serialize PydanticSerializer output in JSON mode - #8570

Merged
leandrodamascena merged 3 commits into
aws-powertools:developfrom
AtulJoshi1206:fix/idempotency-pydantic-serializer-json-mode
Oct 17, 2026
Merged

leandrodamascena merged 3 commits into
aws-powertools:developfrom
AtulJoshi1206:fix/idempotency-pydantic-serializer-json-mode

Conversation

@AtulJoshi1206

Copy link
Copy Markdown
Contributor

Issue number: closes #8569

Summary

Changes

PydanticSerializer.to_dict now returns model_dump(mode="json") instead of model_dump().

save_success stores the result with json.dumps(result, cls=Encoder, sort_keys=True). In Python mode the dict still held UUID, datetime, Enum, bytes and set values, which the Encoder can't encode, so the save failed after the function had already run.

JSON mode is what _prepare_data does for inputs since #8075, and what shared.functions.pydantic_to_dict does. from_dict is unchanged: model_validate parses the JSON values back into the model's types.

Tests (added to tests/functional/idempotency/_pydantic/test_idempotency_with_pydantic.py):

  • test_pydantic_serializer_round_trips_non_json_types: the to_dict output goes through json_serialize, which is the same json.dumps call save_success makes, and from_dict returns an equal model with UUID, datetime, plain Enum and str fields.
  • test_idempotent_function_serialization_pydantic_with_non_json_types, explicit and deduced: the first call saves a COMPLETED record, and the second returns the model without running the function again.

All three fail on develop with Failed to update record state to success in idempotency store - (Object of type datetime is not JSON serializable) and pass with this change.

Checks:

  • tests/functional/idempotency + tests/unit/idempotency: 194 passed.
  • The wider pytest -m "not perf" run: 3745 passed.
  • ruff check and ruff format --check: clean.
  • mypy: no new errors.

Compatibility. For models that already saved, the stored JSON doesn't change: Decimal("1.10") stays "1.10", float infinity stays Infinity, str and int enums stay their values, and tuples stay lists.

Two narrow cases store different bytes but read back to the same values:

  • Decimal("NaN") becomes "NaN". This is only possible with allow_inf_nan=True.
  • Dicts with non-string keys are sorted by their string form.

Stored responses are only parsed back with json.loads and from_dict, never hashed or compared, so records written before or after this change read the same in both versions.

Risks

  • Secret fields: SecretStr/SecretBytes fields are now stored masked ("**********"), so a replay returns the masked value. Before this change these models failed to save at all.
  • Strict models: models with strict=True and datetime/UUID fields now save. Their replay still fails validation, because strict validation doesn't accept the stored strings. The function still runs only once, and fixing the replay would need a from_dict change, so that's out of scope here.
  • JSON-only serializers: custom serializers with when_used="json" now run when the result is saved.

Follow-up. DataclassSerializer.to_dict (asdict) fails the same way. Its from_dict (Model(**data)) can't turn strings back into UUID or datetime, though, so making the save succeed would silently replay strings. That needs a separate change.

No docs change: the output serialization docs already list Pydantic models as supported.

User experience

Before:

@idempotent_function(
    data_keyword_argument="order",
    config=config,
    persistence_store=persistence_layer,
    output_serializer=PydanticSerializer,
)
def charge(order: Order) -> Payment:  # Payment has UUID, datetime and Enum fields
    ...

charge(order=order)
# the charge runs, then:
# IdempotencyPersistenceLayerError: Failed to update record state to success in idempotency store
#   - (Object of type datetime is not JSON serializable)
# the record stays INPROGRESS, so a retry after the in-progress expiry charges again

After: the first call saves a COMPLETED record and returns the Payment. Later calls with the same order return the stored Payment without charging again, with its UUID, datetime and Enum values restored.

AI assistance: prepared with Claude Code; the change was reviewed and the tests were run locally.


By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

🤖 Generated with Claude Code

PydanticSerializer.to_dict used model_dump() in Python mode, but
save_success stores the result with json.dumps and the shared Encoder,
which only handles Decimal, Pydantic models and dataclasses. A returned
model with a UUID, datetime, plain Enum, bytes or set field raised
TypeError after the function had already run. The record stayed
INPROGRESS, so a retry after the in-progress expiry ran the function
again.

Dump in JSON mode, as _prepare_data does for inputs since aws-powertools#8075.
from_dict validates the JSON values back into the model's types, and
models that already worked read back the same as before.
@AtulJoshi1206
AtulJoshi1206 requested a review from a team as a code owner October 17, 2026 05:16
@powertools-for-aws-oss-automation powertools-for-aws-oss-automation Bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Oct 17, 2026
@boring-cyborg boring-cyborg Bot added the tests label Oct 17, 2026
@codecov

codecov Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.93%. Comparing base (ca28871) to head (9a12bc9).
⚠️ Report is 1 commits behind head on develop.

Additional details and impacted files
@@           Coverage Diff            @@
##           develop    #8570   +/-   ##
========================================
  Coverage    96.93%   96.93%           
========================================
  Files          318      318           
  Lines        15999    15999           
  Branches      1419     1419           
========================================
  Hits         15508    15508           
  Misses         349      349           
  Partials       142      142           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@powertools-for-aws-oss-automation powertools-for-aws-oss-automation Bot removed the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Oct 17, 2026
@boring-cyborg boring-cyborg Bot added the documentation Improvements or additions to documentation label Oct 17, 2026
@powertools-for-aws-oss-automation powertools-for-aws-oss-automation Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Oct 17, 2026

@leandrodamascena leandrodamascena left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch @AtulJoshi1206, and thanks for the thorough compatibility analysis, it matches what I verified locally. I pushed a small docs note to your branch about SecretStr being stored masked and strict models failing on replay, since both now fail silently instead of loudly. Approving!

@sonarqubecloud

Copy link
Copy Markdown

@leandrodamascena
leandrodamascena merged commit f58cd87 into aws-powertools:develop Oct 17, 2026
13 of 14 checks passed
@boring-cyborg

boring-cyborg Bot commented Oct 17, 2026

Copy link
Copy Markdown

Awesome work, congrats on your first merged pull request and thank you for helping improve everyone's experience!

@powertools-for-aws-oss-automation

Copy link
Copy Markdown

Awesome work, congrats on your first merged pull request and thank you for helping improve everyone's experience!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation size/L Denotes a PR that changes 100-499 lines, ignoring generated files. tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: PydanticSerializer can't save results with UUID, datetime or Enum fields, so retries run the function again

2 participants