Repository navigation
fix(idempotency): serialize PydanticSerializer output in JSON mode - #8570
Conversation
PydanticSerializer.to_dict used model_dump() in Python mode, but save_success stores the result with json.dumps and the shared Encoder, which only handles Decimal, Pydantic models and dataclasses. A returned model with a UUID, datetime, plain Enum, bytes or set field raised TypeError after the function had already run. The record stayed INPROGRESS, so a retry after the in-progress expiry ran the function again. Dump in JSON mode, as _prepare_data does for inputs since aws-powertools#8075. from_dict validates the JSON values back into the model's types, and models that already worked read back the same as before.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #8570 +/- ##
========================================
Coverage 96.93% 96.93%
========================================
Files 318 318
Lines 15999 15999
Branches 1419 1419
========================================
Hits 15508 15508
Misses 349 349
Partials 142 142 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
leandrodamascena
left a comment
There was a problem hiding this comment.
Nice catch @AtulJoshi1206, and thanks for the thorough compatibility analysis, it matches what I verified locally. I pushed a small docs note to your branch about SecretStr being stored masked and strict models failing on replay, since both now fail silently instead of loudly. Approving!
|
f58cd87
into
aws-powertools:develop
|
Awesome work, congrats on your first merged pull request and thank you for helping improve everyone's experience! |
|
Awesome work, congrats on your first merged pull request and thank you for helping improve everyone's experience! |



Issue number: closes #8569
Summary
Changes
PydanticSerializer.to_dictnow returnsmodel_dump(mode="json")instead ofmodel_dump().save_successstores the result withjson.dumps(result, cls=Encoder, sort_keys=True). In Python mode the dict still heldUUID,datetime,Enum,bytesandsetvalues, which theEncodercan't encode, so the save failed after the function had already run.JSON mode is what
_prepare_datadoes for inputs since #8075, and whatshared.functions.pydantic_to_dictdoes.from_dictis unchanged:model_validateparses the JSON values back into the model's types.Tests (added to
tests/functional/idempotency/_pydantic/test_idempotency_with_pydantic.py):test_pydantic_serializer_round_trips_non_json_types: theto_dictoutput goes throughjson_serialize, which is the samejson.dumpscallsave_successmakes, andfrom_dictreturns an equal model with UUID, datetime, plain Enum and str fields.test_idempotent_function_serialization_pydantic_with_non_json_types, explicit and deduced: the first call saves aCOMPLETEDrecord, and the second returns the model without running the function again.All three fail on
developwithFailed to update record state to success in idempotency store - (Object of type datetime is not JSON serializable)and pass with this change.Checks:
tests/functional/idempotency+tests/unit/idempotency: 194 passed.pytest -m "not perf"run: 3745 passed.ruff checkandruff format --check: clean.mypy: no new errors.Compatibility. For models that already saved, the stored JSON doesn't change:
Decimal("1.10")stays"1.10", float infinity staysInfinity, str and int enums stay their values, and tuples stay lists.Two narrow cases store different bytes but read back to the same values:
Decimal("NaN")becomes"NaN". This is only possible withallow_inf_nan=True.Stored responses are only parsed back with
json.loadsandfrom_dict, never hashed or compared, so records written before or after this change read the same in both versions.Risks
SecretStr/SecretBytesfields are now stored masked ("**********"), so a replay returns the masked value. Before this change these models failed to save at all.strict=Trueanddatetime/UUIDfields now save. Their replay still fails validation, because strict validation doesn't accept the stored strings. The function still runs only once, and fixing the replay would need afrom_dictchange, so that's out of scope here.when_used="json"now run when the result is saved.Follow-up.
DataclassSerializer.to_dict(asdict) fails the same way. Itsfrom_dict(Model(**data)) can't turn strings back intoUUIDordatetime, though, so making the save succeed would silently replay strings. That needs a separate change.No docs change: the output serialization docs already list Pydantic models as supported.
User experience
Before:
After: the first call saves a
COMPLETEDrecord and returns thePayment. Later calls with the same order return the storedPaymentwithout charging again, with itsUUID,datetimeandEnumvalues restored.AI assistance: prepared with Claude Code; the change was reviewed and the tests were run locally.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.
🤖 Generated with Claude Code