Skip to content

fix(logger): keep string messages that are valid JSON scalars as strings - #8838

Merged
leandrodamascena merged 2 commits into
aws-powertools:developfrom
saquibjawedbit:fix/logger-json-scalar-messages
Nov 20, 2026
Merged

leandrodamascena merged 2 commits into
aws-powertools:developfrom
saquibjawedbit:fix/logger-json-scalar-messages

Conversation

@saquibjawedbit

Copy link
Copy Markdown
Contributor

Issue number:
closes #8828

Summary

Changes

LambdaPowertoolsFormatter._extract_log_message (aws_lambda_powertools/logging/formatter.py) tried json.loads on every string message and kept whatever came back. It now keeps the decoded value only when it is a dict or a list; for any other result (null, bool, number, JSON string) it keeps the original string. Decoding JSON objects and arrays into structured output, the documented feature, is unchanged.

Tests added in tests/functional/logger/required_dependencies/test_logger_powertools_formatter.py, through the public Logger API only:

  • test_with_json_scalar_string_message_stays_string, parametrized with "null", "true", "123", "1e3", '"quoted"', "NaN"
  • test_with_json_object_or_array_string_message_is_decoded, a JSON object string and a top-level JSON array string are still decoded

No existing tests were changed.

User experience

Before:

logger.info("null")  # no "message" key in the output
logger.info("true")  # "message": true
logger.info("123")   # "message": 123
logger.info("1e3")   # "message": 1000.0

After, all four log "message": "<the original string>". logger.info('{"a": 1}') and logger.info('[1, 2]') still log structured {"a": 1} / [1, 2].

This is the bug fix as triaged in the issue. Other user-visible effects of the same rule, all covered by tests or checked by hand:

  • logger.info('"quoted"') now keeps the quotes instead of logging quoted.
  • logger.info("NaN") now logs "message": "NaN" instead of a bare NaN, which made the whole line invalid JSON.
  • A custom json_deserializer that returns something other than a dict/list (e.g. a tuple) now leaves the message as the original string.
  • Whitespace-padded objects (' {"a": 1} ') are still decoded; bytes messages are untouched. DatadogLogFormatter inherits the method, so it gets the same fix.

Verification

AI-assisted; I ran the tests below myself and pasted the output.

Local run on macOS, Python 3.13.15, environment from make dev.

New tests on develop, before the fix (uv run --locked pytest tests/functional/logger/required_dependencies/test_logger_powertools_formatter.py -k "json_scalar or json_object_or_array"):

FAILED tests/functional/logger/required_dependencies/test_logger_powertools_formatter.py::test_with_json_scalar_string_message_stays_string[null]
FAILED tests/functional/logger/required_dependencies/test_logger_powertools_formatter.py::test_with_json_scalar_string_message_stays_string[true]
FAILED tests/functional/logger/required_dependencies/test_logger_powertools_formatter.py::test_with_json_scalar_string_message_stays_string[123]
FAILED tests/functional/logger/required_dependencies/test_logger_powertools_formatter.py::test_with_json_scalar_string_message_stays_string[1e3]
FAILED tests/functional/logger/required_dependencies/test_logger_powertools_formatter.py::test_with_json_scalar_string_message_stays_string["quoted"]
FAILED tests/functional/logger/required_dependencies/test_logger_powertools_formatter.py::test_with_json_scalar_string_message_stays_string[NaN]
============ 6 failed, 2 passed, 39 deselected, 1 warning in 0.06s =============

Same tests after the fix (same command with -v):

tests/functional/logger/required_dependencies/test_logger_powertools_formatter.py::test_with_json_scalar_string_message_stays_string[null] PASSED [ 12%]
tests/functional/logger/required_dependencies/test_logger_powertools_formatter.py::test_with_json_scalar_string_message_stays_string[true] PASSED [ 25%]
tests/functional/logger/required_dependencies/test_logger_powertools_formatter.py::test_with_json_scalar_string_message_stays_string[123] PASSED [ 37%]
tests/functional/logger/required_dependencies/test_logger_powertools_formatter.py::test_with_json_scalar_string_message_stays_string[1e3] PASSED [ 50%]
tests/functional/logger/required_dependencies/test_logger_powertools_formatter.py::test_with_json_scalar_string_message_stays_string["quoted"] PASSED [ 62%]
tests/functional/logger/required_dependencies/test_logger_powertools_formatter.py::test_with_json_scalar_string_message_stays_string[NaN] PASSED [ 75%]
tests/functional/logger/required_dependencies/test_logger_powertools_formatter.py::test_with_json_object_or_array_string_message_is_decoded[{"order_id": 123, "items": ["a", "b"]}-expected0] PASSED [ 87%]
tests/functional/logger/required_dependencies/test_logger_powertools_formatter.py::test_with_json_object_or_array_string_message_is_decoded[["a", "b"]-expected1] PASSED [100%]
================= 8 passed, 39 deselected, 1 warning in 0.02s ==================

Logger unit + functional suites (uv run --locked pytest tests/unit/logger tests/functional/logger):

======================= 212 passed, 7 warnings in 0.38s ========================

Lint, format, types (make format-check lint-check, make mypy):

1417 files already formatted
uv run --locked ruff check aws_lambda_powertools tests examples
All checks passed!
uv run --locked mypy --pretty aws_lambda_powertools examples
Success: no issues found in 963 source files

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

@saquibjawedbit
saquibjawedbit requested a review from a team as a code owner November 15, 2026 17:14
@boring-cyborg

boring-cyborg Bot commented Nov 15, 2026

Copy link
Copy Markdown

Thanks a lot for your first contribution! Please check out our contributing guidelines and don't hesitate to ask whatever you need.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

@powertools-for-aws-oss-automation powertools-for-aws-oss-automation Bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Nov 15, 2026
@leandrodamascena leandrodamascena added this to the 3.42.0 milestone Nov 15, 2026

@leandrodamascena leandrodamascena left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @saquibjawedbit! Keeping only dicts and lists is the whole fix, and the parametrized test pins the documented behaviour next to it. Goes out in 3.42.0, no need to rebase, you'll be in the release notes.

@sonarqubecloud

Copy link
Copy Markdown

@codecov

codecov Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.20%. Comparing base (d90afd5) to head (374225b).
⚠️ Report is 1 commits behind head on develop.

Additional details and impacted files
@@           Coverage Diff            @@
##           develop    #8838   +/-   ##
========================================
  Coverage    97.20%   97.20%           
========================================
  Files          327      327           
  Lines        16759    16761    +2     
  Branches      1522     1523    +1     
========================================
+ Hits         16290    16292    +2     
  Misses         328      328           
  Partials       141      141           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@leandrodamascena
leandrodamascena merged commit 840eeab into aws-powertools:develop Nov 20, 2026
14 checks passed
@boring-cyborg

boring-cyborg Bot commented Nov 20, 2026

Copy link
Copy Markdown

Awesome work, congrats on your first merged pull request and thank you for helping improve everyone's experience!

@powertools-for-aws-oss-automation

Copy link
Copy Markdown

Awesome work, congrats on your first merged pull request and thank you for helping improve everyone's experience!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

logger size/M Denotes a PR that changes 30-99 lines, ignoring generated files. tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: string messages that are valid JSON scalars are coerced or dropped

2 participants