Make RawRepresentation serializable, roundtripping as JsonElement

[stephentoub]

Summary

Make RawRepresentation serializable across all AI abstraction types, with JSON roundtripping via JsonElement.

Changes

• Replace [JsonIgnore] with serialization support on RawRepresentation properties across all AI abstraction types (AIContent, ChatMessage, ChatResponse, ChatResponseUpdate, ImageGenerationResponse, SpeechToTextResponse, SpeechToTextResponseUpdate, TextToSpeechResponse, TextToSpeechResponseUpdate, RealtimeConversationItem, RealtimeClientMessage, RealtimeServerMessage, AIAnnotation).
• Add RawRepresentationJsonConverter — a custom JsonConverter<object?> that:
  • On write: serializes the runtime value using the active JsonSerializerOptions. If serialization fails (e.g. circular references, missing type metadata), falls back to writing {}.
  • On read: materializes the JSON payload as a JsonElement.
  • Fast-paths JsonElement and JsonDocument values to avoid redundant re-serialization.
• Update DistributedCachingChatClient to account for RawRepresentation now participating in serialization.
• Update API baseline (Microsoft.Extensions.AI.Abstractions.json).
• Add/update tests covering serialization roundtrips across the affected types.

Microsoft Reviewers: Open in CodeFlow

[stephentoub]

@eiriktsarpalis, I'd appreciate your thoughts on this, whether it's a good idea in general, if so whether it's the right way to achieve it. We just keep getting sporadic complaints about RawRepresentation not persisting in any form through serialization.

[eiriktsarpalis]

With the exception of this eating serialization exceptions, this mostly coincides with how object serialization works out of the box. So maybe this could be simplified by delegating to the default object converter?

[eiriktsarpalis]

Note that in the case of the built-in converter the actual representation can be controlled by the JsonUnknownTypeHandling enum.

[eiriktsarpalis]

I'd probably move the WriteTo call outside the try block since we want to still surface any exceptions related to the underlying writer.

Consider using SerializeToDocument with a using so that pooled buffers are used for the intermediate representation.

[eiriktsarpalis]

@eiriktsarpalis, I'd appreciate your thoughts on this, whether it's a good idea in general, if so whether it's the right way to achieve it. We just keep getting sporadic complaints about RawRepresentation not persisting in any form through serialization.

I always assumed RawRepresentation contained values that were inherently nonserializable. The fact that we try to serialize it regardless doesn't feel great, but at least eating any exceptions means we're not blocking serialization if that fails.