Documentation of HtmlEncoder.EncodeUtf8 fails to mention that OperationStatus.InvalidData is *never* returned.

[AnorZaken]

Type of issue

Missing information

Description

The documentation simply specifies "Returns: OperationStatus A status code that describes the result of the encoding operation."

This leads to the assumption, after one looks at the available enum values, that the method performs some form of validation, i.e. can return OperationStatus.InvalidData. This is not the case. It always substitutes invalid UTF-8 with Rune.ReplacementChar (U+FFFD) and proceeds with the encoding unperturbed, always returning success unless it runs into the other two failure options of DestinationTooSmall or NeedMoreData (the latter only if param isFinalBlock is false).

There is no mention of this either on the method nor at the class description level: https://learn.microsoft.com/en-us/dotnet/api/system.text.encodings.web.textencoder?view=net-10.0

Only way to find out is when unexpected output is encountered in production / tests or if one closely examines the source code.

Page URL

https://learn.microsoft.com/en-us/dotnet/api/system.text.encodings.web.textencoder.encodeutf8?view=net-10.0#system-text-encodings-web-textencoder-encodeutf8(system-readonlyspan((system-byte))-system-span((system-byte))-system-int32@-system-int32@-system-boolean)

Content source URL

https://github.com/dotnet/dotnet-api-docs/blob/main/xml/System.Text.Encodings.Web/TextEncoder.xml

Document Version Independent Id

fe83a6cc-fcae-66ff-b7a5-154fcfdb6d48

Platform Id

9ea117b6-6558-ee75-e0ca-d7451eac5e22

Article author

@dotnet-bot

[AnorZaken]

Aside from the missing documentation, arguably a separate method EncodeUtf8Strict, or a flag parameter, should exist where the encoder aborts and returns OperationStatus.InvalidData when the first instance of invalid UTF-8 is encountered, instead of silently performing the rune-replacement and reporting success.

The documentation for InvalidData:

        /// <summary>
        /// The input contained invalid bytes which could not be processed. If the input is partially processed,
        /// the destination contains the partial result. This guarantees that no additional data appended to the input
        /// will make the invalid sequence valid.
        /// </summary>
        InvalidData

[AnorZaken]

If I'm not mistaken, the relevant logic decision happens at this part:

https://github.com/dotnet/dotnet/blob/main/src/runtime/src/libraries/System.Text.Encodings.Web/src/System/Text/Encodings/Web/OptimizedInboxTextEncoder.cs#L313C26-L313C44

Examples of input that should arguably be invalid in a strict encode:

        private static (string reason, byte[] bytes)[] GetInvalidUtf8Inputs()
            =>
            [
                // A continuation byte cannot start a UTF-8 sequence; it must follow a leading byte.
                ("Lone continuation byte.", [0x80]),

                // `0xE2` starts a 3-byte sequence, but the third byte is missing here.
                ("Truncated 3-byte sequence.", [0xE2, 0x82]),

                // `0xC2` starts a 2-byte sequence, but the continuation byte is missing.
                ("Truncated 2-byte sequence.", [0xC2]),

                // `0xF0` starts a 4-byte sequence, but only three bytes are present.
                ("Truncated 4-byte sequence.", [0xF0, 0x9F, 0x98]),

                // `0xC2` must be followed by a continuation byte in the `10xx_xxxx` range.
                ("2-byte sequence followed by a non-continuation byte.", [0xC2, 0x20]),

                // `0xE2` starts a 3-byte sequence, but `0x28` is ASCII and therefore not a continuation byte.
                ("3-byte sequence with an invalid middle byte.", [0xE2, 0x28, 0xA1]),

                // `0xF0` starts a 4-byte sequence, but the second byte is not a continuation byte.
                ("4-byte sequence with an invalid second byte.", [0xF0, 0x28, 0x8C, 0xBC]),

                // `0xC0 0x80` is an overlong encoding of U+0000, which UTF-8 must reject.
                ("Overlong 2-byte encoding.", [0xC0, 0x80]),

                // `0xE0 0x80 0x80` is another overlong encoding of U+0000.
                ("Overlong 3-byte encoding.", [0xE0, 0x80, 0x80]),

                // `0xF0 0x80 0x80 0x80` is an overlong 4-byte encoding of U+0000.
                ("Overlong 4-byte encoding.", [0xF0, 0x80, 0x80, 0x80]),

                // UTF-8 must not encode UTF-16 surrogate code points such as U+D800.
                ("UTF-16 surrogate encoded as UTF-8.", [0xED, 0xA0, 0x80]),

                // This decodes above U+10FFFF, which is outside the Unicode scalar range.
                ("Code point above Unicode maximum.", [0xF4, 0x90, 0x80, 0x80]),

                // Bytes starting with `11111xxx` are not valid UTF-8 leading bytes at all.
                ("Invalid leading byte in the 5-byte range.", [0xF8, 0x88, 0x80, 0x80, 0x80])
            ];