Editor and BOM (byte order mark)

Editor show BOM (byte order mark) with red dot at begin of file.

UPD: Windows 7, Brackets Sprint 24.

[peterflynn]

@fddima: What encoding is the file using? Brackets currently only supports UTF-8 files.

File is UTF-8 encoded with BOM. I.e. file begins from EF BB BF. Editor show this as red dot at begin of file.

[peterflynn]

Reviewed - low priority @RaymondLim to investigate. Should this be a user story?

Thanks. But what you mean by user story? This issue mostly actual to files which contains non-latin characters, but in fact - even when files contains only latin characters, BOM can be present, and it is valid for any utf-encoded file. I got same very often with similar editors or viewers (red dot at begining of file), but while you target on quality - you must support BOM, even just for utf-8 encoded files.

[RaymondLim]

BOM is used to differentiate between various unicode documents and it has more than the one mentioned by @fddima. See http://en.wikipedia.org/wiki/Byte_order_mark#Representations_of_byte_order_marks_by_encoding for all the encodings that BOM can represent. Since it represents the encoding, maybe it should be part of this non-utf-8 encoding user story https://trello.com/card/support-non-utf-8-encodings/51072abad1b1f8a4560086bf/30.

For a document with BOM, we need to implement the following tasks.

• Detect and strip off BOM on opening files.
• Re-add BOM on saving.
• If other non-utf-8 encodings are supported, then we may need to do encoding conversion on loading/saving.

[kirilloid]

BOM at the beginning of the file is considered as content and for example prevents php header function from working.
I suppose, user should be aware of BOM presence in the file. Unfortunately, solution by @RaymondLim doesn't address that problem.
I'd replace "red dot at begin of file" with, maybe, somthing like "[BOM]" above the first line.

[njx]

@RaymondLim Even in UTF-8, though, the BOM can be present (but only in one form). So we should figure out what to do with it in that case.

[RaymondLim]

@njx Yes, we can initially just focus on handling the only one form EF BB BF for utf-8 document. Then we still need to do the first two tasks that I listed out above.

@kirilloid Thanks for mentioning about php header example. I agree with you that BOM is part of the content. I can also agree that user should be aware of BOM presence in the file. But I don't think showing BOM in the editor is the right thing to do since you don't really want the user to edit it like normal text content. Maybe we should have something in the status bar to show the BOM with some UI that allows the user to add/delete BOM or convert the encoding of BOM.

[peterflynn]

Having just filed a dupe :-) , I agree with @RaymondLim: we shouldn't show the BOM as part of the editor content. I like the idea of indicating it in the status bar, since we'd probably also use the sb to show/edit the overall encoding once we support multiple encodings. For that reason I think we could actually slice this in half: first just stop showing the red dot in the content (but presumably preserve the BOM on save); second, start indicating the BOM presence in the status bar -- we could potentially delay that part all the way until the broader encoding story happens.

[RaymondLim]

@larz0 Can you suggest the UI for BOM in the status bar which can be extended to show the encoding in the future?

[larz0]

@RaymondLim thoughts?

[larz0]

@RaymondLim maybe a tooltip that says, e.g. "UTF-7". Or should it be the other way around? Display encoding in status bar with a tooltop for hex?

[njx]

@larz0 - note that this isn't about showing the encoding (we're still only supporting UTF-8). It's just some way of showing that the file had a BOM (even though we don't show it in the code) and that we'll properly write it out when we save it.

Does Sublime show the BOM somehow? I actually wonder if it's important for us to have any visual indication in the UI that there was a BOM.

FWIW, I think this is only on Windows. I don't see the red dot on Mac--I'm guessing we never see it because we're reading the file as text and the filesystem deals with the encoding somehow.