Wrong width for emoji chars

[santagada]

as reported in xonsh/xonsh#1569 some (maybe all) emojis are reported as being 2 char wide, while most (all?) terminals think they are 1 char wide:

The difference in position is because while editing a command xonsh is using wcwidth but to print the line it just prints and let the terminal position things

[jquast]

The Unicode Consortium has agreed these emoji characters should be allowed 2 cells for display on terminals, and my software repeats this claim to yours. Unfortunately, the terminal emulator used does not agree. Commonly, such software is based upon older "wcwidth", the libc on your system is often woefully out of date.

I think you might notice if you play with this emulator you're displaying here, and set "blinking block cursor", that immediately following the input and echo output of a snake, your blinking cursor occludes the snake -- that is, the snake should be allowed 2 cells, but the emulator advanced only one.

Unless there is newer unicode release data that says the snake emoji may be 1 cell wide, there is no fix for me to make in wcwidth. This was a choice I made, not to chose against any particular emulator. As written,

"This library aims to be forward-looking, portable, and most correct."

and later caveats,

"This library attempts to determine the printable width by an unknown targeted terminal emulator. It does not provide any ability to discern what the target emulator software, version, of level of support is. Results may vary!"

I stress the Results may vary. xterm, iTerm2, rxvt-unicode, whatever windows is doing today with ubuntu, these all vary in level of support for wide unicode characters.

I'm afraid theres no clear resolution from this end.

[jonathanslenders]

I agree with Jeff that we should follow the Unicode Consortium, because that's the only path to make everything consistent. If a terminal emulator does something different, it has to be fixed there.

That said, it could make sense to add a hook to prompt_toolkit to modify the width of a certain character, or monkey patch wcwidth before loading it in your particular situation to handle this case. (Just as a workaround, until they fix the terminal emulator.)

[jquast]

Best resolution is in the emulator, agreed.

I did document a method for determining, by automated test, what level of support a given terminal emulator is. Its not pretty, or fast, but if you put the right effort in, it is possible to have more correct output in more common cases. Maybe by adding some stub code.

But this stub would require changing, following some future releases of the emulators they target, and it could become inverted and incorrect in any next release, I wouldn't go down that path!

[santagada]

best solution is the emulator, but at least vte, iterm2 and terminal.app all share this characteristic and I'm not going around fixing them in the case of vte (by the way I guess terminator is based on vte still) and terminal.app they are mostly part of the operating system and will take a loooong time to change, what we do until then?

Is there any terminal that actually agrees with the unicode standard? If there isn't or even if only those then either we need to fork wcwidth or somehow monkeypatch it, both of which I think is much worse.

I saw some switches for unicode 9 in iterm2, but terminal.app is still the most used terminal in os x (and vte in gnome/terminator) so we will have to do it anyway.

[santagada]

https://gitlab.com/gnachman/iterm2/issues/5003 <--- here is the bug report that someone opened for iterm2. But it is not enough as I said we would have to switch all terminals everywhere which don't make much sense. As this is new in unicode 9 just reverting back to 7 might fix this

[jonathanslenders]

This is painful...
@jquast: did unicode 7 or 8 actually define a width for these characters, or was in undefined before? If all those unicode versions did unambiguously specify a width for all codepoints, then we could maybe ask wcwidth to return the width for a specific unicode version.

From prompt_toolkit, I could then make this configurable in a way.

[jquast]

Yes, we can create tables for any version, so maybe we can accommodate. Will seek a solution soon.

Will probably add new kwarg in next release of the wcwidth* functions.

[blueyed]

FWIW: here is a hack to make the terminal inject its wcwidth via LD_PRELOAD: blueyed/rxvt-unicode#1.
This makes the terminal the controlling instance, as long as wcwidth(3) is used by programs - which is not the case when using this Python lib however..
(the terminal gets the width from the font)

[jquast]

I’ve proposed versioned Unicode level for wcwidth() functions for the next release of wcwidth, I’ve only just briefly played with different version levels, it should be possible even to auto-detect by introspection of cursor movements, pr #23 can be merged soon, and will write autodetection example later when time permits

[jquast]

Hello, I am closing this issue, because the latest version of wcwidth allows selecting any version of Unicode, 5.1.0 through 13.0.0, and a companion/demonstration utility, https://github.com/jquast/ucs-detect/ can automatically detect the version, which is a technique xonsh might try to use until I am able to persuade terminal emulator authors to export the UNICODE_VERSION environment variable.

Also, some emoji's are incorrectly listed as a width of 1 in the Unicode Specification, #27, but I should return 1 for those, as that is how terminals also interpret it, because we both use the same spec, I can't help that issue