Support configurable curses library

[avylove]

We talked in the past about leveraging something like Jinxed to provide terminal info instead of Curses. I'm not sure about wholly swapping it, but, in some cases, it would be good to be able to use Jinxed (or equivalent) even on non-Windows platforms.

My current use case is one where a library can have multiple output formats (terminal, HTML, etc), but uses terminal escapes as a common language because it was initially designed for terminals only. For HTML and the like, I don't need the local system to support all the capabilities, I just need a portable, full featured reference to xterm-256color. It's usually not an issue, but FreeBSD's old school termcap implementation is out there and I'm sure it's not the only one.

Jinxed already works on Linux, but in Blessed we use imports to determine the curses library to use. For most places it's used, the curses library could just be made an argument, using the current values as defaults. The one that seems less clean is ParameterizingString which calls tparm() I guess we could pass term to __new__ and grab it from there.

Thoughts?

[jquast]

I think it’s a good idea, to allow to install and use without curses, or, even with curses, to depend on jinxed instead, /etc/termcap or whatever isn’t maintained anymore, all of the modern terminal emulators are just adding their own new sequences without any concern for it. TERM=“screen” has a lot of issues, some workarounds in blessed for it, and it’s from the very same maintainers of termcap so it’s really hopeless that if they can’t get it right then nobody is, so many modern terminal libraries just assume xterm without checking, and nobody seems to notice anyway! I think the community has basica Lu agreed to bypass termcap especially if you want to add support for recently added sequences in kitty, iterm2, etc.

…

-- Jeff Quast ***@***.***

On Wed, Oct 5, 2022, at 5:28 PM, Avram Lubkin wrote: We talked in the past about leveraging something like Jinxed to provide terminal info instead of Curses. I'm not sure about wholly swapping it, but, in some cases, it would be good to be able to use Jinxed (or equivalent) even on non-Windows platforms. My current use case is one where a library can have multiple output formats (terminal, HTML, etc), but uses terminal escapes as a common language because it was initially designed for terminals only. For HTML and the like, I don't need the local system to support all the capabilities, I just need a portable, full featured reference to xterm-256color. It's usually not an issue, but FreeBSD's old school termcap implementation is out there and I'm sure it's not the only one. Jinxed already works on Linux, but in Blessed we use imports to determine the curses library to use. For most places it's used, the curses library could just be made an argument, using the current values as defaults. The one that seems less clean is `ParameterizingString` which calls `tparm()` I guess we could pass `term` to `__new__` and grab it from there. Thoughts? — Reply to this email directly, view it on GitHub <#243>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHNOKFVUOVT4Q3I3HCRTDLWBXXI3ANCNFSM6AAAAAAQ56EWNI>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[avylove]

I'm torn on if jinxed should be relied on completely. You're right, most other libraries just use hardcoded xterm values and no one seems to notice, though I expect there are some users of Blessed that rely on it because we don't do that.

That said, using terminfo or the like is only useful if it's actually maintained. I think @ThomasDickey does a good job of maintaining that upstream, but not all distros do a good job of keeping it updated or even using it at all. And even if it's there, it doesn't mean Python would use it. On FreeBSD the terminfo database is installable as a separate package, but I didn't see a way to get Python to use it instead of /etc/termcap.

Here are a few things we can do:

1. Make curses optional on all platforms
   • Set curses class as Terminal parameter
     • Use existing behavior (for now) when unset
   • Modify other classes/functions to use the class specified in Terminal
2. Optionally look into adding terminfo parsing directly to Jinxed, so it could fall back to terminfo for uncommon terminals
   • This is a binary format described here
3. Potentially switch the default on non-Windows from curses to jinxed
   • The other stuff would have to be done first
   • I think this would be a major version change

Of course the big challenge here is time which is not a thing either of us have a lot of these days.

[jquast]

I'll just add that I initially started using blessed because I had a hardware terminal at the time, and used TERM=wyse325 or some such, for use in https://github.com/jquast/x84, there were strange artifacts on that terminal when using "ansi" or "xterm" sequences but it worked swimmingly when used through blessed/blessings because it used the right terminal capabilities db

[jquast]

Oh, I'd like to add, there is a way to query a terminal interactively for its terminfo database! In this way, a special termcap like xterm-kitty is compatible on a remote system that doesn't know anything about it. It is called XTGETTCAP, See https://invisible-island.net/xterm/ctlseqs/ctlseqs.html#h3-Device-Control-functions under DCS + q Pt ST

I've done some experiments with this, and naturally it isn't supported by all terminals, I don't know the numbers yet but unless its 100% it isn't really worth perusing in a core library like blessed, and that it would cause some delay for first initialization or first use of each terminal capability.

But it is interesting :)

[jquast]

There was some discussion about using XTGETTCAP in https://gitlab.freedesktop.org/terminal-wg/specifications/-/issues/8#note_529026 as part of moving away from requiring terminfo databases entirely as I also lightly suggested here.

[jquast]

I wrote a lot of throwaway scripts in recent development of wcwidth.py -- to make sure to match all known sequences, and, I ran these tests with TERMCAP entries for many terminals -- kitty, xterm, xterm-256color, alacritty, and more.

And the results were equal for almost all of them, with only very minor differences in preference of some kind of sequence, like \b for cursor_left or \n for cursor_down instead of common SGRs.

It had me thinking, that, even the "derived" popular terminal emulators all still support the base vt220 and xterm sequences, even if it isn't their preferred, because they wish to have compatibility with termcap-naive applications anyway.

And most importantly, at least for every single sequence we generate in our _capabilities.py, I found little to no difference among all of the different termcaps. I think blessed could very safely go "no curses" entirely by having a built in xterm-256color - alike capability database.

Probably the most compatible is to have an if/else around curses module, and only use it when not found, and, no matter the "kind" specified, "xterm-ish" sequences are emitted.

[avylove]

We could fall back to Jinxed for other platforms besides windows if curses can't be found or when forced. I can make you a maintainer if you want.

[jquast]

Yeah, sure, we will need to add some code, some little parser, about parametrized strings, as you say.

I think our library would get used more if we wrote "no curses required" or something, hard to say how many people pass by using it because it has "import curses" and also fnctl/termios/etc. that fails. It would allow to be used to make nice gui telnet or serial applications on embedded devices running python, or more alternate ports/flavors of python that usually don't bother to get curses configured even when they exist, I've seen the problem with anaconda and the like.

[avylove]

I just ran a quick test replacing import curses with import jinxed as curses All the curses functions we use in Blessed should already be reimplimented in Python. Ran the tests for 3.14 and got about 100 failures. It seems like this was mainly because there's no fallback terminal type in Jinxed when it doesn't have it's own terminfo for what TERM is specified as. So in our tests, when we have TERM set for ansi or linux, it just raises an exception. On linux it currently trusts TERM is set for what you want, so we'd need to either add a fallback to jinxed or add a fallback within blessed.

Not sure about the other libs. I think we already have handling for cases they can't be imported and just make the dependent function no-op.

[avylove]

Terminals Jinxed can currently support are here. They all derive from xterm.py and add or delete as necessary.
https://github.com/Rockhopper-Technologies/jinxed/tree/main/jinxed/terminfo

[jquast]

Just another thought, that blessed could run in a constrained environment like micropython, with the use of jinxed, but all of the excess lines of code, features and documentation makes for an overall pretty large program, and that's probably why there hasn't been a lot of demand.

for the purposes that blessed is for -- to provide "rich" interface, and gentle and obvious spelling to otherwise obscure to read and write terminal capability names and parametrize, it does really great. And this comes at a cost, but I think we want to let people have "great tools", the very fine tools can, instead, be put into libraries like wcwidth.

[avylove]

My experience with Micropython is you typically don't have a shell. The displays usually use some library that is converting everything to pixels before transmitting over a bus. There are cases where you can interact with a device from over the terminal, but then Blessed would be used on the client side which can handle it. I'm sure there are some use cases, but I haven't run into them yet.

I think a more likely scenario is running in a stripped down container that proxies its display to some other platform, like the host or a virtual web console.

[jquast]

Oh, ok, wouldn't want a telnet server or serial "server" application? it's so stripped down you would have only basic REPL at best and do a TUI client-side, you say?

[jquast]

@Josverl has recently ported wcwidth to micropython and mentioned a project, jquast/wcwidth#186 (comment) -- i wonder if he would think a library like "blessed" has any place in micropython and his projects -- it would be very easy to make a readline editor above or in blessed, for example. But it's a library so large I think is unpalatable for micropython.

[avylove]

It looks like he's working on the REPL. Typically you'd have REPL access over a serial connection, or, optionally, over a network connection, and there is a web-based REPL. But most production cases will use some kind of client that either uses some sort of API or a lot of the development tools actually just wrap the REPL.