lua: add support for Lua Language Server

[georgiy-belyanin]

Introduction

Lua used by Tarantool has dynamic type system. It may complicate the development of the Tarantool applications because of lack of static checks and absence of IDE/text editor autocomplete support since types of the values can't be inferred before script execution. There are tools such as Lua Language Server (ex. Sumneko Lua Server) [1] allowing providing type annotations and class definitions for performing some static checks. There is a Tarantool VSCode extension [2] for Lua Language Server with type definitions for the most of the Tarantool functions. The problem is that the extension is maintained externally. It means some of the type definitions are deprecated and aren't used in the internal code during the development.

Suggestion

It's suggested to migrate the type definitions inside the Tarantool itself and to provide a straightforward way to supply IDE with type definitions for performing static checks when developing Tarantool. It'd good to reuse the language server checks during the development to make new code more reliable, simplify the development and review process. Also, this type definitions may involve documentation on the functions thus they can be reused for exporting documentation reference.

Implementation

Adding the support includes the following tasks.

• Migrate and actualize type definitions for C-defined Lua functions and Lua modules from the VSCode extension [2]. Since there are a lot of modules it's suggested to work in three iterations.
  • box, clock, config, decimal, fiber, json, log, net.box, string, table, uri, uuid, yaml modules (lua: add basic Lua Language Server support #11038).
  • buffer, csv, datetime, digest, fio, fun, http, merger, pickle, socket, utf8.
  • crypto, iconv, popen, strict, swim, xlog.
• Support exporting definition files for Lua Language Server [3] for developing Tarantool applications (lua: add basic Lua Language Server support #11038).
• Migrate and provide type definitions for popular Tarantool modules.
  • vshard
  • luatest
  • crud
  • queue
  • expirationd
  • kafka
  • Likely, something else.
• Implement building plugins for popular text editors: Neovim, VSCode.
• Bulid a PoC of exporting Lua function documentation reference from type definitions.
• Add type definitions for internal Lua methods and classes.
• Add support for explicitly specifying space schema for providing static checks of the database tuples.
• Include static checks of the Lua Language Server in CI.
• Require the developer to supply type definitions for new Lua functions.

Iterations

Since the task is even too big for a single epic, it's split into iterations.

1. lua: add basic Lua Language Server support #11038

Reference list

• [1] Lua Language Server.
• [2] Tarantool VSCode extension.
• [3] Lua Language Server definition files.

[ligurio]

Just want to say, that some tarantool Lua module already have type annotations in LDoc format (etcd-client, expirationd, metrics) or checks ¹ (luatest, expirationd, crud). Probably these annotations can be reused.

Also, Lua tagging systems are fragmented, see lunarmodules/ldoc#403:

Unfortunately documentation tagging systems have proliferated and fragmented. The current pretty de facto Lua Language Server (Sumneko) uses the LuaCATS annotation system which are derived from EmmyLua annotations which are derived from LDoc which is derived from LuaDoc. Of course LDoc eventually dropped cross-compatibility with LuaDoc, and EmmyLua eventually dropped cross-compatibility with LDoc, and LuaCATS dropped cross-compatibility with EmmyLua.

Footnotes

1. https://github.com/tarantool/checks ↩

[georgiy-belyanin]

After discussing the idea of LSP verbally with @Totktonada and @ochaton we came with a few points that'd make this feature really helpful.

• The following workflow is suggested to work with different Tarantool projects.
  • Definition files are uploaded to S3 with a corresponding Tarantool release.
  • tt utility downloads definition files used by a project into a home subdirectory.
  • tt also configures .luarc.json file for the language server to use pre-downloaded definition files of the used Tarantool version.
• The annotations should be present both for LTS (i.e. 2.11) and for latest versions (i.e. 3.x).
• Since it's unlikely for all IDEs to be tested and supported it'd nice to cover VSCode, NeoVIM and JetBrains IDEs as the most frequent ones.
• The definitions should be tested on the existing projects to check them from the user's point of view.
• Public class names are exported. Thus, it's really imported for them to be consistent and clear.

A few raw points that are reasonable though still remain not so clear.

• Probably we can implement providing annotations to Lua Language Server using HTTP endpoint (i.e. we should support "workspace.library": ["<URI to an archive with annotations>"] ). That may simplify downloading process.
• It may be worth making tt ask for a used IDE and provide configuration for a specific IDE.
• It'd nice to use this definition files for checking out compatibility with newer Tarantool versions. E.g. it may help to test out "What will happen if we upgrade to 3.x? What will become deprecated?".
• It'd nice to mark functions that never return (e.g. box.error) and functions that throw some specific exceptions (similarly to Java throws).
• It'd nice to mark functions as yieldable (and even more make every function using yieldable functions yieldable). And to use this information to help user working in non-MVCC transaction mode.
• It can be really helpful to somehow distinguish memtx/vinyl spaces and different kinds of indices to provide methods specific to them.
• luafun, box.index and box.space are quite painful to be annotated rather well. They should be tested externally.
• Probably, some of the suggested features could be implemented using Lua Language Server plugin system (https://github.com/LuaLS/lua-language-server/wiki/Plugins).
• Russian documentation could be really helpful for the users who don't know English rather well (see https://github.com/LuaLS/lua-language-server/wiki/Translations).
• Lua annotations should be somehow reused inside Tarantool console.
• Probably, Lua annotations could be somehow integrated with possible WASM sandbox.
• Luarocks modules can provide type annotations on their own.

Outside of the scope of the discussion.

• tt lint command that runs Lua Language Server on the code (likely with luacheck) seems reasonable feature to allow users integrate the checks into CI.

[georgiy-belyanin]

Trying to figure out how to annotate box.space properly I've come up with a problem with Lua Language Server generic support. Turn out they're almost completely broken [^1]. After researching a bit more on the topic I've come up with recent PR on the generics [^2] and a mention on a newer Rust-based language server with better [^2] support of generics.

[^1] LuaLS/lua-language-server#1861
[^2] LuaLS/lua-language-server#3055
[^3] https://github.com/CppCXY/emmylua-analyzer-rust