Semantic Tokens LSIF specification.

[gundermanc]

Adds a draft specification for 'semantic tokens' to LSIF. See related ask: #1397

This specification proposal more or less adapts the LSP textDocument/semanticTokens/full protocol message as an edge + vertex off of the LSIF document vertex.

Features

• textDocument/semanticTokens/full edge - Edge from any document vertex to a SemanticTokensResult vertex.
• lsp.SemanticTokens data is pretty much used as-is, though resultId is unnecessary.
• Definition of integer => token types mapping in the capabilities vertex. One implication of this design is the need to standardize the unspec-ed capabilities vertex, currently emitted by both Roslyn and TypeScript generators. See Standardize LSIF capabilities vertex #1753

Reference Implementation and Validation

A reference implementation of this protocol has been accepted into the Roslyn LSIF generator: dotnet/roslyn#65860.

It has been used successfully to demonstrate semantic colorization of C# code, provided a client specific theme resource.

Considerations

• Theming is client specific, much as it is in LSP for VS, VS Mac, VS Code, and other editors.
• I was the original implementor of Semantic Tokens support in VS. Anecdotally a lot of friction came about for language service authors due to lack of a common theme format for LSP between the IDEs. Should we standardize this for LSIF? LSIF can be consumed by any client but... 1) language authors know their color keys best 2) tools that consume LSIF and present colorization info are likely to be polyglot and/or have no other means of integration with language services, and so have limited opportunities for LSIF generator authors to define their experience.