Add support for semanticTokens

[ericdallo]

Closes #430

Add support for textDocument/semanticTokens

Signed-off-by: Eric Dallo ericdallo06@hotmail.com

[ericdallo]

Given this, should we remove or deprecate semantic highlights as semantic tokens replace it?

[eclipse-lsp4j-bot]

Can one of the admins verify this patch?

[kdbeall]

Looks like you beat me to it! Excited to see this. 👍

[ericdallo]

@spoenemann Could you take a look? I don't know who to ping here 😅

[spoenemann]

I have little time this week, maybe next one. If anyone else would like to review, go for it.

[Dufgui]

Thanks for this huge works ! It's really a good job !

[ericdallo]

Thanks for the review, I'll take a look tomorrow!

[ericdallo]

Done @Dufgui

[spoenemann]

Thanks! Two remarks below.

[ericdallo]

Could you review it again @spoenemann ?

[spoenemann]

Almost there.

[spoenemann]

The specification says it's either a SemanticTokensOptions (with legend) or a SemanticTokensRegistrationOptions (with documentSelector), but not both.

Please also add a no-arguments constructor. In some contexts it's easier to construct the protocol classes with setters, e.g. with Xtend:

options = new SemanticTokensWithRegistrationOptions => [
    legend = ...
    range = ...
]

[ericdallo]

I don't get it, what do you suggest?
I thought that this comment suggested merging both classes

[KamasamaK]

SemanticTokensRegistrationOptions extends SemanticTokensOptions in the spec, so legend would be in both.

[spoenemann]

SemanticTokensRegistrationOptions extends TextDocumentRegistrationOptions, which has a documentSelector, but no legend.

https://microsoft.github.io/language-server-protocol/specifications/specification-3-16/#textDocument_semanticTokens

I thought that this comment suggested merging both classes

Yes, but merging the classes is just a workaround because Java's type system is not as flexible as TypeScript. We use a single class to parse / serialize both alternatives of SemanticTokensOptions | SemanticTokensRegistrationOptions.

[spoenemann]

See above, I didn't realize that SemanticTokensRegistrationOptions is actually meant to extend SemanticTokensOptions. Then the current constructors can be kept, but please add one without arguments:

new() {
}

[ericdallo]

Done @spoenemann. I didn't add an empty constructor before because both SemanticTokensOptions and SemanticTokensRegistrationOptions have non-null legend filed, so I thought that it didn't make sense to add a constructor that doesn't initialize the legend field.

[spoenemann]

Thanks!

[nixel2007]

woohoo, good news! gonna implement semanticTokens in next release of our ls. thanks!

[nixel2007]

One small question... I can't find any method for textDocument/semanticTokens/* requests in TextDocumentService interface. Like TextDocumentService#completion() for textDocument/completion and so on. Are they missing from current implementation or should I implement some other new interface?

[ericdallo]

Yeah, I think I forgot about that @nixel2007
I opened this: #450

[nixel2007]

Sorry for pinging closed PR, but shouldn't we migrate List<Integer> to int[]? It's a huge list/array with a lot of Integer boxing/unboxing operations. Yes, List is more comfortable class with all its Collection methods. But is it fast enough?

@ericdallo @spoenemann WDYT?

[ericdallo]

Yeah, I think it makes sense

[spoenemann]

I would do some profiling before optimizing something like that. Java ArrayList and Integer boxing/unboxing is super optimized in the JVM, I don't know whether you'll notice any difference for real-world array sizes.