[TSP] Emit resolvable declarations for special forms, TypeVars, and imported symbols#3716
Open
rchiodo wants to merge 4 commits into
Open
[TSP] Emit resolvable declarations for special forms, TypeVars, and imported symbols#3716rchiodo wants to merge 4 commits into
rchiodo wants to merge 4 commits into
Conversation
|
According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅ |
Contributor
|
@kinto0 has imported this pull request. If you are a Meta employee, you can view this in D107922614. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When Pylance uses the Pyrefly TSP (Type Server Protocol) backend, hovering over imported
symbols —
os,Literal,TypedDict,overload, imported user types/functions, andTypeVars — renders them asunknowninstead of their real names/types. The same symbolsresolve correctly under Pyright's native type server, so this is specifically a gap in how
Pyrefly converts its internal types into the TSP wire format.
This PR fixes that conversion gap so those constructs come across as real, resolvable handles.
Fixes the Pylance-side report:
microsoft/pylance-release#8062 — [TSP] Types and functions imported from other modules show up as unknown when using Pyrefly
Why this is a TSP-specific problem
TSP is a boundary. Pyrefly computes everything in its internal representation
(
pyrefly_types::Type);pyrefly/lib/tsp/type_conversion.rstranslates that into the wire
tsp_types::Typethat Pylance consumes. Pylance rendershover / completion / goto from the wire type — it never sees Pyrefly's internal type.
The critical detail is that a TSP
Typeis not just a display string. For classes, variables,TypeVars, and functions it carries a
Declarationwith aNode { uri, range }— a pointer backto where the symbol is defined. Pylance follows that node to the source to resolve the symbol's
real identity. If the declaration is missing, or the type is an unrecognized "built-in" sentinel,
Pylance has nothing to resolve and falls back to
Unknown.Before this PR, the converter took a shortcut for a whole family of internal types: it emitted
them via
builtin(name), which produces aTspType::BuiltInType { declaration: None, name }.The
builtinslot is meant for the small fixed set of sentinel names Pylance knows(
any,never,none,unknown, …). But we were stuffing real typing constructs into it:SpecialForm(Literal,Final,ClassVar, …)builtin("Literal")UnknownLiteralStringbuiltin("LiteralString")UnknownConcatenatebuiltin("Concatenate")UnknownParamSpecValuebuiltin("ParamSpec")UnknownTypedDictbuiltin("TypedDict")UnknownTypeAlias::Refbuiltin("<name>")UnknownQuantified/ importedTypeVarUnknownVar/Materializationbuiltin("Unknown")(capital U)Pylance doesn't recognize
Literal/TypedDict/ etc. as builtins, and there's no declarationnode to chase, so it renders
Unknown. That is the bug in #8062.What this PR changes
The fix is: stop emitting opaque builtins; emit real, resolvable declarations. To do that the
converter needs a way to map
(module, name) → (source path, range). That requirement drives everychange here.
New
ExportLocationResolvercallback(
type_conversion.rs), threaded as a 4th resolver throughconvert_type_with_resolvers. The converter is stateless and must not touch the type-checkerstate directly, so location lookup is injected as a callback.
TspInterface::resolve_export_location+Serverimpl(
lsp/non_wasm/server.rs). This implements the callback byreusing Pyrefly's existing export machinery —
Transaction::lookup_export_location— whichalready follows re-exports correctly (the same logic goto-definition uses). That method is
made
pub(crate)rather than reimplementing the export walk (DRY).Converter arms rewritten to produce real handles:
SpecialForm,LiteralString,Concatenate,ParamSpecValue, anonymousTypedDict, andTypeAlias::Refnow build a realtyping.<name>ClassType(typing_class) whosedeclaration resolves to typeshed's
typing.pyi. If resolution fails, they still fall back to azero-range
typing.pyideclaration — a typed handle, never an opaque builtin.Quantified/QuantifiedValuego throughconvert_quantified, which uses theQuantifiedIdentity(module + name + origin) carried by the solver TypeVar to resolve the realdeclaration. PEP 695 type params (
def f[T]()) getDeclarationCategory::Typeparam; legacyT = TypeVar("T")getVariable(in the consumer it really is a module-level variable). Thisis what fixes "imported TypeVars show as unknown."
Args,Kwargs,ElementOfTypeVarTuple) gothrough
synthesized_typevar: still a TSP TypeVar (so Pylance renders a TypeVar), just withouta source node — the honest fallback for things that have no user-visible declaration.
Var/Materializationnow emit the lowercasebuiltin("unknown")— the actualprotocol-conformant sentinel — so even true "unknown" renders consistently.
Cold-transaction stdlib panic fix.
resolve_export_location→lookup_export_location→demand(Step::Exports)→get_stdlib. A freshstate.transaction()inherits an empty stdlibmap until a check runs, so
get_stdlibpanicked (Option::unwrap()onNone) on a coldtransaction. We run the target handle for
Require::Exportsfirst, which invokescompute_stdlib(idempotent / cached) before the demand. This is needed only because the newresolver is the first TSP path that demands exports rather than just reading already-computed
state.
Alternatives considered
the client, still yields no declaration/location (goto-def and typeshed hover docs stay broken),
and can't help imported user types/functions/TypeVars. Treats the symptom for a subset.
docs break. We use this only as the fallback (synthesized declarations) for placeholders that
truly have no location.
Typeduring checking. Bloats the core representation withIDE-only data and costs memory/time on every batch check. Locations are inherently per-query (they
depend on the source module's import context), so a lazy callback at conversion time is the right
layer.
resolve_module_uri/resolve_func_def_rangeonly coverModuleTypeand functions with adef_index; they structurally can't cover special forms,QuantifiedTypeVars, or imported functions whoseFuncIdlacks adef_index. A(module, name) → locationresolver was the missing primitive.get_stdlibreturn a bootstrapping stdlib would hide bugs and violatethe repo's "unreachable states must panic" rule; warming the fresh transaction with a cheap,
cached
Require::Exportsrun is the minimal correct fix.Tests
type_conversion.rscovering specialforms,
LiteralString,Concatenate,ParamSpec, anonymousTypedDict,Var/Materialization,and PEP 695 vs. legacy
Quantifiedresolution.test/export_location.rspinning thecold-transaction behavior: running for
Require::Exportsfirst succeeds; skipping it reproduces theoriginal
get_stdlibpanic.cargo test -p pyrefly --libsuite passes.