Skip to content

[TSP] Emit resolvable declarations for special forms, TypeVars, and imported symbols#3716

Open
rchiodo wants to merge 4 commits into
facebook:mainfrom
rchiodo:rchiodo/conversionIssues
Open

[TSP] Emit resolvable declarations for special forms, TypeVars, and imported symbols#3716
rchiodo wants to merge 4 commits into
facebook:mainfrom
rchiodo:rchiodo/conversionIssues

Conversation

@rchiodo

@rchiodo rchiodo commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Summary

When Pylance uses the Pyrefly TSP (Type Server Protocol) backend, hovering over imported
symbols — os, Literal, TypedDict, overload, imported user types/functions, and
TypeVars — renders them as unknown instead of their real names/types. The same symbols
resolve correctly under Pyright's native type server, so this is specifically a gap in how
Pyrefly converts its internal types into the TSP wire format.

This PR fixes that conversion gap so those constructs come across as real, resolvable handles.

Fixes the Pylance-side report:
microsoft/pylance-release#8062 — [TSP] Types and functions imported from other modules show up as unknown when using Pyrefly

Why this is a TSP-specific problem

TSP is a boundary. Pyrefly computes everything in its internal representation
(pyrefly_types::Type); pyrefly/lib/tsp/type_conversion.rs
translates that into the wire tsp_types::Type that Pylance consumes. Pylance renders
hover / completion / goto from the wire type — it never sees Pyrefly's internal type.

The critical detail is that a TSP Type is not just a display string. For classes, variables,
TypeVars, and functions it carries a Declaration with a Node { uri, range } — a pointer back
to where the symbol is defined. Pylance follows that node to the source to resolve the symbol's
real identity.
If the declaration is missing, or the type is an unrecognized "built-in" sentinel,
Pylance has nothing to resolve and falls back to Unknown.

Before this PR, the converter took a shortcut for a whole family of internal types: it emitted
them via builtin(name), which produces a TspType::BuiltInType { declaration: None, name }.
The builtin slot is meant for the small fixed set of sentinel names Pylance knows
(any, never, none, unknown, …). But we were stuffing real typing constructs into it:

Internal type (before) Emitted as Result in Pylance
SpecialForm (Literal, Final, ClassVar, …) builtin("Literal") Unknown
LiteralString builtin("LiteralString") Unknown
Concatenate builtin("Concatenate") Unknown
ParamSpecValue builtin("ParamSpec") Unknown
anonymous TypedDict builtin("TypedDict") Unknown
TypeAlias::Ref builtin("<name>") Unknown
Quantified / imported TypeVar (no resolvable decl) Unknown
Var / Materialization builtin("Unknown") (capital U) inconsistent sentinel

Pylance doesn't recognize Literal / TypedDict / etc. as builtins, and there's no declaration
node to chase, so it renders Unknown. That is the bug in #8062.

What this PR changes

The fix is: stop emitting opaque builtins; emit real, resolvable declarations. To do that the
converter needs a way to map (module, name) → (source path, range). That requirement drives every
change here.

  1. New ExportLocationResolver callback
    (type_conversion.rs), threaded as a 4th resolver through
    convert_type_with_resolvers. The converter is stateless and must not touch the type-checker
    state directly, so location lookup is injected as a callback.

  2. TspInterface::resolve_export_location + Server impl
    (lsp/non_wasm/server.rs). This implements the callback by
    reusing Pyrefly's existing export machinery — Transaction::lookup_export_location — which
    already follows re-exports correctly (the same logic goto-definition uses). That method is
    made pub(crate) rather than reimplementing the export walk (DRY).

  3. Converter arms rewritten to produce real handles:

    • SpecialForm, LiteralString, Concatenate, ParamSpecValue, anonymous TypedDict, and
      TypeAlias::Ref now build a real typing.<name> ClassType (typing_class) whose
      declaration resolves to typeshed's typing.pyi. If resolution fails, they still fall back to a
      zero-range typing.pyi declaration — a typed handle, never an opaque builtin.
    • Quantified / QuantifiedValue go through convert_quantified, which uses the
      QuantifiedIdentity (module + name + origin) carried by the solver TypeVar to resolve the real
      declaration. PEP 695 type params (def f[T]()) get DeclarationCategory::Typeparam; legacy
      T = TypeVar("T") get Variable (in the consumer it really is a module-level variable). This
      is what fixes "imported TypeVars show as unknown."
    • Genuinely locationless solver placeholders (Args, Kwargs, ElementOfTypeVarTuple) go
      through synthesized_typevar: still a TSP TypeVar (so Pylance renders a TypeVar), just without
      a source node — the honest fallback for things that have no user-visible declaration.
    • Var / Materialization now emit the lowercase builtin("unknown") — the actual
      protocol-conformant sentinel — so even true "unknown" renders consistently.
  4. Cold-transaction stdlib panic fix. resolve_export_locationlookup_export_location
    demand(Step::Exports)get_stdlib. A fresh state.transaction() inherits an empty stdlib
    map until a check runs, so get_stdlib panicked (Option::unwrap() on None) on a cold
    transaction. We run the target handle for Require::Exports first, which invokes
    compute_stdlib (idempotent / cached) before the demand. This is needed only because the new
    resolver is the first TSP path that demands exports rather than just reading already-computed
    state.

Alternatives considered

  • Fix it in Pylance (teach the client more builtin names). Pushes type-system knowledge into
    the client, still yields no declaration/location (goto-def and typeshed hover docs stay broken),
    and can't help imported user types/functions/TypeVars. Treats the symptom for a subset.
  • Emit the right name but no location. Pylance would show the name but goto-definition and hover
    docs break. We use this only as the fallback (synthesized declarations) for placeholders that
    truly have no location.
  • Bake locations into the core Type during checking. Bloats the core representation with
    IDE-only data and costs memory/time on every batch check. Locations are inherently per-query (they
    depend on the source module's import context), so a lazy callback at conversion time is the right
    layer.
  • Reuse the existing resolvers. resolve_module_uri / resolve_func_def_range only cover
    ModuleType and functions with a def_index; they structurally can't cover special forms,
    Quantified TypeVars, or imported functions whose FuncId lacks a def_index. A
    (module, name) → location resolver was the missing primitive.
  • For the panic: making get_stdlib return a bootstrapping stdlib would hide bugs and violate
    the repo's "unreachable states must panic" rule; warming the fresh transaction with a cheap,
    cached Require::Exports run is the minimal correct fix.

Tests

  • New conversion tests in type_conversion.rs covering special
    forms, LiteralString, Concatenate, ParamSpec, anonymous TypedDict, Var/Materialization,
    and PEP 695 vs. legacy Quantified resolution.
  • New regression tests in test/export_location.rs pinning the
    cold-transaction behavior: running for Require::Exports first succeeds; skipping it reproduces the
    original get_stdlib panic.
  • Full cargo test -p pyrefly --lib suite passes.

@meta-cla meta-cla Bot added the cla signed label Jun 8, 2026
@rchiodo rchiodo marked this pull request as ready for review June 8, 2026 18:28
@github-actions

github-actions Bot commented Jun 8, 2026

Copy link
Copy Markdown

According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅

@kinto0 kinto0 self-assigned this Jun 8, 2026
@meta-codesync

meta-codesync Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

@kinto0 has imported this pull request. If you are a Meta employee, you can view this in D107922614.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants