Building JSONiq Language Tools, Part 5

In the previous chapters, we built a parsing and analysis pipeline that produces the following result:

typescript

type AnalysisResult = {
    ast: ModuleNode;            // The root of the analysis AST
    scope: Scope;               // The root scope
    diagnostics: Diagnostic[];  // Semantic diagnostics found during analysis
};

This chapter explains how this shared semantic model is used to implement Go to Definition, Find References, Rename, and Completion.

Since all these features query the same analysis results, they are cached by document URI and version. Several LSP requests may be issued for the same version of a document, and rebuilding the analysis each time would repeat the same work. When the document version changes, the cached result is replaced with a new one.

Finding the semantic node at the cursor

Many LSP features are position-based. For example, when a user invokes Go to Definition, the client sends the URI of the current document and the cursor position.

An LSP position consists of a zero-based line and character:

typescript

interface Position {
    line: number;
    character: number;
}

The analysis AST already stores source ranges using the same coordinate system¹. Therefore, locating the node at the cursor is mainly a tree search: discard nodes whose ranges do not contain the position, then continue through their children until the most specific matching node is found.

typescript

function findContainingNodePath(
    node: AstNode,
    position: Position,
): AstNode[] | undefined {
    // Stop searching this branch if it does not contain the cursor.
    if (!rangeContainsPosition(node.range, position)) {
        return undefined;
    }

    for (const child of node.children) {
        const match = findContainingNodePath(child, position);

        if (match !== undefined) {
            return [node, ...match];
        }
    }

    return [node];
}

This function returns the complete path from the root to the innermost matching node rather than only returning the leaf. For example, consider a cursor inside the second argument of a function call:

jsoniq

local:add($x, $y)

The result may contain the following nodes:

module
└── function-call: local:add#2
    └── argument: index 1
        └── variable-reference: $y

Keeping the ancestor nodes preserves the context around the cursor. In this example, the variable-reference node identifies $y, while the enclosing argument and function-call nodes show that it is the second argument of local:add#2. This surrounding context will be used by type-aware features such as signature help in the next chapter.

Finding symbols

Navigation features need a more specific query. They should only activate when the cursor is on the name of a declaration or reference.

A declaration node has two ranges. Its range covers the entire declaration, while its selectionRange covers only the declared name:

jsoniq

declare function local:double($value) {
    $value * 2
};

For this declaration, the full range covers the complete function, but the selection range only covers local:double. If symbol lookup used the full range, invoking Go to Definition from almost anywhere in the function body could incorrectly select the function declaration.

References already have a narrow range covering the referenced name. A symbol-specific traversal can therefore inspect declaration selection ranges and reference ranges separately. The relevant part of the search can be summarized as follows:

typescript

function findSymbolOccurrenceAtPosition(
    node: AstNode,
    position: Position,
): SymbolOccurrence | undefined {
    if (!rangeContainsPosition(node.range, position)) {
        return undefined;
    }

    for (const child of node.children) {
        const match = findSymbolOccurrenceAtPosition(child, position);
        if (match !== undefined) {
            return match;
        }
    }

    if (
        node.kind === "declaration" &&
        rangeContainsPosition(node.declaration.selectionRange, position)
    ) {
        return {
            range: node.declaration.selectionRange,
            declaration: node.declaration,
            reference: undefined,
        };
    }

    if (node.kind === "reference" && node.resolution !== undefined) {
        return {
            range: node.range,
            declaration: node.resolution.declaration,
            reference: node.resolution,
        };
    }

    return undefined;
}

Children are searched before their parents so that traversal reaches the narrowest node containing the cursor before inspecting the current node. If no child represents a symbol occurrence, the current declaration or reference is checked.

The result is normalized by the SymbolOccurrence interface:

typescript

interface SymbolOccurrence {
    range: Range;
    declaration: Definition;
    reference: ResolvedReference | undefined;
}

When the cursor is on a declaration, reference is undefined and declaration is the declaration itself. When it is on a resolved reference, declaration is the definition found during semantic analysis and reference records the selected occurrence.

This small query becomes the common entry point for Go to Definition, Find References, and Rename. Those features do not need to resolve scopes again. They only need to locate the occurrence under the cursor and use the semantic links that were already established while building the analysis AST.

Unresolved references remain in the analysis AST so that diagnostics and surrounding editor features can still operate. However, because they are not connected to a declaration, they cannot be used as targets for Go to Definition, Find References, or Rename.

Go to Definition

With symbol lookup in place, implementing Go to Definition becomes straightforward. The analysis stage has already resolved each reference to its declaration, so the feature only needs to find the symbol at the cursor and convert its declaration into an LSP location:

typescript

function findDefinitionLocation(
    document: TextDocument,
    position: Position,
): Location | null {
    const analysis = getAnalysis(document);
    const occurrence = findSymbolAtPosition(analysis, position);
    const declaration = occurrence?.declaration;

    if (!isSourceDefinition(declaration)) {
        // Check that the declaration exists and comes from the source document.
        return null;
    }

    return {
        uri: document.uri,
        range: declaration.selectionRange,
    };
}

Name collisions are not an issue because they were already resolved according to the scope rules during analysis.

Find References

Find References performs the inverse operation: given a declaration or one of its references, it returns every occurrence associated with that declaration.

The analysis stage already maintains this relationship in both directions. A resolved reference points to its declaration, and each definition stores the references that resolved to it:

typescript

interface AbstractDefinition<K extends DefinitionKind> {
    name: DefinitionNameByKind[K];
    kind: K;
    references: ResolvedReference[]; 
    isBuiltin: boolean;
}

As each reference is resolved, it is added to the definition's reference list. Find References can therefore reuse these links without traversing and resolving the entire AST again.

typescript

function findReferenceLocations(
    document: TextDocument,
    position: Position,
    includeDeclaration: boolean,
): Location[] {
    const analysis = getAnalysis(document);
    const occurrence = findSymbolAtPosition(analysis, position);
    const targetDeclaration = occurrence?.declaration;

    if (!isSourceDefinition(targetDeclaration)) {
        return [];
    }

    const locations: Location[] = [];

    if (includeDeclaration) {
        locations.push({
            uri: document.uri,
            range: targetDeclaration.selectionRange,
        });
    }

    for (const reference of targetDeclaration.references) {
        locations.push({
            uri: document.uri,
            range: reference.range,
        });
    }

    return locations;
}

For example, consider the following query with two declarations of $value:

jsoniq

let $value := 1
return (
    $value,
    let $value := 2
    return $value
)

When references are requested for the inner declaration, only the final $value is returned. The reference before the inner let belongs to the outer declaration and is excluded.

The screenshot shows the same behavior in VS Code: references are grouped by their resolved declaration rather than by their text.

Rename

The Rename feature allows a declaration and all its references to be changed at the same time. It uses the same analysis information as Go to Definition and Find References.

The operation has two steps. First, the editor asks whether the symbol under the cursor can be renamed (prepareRename request). The language server returns the range that may be renamed and a placeholder containing its current name, or null when Rename is unavailable. The editor then asks the user for a new name and sends it to the server.

Before applying it, the server checks that the new name is valid. A JSONiq variable name must start with $; for example, $value and $input:value are valid, while value and $ are rejected.

Once the name has been validated, the server creates one text edit for the declaration and one for each resolved reference. These edits are returned to the editor as a WorkspaceEdit object.

Completion

The completion system provides useful suggestions at the current cursor position, including:

Variables and parameters visible at this position
User-defined functions and types
Built-in functions and types
Object fields inferred from the expression type
Valid keywords

To produce context-sensitive suggestions, the system is divided into two parts.

Grammar-based completion

Completion requests are usually triggered while the user is still typing, meaning that the current document may not yet be syntactically correct. Therefore, the first question that needs to be answered is not which symbols exist, but rather what kinds of constructs are syntactically valid at the current cursor position.

This also means that completion depends on the parser's error recovery. As discussed in Part 3, ANTLR can preserve a useful parse tree around missing or unexpected tokens. Declarations recovered before the error can still be included in the analysis and offered as completion items.

To obtain this information, we rely on the language grammar and the antlr4-c3 library. Given a token stream and a cursor position, antlr4-c3 traverses the parser automaton and determines which tokens can legally appear next according to the grammar.

For example, in the following incomplete query, where | denotes the cursor position:

jsoniq

let $value := 1
return |

antlr4-c3 can determine that an expression is expected after the return keyword and return tokens that can begin an expression, such as $, if, or (.

Token candidates are sufficient for keywords and symbols, but not for semantic entities such as variables, functions, and types. For example, $ can begin both a variable reference and a declaration. To suggest visible variables such as $value and $count, the completion system must know which construct is expected.

To obtain this higher-level information, antlr4-c3 supports a preferred rules mechanism. When it reaches one of these rules, it returns the rule as a completion candidate instead of continuing to its individual tokens. The candidate also records the rule stack that led to it.

In the previous example, a possible rule stack could be:

exprSingle
└── exprSimple
    └── orExpr
        └── … expression-precedence rules …
            └── postfixExpr
                └── primaryExpr
                    └── varRef

Without preferred rules, C3 continues through varRef and returns DOLLAR as a possible token. When varRef is preferred, C3 returns the rule itself instead. The language server can interpret this candidate as a request for visible variables and suggest $value. Candidates from other grammar paths are still collected, so keyword tokens such as KW_IF can be offered alongside those semantic suggestions.

Preferred rules should be used selectively. Each preferred rule replaces token candidates found through its grammar path, so broad or overlapping rules can hide more specific information, including valid keyword candidates. They also make the returned candidates depend more heavily on the structure and traversal of the rule stack. For this reason, the language server limits the preferred set to rules that provide useful semantic information.

The language server also derives a lightweight token context from the tokens immediately before the cursor. This recognizes positions with a clear local pattern without requiring another traversal of the parser automaton. For example, a cursor after AS expects a type name, while one after let $ expects a new variable declaration rather than a reference.

Together, the C3 candidates and token context are translated into a completion intent describing whether the current position allows:

Variable declarations
Variable references
Function calls
Type references
Object field lookups (to suggest available object fields)
Keywords (and which ones are accepted in the current position)

The grammar layer can recognize an object-field lookup position, but producing the field names also requires the type information, which will be introduced in the next chapter.

The completion pipeline is shared by the JSONiq and XQuery adapters. Each adapter provides its grammar-specific token-context analyzer, ignored tokens, preferred rules, keyword list, and functions that identify preferred rules such as variable references, function calls, and object lookups. The shared code combines this configuration into the same CompletionIntent representation.

typescript

getCompletionIntent(parsed, cursorOffset, {
    tokenContextAnalyzer,
    ignoredTokens,
    preferredRules,
    languageKeywords,
    isVariableReferenceRule,
    isFunctionCallRule,
    isObjectLookupRule,
});

This is another benefit of the parser-adapter boundary introduced in Part 3. JSONiq and XQuery use different generated parser classes, but the shared completion code converts their candidates into the same CompletionIntent representation. The remaining completion service can therefore operate independently of the selected grammar.

Analysis-based completion

The analysis-based part provides completion items for declarations in the source document, such as variables, parameters, functions, and custom types. Unlike Go to Definition, it does not begin with an existing reference. Instead, it must determine which declarations are visible where the user is writing.

Consider the following incomplete query:

jsoniq

let $outer := 1
return (
    let $inner := 2
    return $|
)

At the cursor position, the completion list should contain $outer and $inner. Both variables have already been declared and are visible from the scope containing the cursor.

To obtain this result, the language server first finds the innermost scope containing the cursor by traversing the scope tree. It then collects declarations whose visibleFrom position is before the cursor. If a name is not declared in the current scope, declarations from its parent scopes are also considered. This prevents a declaration from being suggested before it becomes available while still making outer declarations accessible.

For example, $double should not be suggested in its own initialization expression because the cursor is before its visibleFrom position.

jsoniq

let $double := $|

However, it should be available in the following return expression:

jsoniq

let $double := 2
return $|

If two visible declarations have the same name, the declaration in the closest scope is used:

jsoniq

let $value := 1
return (
    let $value := 2
    return $|
)

Here, completion should contain only the inner $value. Suggesting both declarations would not be useful because a reference written at this position resolves to the inner declaration.

Function declarations use a slightly different rule. A function is registered before its body is analyzed, allowing references in the body to resolve recursively. Its visibleFrom value is the end of its selectionRange, so it can also appear in the completion list inside its own body. For most lexical bindings, visibility begins after the complete binding.

Combining both parts

flowchart LR
    A["Grammar candidates and token context"] --> B["Allowed categories<br/>variables, functions, types, keywords"]
    C["Scope analysis"] --> D["Visible declarations"]
    B --> E["Combine and filter"]
    D --> E
    E --> F["Final completion items"]

The completion service filters visible declarations according to the categories permitted by the grammar. Built-in functions and types are stored separately from the source analysis, but they are added when their corresponding category is allowed.

For example, consider the following query:

jsoniq

let $value := 1
return $|

At the cursor position, the grammar indicates that a variable reference is expected, while the semantic analysis determines that $value is visible. Consequently, $value is included in the completion list.

The distinction is clearer at a declaration position:

jsoniq

let $|

The scope analysis may still find variables declared in an outer scope, but the grammar indicates that the cursor expects a new variable declaration rather than a reference. Existing variable names are therefore not returned. In the current implementation, the server only suggests $ as the beginning of the declaration.

Neither component is sufficient on its own. The grammar determines what kind of construct is valid, while the analysis determines which source declarations can be referenced. Combining them produces suggestions that are both syntactically appropriate and semantically visible.

Conclusion

The features discussed in this chapter all reuse the semantic model built during analysis. Go to Definition follows the link from a reference to its declaration, Find References uses the reverse link stored by each definition, and Rename converts those same relationships into text edits. Completion additionally queries the scope tree and combines visible declarations with the syntactic expectations reported by antlr4-c3.

The testing infrastructure used to verify these features—including incomplete documents, shadowed declarations, both parser adapters, and wrapper-backed requests—will be discussed in a later chapter.

The current implementation focuses on declarations and references within one document. Supporting navigation and rename across imported modules will require a workspace-level index that connects analysis results from multiple files. The next chapter will move to type-aware features, including static diagnostics and signature help, which require information from the Java wrapper around RumbleDB.

The exception is visibleFrom, which stores a source offset as a number. The LSP utility library provides conversions between offsets and positions. ↩