Building JSONiq Language Tools, Part 1

As part of my master's thesis, I’m currently building language tools for JSONiq, a declarative query language made for denormalized data. There are multiple implementations of this language, and I will use RumbleDB as a reference.

In this post, the goal is simply to get the first development loop working: a basic language server, a VS Code extension, and enough plumbing to iterate on features later.

An example of a JSONiq query is the following:

let $sequence := 1 to 10
for $value in $sequence
let $double := $value * 2
where $double < 10
return $double

Before I started my work, there was already an existing set of language tools built by previous students:

• JSONiq VSCode Extension
• JSONiq Language Server 1.1.3

However, they had not been updated for more than a year, which is why I decided to pick the project up again and update it to keep up with the latest language features.

The current implementation of RumbleDB does not provide native support for a language server. It does have CLI flags like --print-iterator-tree yes, which give the internal shape of an input query, but that is not enough for language server protocol features¹.

The goal is to create a language server with as many features as possible, aligning with the functionality of the query engine.

It’s my first time building language tools, so I’m learning while building, and there are probably things that I do which are not best practice. Feel free to write me an email if you find any mistake, or if you find this post useful. This series of posts serves as notes for myself, so it will contain details which are not very technical, like setting up the CI pipeline and dealing with API tokens.

GitHub repository: RumbleDB/jsoniq-lsp: The official JSONiq language server

1. Initial tool stack

I’m more familiar with JavaScript, and the implementation of microsoft/vscode-languageserver-node is rather standard and easy to integrate with a VS Code extension later, so I decided to go with Node.js for the server implementation. There shouldn’t be any problem using a different language though, since the protocol itself is language agnostic.

The original grammar is defined using ANTLR, and antlr-ng can generate parsers and lexers for the runtime. We will also use this tool later to generate visitor / listener pattern abstract classes.

These two tools should be good enough to start building basic features like go to definition and unresolved reference diagnostics.

As for the code structure, I decided to go with a monorepo, and place both the language server package and the VS Code extension in the same repository. The extension package declares the language server as a workspace dependency, and pnpm will handle the symlink automatically. This is very useful for reusing type declarations and making sure that the versions of both stay in sync.

The resulting folder structure:

├── .vscode
│   ├── launch.json					# Define a launch configuration for the extension
├── packages
│   ├── language-server
│   ├── rumble-lsp-wrapper			# Java backend for richer diagnosis
│   └── vscode-extension
├── package.json

In the root package.json, define a global build task which triggers a rebuild of all subpackages. In this way, I can be sure that each time I start the extension, it will be using the latest version of the code (or fail to start because there’s some error).

2. Syntax highlighting in VSCode

For VS Code, syntax highlighting (based on tokenization) is defined using a TextMate grammar, which is a collection of regular expressions written in JSON files and referenced through the extension package.json:

{
    "name": "jsoniq-vscode",
    "version": "1.1.0",
    "private": true,
    "main": "./dist/extension.js",
    "contributes": {
        "languages": [
            {
                "id": "jsoniq",
                "aliases": [
                    "JSONiq"
                ],
                "extensions": [
                    ".jq",
                    ".jsoniq"
                ],
                "configuration": "./language-configuration.json"
            }
        ],
        "grammars": [
            {
                "language": "jsoniq",
                "scopeName": "source.jsoniq",
                "path": "./syntaxes/jsoniq.tmLanguage.json"
            }
        ],
    },
    "engines": {
        "vscode": "^1.120.0"
    }
}

This is a simplified version of jsoniq.tmLanguage.json, with rules for variable and type declaration:

{
    "$schema": "https://raw.githubusercontent.com/martinring/tmlanguage/master/tmlanguage.json",
    "name": "JSONiq",
    "scopeName": "source.jsoniq",
    "patterns": [
        {
            "include": "#variables"
        },
        {
            "include": "#types"
        }
    ],
    "repository": {
        "variables": {
            "patterns": [
                {
                    "name": "variable.other.jsoniq",
                    "match": "\\$(?:[A-Za-z_][A-Za-z0-9._-]*:)?[A-Za-z_][A-Za-z0-9._-]*"
                }
            ]
        },
        "types": {
            "patterns": [
                {
                    "name": "storage.type.jsoniq",
                    "match": "\\b(?:item|node|object-node|array-node|string|integer|decimal|double|float|boolean|null|document-node|text|comment|binary)\\b"
                }
            ]
        }
    }
}

The regex for a variable name depends strongly on the language grammar. In our case, it must start with a dollar $ sign, and can be followed with a namespace prefix (?:[A-Za-z_][A-Za-z0-9._-]*:)?, and finish with the local name of the variable.

On the other hand, the pattern for types includes a list of built-in type keywords. This is not enough, because we can also define custom types in the query. We will complete the syntax highlighting with semantic highlighting, which gives richer information. The two are complementary: static highlighting is faster, while semantic highlighting is more accurate.

We also have language-configuration.json, which defines features like:

• Comment toggling
• Brackets definition
• Autoclosing
• Autosurrounding
• Folding
• Word pattern
• Indentation Rules

These are more straightforward to define. See the official VS Code documentation for reference: Language Configuration Guide

3. Setting up the basic language server

The main file of the language server should create a new language server connection, and declare all the capabilities of the server. Since this file depends a lot on the features implemented by the server, here is a very basic example, where the server can return syntax diagnostics (e.g. syntax errors) and document symbols for the outline view in VS Code.

import { TextDocument } from "vscode-languageserver-textdocument";
import {
    TextDocumentSyncKind,
    createConnection,
    ProposedFeatures,
    TextDocuments,
    type InitializeParams,
    type InitializeResult,
} from "vscode-languageserver/node.js";

import { parseDocument } from "./parser/index.js";
import { supportsDocument } from "./parser/registry.js";

const connection = createConnection(ProposedFeatures.all);
const documents = new TextDocuments(TextDocument);

connection.onInitialize(async (_params: InitializeParams): Promise<InitializeResult> => {
    return {
        capabilities: {
            /// Add more capabilities here as we implement them
            textDocumentSync: TextDocumentSyncKind.Incremental,
            documentSymbolProvider: true,
        },
        serverInfo: {
            name: "JSONiq Language Server",
            version: require("../package.json").version,
        },
    };
});

documents.onDidOpen(async (event) => {
    await refreshDiagnostics(event.document.uri);
});

documents.onDidChangeContent(async (event) => {
    await refreshDiagnostics(event.document.uri);
});

documents.onDidClose((event) => {
    connection.sendDiagnostics({
        uri: event.document.uri,
        diagnostics: [],
    });
});

documents.listen(connection);
connection.listen();

Running this file in the terminal starts the language server.

4. Connecting the VS Code extension to the language server

Assuming that the compiled server entrypoint is saved in packages/language-server/dist/bundled/main.mjs, and it’s exported in the package.json of the language server:

{
    "exports": {
        "./bundled": "./dist/bundled/main.mjs"
    }
}

In the VS Code extension package, declare the language server as a workspace dependency:

{
    "dependencies": {
        "jsoniq-language-server": "workspace:*",
        "vscode-languageclient": "^9.0.1"
    }
}

Then in extension.ts, we can resolve the path to the language server by using require.resolve:

const serverModule = require.resolve("jsoniq-language-server/bundled");

It’s also possible to manually build the path string as shown in Extension Guide:

typescript

let serverModule = context.asAbsolutePath(path.join('server', 'out', 'server.js'));

But with this monorepo setup I believe it’s easier to do module resolution, so in both production and development environments we can use the same logic.

This would be an example of the main script of the extension, where we define the location of the language server, and the new language that we want to support:

import * as vscode from "vscode";
import {
    LanguageClient,
    type LanguageClientOptions,
    type ServerOptions,
    TransportKind,
} from "vscode-languageclient/node.js";

import { JSONIQ_LANGUAGE_ID } from "./const.js";

let client: LanguageClient | undefined;

export async function activate(context: vscode.ExtensionContext): Promise<void> {
    const serverModule = require.resolve("jsoniq-language-server/bundled");

    const serverOptions: ServerOptions = {
        run: {
            module: serverModule,
            transport: TransportKind.stdio,
        },
        debug: {
            module: serverModule,
            transport: TransportKind.stdio,
            options: {
                execArgv: ["--nolazy", "--inspect=6009"],
                env: {
                    ...process.env,
                    JSONIQ_LSP_DEBUG: "1",
                },
            },
        },
    };

    const clientOptions: LanguageClientOptions = {
        documentSelector: [
            { scheme: "file", language: JSONIQ_LANGUAGE_ID },
        ],
    };

    client = new LanguageClient(
        "jsoniqLanguageServer",
        "JSONiq Language Server",
        serverOptions,
        clientOptions,
    );

    context.subscriptions.push(client);
    await client.start();
}

export async function deactivate(): Promise<void> {
    if (client !== undefined) {
        await client.stop();
        client = undefined;
    }
}

5. Wrapping up

In .vscode/launch.json, define a launch configuration that starts the development window of the VS Code extension:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Run JSONiq Client",
            "type": "extensionHost",
            "request": "launch",
            "runtimeExecutable": "${execPath}",
            "args": ["--extensionDevelopmentPath=${workspaceFolder}/packages/vscode-extension"],
            "outFiles": ["${workspaceFolder}/packages/vscode-extension/dist/**/*.js"],
            "preLaunchTask": "npm: build"
        }
    ]
}

If everything goes well, a new window should appear, with our new language available in the language selection, and the original window now showing the debug toolbar:

I recommend creating a new profile (bottom-left corner -> profile) and selecting it in the extension development host window, so you can disable all the other extensions without affecting your main environment. Also use a different color theme to easily distinguish the two windows (in my case, I have set the extension development host window to use a yellow theme).

What’s next…

I’ll probably write a little bit about my CI pipeline setup and the publishing process in the next post, and then dive into more technical details of different feature implementations.

1. For example, RumbleDB does not currently store the full range (start and end position) of a node, which makes language server protocol functionalities like finding the symbol at a position hard or impossible to implement. ↩