treesitter

Reports what a source file declares, in one vocabulary, across ten languages.

p, err := golang.New()   // or python.New(), csharp.New(), ...
defer p.Close()

symbols, err := p.Declares("pkg/service.go", content)

Declares takes content rather than a path to read, so a buffer that is not on a filesystem works the same as a file.

Modules

Directory	Module path	Grammar
treesitter/	go.dokimi.dev/treesitter	—
treesitter-go/	go.dokimi.dev/treesitter/go	tree-sitter-go v0.25.0
treesitter-python/	go.dokimi.dev/treesitter/python	tree-sitter-python v0.25.0
treesitter-java/	go.dokimi.dev/treesitter/java	tree-sitter-java v0.23.5
treesitter-rust/	go.dokimi.dev/treesitter/rust	tree-sitter-rust v0.24.2
treesitter-typescript/	go.dokimi.dev/treesitter/typescript	tree-sitter-typescript v0.23.2
treesitter-javascript/	go.dokimi.dev/treesitter/javascript	tree-sitter-javascript v0.25.0
treesitter-ruby/	go.dokimi.dev/treesitter/ruby	tree-sitter-ruby v0.23.1
treesitter-c/	go.dokimi.dev/treesitter/c	tree-sitter-c v0.24.2
treesitter-csharp/	go.dokimi.dev/treesitter/csharp	tree-sitter-c-sharp v0.23.5
treesitter-scala/	go.dokimi.dev/treesitter/scala	tree-sitter-scala v0.26.2

A grammar is cgo, so importing a language module means a C toolchain. Importing only the ones you need keeps the rest out of the build.

What a symbol carries

type Symbol struct {
    Name        string
    Kind        Kind
    Language    Language
    Span        Span
    Visibility  Visibility
    Parent      int          // index into the slice, -1 at the top level
    Modifiers   []string     // static, final, async, pub, export, readonly
    Annotations []Annotation // @Injectable, #[derive], `json:"id"`
    Doc         string
    Snippet     string
}

Parent is computed by span containment rather than by query patterns, so it holds for every grammar. It is an index rather than a name because names repeat: two classes in one file can both declare get.

Annotations are not symbols

A Java @interface declares an annotation type, and that is a symbol of KindAnnotation. Applying one — @Injectable, #[derive(Debug)], @property, `json:"id"` — binds no name, so it is not a symbol. It reaches the caller as an Annotation on the thing it is attached to, because it decides what a generator should emit.

if sym.Annotated("Injectable") { ... }
if sym.Modified("static")      { ... }

Four grammars spell this four ways — hung off the declaration, inside a wrapper node, as a preceding sibling, or nested in the modifiers node — and all four are read.

The vocabulary

Kind carries 23 values: module, package, file, type, struct, union, enum, enum-member, interface, annotation, function, method, constructor, property, macro, implementation, field, variable, constant, parameter, type-parameter, import, label.

Every value is a node kind at least one supported grammar declares; the set was taken from the grammars rather than guessed. Where a language draws a distinction the set does not carry, both sides map onto the nearest value. A Java class, a Python class, a Ruby class, a Scala object and a Go struct are all struct: they are one shape here, and a caller looking for that shape should not have to know which language answered.

Kind.Declares() reports whether a kind names something other code can refer to, so a caller listing an API can drop parameters, labels and imports in one check.

Visibility is three-valued. Go, Python and Ruby spell visibility in the name or by convention; the rest spell it as a modifier, which a name carries nothing of, so those report unknown rather than guessing exported. The modifier is still in Modifiers.

The queries are not upstream's

Each module ships a tags query derived from the grammar's own and extended. Upstream aims at code navigation and stops well short of a declaration set. Measured against each language's own parser, upstream's Go query reported no constant and no interface, and upstream's Python query had no method at all. Ruby ships no constant, C no struct, field or enum, and C# no enum, property or constructor.

Two facts about tree-sitter shape every query, both established by parsing rather than assumed:

• A pattern cannot say what it is not, so the general pattern for a shape also matches the specific ones. Outranks decides which kind survives.
• A declaration can bind more names than one pattern can capture, as Go's const a, b = 1, 2 does, so the remaining names are read off the node.

Trusting it

conformance.Run pins the whole outline of a fixture as an exact set: a symbol found and not listed fails as surely as one listed and not found.

conformance.Measure is the check that matters. It runs a real repository past the language's own parser and past this library, and reports recall, misses and extras by kind. What the language's parser says is true by construction.

Language	Oracle	Corpus	Result
Go	go/ast	Go standard library, 1,500 files	57,711 declarations, 0 missed
Python	CPython ast	Django, 2,929 files	51,565 declarations, 3 missed
Java, Rust, TypeScript, JavaScript, Ruby, C, C#, Scala	not yet written	—	fixture only

A fixture proves a query finds what somebody thought to write down. Only the corpus check proves it finds what the language has.

What it is not

A parser matched text. It did not resolve a name, follow an import or check a type. Two declarations sharing a name are indistinguishable, and a file that declares nothing looks like one that could not be understood. A caller that needs to know a name refers to one particular declaration needs a type checker.