feat(completion): extend filter completion beyond Identity/Object Index/Array Index

[ynqa]

Background

Current completion is intentionally scoped to a JSON-path-level subset:

• Identity
• Object Identifier-Index
• Array Index

This subset was chosen pragmatically: these completions can be derived directly from runtime JSON shape (path enumeration) without introducing a dedicated jq parser, error-tolerant AST recovery, or other complex semantic analysis.

Current design and why

Following that scoped approach, the current completion pipeline has two key properties:

1. jnv loads completion candidates incrementally (lazy-loading style) so startup and readline interaction stay responsive, even for large JSON inputs.

2. The path-enumeration logic itself is delegated to promkit (get_all_paths), and jnv builds completion behavior on top of that result set.

Lazy loading for JSON Paths

jnv uses a shared SuggestionStore so the editor can consume completion candidates while loading is still in progress.

• Implementation:
  • https://github.com/ynqa/jnv/blob/v0.7.1/src/completion.rs#L52-L102

How it works:

• The store is wrapped as Arc<Mutex<...>> (SharedSuggestionStore), so loading tasks and UI/editor logic can access the same state safely.
• A background task incrementally inserts discovered paths into store.paths.
• Progress (loaded_path_count, is_complete) is updated in the same shared state.
• The editor side reads from this shared store (collect_matches) and can show partial results immediately.

Path enumeration delegated to promkit (get_all_paths)

jnv does not implement its own path-enumeration engine.
Instead, it delegates JSON path extraction to promkit and builds completion behavior on top of those paths.

• jnv wrapper (get_all_paths)
• jnv call site to jsonz::get_all_paths
• promkit implementation (get_all_paths)

What we want to improve

• Expand completion beyond JSON-path-only cases into broader jq contexts.
• Support context-aware completion in pipeline/editing flows (e.g. after |, inside partially typed expressions).
• Optimize candidate ordering/ranking so the most relevant items appear first.

Constraints

• Preserve current responsiveness characteristics (non-blocking interaction, incremental loading).

References

• jq Shell Autocomplete jqlang/jq#1417
• https://github.com/wader/jq-lsp
• duckdb's code?
• Reddit conversations

[01mf02]

@ynqa wrote:

Are there any recommended ways to obtain completion candidates for jq queries in jaq?

Currently, jnv only supports completion for JSON paths, but I’m interested in providing richer jq-aware completions. I was wondering if there’s anything available at the parser level, or any other approach you’d suggest.

My original answer was:

Your completion candidate problem is fascinating, and quite non-trivial. I just checked, and you seem to support completion candidates only for a very restricted subset of jq filters; for example, .name yields candidates, whereas inserting a space before .name and . | .name does not. So I suppose that you have some kind of custom parser for jq path expressions (compound paths in jaq parlance). The general case --- auto-completing arbitrary jq filters --- is probably pretty difficult.

For example, what do you do when the user types: [.[].nam? Here's a rough draft of what might work: You might need to ignore parse errors (missing ] at the end!) and get an incomplete parse tree (using something like jaq_core::load::Parser::parse, but instead returning an incomplete parse tree instead of just errors). Then, you'd need to find the position in the parse tree where the cursor is, and probably adapt the parse tree a bit at that position, i.e. remove the .nam. At that position, you could insert a call to a native filter that writes outputs to a Rc<RefCell<Vec<...>>> of your liking, and you can then analyse the contents of that to perform the actual syntax completion. I would be lying if I'd say that this is easy. But if you are interested in going through with such a solution, I might be able to assist you. We could discuss this stuff in an issue in the jnv repository. Perhaps Mattias Wadman (@wader) also has some ideas.

I have since thought a bit more about this, and the process above could be simplified a bit. You do not need to create a new native filter and insert a call to it at the position of the cursor. Instead, you can basically "unpack" or "take the derivative of" the filter at the current position. This might be illustrated by an example:

.a[] | [f | .b<CURSOR>]

So here, the user wants to auto-complete the .b filter, perhaps to mean .bitrate or whatever is in the data at that point.
The solution here is to rewrite this filter to just .a[] | f --- that is the "derivative" of the filter with respect to the current position.
Then we can just get the outputs of that derivative, analyse it, and recognise that its first output is an object with a field bitrate.

This is the hard part. For autocompleting things like keys_ to keys_unsorted or $x to $xyz, this is quite a different, but probably an easier problem.

[01mf02]

I'm wondering whether you need to "get all paths" for autocompletion ... It would seem to me that it would suffice to only get paths relevant to the current autocompletion context, i.e. cursor position. That is, if you have .a.b | [.c<CURSOR>], you should not need to care about all other properties of the root object --- you only care about the paths in .a.b.

[01mf02]

I guess that the all paths strategy is useful to get a quick response when you only consider a simple set of compound paths as input filters, but when you consider more general jq filters, that does not cut it anymore: For example, consider the filter:

{first: 1, second: 2} | .fi<CURSOR>

Here, autocompletion should probably suggest .first, even if this might not even be part of the original input fed to jnv.

[ynqa]

I'm wondering whether you need to "get all paths" for autocompletion

So far I had only been thinking in terms of:

1. collect all paths first
2. then filter them

But you’re absolutely right — as long as we don’t compromise responsiveness (non-blocking interaction, incremental loading), “get all paths” is not necessarily a requirement.

{first: 1, second: 2} | .fi

This example also makes perfect sense — completion candidates can come from values that are not even present in the original input, which clearly goes beyond what path enumeration can provide.

[ynqa]

.a[] | [f | .b] => .a[] | f

• The “derivative” approach makes a lot of sense to me, but generalizing it into a robust strategy seems quite challenging...
• Also, even if we can derive such a filter, there might be performance concerns with large JSON inputs. If executing the resulting (derivative) jq filter takes too long and delays completion responses, we might need some kind of fallback or lightweight completion strategy to keep the interaction responsive.

[ynqa]

This is an example of using jq itself to implement jq query completion.

• https://github.com/wader/fq/blob/v0.17.0/pkg/interp/repl.jq#L50

Basically, in some SQL TUIs, the system infers a type from the current context and then selects a corresponding set of candidate completions based on that type:

• Maxteabag/sqlit
  • https://github.com/Maxteabag/sqlit/blob/v1.3.1.1/sqlit/domains/query/completion/core.py
  • https://github.com/Maxteabag/sqlit/blob/v1.3.1.1/sqlit/domains/query/completion/completion.py
• duckdb/duckdb
  • https://github.com/duckdb/duckdb/blob/v1.5.1/extension/autocomplete/autocomplete_extension.cpp#L47

[01mf02]

This is an example of using jq itself to implement jq query completion.

* https://github.com/wader/fq/blob/v0.17.0/pkg/interp/repl.jq#L50

That's cool! Note that it also evaluates queries to provide completion.

Basically, in some SQL TUIs, the system infers a type from the current context and then selects a corresponding set of candidate completions based on that type:

I suppose that by "type", you mean something like "function call", "variable" or "path expression", rather than "string", "integer", or "object with key 'a'"? That is, you are talking about types of expressions, not types of values?

[01mf02]

The “derivative” approach makes a lot of sense to me, but generalizing it into a robust strategy seems quite challenging...

I had yet another idea about this. In a nutshell, a simple strategy is to get values that are produced at some position in a filter, and infer completions from these values. The problem is: How to get these values?

The derivate idea is one way, but it might be a bit difficult to implement. This becomes evident when you consider autocompletion within function definitions. How to derive a filter inside a function? I have ideas how to do it, but when you consider recursive functions, it becomes quite hairy ...

Fortunately, I remembered something that I had thought about some time ago already. The idea is that we could introduce a new filter type yield $x, similar to break $x. Whereas break $x jumps to the corresponding label $x without returning any value, the idea is that yield $x would cause the corresponding label $x to return (yield) the value passed to yield $x, and then, the filter would be resumed at yield $x. That is, yield $x allows us to return all values encountered some position in the filter.

To apply this to the example: .a[] | [f | .b<CURSOR>] could be rewritten into:

label $JNV_CURSOR |
.a[] | [f | yield $JNV_CURSOR]
| empty # so that we do not return the final results of the filter, only those that `yield` produced

Bingo! That gives us exactly those values produced at the cursor position. And, what is better, the same technique works completely the same for definitions; that is, def f: .[0].<CURSOR> as $x | .[1:] | f; f could be translated into:

label $JNV_CURSOR |
def f: (.[0] | yield $JNV_CURSOR) as $x | .[1:] | f; f
| empty

I estimate that implementing yield $x as filter type in jaq would be pretty straightforward. It's a rather small modification from the existing break $x implementation. I would be motivated to implement that, if you are interested in going that route.

Also, even if we can derive such a filter, there might be performance concerns with large JSON inputs. If executing the resulting (derivative) jq filter takes too long and delays completion responses, we might need some kind of fallback or lightweight completion strategy to keep the interaction responsive.

I would personally not worry about that too much. As long as you keep the input(s) in memory as jaq_json::Val, I suspect that a very vast majority of user queries will yield results quickly enough to feel "real-time". Of course, if you try to do autocompletion after a filter that calculates the 1000000-th Fibonacci value, then yes, completion will take some time. But, to be honest, I expect that 90% of users will use jq/jaq/jnv mostly using path expressions, that is, something like .a[].b, perhaps sprinkled with some select. And in such scenarios, the approach shown above should yield results almost instantly.

In my experience, it pays off for most problems to concentrate only on the most general case, and to make that really fast, instead of creating two or more "parallel" structures. There are exceptions, of course, but this is my general experience, and this approach has served me very well in jaq. That's why I would advise to first try to go with a single, most general autocompletion strategy, instead of thinking already about lightweight fallback strategies, which will also take time to implement and maintain. I believe that we can make the general autocompletion strategy pretty fast.

[ynqa]

you are talking about types of expressions, not types of values?

Yes, that’s how I understand it. Here, “type” doesn’t seem to refer to a value type such as string or integer, but rather to the category of completion expected in the current context — for example, a table name, column reference, keyword, procedure, or operator.

In other words, it appears that the code infers the kind of syntactic or semantic element valid at the cursor position, and then selects candidates from that category.

And also I was thinking that Expect in parse.rs might be a natural starting point for completion, since it already captures what kind of token or construct the parser expects next (Var, Key, Ident, Str, Just(...), etc.).
My understanding is that this may not be sufficient for final candidate selection on its own—especially for broad cases like Expect::Term—but it seems like a useful parser-level signal from which completion categories could be derived.

---

[edited]

That's why I would advise to first try to go with a single, most general autocompletion strategy, instead of thinking already about lightweight fallback strategies, which will also take time to implement and maintain.

So if I understand you correctly, your suggestion is to focus first on a single general strategy based on context-/value-aware completion (i.e. observing the values flowing at the cursor position), rather than starting with parser-level signals like Expect or introducing separate fallback strategies too early—does that sound right?

---

[edited]

That said, I feel it might still be desirable to mix in at least some minimal function or keyword completions—for example, suggesting something like select when the user types se.

[ynqa]

yield $x

Sorry if this is a naive question, but how would we capture the values produced at yield $x from Rust?

[01mf02]

Yes, that’s how I understand it. Here, “type” doesn’t seem to refer to a value type such as string or integer, but rather to the category of completion expected in the current context — for example, a table name, column reference, keyword, procedure, or operator.

Perfect --- then we're on the same page. :)

And also I was thinking that Expect in parse.rs might be a natural starting point for completion, since it already captures what kind of token or construct the parser expects next (Var, Key, Ident, Str, Just(...), etc.).
My understanding is that this may not be sufficient for final candidate selection on its own—especially for broad cases like Expect::Term—but it seems like a useful parser-level signal from which completion categories could be derived.

I'm not sure whether using Expect would work out the way you expect (haha!) it. Expect is returned if the parsed expression is invalid; however, when we are parsing something like {first: 1} | .fi<CURSOR>, this is a perfectly parseable expression that does not yield any errors. So I do not think that Expect can be used here for autocompletion purposes.

So if I understand you correctly, your suggestion is to focus first on a single general strategy based on context-/value-aware completion (i.e. observing the values flowing at the cursor position), rather than starting with parser-level signals like Expect or introducing separate fallback strategies too early—does that sound right?

My suggestion would be to do autocompletion based on the type of expression at the cursor position, similar to what you suggest. That is, we could have syntactic and value-based analysis.

• Syntactic analysis: keys_un should autocomplete keys_unsorted, $fi should autocomplete $first. This does not require any filter execution.
• Value-based analysis: .fi should autocomplete .first. This requires filter execution.

My point about the "single general strategy" applies only to value-based analysis. That is, I would concentrate on implementing a single approach that covers autocompletion of .fi in all positions, instead of, let's say, introducing a special case for filters that consist uniquely of a single path expression (the way that jnv handles autocompletion today).

[01mf02]

yield $x

Sorry if this is a naive question, but how would we capture the values produced at yield $x from Rust?

Do not worry, this is not a naive question --- I think this is the first time that I write about yield $x after all. ;)

Getting the values produced by yield $x can be obtained simply by running the filter. It's as simple as that. ;)
For example:

label $x | ((1, (2 | yield $x), 3, (4 | yield $x)) | empty)

This would yield 2 and 4. It's as if yield $x "sends" its input value to the corresponding label, which returns the sent value. Like a pipe.

An important part here is the empty at the end. Without empty, we would get out 1, 2, 3, 4. For autocompletion, that is not desirable, because we want to only get the values at the place where the cursor is, not the overall output of the filter.

[01mf02]

Another example for yield:

label $x | (((1, (2 | yield $x)) | . - 1)

This returns 0 and 2. That is, the 1 is impacted by the . - 1, whereas 2 is returned as-is by the label.

[ynqa]

My point about the "single general strategy" applies only to value-based analysis.

Got it, that makes perfect sense now.

label $x

Awww — that clears it up. label $x is basically a jump target.

I thought I understood yield (probably), but it turns out I was missing how label works. I had assumed that:

• values are somehow assigned to label $x
• and that I would need to retrieve them from $x

But now I see that label is just a control-flow target, and yield is what actually emits values.