Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 14 additions & 2 deletions src/content/docs/reference/operators.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -168,13 +168,17 @@ operators:
example: 'context::inspect "ctx"'
path: 'reference/operators/context/inspect'
- name: 'context::list'
description: 'Lists all contexts'
description: 'Lists all contexts.'
example: 'context::list'
path: 'reference/operators/context/list'
- name: 'context::load'
description: 'Loads context state.'
example: 'context::load "ctx"'
path: 'reference/operators/context/load'
- name: 'context::lookup'
description: 'Performs unified matching.'
example: 'context::lookup "ctx", key=field'
path: 'reference/operators/context/lookup'
- name: 'context::remove'
description: 'Deletes a context.'
example: 'context::remove "ctx"'
Expand Down Expand Up @@ -972,7 +976,7 @@ context::inspect "ctx"

</ReferenceCard>

<ReferenceCard title="context::list" description="Lists all contexts" href="/reference/operators/context/list">
<ReferenceCard title="context::list" description="Lists all contexts." href="/reference/operators/context/list">

```tql
context::list
Expand All @@ -988,6 +992,14 @@ context::load "ctx"

</ReferenceCard>

<ReferenceCard title="context::lookup" description="Performs unified matching." href="/reference/operators/context/lookup">

```tql
context::lookup "ctx", key=field
```

</ReferenceCard>

<ReferenceCard title="context::remove" description="Deletes a context." href="/reference/operators/context/remove">

```tql
Expand Down
127 changes: 127 additions & 0 deletions src/content/docs/reference/operators/context/lookup.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
---
title: context::lookup
category: Contexts
example: 'context::lookup "ctx", key=src_ip'
---

Performs live filtering of the import feed using a context and translates
context updates into historical queries.

```tql
context::lookup name:string, key=field|list, [into=field, yield=field,
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Align lookup key type between synopsis and argument docs

The synopsis lists key=field|list, but the dedicated key section later documents key = field|list[field]. Because these are conflicting type shapes for multi-key lookup, users copying the top-level syntax may provide the wrong form and get invalid queries; the page should expose one canonical type in both places.

Useful? React with 👍 / 👎.

separate=bool, live=bool, retro=bool, parallel=int]
```

## Description

The `lookup` operator performs two actions simultaneously:

1. Translate context updates into historical queries
2. Filter all data newly ingested into the node against any context updates

These two operations combined offer *unified matching*, i.e., automated retro
matching by turning context updates into historical queries, and live matching
with a context on the import feed.

The diagram below illustrates how the operator works:

![lookup](lookup.svg)

### `name: string`

The name of the context to lookup with.

### `key = field|list[field]`

A field, or list of fields to use for matching.

### `into = field (optional)`

The name of the field in which to store the context's enrichment.

Defaults to the name of the context.

### `separate = bool (optional)`

When multiple fields are provided, duplicate the event for every provided field
and enrich them individually.

When using the option, the context moves from `<into>.context.<field...>` to
`<into>` in the resulting event, with a new field `<output>.field` containing
Comment on lines +49 to +50
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Replace undefined <output> placeholder in separate docs

The separate option description says enriched data moves to <into> and adds a new field at <output>.field, but output is not documented as a parameter anywhere in this operator signature. This leaves users with an unusable field path and suggests an option that does not exist, so readers cannot reliably query the emitted metadata after enabling separate.

Useful? React with 👍 / 👎.

the enriched path.

### `live = bool (optional)`

Enables live lookup for incoming events.

By default, both retro and live lookups are enabled. Specifying either `retro=true`
or `live=true` explicitly disables the other.

### `retro = bool (optional)`

Enables retrospective lookups for previously imported events. The `lookup`
operator will then apply a context after a context update.

By default, both retro and live lookups are enabled. Specifying either `retro=true`
or `live=true` explicitly disables the other.

### `yield = field (optional)`

Provide a field into the context object to use as the context instead. If the
key does not exist within the context, a `null` value is used instead.

### `parallel = int (optional)`

The number of partitions to open in parallel for retrospective lookups. This
number directly correlates with memory usage and performance of the `lookup`
operator.

Defaults to `3`.

## Examples

### Apply context to incoming and historical events

Apply the context `feodo` to incoming `suricata.flow` events as well as retro-match
any historical event to context updates:

```tql
context::lookup "feodo", key=src_ip
where @name == "suricata.flow"
```

### Apply context to incoming events

Apply the context `feodo` to incoming `suricata.flow` events. In this example,
only events that are freshly imported will be matched against the `fedo` context.

```tql
context::lookup "feodo", key=src_ip, live=true
where @name == "suricata.flow"
```

### Apply context to historical events

Apply the context `feodo` to historical `suricata.flow` events with every update
to `feodo`. In this example, on any `context::update` to the `fedo` context,
historical data persisted in the node will be retro-matched against the update.

```tql
context::lookup "feodo", key=src_ip, retro=true
where @name == "suricata.flow"
```

## See Also

- <Op>context::create_bloom_filter</Op>
- <Op>context::create_geoip</Op>
- <Op>context::create_lookup_table</Op>
- <Op>context::enrich</Op>
- <Op>context::inspect</Op>
- <Op>context::list</Op>
- <Op>context::load</Op>
- <Op>context::remove</Op>
- <Op>context::reset</Op>
- <Op>context::save</Op>
- <Op>context::update</Op>
- <Guide>enrichment/work-with-lookup-tables</Guide>
21 changes: 21 additions & 0 deletions src/content/docs/reference/operators/context/lookup.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading