Implement JSON filter parsing for Qdrant queries

[leynos]

Description

The parse_filter method in crates/dear-diary-mcp/src/server.rs currently cannot parse arbitrary JSON filters into Qdrant Filter objects because Qdrant's protobuf Filter type does not implement serde's Deserialize trait.

Current Behaviour

When allow_arbitrary_filter is set to true in the configuration, the server returns an error:

InvalidFilter("Arbitrary JSON filter parsing is not yet implemented")

See: crates/dear-diary-mcp/src/server.rs lines 63-81

Desired Behaviour

The server should be able to:

1. Accept JSON filter structures via the filter parameter in qdrant_find requests
2. Parse these JSON structures into Qdrant Filter objects
3. Apply these filters when querying the vector database

Potential Approaches

1. Manual JSON→Filter construction: Write custom deserialisation logic to manually construct Filter objects from JSON structures by traversing the JSON and building the protobuf types directly.

2. Wrapper types with serde: Create wrapper types that implement serde traits and provide conversion methods to/from Qdrant's protobuf types.

3. Upstream serde support: Investigate whether Qdrant's Rust client could add serde support for protobuf types, or contribute this feature upstream.

4. Alternative filter API: Design a simplified, serde-compatible filter DSL specific to Dear Diary's use cases that can be translated to Qdrant filters.

5. serde_protobuf crate: Explore using the serde_protobuf crate for dynamic schema deserialisation, though this may require additional runtime overhead.

Technical Background

Protobuf-generated Rust types (using prost) typically do not implement serde traits by default. The Qdrant Rust client uses prost-generated types for its protobuf definitions, which means Filter lacks Deserialize implementation.

Possible technical solutions:

• Custom conversion logic from JSON → protobuf structures
• Intermediate representation that is serde-compatible
• Wrapper types with manual trait implementations

References

• PR: Initial implementation #1
• Comment discussion: Initial implementation #1 (comment)
• Qdrant Rust client: https://github.com/qdrant/rust-client
• serde_protobuf crate: https://docs.rs/serde-protobuf/latest/serde_protobuf/

Related Code

fn parse_filter(&self, filter: Option<Value>) -> Result<Option<Filter>, McpServerError> {
    let Some(_filter_value) = filter else {
        return Ok(None);
    };

    if !self.settings.qdrant.allow_arbitrary_filter {
        return Err(McpServerError::InvalidFilter(
            "Arbitrary filters are not enabled".to_owned(),
        ));
    }

    // TODO: Implement proper filter parsing from JSON
    Err(McpServerError::InvalidFilter(
        "Arbitrary JSON filter parsing is not yet implemented".to_owned(),
    ))
}

Impact

• Users cannot currently use the allow_arbitrary_filter configuration option
• Limited to filterable_fields configuration for query filtering
• Reduces flexibility for advanced filtering use cases

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

---

🔗 Similar Issues

Related Issues

• Implement recursive parser for improved delimiter token handling ddlint#54

👤 Suggested Assignees

• coderabbitai

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[traycerai]

Plan

Observations

The parse_filter method in file:crates/dear-diary-mcp/src/server.rs currently returns an error when allow_arbitrary_filter is enabled because Qdrant's protobuf-generated Filter type lacks serde's Deserialize trait. The Qdrant Rust client uses prost-generated types which don't implement serde by default. The codebase already has a configuration option (allow_arbitrary_filter) and error handling (McpServerError::InvalidFilter) in place, indicating the infrastructure is ready for this feature.

Approach

Implement manual JSON→Filter construction by creating a custom deserialization layer that traverses JSON structures and builds Qdrant protobuf Filter objects directly. This approach provides full control, avoids external dependencies on upstream changes, and aligns with the existing codebase structure. The solution will use serde_json::Value for JSON parsing and construct Qdrant's Filter, Condition, and related types programmatically through their builder APIs or direct field assignment.

Implementation Steps

1. Research Qdrant Filter Structure

Objective: Understand the complete structure of Qdrant's Filter protobuf types

• Examine the Qdrant Rust client documentation at https://docs.rs/qdrant-client/latest/qdrant_client/
• Review the Filter, Condition, FieldCondition, Match, Range, and other filter-related types
• Document the hierarchical structure: Filter contains should, must, must_not arrays of Condition objects
• Note that Condition can contain FieldCondition, nested Filter, HasId, IsEmpty, IsNull, etc.
• Identify all supported field condition types: Match, Range, GeoBoundingBox, GeoRadius, ValuesCount, etc.

2. Define JSON Filter Schema

Objective: Create a clear specification for the expected JSON filter format

• Design a JSON schema that mirrors Qdrant's filter structure but is user-friendly
• Structure should support:
  • Top-level keys: must, should, must_not (arrays of conditions)
  • Condition types: field, nested, has_id, is_empty, is_null
  • Field condition operators: match, range, geo_bounding_box, geo_radius, values_count
  • Match types: exact value, array of values, text matching
  • Range operators: gt, gte, lt, lte
• Document this schema in code comments or a separate documentation file
• Example JSON structure:

  {
    "must": [
      {"field": "status", "match": {"value": "active"}},
      {"field": "age", "range": {"gte": 18, "lt": 65}}
    ],
    "should": [
      {"field": "category", "match": {"any": ["tech", "science"]}}
    ]
  }

3. Create Filter Parsing Module

Objective: Implement the core JSON-to-Filter conversion logic

• Create a new module file:crates/dear-diary-mcp/src/filter_parser.rs or add to existing modules
• Implement main function: pub fn parse_json_filter(json: &Value) -> Result<Filter, FilterParseError>
• Create custom error type FilterParseError with variants for:
  • Invalid JSON structure
  • Unsupported condition type
  • Missing required fields
  • Invalid field values
  • Type mismatches
• Implement helper functions:
  • parse_conditions(json: &Value) -> Result<Vec<Condition>, FilterParseError> - parse condition arrays
  • parse_condition(json: &Value) -> Result<Condition, FilterParseError> - parse single condition
  • parse_field_condition(json: &Value) -> Result<FieldCondition, FilterParseError> - parse field conditions
  • parse_match(json: &Value) -> Result<Match, FilterParseError> - parse match conditions
  • parse_range(json: &Value) -> Result<Range, FilterParseError> - parse range conditions

4. Implement Condition Type Parsers

Objective: Handle each Qdrant condition type with specific parsing logic

• Field Condition Parser:

  • Extract field key (required string)
  • Detect operator type: match, range, geo_bounding_box, geo_radius, values_count
  • Construct FieldCondition with appropriate operator
  • Handle nested field paths (dot notation)

• Match Condition Parser:

  • Support single value: {"value": "text"} or {"value": 123}
  • Support multiple values: {"any": ["val1", "val2"]}
  • Support text matching: {"text": "search term"}
  • Handle different value types: string, integer, boolean
  • Create appropriate MatchValue enum variant

• Range Condition Parser:

  • Parse gt, gte, lt, lte operators
  • Support integer and float ranges
  • Validate that at least one bound is specified
  • Construct Range with appropriate bounds

• Nested Filter Parser:

  • Detect nested filter structure with nested key
  • Recursively call parse_json_filter for nested filters
  • Handle nested field paths

• Special Condition Parsers:

  • has_id: Parse ID list from JSON array
  • is_empty: Parse field name
  • is_null: Parse field name

5. Integrate with parse_filter Method

Objective: Replace the TODO in file:crates/dear-diary-mcp/src/server.rs

• Update the parse_filter method implementation:
  • Remove the error return for unimplemented parsing
  • Call parse_json_filter(&filter_value)
  • Map FilterParseError to McpServerError::InvalidFilter
  • Wrap result in Some(filter)
• Add appropriate error context with descriptive messages
• Ensure the method signature remains unchanged: fn parse_filter(&self, filter: Option<Value>) -> Result<Option<Filter>, McpServerError>

6. Add Validation and Error Handling

Objective: Provide clear error messages for invalid filter JSON

• Validate JSON structure before parsing:
  • Check that top-level is an object
  • Verify must, should, must_not are arrays if present
  • Ensure at least one condition array is non-empty
• Add detailed error messages indicating:
  • Which field caused the error
  • What was expected vs. what was received
  • Suggestions for fixing common mistakes
• Implement Display trait for FilterParseError with user-friendly messages
• Consider adding a validation mode that checks filter structure without executing

7. Add Unit Tests

Objective: Ensure robust parsing with comprehensive test coverage

• Create test module in file:crates/dear-diary-mcp/src/filter_parser.rs or separate test file
• Test cases for valid filters:
  • Simple field match conditions
  • Range conditions with various operators
  • Multiple conditions in must, should, must_not
  • Nested filters
  • Complex filters combining multiple condition types
  • Different value types (string, int, float, bool)
• Test cases for invalid filters:
  • Malformed JSON structures
  • Missing required fields
  • Invalid operator names
  • Type mismatches
  • Empty condition arrays
• Test edge cases:
  • Empty filter object {}
  • Single condition
  • Deeply nested filters
  • Large number of conditions
• Verify error messages are descriptive and helpful

8. Update Configuration Documentation

Objective: Document the new filter JSON format for users

• Update configuration documentation to explain allow_arbitrary_filter option
• Provide comprehensive examples of filter JSON structures
• Document all supported condition types and operators
• Include common use cases and patterns
• Add troubleshooting section for common errors
• Reference Qdrant's filter documentation for advanced users
• Consider adding JSON schema file for IDE autocompletion

9. Add Integration Tests

Objective: Verify end-to-end functionality with actual Qdrant queries

• Create integration tests that:
  • Set up test Qdrant collection
  • Insert test data with various field types
  • Execute qdrant_find requests with JSON filters
  • Verify correct results are returned
  • Test filter combinations
• Test with allow_arbitrary_filter enabled and disabled
• Verify error handling when filters are disabled
• Test interaction with existing filterable_fields configuration

10. Performance Considerations

Objective: Ensure parsing doesn't introduce significant overhead

• Profile the parsing logic with various filter complexities
• Consider caching parsed filters if the same filter is used repeatedly
• Optimize JSON traversal to minimize allocations
• Add benchmarks for common filter patterns
• Document any performance limitations or recommendations

Architecture Diagram

sequenceDiagram
    participant Client
    participant Server as parse_filter
    participant Parser as filter_parser
    participant Qdrant as Qdrant Client

    Client->>Server: qdrant_find with JSON filter
    Server->>Server: Check allow_arbitrary_filter
    alt arbitrary filter disabled
        Server-->>Client: Error: Arbitrary filters not enabled
    else arbitrary filter enabled
        Server->>Parser: parse_json_filter(json_value)
        Parser->>Parser: Validate JSON structure
        Parser->>Parser: Parse top-level (must/should/must_not)
        loop For each condition
            Parser->>Parser: parse_condition()
            alt Field Condition
                Parser->>Parser: parse_field_condition()
                Parser->>Parser: parse_match/range/geo/etc()
            else Nested Filter
                Parser->>Parser: Recursive parse_json_filter()
            else Special Condition
                Parser->>Parser: parse_has_id/is_empty/is_null()
            end
        end
        Parser-->>Server: Filter object
        Server->>Qdrant: Query with Filter
        Qdrant-->>Server: Results
        Server-->>Client: Query results
    end

Loading

Key Components

Component	File	Purpose
parse_filter	file:crates/dear-diary-mcp/src/server.rs	Entry point, validates config and calls parser
filter_parser module	file:crates/dear-diary-mcp/src/filter_parser.rs	Core parsing logic and helper functions
FilterParseError	file:crates/dear-diary-mcp/src/filter_parser.rs	Custom error type for parsing failures
Unit tests	file:crates/dear-diary-mcp/src/filter_parser.rs	Test coverage for all parsing scenarios
Integration tests	file:crates/dear-diary-mcp/tests/filter_integration.rs	End-to-end testing with Qdrant

Filter Structure Mapping

JSON Structure	Qdrant Type	Notes
{"must": [...]}	Filter.must	Array of conditions that must match
{"should": [...]}	Filter.should	Array of conditions where at least one must match
{"must_not": [...]}	Filter.must_not	Array of conditions that must not match
{"field": "name", "match": {...}}	FieldCondition	Field-based condition
{"value": "text"}	Match::Value	Exact match
{"any": ["a", "b"]}	Match::Any	Match any of the values
{"range": {"gte": 10, "lt": 20}}	Range	Numeric range
{"nested": {...}}	Nested Filter	Recursive filter structure

Import In IDE

🤖 Prompt for AI Agents

## Observations

The `parse_filter` method in `file:crates/dear-diary-mcp/src/server.rs` currently returns an error when `allow_arbitrary_filter` is enabled because Qdrant's protobuf-generated `Filter` type lacks serde's `Deserialize` trait. The Qdrant Rust client uses prost-generated types which don't implement serde by default. The codebase already has a configuration option (`allow_arbitrary_filter`) and error handling (`McpServerError::InvalidFilter`) in place, indicating the infrastructure is ready for this feature.

## Approach

Implement **manual JSON→Filter construction** by creating a custom deserialization layer that traverses JSON structures and builds Qdrant protobuf `Filter` objects directly. This approach provides full control, avoids external dependencies on upstream changes, and aligns with the existing codebase structure. The solution will use `serde_json::Value` for JSON parsing and construct Qdrant's `Filter`, `Condition`, and related types programmatically through their builder APIs or direct field assignment.

## Implementation Steps

### 1. Research Qdrant Filter Structure

**Objective**: Understand the complete structure of Qdrant's Filter protobuf types

- Examine the Qdrant Rust client documentation at https://docs.rs/qdrant-client/latest/qdrant_client/
- Review the `Filter`, `Condition`, `FieldCondition`, `Match`, `Range`, and other filter-related types
- Document the hierarchical structure: `Filter` contains `should`, `must`, `must_not` arrays of `Condition` objects
- Note that `Condition` can contain `FieldCondition`, nested `Filter`, `HasId`, `IsEmpty`, `IsNull`, etc.
- Identify all supported field condition types: `Match`, `Range`, `GeoBoundingBox`, `GeoRadius`, `ValuesCount`, etc.

### 2. Define JSON Filter Schema

**Objective**: Create a clear specification for the expected JSON filter format

- Design a JSON schema that mirrors Qdrant's filter structure but is user-friendly
- Structure should support:
  - Top-level keys: `must`, `should`, `must_not` (arrays of conditions)
  - Condition types: `field`, `nested`, `has_id`, `is_empty`, `is_null`
  - Field condition operators: `match`, `range`, `geo_bounding_box`, `geo_radius`, `values_count`
  - Match types: exact value, array of values, text matching
  - Range operators: `gt`, `gte`, `lt`, `lte`
- Document this schema in code comments or a separate documentation file
- Example JSON structure:
  ```json
  {
    "must": [
      {"field": "status", "match": {"value": "active"}},
      {"field": "age", "range": {"gte": 18, "lt": 65}}
    ],
    "should": [
      {"field": "category", "match": {"any": ["tech", "science"]}}
    ]
  }
  ```

### 3. Create Filter Parsing Module

**Objective**: Implement the core JSON-to-Filter conversion logic

- Create a new module `file:crates/dear-diary-mcp/src/filter_parser.rs` or add to existing modules
- Implement main function: `pub fn parse_json_filter(json: &Value) -> Result<Filter, FilterParseError>`
- Create custom error type `FilterParseError` with variants for:
  - Invalid JSON structure
  - Unsupported condition type
  - Missing required fields
  - Invalid field values
  - Type mismatches
- Implement helper functions:
  - `parse_conditions(json: &Value) -> Result<Vec<Condition>, FilterParseError>` - parse condition arrays
  - `parse_condition(json: &Value) -> Result<Condition, FilterParseError>` - parse single condition
  - `parse_field_condition(json: &Value) -> Result<FieldCondition, FilterParseError>` - parse field conditions
  - `parse_match(json: &Value) -> Result<Match, FilterParseError>` - parse match conditions
  - `parse_range(json: &Value) -> Result<Range, FilterParseError>` - parse range conditions

### 4. Implement Condition Type Parsers

**Objective**: Handle each Qdrant condition type with specific parsing logic

- **Field Condition Parser**:
  - Extract `field` key (required string)
  - Detect operator type: `match`, `range`, `geo_bounding_box`, `geo_radius`, `values_count`
  - Construct `FieldCondition` with appropriate operator
  - Handle nested field paths (dot notation)

- **Match Condition Parser**:
  - Support single value: `{"value": "text"}` or `{"value": 123}`
  - Support multiple values: `{"any": ["val1", "val2"]}`
  - Support text matching: `{"text": "search term"}`
  - Handle different value types: string, integer, boolean
  - Create appropriate `MatchValue` enum variant

- **Range Condition Parser**:
  - Parse `gt`, `gte`, `lt`, `lte` operators
  - Support integer and float ranges
  - Validate that at least one bound is specified
  - Construct `Range` with appropriate bounds

- **Nested Filter Parser**:
  - Detect nested filter structure with `nested` key
  - Recursively call `parse_json_filter` for nested filters
  - Handle nested field paths

- **Special Condition Parsers**:
  - `has_id`: Parse ID list from JSON array
  - `is_empty`: Parse field name
  - `is_null`: Parse field name

### 5. Integrate with parse_filter Method

**Objective**: Replace the TODO in `file:crates/dear-diary-mcp/src/server.rs`

- Update the `parse_filter` method implementation:
  - Remove the error return for unimplemented parsing
  - Call `parse_json_filter(&filter_value)`
  - Map `FilterParseError` to `McpServerError::InvalidFilter`
  - Wrap result in `Some(filter)`
- Add appropriate error context with descriptive messages
- Ensure the method signature remains unchanged: `fn parse_filter(&self, filter: Option<Value>) -> Result<Option<Filter>, McpServerError>`

### 6. Add Validation and Error Handling

**Objective**: Provide clear error messages for invalid filter JSON

- Validate JSON structure before parsing:
  - Check that top-level is an object
  - Verify `must`, `should`, `must_not` are arrays if present
  - Ensure at least one condition array is non-empty
- Add detailed error messages indicating:
  - Which field caused the error
  - What was expected vs. what was received
  - Suggestions for fixing common mistakes
- Implement `Display` trait for `FilterParseError` with user-friendly messages
- Consider adding a validation mode that checks filter structure without executing

### 7. Add Unit Tests

**Objective**: Ensure robust parsing with comprehensive test coverage

- Create test module in `file:crates/dear-diary-mcp/src/filter_parser.rs` or separate test file
- Test cases for valid filters:
  - Simple field match conditions
  - Range conditions with various operators
  - Multiple conditions in `must`, `should`, `must_not`
  - Nested filters
  - Complex filters combining multiple condition types
  - Different value types (string, int, float, bool)
- Test cases for invalid filters:
  - Malformed JSON structures
  - Missing required fields
  - Invalid operator names
  - Type mismatches
  - Empty condition arrays
- Test edge cases:
  - Empty filter object `{}`
  - Single condition
  - Deeply nested filters
  - Large number of conditions
- Verify error messages are descriptive and helpful

### 8. Update Configuration Documentation

**Objective**: Document the new filter JSON format for users

- Update configuration documentation to explain `allow_arbitrary_filter` option
- Provide comprehensive examples of filter JSON structures
- Document all supported condition types and operators
- Include common use cases and patterns
- Add troubleshooting section for common errors
- Reference Qdrant's filter documentation for advanced users
- Consider adding JSON schema file for IDE autocompletion

### 9. Add Integration Tests

**Objective**: Verify end-to-end functionality with actual Qdrant queries

- Create integration tests that:
  - Set up test Qdrant collection
  - Insert test data with various field types
  - Execute `qdrant_find` requests with JSON filters
  - Verify correct results are returned
  - Test filter combinations
- Test with `allow_arbitrary_filter` enabled and disabled
- Verify error handling when filters are disabled
- Test interaction with existing `filterable_fields` configuration

### 10. Performance Considerations

**Objective**: Ensure parsing doesn't introduce significant overhead

- Profile the parsing logic with various filter complexities
- Consider caching parsed filters if the same filter is used repeatedly
- Optimize JSON traversal to minimize allocations
- Add benchmarks for common filter patterns
- Document any performance limitations or recommendations

## Architecture Diagram

```mermaid
sequenceDiagram
    participant Client
    participant Server as parse_filter
    participant Parser as filter_parser
    participant Qdrant as Qdrant Client

    Client->>Server: qdrant_find with JSON filter
    Server->>Server: Check allow_arbitrary_filter
    alt arbitrary filter disabled
        Server-->>Client: Error: Arbitrary filters not enabled
    else arbitrary filter enabled
        Server->>Parser: parse_json_filter(json_value)
        Parser->>Parser: Validate JSON structure
        Parser->>Parser: Parse top-level (must/should/must_not)
        loop For each condition
            Parser->>Parser: parse_condition()
            alt Field Condition
                Parser->>Parser: parse_field_condition()
                Parser->>Parser: parse_match/range/geo/etc()
            else Nested Filter
                Parser->>Parser: Recursive parse_json_filter()
            else Special Condition
                Parser->>Parser: parse_has_id/is_empty/is_null()
            end
        end
        Parser-->>Server: Filter object
        Server->>Qdrant: Query with Filter
        Qdrant-->>Server: Results
        Server-->>Client: Query results
    end
```

## Key Components

| Component | File | Purpose |
|-----------|------|---------|
| `parse_filter` | `file:crates/dear-diary-mcp/src/server.rs` | Entry point, validates config and calls parser |
| `filter_parser` module | `file:crates/dear-diary-mcp/src/filter_parser.rs` | Core parsing logic and helper functions |
| `FilterParseError` | `file:crates/dear-diary-mcp/src/filter_parser.rs` | Custom error type for parsing failures |
| Unit tests | `file:crates/dear-diary-mcp/src/filter_parser.rs` | Test coverage for all parsing scenarios |
| Integration tests | `file:crates/dear-diary-mcp/tests/filter_integration.rs` | End-to-end testing with Qdrant |

## Filter Structure Mapping

| JSON Structure | Qdrant Type | Notes |
|----------------|-------------|-------|
| `{"must": [...]}` | `Filter.must` | Array of conditions that must match |
| `{"should": [...]}` | `Filter.should` | Array of conditions where at least one must match |
| `{"must_not": [...]}` | `Filter.must_not` | Array of conditions that must not match |
| `{"field": "name", "match": {...}}` | `FieldCondition` | Field-based condition |
| `{"value": "text"}` | `Match::Value` | Exact match |
| `{"any": ["a", "b"]}` | `Match::Any` | Match any of the values |
| `{"range": {"gte": 10, "lt": 20}}` | `Range` | Numeric range |
| `{"nested": {...}}` | Nested `Filter` | Recursive filter structure |

---

Execution Information

Branch: main
Commit: 8d1f6af

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.