feat: Support JSON as data type

[eddyxu]

Support JSON as data type

Users are expecting to store schema less json blobs in the dataset. In other systems,

• Postgres can use jsonb type
• Spark offers functions, such as get_json_object / json_object_keys function on json strings.
• Pyarrow has JsonType now

It'd be beneficial for Lance to have capability to store, access and search json objects.

• Store JSON as one of the arrow field.
• Implement reading values by key or json path.
• Create index on one of the scalar field.
• Search a scalar json field
• Create vector index on one of vector field.
• Search a vector json field
• Create full text index on one of the string field.
• Search a fulltext json field.

[westonpace]

I think adding JSON to file format (e.g. variant or something, if needed) and adding JSON to the table format (JSON indexes) and adding JSON to the compute engine (JSON functions) can all be tackled independently.

The file format and compute engine changes are probably more well understood.

The table format changes are a little less understood to me. There is maybe an easy path for some kind of read-only JSON column where a write means rewriting the entire JSON value. Making a full schema-less table will be quite different.

Is a user going to want to change a JSON value for a row (e.g. an update)? Will they want to add a new JSON column (to an existing JSON object)?

[westonpace]

I assume if we are training a vector index on a vector field and that field isn't a vector type on some row (JSON does not enforce homogenous types) or if the vector has different dimensions then we just skip the field?

What should we use to determine the dimension of the JSON vector field? Should it be the first value we encounter?

[wjones127]

One good option is to join the effort in DataFusion to support the Parquet Variant data type. (It's just called Parquet Variant because that's where the standard was defined, not because it's tied to parquet.)

Similar to how we can add geospatial support by just including a few plugins (see demo from #4482), we could probably add variant support with a few plugins.

Relevant packages:

• parquet-variant
• parquet-variant-json
• parquet-variant-compute

[wjones127]

If we want something more immediate, there is:

https://github.com/datafusion-contrib/datafusion-functions-json

[Xuanwo]

I think the variant approach works well for shredding support. Our ultimate goal is to support variants in Lance. However, we can begin by adding JSON support through the introduction of jsonb first. We have a mature jsonb crate and it has nearly everthing we need so far.

My current plan:

• Allow users to write data in json
• We encode json into jsonb binary as a logic type
• We store the binary as LargeBinary in lance

The first MVP will allow users to:

• write json data in arrow
• read json data by get_json_path