PrimitiveLiteral of of Decimal

[Fokko]

With the PyIceberg <> Iceberg-Rust integration I'm running into issues with the Decimal literal.

Problem

The decimal comes with:

• Precision: is the number of digits in a number.
• Scale: the number of digits to the right of the decimal point in a number.

However, internally it is stored as a i128, which is fine for the Precision, since everything fits in there. Unfortunately, you cannot use this primitive on its own since you don't know the scale.

Potential solution

My first instinct was to introduce a separate primitive that would maintain the decimal scale. Unfortunately, Rust does not have an internal type; therefore, we now use rust_decimal. However, I don't like the idea of exposing third party types in the public Iceberg-Rust API. As pointed out in issue #669, rust_decimal does not support full precision. Switching libraries would already entail breaking the API.

I wanted to start a discussion to see if more folks have run into this and if there are any solutions I've missed.

[liurenjie1024]

Thanks @Fokko for raising this discussion. In iceberg-rust, Datum contains two parts: a data type and a literal. PrimitiveLiteral is used for physical data of a type. I think the right direction maybe introduce a PyDatum data struct for pyiceberg?

[kevinjqliu]

#669 is now merged so now rust_decimal is replaced with fastnum

I like what Renjie suggested. we're using PrimitiveLiteral as PyPrimitiveLiteral in the python binding right now,

iceberg-rust/bindings/python/src/data_file.rs

Lines 26 to 48 in 6165cd9

	#[pyclass()]
	pub struct PyPrimitiveLiteral {
	inner: PrimitiveLiteral,
	}
	#[pymethods]
	impl PyPrimitiveLiteral {
	pub fn value(&self, py: Python<'_>) -> PyResult<Py<PyAny>> {
	match &self.inner {
	PrimitiveLiteral::Boolean(v) => v.into_py_any(py),
	PrimitiveLiteral::Int(v) => v.into_py_any(py),
	PrimitiveLiteral::Long(v) => v.into_py_any(py),
	PrimitiveLiteral::Float(v) => v.0.into_py_any(py), // unwrap OrderedFloat
	PrimitiveLiteral::Double(v) => v.0.into_py_any(py),
	PrimitiveLiteral::String(v) => v.into_py_any(py),
	PrimitiveLiteral::Binary(v) => PyBytes::new(py, v).into_py_any(py),
	PrimitiveLiteral::Int128(v) => v.into_py_any(py), // Python handles big ints
	PrimitiveLiteral::UInt128(v) => v.into_py_any(py),
	PrimitiveLiteral::AboveMax => Err(PyValueError::new_err("AboveMax is not supported")),
	PrimitiveLiteral::BelowMin => Err(PyValueError::new_err("BelowMin is not supported")),
	}
	}
	}

seems like Datum and PyDatum might be the right abstraction