Describe the enhancement requested
Casting a floating-point array to string writes integral values without a decimal point:
>>> import pyarrow as pa, pyarrow.compute as pc
>>> pc.cast(pa.array([20.0, 20.5, 1e10]), pa.string()).to_pylist()
['20', '20.5', '1e+10']
The CSV writer formats columns through the same cast, so float64 columns are written the same way. Because the text no longer says the value was a float, a reader that infers types (including pyarrow.csv.read_csv) reads the column back as int64. Whether a column round-trips as double or int64 then depends on the values in it, not on its type.
Example
Writing a time series with one CSV per timestep, where the first timestep happens to hold whole numbers:
import pyarrow as pa, pyarrow.csv as csv, pyarrow.dataset as ds
csv.write_csv(pa.table({"x": [20.0, 21.0]}), "ts/t0.csv") # written as 20, 21
csv.write_csv(pa.table({"x": [20.5, 21.2]}), "ts/t1.csv")
ds.dataset("ts", format="csv").to_table()
# ArrowInvalid: Could not open CSV input source 'ts/t1.csv': Invalid: In CSV column #0:
# Row #2: CSV conversion error to int64: invalid value '20.5'Both tables are float64, but t0.csv is inferred as int64. pandas to_csv and Polars write_csv both write 20.0.
For context, this came up downstream in Narwhals, where cast(String) on a float column gives different text on the PyArrow backend than with pandas or Polars: narwhals-dev/narwhals#4027
Component(s)
C++, Python
Describe the enhancement requested
Casting a floating-point array to string writes integral values without a decimal point:
The CSV writer formats columns through the same cast, so float64 columns are written the same way. Because the text no longer says the value was a float, a reader that infers types (including
pyarrow.csv.read_csv) reads the column back asint64. Whether a column round-trips asdoubleorint64then depends on the values in it, not on its type.Example
Writing a time series with one CSV per timestep, where the first timestep happens to hold whole numbers:
Both tables are
float64, butt0.csvis inferred asint64. pandasto_csvand Polarswrite_csvboth write20.0.For context, this came up downstream in Narwhals, where
cast(String)on a float column gives different text on the PyArrow backend than with pandas or Polars: narwhals-dev/narwhals#4027Component(s)
C++, Python