Example of problem with dates - full recovery only possible when "iso" string output is used instead of epoch, and col is cast from str to datetime by pandas:
import pandas as pd
from IPython.display import display
df = pd.DataFrame({'a': [pd.Timestamp('20200309'),
pd.Timestamp('20200309')],
'b': [1, 2]})
# Also works with time-zone aware timestamps
# df = pd.DataFrame({'a': [pd.Timestamp('20200309T120000.000000-0000'),
# pd.Timestamp('20200309T130000.000000-0000')],
# 'b': [1, 2]})
display(df)
df_json_1 = df.to_json()
display(df_json_1)
df_deser_1 = pd.read_json(df_json_1)
display(df_deser_1)
display(df_deser_1.dtypes)
df_deser_1 = df_deser_1.astype({'a': pd.api.types.DatetimeTZDtype(unit='ns', tz='UTC')})
display(df_deser_1)
display(df_deser_1.dtypes)
df_json_2 = df.to_json(date_format='iso')
display(df_json_2)
df_deser_2 = pd.read_json(df_json_2)
display(df_deser_2)
display(df_deser_2.dtypes)
display(type(df_deser_2.loc[0,'a']))
df_deser_2 = df_deser_2.astype({'a': pd.api.types.DatetimeTZDtype(unit='ns', tz='UTC')})
display(df_deser_2)
display(df_deser_2.dtypes)
display(type(df_deser_2.loc[0,'a']))
Inspired by #600, further discussion there
Problem:
Proposal:
Example of problem with dates - full recovery only possible when "iso" string output is used instead of epoch, and col is cast from str to datetime by pandas: