Skip to content

Reevaluate dtypes on importing from SBDF file #75

@bbassett-tibco

Description

@bbassett-tibco

Given the following script:

from spotfire import sbdf

x = sbdf.import_data("spotfire/test/files/sbdf/10001.sbdf")
print("column Boolean")
print(x['Boolean'].loc[[0,1,2]])
print("column String")
print(x['String'].loc[[0,1,2]])

Output:

column Boolean
0    False
1     True
2     None
Name: Boolean, dtype: object
column String
0      The
1    quick
2     None
Name: String, dtype: object

Note how both of these columns report as being dtype object. However, it's probably better to use the nullable boolean (https://pandas.pydata.org/docs/user_guide/boolean.html) and string (https://pandas.pydata.org/docs/user_guide/text.html#text) dtypes for these columns. (We're actually using the nullable integer and float dtypes for other column types, so our current requirements on Pandas should be untouched.)

Current workarounds are to manually cast the columns using Pandas' astype method:

>>> x['Boolean'].loc[[0,1,2]].astype('boolean')
0    False
1     True
2     <NA>
Name: Boolean, dtype: boolean
>>> x['String'].loc[[0,1,2]].astype('string')
0      The
1    quick
2     <NA>
Name: String, dtype: string

Make this the native result of the import_data function, and make sure that these dtypes correctly export to SBDF in the export_data function.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions