Skip to content

readParquetSchema without loading entire file#7

Open
AshyIsMe wants to merge 1 commit intointerregna:mainfrom
AshyIsMe:feat/parquet-schema-efficiency
Open

readParquetSchema without loading entire file#7
AshyIsMe wants to merge 1 commit intointerregna:mainfrom
AshyIsMe:feat/parquet-schema-efficiency

Conversation

@AshyIsMe
Copy link
Contributor

Intended to allow reading just the schema from a parquet file without having to read all of the data into memory.

Unfortunately this approach still reads the entire file into memory (and then seems to leak that memory immediately...).

It looks like this is a limitation of the arrow glib C binding but that seems so brain-dead that I surely must be wrong...

@interregna
Copy link
Owner

Sounds like this doesn't solve the problem. Will take a look and see if there's some way to release the memory and not read the whole file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants