[Optional] In cases where a single scan reads from both local and remote paths, since we've already read and cached the parquet info, is there a good reason to throw it away if some path has not been cached (is not remote)? Can we not apply the ones we have anyway, e.g., cached = [cached_parquet_info_map.get(path, None) for path in scan.paths], with appropriate subsequent handling of None by splitting the scan into two reads or adding a signal to libcudf to read a footer on-demand? Or could we at least skip the remote metadata fetch for paths that only appear in such scans, where we know we'd be throwing it away?
Originally posted by @igorpeshansky in #23558 (comment)
[Optional] In cases where a single scan reads from both local and remote paths, since we've already read and cached the parquet info, is there a good reason to throw it away if some path has not been cached (is not remote)? Can we not apply the ones we have anyway, e.g.,
cached = [cached_parquet_info_map.get(path, None) for path in scan.paths], with appropriate subsequent handling ofNoneby splitting the scan into two reads or adding a signal to libcudf to read a footer on-demand? Or could we at least skip the remote metadata fetch for paths that only appear in such scans, where we know we'd be throwing it away?Originally posted by @igorpeshansky in #23558 (comment)