batch decoding for bq and vbq readers with zero-copy API

[noamteyssier]

No description provided.

[gemini-code-assist]

Summary of Changes

Hello @noamteyssier, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces significant performance enhancements to the BQ and VBQ sequence data readers by implementing batch decoding and a zero-copy API. These changes aim to minimize memory allocations and improve data access efficiency, especially in parallel processing contexts, by allowing direct referencing of decoded sequence and header data rather than copying it into intermediate buffers.

Highlights

• Batch Decoding Implementation: Introduced batch decoding capabilities for both BQ and VBQ readers. For BQ, a new BatchRecord type facilitates processing multiple records simultaneously. For VBQ, the RecordBlock can now decode all sequences within a block into a single buffer.
• Zero-Copy API for Sequences and Headers: Enhanced the BinseqRecord trait with new sseq() and xseq() methods, providing direct &[u8] slices to decoded sequence data. Similarly, sheader() and xheader() methods now return &[u8], eliminating the need for mutable buffer arguments and reducing memory allocations.
• Optimized Parallel Processing: Refactored the parallel reader implementations for both BQ and VBQ to leverage the new batch decoding and zero-copy mechanisms. This significantly reduces memory allocations and improves data throughput by processing data in larger chunks.
• Configurable VBQ Batch Decoding: Added a set_decode_block method to the VBQ MmapReader and BinseqReader enum, allowing users to enable or disable block-level batch decoding based on their performance needs.
• Example Code Updates: All relevant example files (parallel_range.rs, read_write.rs, src/vbq/mod.rs, src/vbq/reader.rs) have been updated to demonstrate and utilize the new zero-copy API, removing previous patterns that involved mutable Vec<u8> buffers for decoding.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces batch decoding and a zero-copy API for bq and vbq readers, which is a significant performance enhancement. The changes are well-structured, with the BinseqRecord trait updated to support returning slices (&[u8]) for headers and sequences, avoiding buffer allocations and copies.

For vbq, the RecordBlock is now capable of decoding all its sequences at once, and RefRecord is updated to provide zero-copy access to this pre-decoded data, with a graceful fallback to on-the-fly decoding.

For bq, a new BatchRecord struct is introduced for parallel processing to handle batch-decoded data.

I've found a critical issue in the implementation of sseq and xseq for bq::BatchRecord that could lead to incorrect data being returned when flags are enabled. I've also found a high-severity issue in bq::RefRecord's xheader implementation. My review includes suggestions to fix these issues.

Overall, this is a great improvement, and with the suggested fixes, it will be a solid enhancement to the library.