Add fasta support

[georgia-hf]

This PR adds support for FASTA files conversion to Parquet.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[lhoestq]

Looks all good ! I just left minor comments

[lhoestq]

You can also add the upper versions of the extensions here

[georgia-hf]

Huh, but why do none of the other extensions have uppers?

[lhoestq]

No need for xopen, you can use regular open (it's replaced by xopen anyway when needed due to an old logic)

[apcamargo]

A few comments:

• Have you tried using this with longer sequences? @UriNeri developed something similar internally and used it with viral genomes. He got some Parquet errors due to genomes not fitting in a utf8 column. This was fixed by using large_utf8.
• If you're only using it to read FASTA files, I think that having BioPython as a dependency is overkill. The library is very large and the FASTA parser isn't particularly fast. I have an example of a fast parser with no external references here (this is actually based on readfq.py, with a couple of extra functions that might not be useful in the context of this PR)

[UriNeri]

• If you're only using it to read FASTA files, I think that having BioPython as a dependency is overkill. The library is very large and the FASTA parser isn't particularly fast. I have an example of a fast parser with no external references here (this is actually based on readfq.py, with a couple of extra functions that might not be useful in the context of this PR)

What @apcamargo said, plus FWIW in our approach (so might not be relevant here) we use polars (with custom fasta io parser) or polars-bio (that has a scan_fasta function) and we foudn out that the page size sometimes need to be adjusted:

envs/default/lib/python3.9/site-packages/polars/lazyframe/frame.py:2422, in LazyFrame.collect(self, type_coercion, predicate_pushdown, projection_pushdown, simplify_expression, slice_pushdown, comm_subplan_elim, comm_subexpr_elim, cluster_with_columns, collapse_joins, no_optimization, engine, background, optimizations, **_kwargs)
   2420 # Only for testing purposes
   2421 callback = _kwargs.get("post_opt_callback", callback)
-> 2422 return wrap_df(ldf.collect(engine, callback))
ComputeError: parquet: File out of specification: A page can only contain i32::MAX uncompressed bytes. This one contains 4544943557

Which in polars can be solved with:

df.write_parquet(
    "test1.patquet",
   compression="zstd",
    row_group_size=10_000,  #  smaller row groups
     data_page_size=1024*1024  # 1MB page size
)

[lhoestq]

coming back to scientific formats datasets :)

just a little comment before and checking the CI is green before we merge

[lhoestq]

can you move this import in _iter_fasta_records ? since it's an optional dependency

[lhoestq]

are there fasta datasets on the Hub that would be cool