Repository navigation

Add Parquet Modular encryption support (write) - #7111

Merged

alamb merged 101 commits into

apache:mainfrom

rok:encryption-basics-fork

Apr 1, 2025

Merged

Add Parquet Modular encryption support (write)#7111
alamb merged 101 commits into
apache:mainfrom
rok:encryption-basics-fork

Commits on Mar 16, 2025
- Start encryption
  
  ggershinsky
  authored and
  rok
  committed
  11e083e
- Work
  rok
  committed
  0b31469
- Work
  rok
  committed
  d2a73ef
- Pass Encryptor to SerializedRowGroupWriter
  rok
  committed
  3b87a04
- Expand test, pass FileEncryptionProperties instead of FileEncryptor to SerializedPageWriter, encrypt pages
  rok
  committed
  5c40e40
- Add encrypt_object helper
  rok
  committed
  6655d21
- Implement serialization of column crypto metadata
  
  adamreeve
  authored and
  rok
  committed
  804e903
- Fix writing Parquet magic bytes
  
  adamreeve
  authored and
  rok
  committed
  d328396
- Add key metadata to file encryption properties
  
  adamreeve
  authored and
  rok
  committed
  a5ffb49
- Generate unique file aad and add prefix if set
  
  adamreeve
  authored and
  rok
  committed
  ae42dd4
- Add aad param to encrypt_object and don't require TrackedWrite
  
  adamreeve
  authored and
  rok
  committed
  9d5db21
- Write file crypto metadata
  
  adamreeve
  authored and
  rok
  committed
  c92559d
- Set column crypto metadata
  
  adamreeve
  authored and
  rok
  committed
  bead72a
- Store file_aad and aad_file_unique in FileEncryptor
  
  adamreeve
  authored and
  rok
  committed
  27e095b
- Work towards using correct AADs
  
  adamreeve
  authored and
  rok
  committed
  c3a3ac7
- Fix writing ciphertext length
  
  adamreeve
  authored and
  rok
  committed
  5c04f1c
- Add check of ciphertext length
  
  adamreeve
  authored and
  rok
  committed
  84b4d50
- Ugly workaround for setting compressed page size
  
  adamreeve
  authored and
  rok
  committed
  a609111
- Fix test logic
  
  adamreeve
  authored and
  rok
  committed
  8c64215
- Add page_ordinal
  rok
  committed
  3a79993
- Add some feature flags
  rok
  committed
  07dc037
- Add page_ordinal, row_group_ordinal and column_ordinal to SerializedPageWriter
  rok
  committed
  33ffb97
- minor changes
  rok
  committed
  73ee0fa
- clippy fixes
  rok
  committed
  3a867fb
- Encapsulate page encryption context in a PageEncryptor struct
  
  adamreeve
  authored and
  rok
  committed
  24e73ed
- SerializedRowGroupWriter.column_index starts at 1 not 0
  rok
  committed
  dd59b46
- Fix handling dictionary pages and update test
  
  adamreeve
  authored and
  rok
  committed
  bbd3587
- Fix clippy issues
  rok
  committed
  3f43fee
- clippy
  rok
  committed
  877805a
- Use PageEncryptor in ArrowPageWriter
  
  adamreeve
  authored and
  rok
  committed
  e390d7b
- Tidy up feature handling and reduce duplication
  
  adamreeve
  authored and
  rok
  committed
  8facaca
- Test fixes
  
  adamreeve
  authored and
  rok
  committed
  dc0ce9e
- Fix setting Arrow page writer for byte typed columns
  
  adamreeve
  authored and
  rok
  committed
  a14fdf4
- WIP Add per-column encryption keys
  
  adamreeve
  authored and
  rok
  committed
  08958af
- Add test_non_uniform_encryption
  rok
  committed
  69f557a
- lint
  rok
  committed
  8eb841a
- lint
  rok
  committed
  96d09a7
- Add SchemaRef ArrowColumnWriterFactory to get column_path via column_index
  rok
  committed
  69969dc
- Get column path from descriptor rather than Arrow schema
  
  adamreeve
  authored and
  rok
  committed
  65fc410
- Fix writing multiple encrypted pages with ArrowPageWriter
  
  adamreeve
  authored and
  rok
  committed
  3f8907a
- Return encryptors as a Result<Box<dyn BlockEncryptor>>
  
  adamreeve
  authored and
  rok
  committed
  7a4d6c0
- Get per-column encryption working and various tidy ups
  
  adamreeve
  authored and
  rok
  committed
  36f3163
- Handle non-encrypted columns
  
  adamreeve
  authored and
  rok
  committed
  b3f24e3
- Tidy up some duplication
  
  adamreeve
  authored and
  rok
  committed
  cbf7d74
- Add encryption_util module for tests
  
  adamreeve
  authored and
  rok
  committed
  d862c63
- Add uniform encryption test
  rok
  committed
  a384663
- lint
  rok
  committed
  0f8d6f8
- post rebase
  rok
  committed
  3e7efe4
- Check if columns to encrypt are in schema
  rok
  committed
  854b257
- Apply suggestions from code review
  
  rok
  and
  adamreeve
  committed
  ea2db66
- Move tests to tests/. Post rebase fixes.
  rok
  committed
  2b2a3ef
- Review feedback
  rok
  committed
  22135ba
- Review feedback
  rok
  committed
  32673c1
- Minor changes
  rok
  committed
  f479df1
- Raise if writing plaintext footer
  rok
  committed
  8ac232b
- Docs for crypto methods
  rok
  committed
  75b98cd
Commits on Mar 17, 2025
- More practical key API
  rok
  committed
  0460424
Commits on Mar 18, 2025
- Refactor PageEncryptor use
  adamreeve
  committed
  4032b4f
Commits on Mar 19, 2025
- Simplify with_new_compressed_buffer method
  adamreeve
  committed
  e2790dd
- Apply suggestions from code review
  
  rok
  and
  adamreeve
  authored
  6991943
Commits on Mar 20, 2025
- Review feedback
  rok
  committed
  b01b285
- Lint and remove redundant test.
  rok
  committed
  ffd2fc9
- Docs
  rok
  committed
  a522ab1
- Add async writer test for encrypted data
  rok
  committed
  70e7190
- Test struct array encryption, column name with '.'
  rok
  committed
  a4fbaf2
- Review feedback
  rok
  committed
  c46a259
- First round of changes, add accessors and return result for encryption builder.
  corwinjoy
  committed
  3d2054c
- Add
  corwinjoy
  committed
  4ddbc4c
Commits on Mar 21, 2025
- Update parquet/src/encryption/encrypt.rs
  
  rok
  and
  adamreeve
  authored
  159b3df
- Move encryption tests
  rok
  committed
  ce8d2a9
- Backout change to arrow/async_reader/mod.rs. TODO, put this in a separate PR.
  corwinjoy
  committed
  4f15a96
- Update notes on changes to writer.rs
  corwinjoy
  committed
  cf57871
- Merge branch 'encryption-basics-fork' into encryption-basics-fork-pr-cj
  corwinjoy
  committed
  8baa777
- Merge pull request #3 from rok/encryption-basics-fork-pr-cj
  corwinjoy
  authored
  3e02f74
- Fix struct array encryption
  rok
  committed
  614d5e8
- Minor fixes
  rok
  committed
  b1db051
- Lint
  rok
  committed
  ae4c089
- Fix test
  rok
  committed
  6240644
- Add '.' to struct array name
  rok
  committed
  df640cd
Commits on Mar 23, 2025
- Fix required features for encryption tests
  adamreeve
  committed
  cdc3246
- Fix reading encrypted struct columns and writer test
  adamreeve
  committed
  05f5982
Commits on Mar 24, 2025
- Tidy ups
  adamreeve
  committed
  9811c76
- Remove unnecessary clone of all row group metadata in unencrypted case
  adamreeve
  committed
  a15816d
- Remove overly broad error remapping
  adamreeve
  committed
  0ccbc03
- Tidy up duplicated test function
  adamreeve
  committed
  2a3e905
- Suppress unused mut error
  adamreeve
  committed
  5a6fac8
- Re-use block encryptors in PageEncryptor
  adamreeve
  committed
  9e79d75
- Merge remote-tracking branch 'upstream/main' into encryption-basics-fork
  adamreeve
  committed
  b55a1b3
Commits on Mar 25, 2025
- Slightly update error message for missing column key.
  corwinjoy
  committed
  5600e61
- Refactor PageEncryptor to reduce use of cfg(feature)
  adamreeve
  committed
  c7181b8
- Refactor PageEncryptor construction in SerializedPageWriter
  adamreeve
  committed
  6c4d5b3
Commits on Mar 26, 2025
- Reduce use of inline #[cfg(feature = "encryption")]
  adamreeve
  committed
  78ac7b2
- Refactor ThriftMetadataWriter to reduce use of feature checks within functions
  adamreeve
  committed
  fd6c30b
Commits on Mar 27, 2025
- Tidy ups
  adamreeve
  committed
  26279eb
- Refactor ArrowRowGroupWriter creation
  adamreeve
  committed
  4e9e157
Commits on Mar 30, 2025
- Make pub(crate) more explicit on some structs
  adamreeve
  committed
  c11dc01
- Check for length mismatch in with_column_keys
  adamreeve
  committed
  b9b860b
- Comment and error message tidy ups
  adamreeve
  committed
  754e8bd
- Add test to verify column statistics are usable after write
  adamreeve
  committed
  bfe49c8
Commits on Mar 31, 2025
- Merge remote-tracking branch 'apache/main' into encryption-basics-fork
  alamb
  committed
  70d3a9b
Commits on Apr 1, 2025
- Merge remote-tracking branch 'apache/main' into encryption-basics-fork
  alamb
  committed
  04a8dad

Commits

Commits on Mar 16, 2025

Commits on Mar 17, 2025

Commits on Mar 18, 2025

Commits on Mar 19, 2025

Commits on Mar 20, 2025

Commits on Mar 21, 2025

Commits on Mar 23, 2025

Commits on Mar 24, 2025

Commits on Mar 25, 2025

Commits on Mar 26, 2025

Commits on Mar 27, 2025

Commits on Mar 30, 2025

Commits on Mar 31, 2025

Commits on Apr 1, 2025