Bechmark batched writes to nilDB and owned schemas

[vishakh]

• Review https://github.com/NillionNetwork/secretvaults-py/blob/main/examples/standard_data_example.py and https://github.com/NillionNetwork/secretvaults-py/blob/main/examples/owned_data_example.py
• Review https://github.com/Cryptonomic/MonadicDNA/tree/feature/nillion-load-test/nillion-load-test
• Create a new branch off nillion-load-test
• Modify the code so that user data is uploaded to nilDB in batches while still getting good upload times
• Add a comment here with the bechmarking results from above
• Switch the data schema from standard to owned and simulate uploading data from a hundred users
• Run a nilDB query to get the values of the same rsid for all one hundred users using an independent Monadic DNA account
• Add a comment here with the query used and the performance details

[amardeepio]

9️⃣ Creating standard data in batches...
📖 Reading data from ../services/nillion-interactor/testdata/hu278AF5_20210124151934.txt...
🧬 Total records to process: 631955
📦 Uploading data in 64 batches of size 10000...
Uploading data: 100%|██████████████████████████████████████| 631955/631955 [35:53<00:00, 293.51records/s]

✅ Successfully created 631955 data records in 64 batches.
⏱️ **Total upload time**: 2153.09 seconds

with batch size of 10,000, it took 35 min and 35 seconds.

I did the benchmark with 20,000 batch size.

9️⃣ Creating standard data in batches...
📖 Reading data from ../services/nillion-interactor/testdata/hu278AF5_20210124151934.txt...
🧬 Total records to process: 631955
📦 Uploading data in 32 batches of size 20000...
Uploading data: 100%|████████████████████████████████████████████████|631955/631955 [24:25<00:00, 431.31records/s]

✅ Successfully created 631955 data records in 32 batches.
⏱️ Total upload time: 1465.17 seconds or 24.41667 min
📈 Average records/sec: 431.32

Above 20,000 batch size, i got into error. so it seems like 20k is the sweet spot

[amardeepio]

Simulation result for100 users with 100 records each :

✅ Builder registered successfully!

7️⃣ Creating owned collection...
✅ Owned collection created with ID: 353e55ad-bf24-42e3-ae1c-237a8ce47a9f

8️⃣ Simulating data upload for 100 users, with 100 records each...
📖 Reading data from ../services/nillion-interactor/testdata/hu278AF5_20210124151934.txt...
ℹ️  Granting read access to the builder itself: did:nil:023799f849fe727ecb0687b7e45b945f515463a4afe29eb32b10db7a43e1871732
Uploading user data: 100%|███████████████████████████████████████| 100/100 [03:35<00:00,  2.15s/users]

✅ Successfully simulated 100 users uploading 100 records each.
⏱️ Total upload time: 215.47 seconds
📈 Average records/sec: 46.41

Simulation result for100 users with 6000 records each :

✅ Builder registered successfully!

7️⃣ Creating owned collection...
✅ Owned collection created with ID: 205cd8f8-6482-4912-b4f4-2512f359ee16

8️⃣ Simulating data upload for 100 users, with 6000 records each...
📖 Reading data from ../services/nillion-interactor/testdata/hu278AF5_20210124151934.txt...
ℹ️  Granting read access to the builder itself: did:nil:023799f849fe727ecb0687b7e45b945f515463a4afe29eb32b10db7a43e1871732
Uploading user data: 100%|███████████████████████████████████████| 100/100 [13:41<00:00,  8.22s/users]

✅ Successfully simulated 100 users uploading 6000 records each.
⏱️ Total upload time: 821.72 seconds
📈 Average records/sec: 730.18