Skip to content

#425: Improve performance of INI/BC serialization#426

Open
priscavdsluis wants to merge 80 commits into
mainfrom
refactor/425-improve-performance-ini-serialization
Open

#425: Improve performance of INI/BC serialization#426
priscavdsluis wants to merge 80 commits into
mainfrom
refactor/425-improve-performance-ini-serialization

Conversation

@priscavdsluis

Copy link
Copy Markdown
Contributor

No description provided.

priscavdsluis and others added 30 commits November 1, 2022 16:33
@priscavdsluis priscavdsluis linked an issue Nov 22, 2022 that may be closed by this pull request
@sonarqubecloud

Copy link
Copy Markdown

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 6 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

@priscavdsluis

priscavdsluis commented Nov 22, 2022

Copy link
Copy Markdown
Contributor Author

With these changes the @veenstrajelmer tests will be about 33% faster (from ~30s to ~20s on 100000 rows)
There is still a problem with the AutoStart option (hence the failing tests), which may be solved in a different issue.

@priscavdsluis

Copy link
Copy Markdown
Contributor Author

I think this PR could still be useful for the performance of writing BC/INI files.

At the moment, hydrolib-core creates all kinds of objects before writing/serializig them to a file. For example Property and Section objects for INI files.
But all of these objects are validated by Pydantic. This is desired and useful for reading files to validate user input, but not when writing them (we know the data that we write will be valid). So the validation during writing is not needed and reduces the performance.

According to Pydantic, creating objects can be up to 30x faster when object are created via the construct() function instead of the default initializer. So it might be worth looking into again.

https://docs.pydantic.dev/1.10/usage/models/#creating-models-without-validation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve performance writing BC/INI files

2 participants