feature: ATL-51: cache haplotype fetching#1443
feature: ATL-51: cache haplotype fetching#1443SergeySalinAtDataart wants to merge 4 commits intomasterfrom
Conversation
760c0e7 to
a1c7347
Compare
There was a problem hiding this comment.
Pull request overview
This PR adds caching for active haplotype-frequency-set lookups in the Match Prediction stack and introduces stronger options binding/validation for several settings classes. It fits into the Match Prediction services by reducing repeated database reads for active set selection and wiring the new cache TTL through app configuration.
Changes:
- Added
HaplotypeFrequencySetCachesettings, Terraform wiring, and docs for configuring the active-set cache TTL. - Updated
HaplotypeFrequencyServiceto cache all active HF sets and invalidate that cache after imports. - Added/expanded settings validation annotations and worker-side validated options registration, plus unit/integration test config updates.
Reviewed changes
Copilot reviewed 30 out of 30 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| terraform/core/variables.tf | Adds Terraform variable for active HF-set cache expiry. |
| terraform/core/modules/match_prediction/variables_release.tf | Exposes cache expiry variable to the match prediction module. |
| terraform/core/modules/match_prediction/function.tf | Passes cache expiry into Match Prediction Function app settings. |
| terraform/core/main.tf | Wires root Terraform variable into the module input. |
| README_Configuration.md | Documents the new configuration setting. |
| Atlas.MultipleAlleleCodeDictionary/Settings/MacDictionarySettings.cs | Adds data-annotation validation to MAC settings. |
| Atlas.MatchPrediction/Services/HaplotypeFrequencies/HaplotypeFrequencyService.cs | Implements active-set caching and cache invalidation on import. |
| Atlas.MatchPrediction/ExternalInterface/Settings/MessagingServiceBusSettings.cs | Adds validation attributes to messaging settings. |
| Atlas.MatchPrediction/ExternalInterface/Settings/MatchPredictionRequestsSettings.cs | Adds validation attributes to request topic settings. |
| Atlas.MatchPrediction/ExternalInterface/Settings/HaplotypeFrequencySetCacheSettings.cs | Introduces cache TTL settings type. |
| Atlas.MatchPrediction/ExternalInterface/Settings/AzureStorageSettings.cs | Adds validation attributes to storage settings. |
| Atlas.MatchPrediction.Worker/Startup.cs | Switches worker startup to validated options registration. |
| Atlas.MatchPrediction.Worker/Settings/SettingsValidationExtensions.cs | Adds worker options binding/validation helpers and validators. |
| Atlas.MatchPrediction.Worker/Settings/MatchPredictionWorkerSettings.cs | Adds validation attributes for worker settings. |
| Atlas.MatchPrediction.Worker/Program.cs | Passes configuration into startup. |
| Atlas.MatchPrediction.Worker/appsettings.json | Adds cache settings and validated worker config values. |
| Atlas.MatchPrediction.Test/Services/HaplotypeFrequencies/HaplotypeFrequencyServiceTests.cs | Adds unit tests for caching and invalidation behavior. |
| Atlas.MatchPrediction.Test.Verification/Startup.cs | Registers new cache settings in verification tests. |
| Atlas.MatchPrediction.Test.Verification/local.settings.template.json | Adds cache setting to verification local config. |
| Atlas.MatchPrediction.Test.Validation/Startup.cs | Registers new cache settings in validation tests. |
| Atlas.MatchPrediction.Test.Validation/local.settings.template.json | Adds cache setting to validation local config. |
| Atlas.MatchPrediction.Test.Integration/DependencyInjection/ServiceConfiguration.cs | Registers cache settings for integration tests. |
| Atlas.MatchPrediction.Test.Integration/appsettings.json | Adds cache setting to integration config. |
| Atlas.MatchPrediction.Functions/Startup.cs | Registers cache settings in Match Prediction Functions host. |
| Atlas.MatchPrediction.Functions/local.settings.template.json | Adds cache setting to local function config. |
| Atlas.MatchPrediction.Data/Repositories/HaplotypeFrequencySetRepository.cs | Adds query to fetch all active sets. |
| Atlas.HlaMetadataDictionary/ExternalInterface/Settings/SearchRelatedMetadataServiceSettings.cs | Adds validation attribute to nested metadata cache setting. |
| Atlas.HlaMetadataDictionary/ExternalInterface/Settings/HlaMetadataDictionarySettings.cs | Adds validation attributes to dictionary settings. |
| Atlas.Functions/Startup.cs | Registers cache settings in top-level ATLAS-FUNCTIONS host. |
| Atlas.Common/Notifications/NotificationsServiceBusSettings.cs | Adds validation attributes to notification settings. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| @@ -90,6 +90,7 @@ private static void RegisterSettings(IServiceCollection services) | |||
|
|
|||
| // Match Prediction Algorithm | |||
| services.RegisterAsOptions<MatchPrediction.ExternalInterface.Settings.AzureStorageSettings>("MatchPrediction:AzureStorage"); | |||
There was a problem hiding this comment.
Need to add the setting to appsettings.json? And the terraform?
| try | ||
| { | ||
| await frequencySetImporter.Import(file, importBehaviour); | ||
| cache.Remove(ActiveHaplotypeFrequencySetsCacheKey); |
daria-sorokina-da
left a comment
There was a problem hiding this comment.
@daria-sorokina-da reviewed 30 files and all commit messages, and made 1 comment.
Reviewable status: all files reviewed, 2 unresolved discussions (waiting on daria-sorokina-da and pchenery).
| @@ -90,6 +90,7 @@ private static void RegisterSettings(IServiceCollection services) | |||
|
|
|||
| // Match Prediction Algorithm | |||
| services.RegisterAsOptions<MatchPrediction.ExternalInterface.Settings.AzureStorageSettings>("MatchPrediction:AzureStorage"); | |||
There was a problem hiding this comment.
Need to add the setting to appsettings.json? And the terraform?
a1c7347 to
52606b5
Compare
daria-sorokina-da
left a comment
There was a problem hiding this comment.
@daria-sorokina-da reviewed 3 files and all commit messages.
Reviewable status: all files reviewed, 2 unresolved discussions (waiting on pchenery).
^ Conflicts: ^ terraform/core/modules/match_prediction/variables_release.tf
52606b5 to
d2dff37
Compare
daria-sorokina-da
left a comment
There was a problem hiding this comment.
@daria-sorokina-da reviewed 2 files and all commit messages, and resolved 2 discussions.
Reviewable status:complete! all files reviewed, all discussions resolved (waiting on pchenery).
This change is