Skip to content

[COST-7252] Implement constant currency Phase 1#6005

Open
ELK4N4 wants to merge 122 commits into
project-koku:mainfrom
ELK4N4:COST-7252/constant-currency-phase1
Open

[COST-7252] Implement constant currency Phase 1#6005
ELK4N4 wants to merge 122 commits into
project-koku:mainfrom
ELK4N4:COST-7252/constant-currency-phase1

Conversation

@ELK4N4
Copy link
Copy Markdown
Contributor

@ELK4N4 ELK4N4 commented Apr 15, 2026

Summary

Implements Phase 1 of the constant currency feature, enabling per-month exchange rate resolution in report queries and forecasts instead of using a single daily rate.

  • New models (cost_models app, tenant-scoped): StaticExchangeRate (user-defined rates with monthly validity), MonthlyExchangeRate (single source of truth per month), EnabledCurrency (UI dropdown control)
  • New API endpoints: exchange-rate-pairs/ (static rate CRUD), settings/currency/enabled-currencies/ (list/update enablement), settings/currency/available-currencies/ (dropdown source)
  • Pipeline changes: get_daily_currency_rates Celery task now discovers currencies, creates EnabledCurrency records, and upserts dynamic rates into MonthlyExchangeRate (respecting static overrides)
  • Query handler changes: Base, OCP, and forecast query handlers use Subquery annotation from MonthlyExchangeRate for per-month rate resolution with earliest-rate fallback
  • Report metadata: Responses include exchange_rates_applied in meta for transparency
  • Cache invalidation: Both static rate writes and Celery task updates invalidate tenant view caches

Test plan

  • Unit tests for StaticExchangeRate serializer (validation, versioning, MonthlyExchangeRate side effects)
  • Unit tests for StaticExchangeRateViewSet (CRUD operations, filtering, error cases)
  • Unit tests for EnabledCurrencyView and AvailableCurrencyView
  • Unit tests for MonthlyExchangeRate and EnabledCurrency model constraints
  • Integration test: report queries return per-month exchange rates from MonthlyExchangeRate
  • Integration test: static rate overrides dynamic rate for overlapping months
  • Migration test: seed data populates MonthlyExchangeRate from ExchangeRateDictionary
  • Manual test: full on-prem dev env with cost model rates + currency conversion

Made with Cursor

@ELK4N4 ELK4N4 added the flightpath-pr Issues being worked on by the flight path team label Apr 15, 2026
@ELK4N4 ELK4N4 changed the title COST-7252: Implement constant currency Phase 1 [COST-7252] Implement constant currency Phase 1 Apr 15, 2026
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements the Constant Currency feature, introducing static exchange rate management and dynamic rate locking to ensure historical reporting accuracy. Key architectural changes include new tenant-scoped models, a CRUD API for rate pairs, and a transition to subquery-based rate resolution in the reporting pipeline. The review feedback primarily identifies opportunities to adhere to DRY principles by refactoring duplicated exchange rate annotation logic across query and forecast handlers. Additionally, it is recommended to enhance error reporting by specifying missing currency pairs to improve system debuggability.

Comment thread koku/api/report/queries.py Outdated
Comment thread koku/api/report/queries.py Outdated
Comment thread koku/forecast/forecast.py Outdated
Comment thread koku/forecast/forecast.py Outdated
Add static exchange rate CRUD, per-month rate storage via
MonthlyExchangeRate, currency enablement settings, and Subquery-based
exchange rate resolution in query handlers and forecasts.

New models (tenant-scoped in cost_models app):
- StaticExchangeRate: user-defined rates with monthly validity periods
- MonthlyExchangeRate: single source of truth for all months
- EnabledCurrency: controls dropdown visibility per tenant

New endpoints:
- GET/POST/PUT/DELETE /exchange-rate-pairs/ (static rate CRUD)
- GET/PUT /settings/currency/enabled-currencies/
- GET /settings/currency/available-currencies/

Pipeline changes:
- get_daily_currency_rates: currency discovery, per-tenant
  MonthlyExchangeRate upsert, CURRENCY_URL skip when empty
- QueryHandler/ReportQueryHandler: Subquery annotation from
  MonthlyExchangeRate with earliest-rate fallback
- OCP query handler: dual Subquery (cost model + infra currency)
- Forecast handlers: same Subquery pattern
- Report responses: exchange_rates_applied metadata
- Cache invalidation on both write paths

Made-with: Cursor
@ELK4N4 ELK4N4 force-pushed the COST-7252/constant-currency-phase1 branch from 7aab645 to 48f49de Compare April 15, 2026 16:25
ELK4N4 and others added 2 commits April 16, 2026 18:13
- Renumber migrations 0012-0014 to 0013-0015 to resolve conflict
  with 0012_add_rate_model from main
- Remove unused ValidationError import in test_static_exchange_rate_serializer
- Extract exchange rate annotation logic into cost_models/exchange_rate_annotations.py
  to eliminate duplication across QueryHandler, ReportQueryHandler, Forecast,
  OCPReportQueryHandler, and OCPForecast (DRY)
- Include missing base currencies in _validate_exchange_rates error message
  for easier debugging

Made-with: Cursor
@github-actions github-actions Bot added the smokes-required Label to show that smokes tests should be run against these changes. label Apr 16, 2026
ELK4N4 and others added 11 commits April 16, 2026 18:31
Remove unused imports flagged by flake8 F401 across serializer, view,
and test files. Apply black formatting fixes for line wrapping.

Made-with: Cursor
- Replace TruncMonth(OuterRef()) with ExtractYear/ExtractMonth to avoid
  Django's ResolvedOuterRef.output_field AttributeError
- Use **self.headers in view tests instead of force_authenticate
  (Koku's IdentityHeaderMiddleware requires x-rh-identity)
- Restore log_json import removed in prior cleanup (F821)
- Refactor get_daily_currency_rates to reduce cyclomatic complexity (C901)

Made-with: Cursor
…lookup

Django auto-generates the reverse accessor without underscores for
CostModelMap -> CostModel FK. All existing code uses costmodelmap.

Made-with: Cursor
Use OuterRef(OuterRef("source_uuid")) to correctly reference the
outermost queryset when the CostModel subquery is nested inside
the MonthlyExchangeRate subquery.

Made-with: Cursor
Django 5.2 resolves OuterRef to ResolvedOuterRef without output_field,
which breaks datetime Trunc* transforms. Pass DateField on OuterRef so
TruncMonth can resolve the lhs field type.

Made-with: Cursor
OuterRef inherits F; output_field is not a valid kwarg in our Django version,
causing TypeError across report and forecast tests.

Made-with: Cursor
Replace TruncMonth(OuterRef("usage_start")) with ExtractYear/ExtractMonth
field lookups. TruncBase.resolve_expression accesses copy.lhs.output_field
directly, which crashes on ResolvedOuterRef. Extract.resolve_expression uses
getattr with a None fallback, avoiding the AttributeError.

Also fix CostModel reverse FK lookup: cost_model_map -> costmodelmap
(Django auto-generates the accessor without underscores).

These two bugs caused all 688 unit test failures in CI.

Made-with: Cursor
_get_exchange_rates_applied queries MonthlyExchangeRate (a tenant-scoped
model in cost_models app) but runs outside the tenant_context block in
execute_query. Wrap the query with tenant_context(self.tenant) so
PostgreSQL finds the table in the correct tenant schema.

Made-with: Cursor
@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 16, 2026

Codecov Report

❌ Patch coverage is 91.52174% with 39 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.3%. Comparing base (9da32c9) to head (f078f46).

Additional details and impacted files
@@           Coverage Diff           @@
##            main   #6005     +/-   ##
=======================================
- Coverage   94.4%   94.3%   -0.0%     
=======================================
  Files        362     367      +5     
  Lines      32101   32448    +347     
  Branches    3538    3585     +47     
=======================================
+ Hits       30290   30613    +323     
- Misses      1176    1186     +10     
- Partials     635     649     +14     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

ELK4N4 and others added 10 commits April 19, 2026 13:42
- Remove unused VALID_RATE_TYPES constant (superseded by RateType enum)
- Remove uncalled _validate_exchange_rates method and its ValidationError import
- Remove unused logging imports and LOG variables from price_list_serializer,
  price_list_view, and static_exchange_rate_view

Made-with: Cursor
This helper is only used within exchange_rate_annotations.py, so prefix
it with an underscore to signal internal-only usage.

Made-with: Cursor
Accept a pre-built Django expression for base_currency instead of a
field name string, eliminating duplicated subquery logic in
build_ocp_exchange_rate_annotation_dict.

Made-with: Cursor
Static rate CRUD now enables currencies in EnabledCurrency on
create/update, so AvailableCurrencyView only needs to query
EnabledCurrency instead of unioning with StaticExchangeRate.

Made-with: Cursor
ELK4N4 and others added 2 commits May 7, 2026 18:13
DRF permission classes check request.user.is_authenticated, which
defaults to False on non-Django-auth models. Returning True is safe
because User instances are only created after middleware authentication.

Co-authored-by: Cursor <cursoragent@cursor.com>
bacciotti
bacciotti previously approved these changes May 9, 2026
@ELK4N4 ELK4N4 enabled auto-merge (squash) May 11, 2026 09:50
Validate that exchange rate values are non-null and positive before
storing them, preventing bad data from corrupting the rates table.

Co-authored-by: Cursor <cursoragent@cursor.com>
bacciotti
bacciotti previously approved these changes May 11, 2026
Copy link
Copy Markdown
Contributor

@myersCody myersCody left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should just combine the migrations.

Comment on lines +20 to +21

_ISO_4217_CURRENCIES = get_global("all_currencies")
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It appears you are allowing the customer to select any currency that is in the get_global("all_currencies") library. However, for our dynamic rates logic pulls pulls the exchange rates from: https://open.er-api.com/v6/latest/USD

@ELK4N4 @bacciotti Has anyone done any checking to ensure the list provided by get_global does not include currencies that can not be found in URL we use to grab dynamic exchange rates?

Copy link
Copy Markdown
Contributor Author

@ELK4N4 ELK4N4 May 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A user can defines a currency that doesn't exist in https://open.er-api.com/v6/latest/USD using static rates

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current process from what I can see allows user to enable a currency that can not be found in the CURRENCY_URL. Then if they try to utilize it they git a:

{"currency": ["No exchange rate available for <CODE>. Ask your administrator to configure static exchange rates or enable dynamic exchange rates."]}

The problem is that they did enable the dynamic rate, it just can't be supported.

I think we may need some type of validation for the dynamic flow that ensure they currency code is found in whatever url they decide to use? Thoughts?

That way they can "enable a dynamic rate" without being able to get a rate if that makes sense.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can change the error message to

No exchange rate available for <CODE>. Ask your administrator to configure static exchange rates

if CURRENCY_URL is defined.

In general, I think the enabled currencies capability should be decoupled from the actual available rates as long the list is ISO 4217

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have checked the differences between Babel and the API.
Babel support all ISO 4217 currencies(~300) and the API support only 162.
The exchange rate API (https://open.er-api.com/v6/latest/USD) returns 6 currency
codes that are not in babel's ISO 4217 registry. These are silently skipped
during the exchange rate fetch in masu/celery/tasks.py.

Code Currency Territory Official Currency Pegged To
FOK Faroese Króna Faroe Islands DKK (Danish Krone) 1:1 DKK
GGP Guernsey Pound Guernsey GBP (British Pound) 1:1 GBP
IMP Isle of Man Pound Isle of Man GBP (British Pound) 1:1 GBP
JEP Jersey Pound Jersey GBP (British Pound) 1:1 GBP
KID Kiribati Dollar Kiribati AUD (Australian Dollar) 1:1 AUD
TVD Tuvaluan Dollar Tuvalu AUD (Australian Dollar) 1:1 AUD

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the design is correct. decoupling "which currencies are enabled" from "which currencies have a dynamic rate" is the right call. however id say we have two things that worth addressing:

  1. fix the error message: if saas customer enables a currency thats not in the er-api and tries to use it, they get told to "enable dynamic exchange rates" -> but they already did

  2. would be good to, at least, warn the admin at "enable-time". Foe example: if CURRENCY_URL is configured and the currency being enabled isnt in ExchangeRateDictionary, return a warnings field in the response saying, maybe, "this currency has no dynamic rate available, you'll need to configure a static rate before users can use it."

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From the SaaS perspective they don't get to control the CURRENCY url. Therefore, there is no way for them to "fix" it to where they could provide a dynamic rate to the currencies not in the URL. It is probably worth a discussion a team level on how to handle it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

after the team discussion, i think we landed on two things that need to be addressed before merge:

  1. error message: if a currency has no rate available, the current message suggests enabling dynamic exchange rates, but that won't help if the currency simply isn't in the er-api. the message needs to be clear and actionable, something like "this currency requires a static exchange rate to be configured by your administrator."

  2. static-only signal at enable-time: if the currency being enabled isn't in the er-api, the admin should know upfront that it's static only. without a static rate configured, users will hit an error in reports.

the dynamic fallback only makes sense for currencies that exist in the er-api. for everything else, static is the only path, and both the admin and the end user need to know that clearly.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

code = code.upper()
try:
name = get_currency_name(code, locale="en_US")
symbol = get_currency_symbol(code, locale="en_US")
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The symbol and description are directly utilize by our UI. The previous approach utilize Unicode escape sequence for the more complex currency symbols. The purpose around this is because the UX supports multiple locals.

For example:

{
        "code": "CNY",
        "name": "Chinese Yuan",
        "symbol": "CN\u00a5",
        "description": "CNY (CN\u00a5) - Chinese Yuan",
},

Would return CN¥ for en_US local; however, if your local is Chinese it just returns ¥.

This approach forces all customers into the en_US local for currency. A minor change, but also different functionality from our previous approach. Let me ping the team to see if there is any impact to this change that I don't recognize real quick.

Copy link
Copy Markdown
Contributor Author

@ELK4N4 ELK4N4 May 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Today the UI converts from code to symbol? If that the case I can just returns the code and the UI will do the rest

ELK4N4 and others added 7 commits May 12, 2026 13:31
Only suggest enabling dynamic exchange rates when CURRENCY_URL is
configured. On-prem deployments without a rate API get a simpler
message pointing to static exchange rates only.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Return a warning in the POST response when CURRENCY_URL is configured
but the requested currency has no dynamic rate available, prompting
the admin to configure a static rate. Also default CURRENCY_URL to
None in on-prem mode since external rate APIs are typically unavailable.

Co-authored-by: Cursor <cursoragent@cursor.com>
@ELK4N4 ELK4N4 force-pushed the COST-7252/constant-currency-phase1 branch from b83a806 to 2bb4ecf Compare May 18, 2026 10:12
ELK4N4 and others added 2 commits May 18, 2026 14:28
Co-authored-by: Cursor <cursoragent@cursor.com>
Swap the error messages so that when CURRENCY_URL is configured,
we only suggest static rates (dynamic is already available), and
when it's not configured, we suggest both options.

Co-authored-by: Cursor <cursoragent@cursor.com>
@koku-ci-triager-bot
Copy link
Copy Markdown
Collaborator

🤖 CI Triager — Warning

Check: Migration convention
Root cause: This PR adds 2 migration files across 2 different apps, which deviates from the one-migration-per-PR convention.

Migrations found:

koku/api/migrations/0072_alter_exchangerates_currency_type.py
koku/cost_models/migrations/0014_constant_currency.py

Action: Since these migrations are in different apps (api and cost_models), they cannot be squashed together via squashmigrations. Please verify that both migrations are necessary for this PR. If one is preparatory or could land separately, consider splitting the PR. If both are intentional and needed together for the constant-currency feature, no action is required — but please confirm with a reviewer.

Generated automatically. Review before applying.

Check MonthlyExchangeRate (the actual source of truth for reports)
instead of ExchangeRateDictionary, and always warn regardless of
whether CURRENCY_URL is configured.

Co-authored-by: Cursor <cursoragent@cursor.com>
@koku-ci-triager-bot
Copy link
Copy Markdown
Collaborator

🤖 CI Triager — Warning

Check: Migration convention
Root cause: This PR adds 2 migration files across 2 apps, violating the one-migration-per-PR convention.
Evidence:

koku/api/migrations/0072_alter_exchangerates_currency_type.py
koku/cost_models/migrations/0014_constant_currency.py

Action: Please squash the migrations within each app into one. Run:

python koku/manage.py squashmigrations api 0072 0072
python koku/manage.py squashmigrations cost_models 0014 0014

If the migrations are in separate apps and each app has only one new migration, no squash is needed — but confirm that each app has exactly one new migration file, not multiple. In this case, two separate apps each have one new migration, which may be acceptable if they are both intentionally part of this PR. Please confirm with the team that both migration files are required in the same PR.

Generated automatically. Review before applying.

@koku-ci-triager-bot
Copy link
Copy Markdown
Collaborator

🤖 CI Triager — Diagnosis

Check: Units - 3.11
Run: 26178594573
Root cause: test_get_cost_model_rate_rbac_access fails with a DUPLICATE_AUTH error because initialize_request uses self.fake.word() for the OCP cluster_id. Faker's word list has ~40–50 entries; with the test DB preserved across CI runs, providers accumulate until a collision occurs.
Evidence:

ERROR: test_get_cost_model_rate_rbac_access (cost_models.test.test_view.CostModelViewTests)
...
  File "koku/cost_models/test/test_view.py", line 47, in initialize_request
    self.provider = serializer.save()
  File "koku/api/provider/serializers.py", line 318, in create
    raise serializers.ValidationError(error_obj(ProviderErrors.DUPLICATE_AUTH, message))
ValidationError: {'source.duplicate': [ErrorDetail(
  string='Cost management does not allow duplicate accounts. An integration
          already exists with these details.', code='invalid')]}

Action: In koku/cost_models/test/test_view.py, change initialize_request to use a UUID for the cluster_id so it is always unique, regardless of DB state:

# Line ~41 in initialize_request():
# Before:
"authentication": {"credentials": {"cluster_id": self.fake.word()}},
# After:
"authentication": {"credentials": {"cluster_id": str(self.fake.uuid4())}},

This file is not in the PR diff, but the fix is a one-line change. Alternatively, re-run CI with a clean test database (--keepdb=False) to unblock this run.

Generated automatically. Review before applying.

@koku-ci-triager-bot
Copy link
Copy Markdown
Collaborator

🤖 CI Triager — Warning

Check: Migration convention
Root cause: This PR adds 2 new migration files across two apps. The convention is one migration per PR; multiple migrations in the same app should be squashed.
Evidence:

koku/api/migrations/0072_alter_exchangerates_currency_type.py
koku/cost_models/migrations/0014_constant_currency.py

Action: These migrations are in different apps (api and cost_models), so cross-app squashing is not applicable. However, if either app has intermediate migration files, squash them with:

python koku/manage.py squashmigrations api <first> 0072
python koku/manage.py squashmigrations cost_models <first> 0014

If each app has exactly one new migration file, no action is needed — this warning is informational.

Generated automatically. Review before applying.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

flightpath-pr Issues being worked on by the flight path team smoke-tests pr_check will run minimal required smokes. Used when changes hit multiple providers. smokes-required Label to show that smokes tests should be run against these changes.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants