Codex/merge upstream 0.7 - #3
Merged
Merged
Conversation
LCAConfig has a foreground_db_name field, but LCADataProcessor.__init__ declared its own foreground_db_name parameter defaulting to "foreground" and never consulted the config. Setting the name only on the config was therefore ignored: the processor read whatever database happened to be called "foreground". The failure is silent. _construct_foreground_tensors skips every node whose type is not "process", so pointing at the wrong database yields empty tensors, logged as "(0 processes, 0 flows, 0 years)", and the run continues until something unrelated fails much later. The parameter now defaults to None and falls back to config.foreground_db_name, so a config naming a missing database raises instead of silently falling back. An explicitly passed name still wins, keeping existing callers working.
…und-db-name fix: honor LCAConfig.foreground_db_name in LCADataProcessor
Solve all intermediate flows of a background database against a single factorization and keep only the aggregated elementary flow vector (B @ supply_array) instead of building the full per-process inventory matrix and annotating it through to_dataframe() once per flow and method. The inventory never depended on the LCIA method, so it is computed once. Activities and biosphere flows are resolved from one pass over each database, results are cached per (project, database, modified token, activity, cutoff), and databases are calculated in parallel by default. Dynamic characterization now runs in a single characterize() call for all elementary flows, with the method's characterization functions built once and cached. cutoff no longer defaults to 1e4: it truncated the unaggregated inventory before summing, which biased flow amounts low. Flows without a characterization factor in any category are dropped instead, since they cannot affect any impact; retain_flows keeps the ones needed for flow limits.
…ase study Documents that uncharacterized elementary flows are dropped and how to keep them with retain_flows, plus the entry point guard needed for the parallel calculation in scripts. methanol_and_iron_v0.5.0.ipynb preserves the case study exactly as it was run for the paper with optimex 0.5.0, whose background inventories were truncated.
The expression rules read the background inventory, the database mapping and the characterization factors through Pyomo Params, one indexed lookup with full index validation per scalar. None of those tensors carry decision variables, so they are now combined into plain Python numbers before the expressions are built: - the background inventory is collapsed over background databases once, per (intermediate flow, elementary flow, year) - the sum over elementary flows is folded into the characterization factors, so the impact expressions run over intermediate flows instead of every elementary flow -- mathematically the same, but a much smaller tree, which the LP writer also has to walk - installation- and operation-driven flows, always used together, become one Expression indexed only over the (process, flow) pairs that carry data - production rates, vintage flow rates and the active vintages per (process, year) are cached instead of recomputed per rule call get_dynamic_inventory now assembles the inventory from solved flow values and the same data instead of evaluating the scaled_inventory expression once per (process, flow, year), and maps flow codes to ids from one pass over the biosphere database rather than a query per row. Rows that are exactly zero are no longer emitted. On the paper case study: model build 103.6 s -> 1.7 s, solve wall clock 55.7 s -> 0.1 s, dynamic inventory 19.8 s -> 0.2 s, with identical objective, identical variable values and identical model size.
Building and factorizing the technosphere matrix of a background database is the bulk of the LCA processing, and a new Python session used to redo it for every database. Calculated inventories are now also written to optimex-inventory-cache inside the Brightway project, keyed by the database's modified token, the activity identity and the cutoff, so a stale entry misses instead of being served. On the methanol & iron case study the processor drops from 37 s to 0.3 s on the second run, i.e. the whole pipeline from 38 s to 1.5 s. Configurable through background_inventory.use_disk_cache and disk_cache_dir; clear_lca_caches(include_disk=True) removes the files.
Constraining an elementary flow that was dropped for lacking a characterization factor failed with a bare list of valid keys. The error now names the setting that keeps such a flow. Adds end-to-end coverage that a retained, uncharacterized background flow still drives a cumulative flow limit, and gives the notebook cells the ids nbformat 4.5 expects.
The background inventory collapsed over databases was held three times: the flat (intermediate flow, elementary flow, year) dict plus the two groupings built from it. Building the impact factors from a grouping instead lets the flat form go, cutting create_model's peak memory on the paper case study from +192 MB to +139 MB.
The constraint rules are indexed over the model's own sets, so a limit naming a flow the model does not have, or a year outside the horizon, was dropped without a word. OptimizationModelInputs rejects those on construction, but assigning a limit to an already-built instance bypasses that. create_model now warns about limits it ignores, pointing at retain_flows for the dropped-flow case, and about limits on flows that no process exchanges, which can never bind.
Group the paper version of the methanol & pig iron case study with the data
and plotting code it needs, so the reproducibility artifacts live next to the
notebook that produces them:
notebooks/paper/
├── methanol_and_iron_v0.5.0.ipynb
├── data/ (was notebooks/data/paper/)
└── plots/ (was notebooks/plots/)
Shared inputs (model_inputs_2050.json, product_system.svg) stay in
notebooks/data/ and are referenced as ../data/.
The docs example notebook (methanol_and_iron.ipynb) no longer reaches into
paper/: it loads its own saved model inputs, and its result exports are
commented out since only the paper needs those files on disk. The
baseline_design/evolution_results round-trip stays active - it is what lets
Scenario 4 run without keeping the earlier models in RAM - and now writes to
notebooks/data/.
perf: batch the background inventory and build the model from precomputed data
Factorizing an ecoinvent-sized technosphere matrix (~44k x 44k) costs ~3.8 s with UMFPACK and cuts each subsequent solve from ~0.38 s to ~0.03 s, so it only amortizes above ~11 intermediate flows. Databases contributing just a handful of flows were paying the full factorization for a few solves. Gate it behind a flow-count threshold and add scikit-umfpack as a dependency so the fast factorization is actually available. With PARDISO the flag is a no-op anyway: decompose_technosphere warns and returns, and PARDISO reuses its own factorization keyed on the matrix.
scikit-umfpack was added unconditionally, which is wrong on x86: MKL via pypardiso is the faster and better-supported option there, and it is what brightway uses when available. UMFPACK is only the right choice on ARM macs, where MKL does not exist. Gate both behind environment markers - pypardiso on x86_64/AMD64, scikit-umfpack on Darwin arm64 - so neither is installed where it does not belong. Everything else falls back to scipy's SuperLU.
scikit-umfpack ships no wheels, so installing it builds from sdist and needs
swig and SuiteSparse on the machine. As a platform-conditional dependency it
therefore broke every install on ARM macs - including CI on macos-latest,
which is aarch64:
../meson.build:21:7: ERROR: Program 'swig' not found or not executable
Move it to an `umfpack` extra for those who have the build tools. Without it
ARM macs fall back to scipy's SuperLU, which is slower but always available;
x86 keeps pypardiso, which does ship wheels.
pypardiso's `spsolve` squeezes its result, so a background database whose
technosphere holds a single process returns a 0-d array instead of a length-1
vector. Sparse `@` rejects that as a scalar operand:
ValueError: Scalar operands are not allowed, use '*' instead
This surfaced on x86 CI once pypardiso became a dependency there; the small
test systems are the ones that hit it, since a real database never solves to a
single element. Reshape the solution to 1-d before multiplying, and cover it
with a test that emulates the squeeze on a single-process background.
Bump __version__ to 0.6.0 and close out the Unreleased changelog section with today's date.
Release 0.6.0
Resolve conflicts against the v0.6.0 performance refactor (RWTH-LTT#64): - optimizer.py: main moved the flow and production rates into cached helpers (`flow_rate`, `production_rate`) that summed the tensor over the whole operation window. Both now take the lifecycle stage `tau` and return the per-tau entry, i.e. the flow/output per operating unit and year; the cache key carries `tau`. Call sites pass `t - v`. - optimizer.py: `OperationCapacity` keeps the PR's unit-count form and is indexed by ACTIVE_VINTAGE_TIME only, so `get_production_value` is now unused and removed. - tests/test_single_route_lca_comparison.py: main rewrote the file with LF endings and added `test_background_inventory_handles_squeezed_solver_result`, so the whole file conflicted; rebuilt as main's version plus the PR's restated single-year test and new multi-period test. - CHANGES.md: PR entries moved to [Unreleased] above the released 0.6.0.
…n-semantics fix: amortize installation impacts over a unit's lifetime output
Bump __version__ to 0.7.0 and close out the Unreleased changelog section with today's date. Minor rather than patch: the installation-semantics fix changes results for any process with a multi-year operation window, and existing_capacity and the deployment and operation limits now carry a different unit, so existing setups need their inputs rescaled.
Release 0.7.0
`existing_capacity` counts process units, not annual capacity, so the brownfield fleet of 0.5e6 kg/yr per vintage was read as 1/26 of its intended size and the optimizer replaced the missing capacity with new build. Multiply the annual figures by the number of operating years. Switch the scenarios back to Gurobi. The installation variables carry objective coefficients of 1e-9 to 1e-5, at or below GLPK's default optimality tolerance, so GLPK reports `optimal` while returning a solution 0.3% off with an essentially arbitrary deployment schedule (175.93 against 175.41 from `glpsol --exact`). Re-derive the iridium budget. All iridium in this system sits in PEM stack construction, whose material demand per kg of hydrogen was under-counted by the 9-step operation window, so 0.125 kg no longer represents the share of the unconstrained requirement it was chosen to represent. 1.125 kg restores it to roughly 55%. Clear the stale scenario outputs: they mixed two runs and carried a NameError. Refresh the saved model inputs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.