Skip to content

Energy demand profile: to take heating into account#28232

Open
daniel309 wants to merge 40 commits intoevcc-io:masterfrom
daniel309:feature/temperature-correction
Open

Energy demand profile: to take heating into account#28232
daniel309 wants to merge 40 commits intoevcc-io:masterfrom
daniel309:feature/temperature-correction

Conversation

@daniel309
Copy link
Contributor

@daniel309 daniel309 commented Mar 15, 2026

Temperature-Based Household Load Correction with Heater Profile Separation

Base PR: #27780 (Weather Tariff)
This PR: Heater Profile Separation for Temperature Correction

note: this PR doesnt change the database schema (only intermediate dev states did). Its independent of the "history for meters" work going on in #23185.

TL/DR

With this PR optimizer forecasts for the household load are finally useable when you have a heatpump device registered in evcc.

Before that they were unuseable because:

  1. heater loadpoints got substracted from profile gt, and
  2. no temperature correction applied.

evidence of the effects of this PR: #28232 (comment)
explanation of the mathematical function to estimate corrections: #28232 (comment)

Problem

The evopt energy optimizer uses a household load profile (gt) to predict how much energy the home will consume in each future 15-minute slot. This profile is currently computed as a 30-day historical average — the same flat pattern is repeated regardless of weather conditions.

This is a significant blind spot: heating and cooling loads are strongly temperature-dependent. On a cold winter day, a home with a heat pump or electric heating can consume 30–80% more energy than on a mild day. When the optimizer doesn't know this, it:

  • Under-estimates household demand on cold days → schedules EV charging at times when grid power is actually needed for heating
  • Over-estimates household demand on warm days → unnecessarily avoids cheap/green charging windows
  • Misses opportunities to pre-charge the battery before a cold night when heating demand will spike

Solution Overview

This PR builds on #27780 (which added the open-meteo weather tariff) and implements temperature-based correction of the household load profile with a critical improvement: the correction is applied only to heating device loads that are explicitly marked as temperature-sensitive.

Key Innovation: Selective Temperature Correction

Not all heating devices have temperature-dependent loads. For example:

  • Heat pumps are highly temperature-dependent
  • Auxiliary electric heaters may be temperature-dependent (more electricity because its colder outside) or not when used for warm water.

This PR implements a four-step process:

  1. Identify: Detect which heating devices are temperature-sensitive via the OutdoorTemperatureSensitive feature flag
  2. Separate: Extract temperature-sensitive and non-sensitive heater profiles separately
  3. Correct: Apply temperature adjustment only to temperature-sensitive heaters
  4. Merge: Combine base load + corrected temp-sensitive heaters + uncorrected non-sensitive heaters
Total Household = Base Load + Temp-Sensitive Heaters + Non-Sensitive Heaters
                  (gt_base)   (gt_temp_corrected)      (gt_non_sensitive)
                      ↓                ↓                        ↓
                 [unchanged]   [temp corrected]           [unchanged]
                      ↓                ↓                        ↓
                  Final Profile = gt_base + gt_temp_corrected + gt_non_sensitive

Temperature Correction Algorithm

The correction algorithm uses a physics-based model that relates heating load to the temperature difference between indoor and outdoor conditions:

image

which becomes:

load[i] = load_avg[i] × ((T_room − T_forecast[i]) / (T_room − T_past_avg[h]))

where:

  • T_room = 21°C (constant room temperature)
  • T_past_avg[h] = average temperature at hour-of-day h over the past 7 days
  • T_forecast[i] = forecast temperature at the wall-clock time of slot i

This formula models heating load as proportional to the temperature difference that must be maintained. The correction factor represents the ratio of future heating demand to historical average demand.

Safety Check: If historical temperature is within 0.5°C of room temperature (indicating heating was likely off), the correction is skipped for that time slot.

Example: With a 7-day historical average of 8°C at a given hour:

  • Forecast 8°C → no correction (factor = (21-8)/(21-8) = 1.0)
  • Forecast 3°C → +38% heater load (factor = (21-3)/(21-8) = 1.38)
  • Forecast −2°C → +77% heater load (factor = (21-(-2))/(21-8) = 1.77)
  • Forecast 13°C → −38% heater load (factor = (21-13)/(21-8) = 0.62)

Files Changed

File Change
api/feature.go Added OutdoorTemperatureSensitive feature flag
api/feature_enumer.go Generated enum code for new feature
core/site.go Removed heatingCoefficient parameter and added loadpoint energy tracking infrastructure (~50 lines)
core/metrics/db.go Added PersistLoadpoint() function for per-loadpoint tracking using meter IDs
core/site_optimizer.go Refactored to separate temp-sensitive/non-sensitive heaters and apply physics-based correction (~150 lines)
core/site_optimizer_temperature_test.go Updated test to remove heatingCoefficient

Configuration

The feature is fully opt-in — no changes needed for existing setups. Temperature correction is only active for heating devices explicitly marked as temperature-sensitive.

Basic Setup (requires base PR #27780)

tariffs:
  temperature:
    type: template
    template: open-meteo-temperature
    latitude: 48.1
    longitude: 11.6

chargers:
  - name: heatpump
    type: template
    template: luxtronik
    host: 192.168.1.10
    features:
      - outdoortemperaturesensitive   # Enable temperature correction. 
                                                             # Some templates, like the Luxtronik, have this set by default

  - name: water_heater
    type: custom
    features:
      - heating                                             # Mark as heating device
      # No outdoortemperaturesensitive  # No correction applied

Configuration Details

Temperature Tariff (required for correction):

  • Must be configured as shown above
  • Provides historical and forecast temperature data from Open-Meteo

Heating Device Features:

  • heating - Marks the device as a heating system (required for all heating devices)
  • outdoortemperaturesensitive - Enables temperature correction for this specific device (optional)

Important:

  • Devices with only heating feature will have their consumption included in forecasts but without temperature correction
  • Devices with both heating and outdoortemperaturesensitive will have temperature correction applied
  • This allows mixing temperature-dependent devices (space heating heat pumps) with schedule-based devices (water heaters, pool heaters)

Backward Compatibility

The feature is completely opt-in and requires explicit configuration. Existing setups are unaffected.

Temperature Correction Conditions

Heating device consumption is always added to the household profile. Temperature correction is only applied to specific heating loads when ALL of the following conditions are met:

  1. ✅ Weather tariff configured (tariffs.temperature)
  2. ✅ Heating device has api.Heating feature flag
  3. ✅ Heating device has api.OutdoorTemperatureSensitive feature flag
  4. ✅ Historical heater consumption data available (96 slots = 24 hours)
  5. ✅ Weather data available from Open-Meteo
  6. ✅ Historical temperature not too close to room temperature (>0.5°C difference)

If any condition is not met for a specific heater, its uncorrected profile is used, but consumption is still added to the total household forecast.

The algorithm works as follows:

Total Household = Base Load + Temp-Sensitive Heating (corrected) + Non-Sensitive Heating
                  (meter=1)   (meter=lpID+1000, corrected)        (meter=lpID+1000, as-is)

Multiple Heater Support

The implementation fully supports multiple heating devices with selective correction:

  1. Automatic Detection: All loadpoints with api.Heating feature are automatically identified
  2. Selective Correction: Only devices with api.OutdoorTemperatureSensitive get temperature correction
  3. Individual Tracking: Each heater's consumption is tracked separately in the database using its loadpoint ID
  4. Slot-by-Slot Aggregation: Multiple heater profiles are summed together for each 15-minute slot
  5. Separate Processing: Temperature-sensitive and non-sensitive heaters are processed independently

Example with 3 heaters:

Heat Pump (temp-sensitive):     [2.0, 2.5, 3.0, ...] kWh per 15min
                                         ↓
                                Temperature Correction
                                         ↓
                                [2.8, 3.5, 4.2, ...] (corrected)

Water Heater (not temp-sensitive): [0.5, 0.3, 0.8, ...] kWh per 15min (unchanged)

Aux Heater (temp-sensitive):        [0.3, 0.4, 0.5, ...] kWh per 15min
                                         ↓
                                Temperature Correction
                                         ↓
                                [0.4, 0.6, 0.7, ...] (corrected)

Combined Corrected:             [2.8, 3.5, 4.2, ...] + [0.4, 0.6, 0.7, ...] = [3.2, 4.1, 4.9, ...]
Combined Non-Corrected:         [0.5, 0.3, 0.8, ...]
                               ─────────────────────────────────────────────────────────────────
Final = Base Load + [3.2, 4.1, 4.9, ...] + [0.5, 0.3, 0.8, ...]

This approach ensures that:

  • Each heater's historical pattern is preserved
  • Only temperature-dependent loads are adjusted for weather
  • Schedule-based heaters remain predictable
  • Total heating load is accurately represented
  • Works with any number and combination of heating devices

Database Schema

No schema changes required. The implementation uses the existing meter table structure with different meter IDs to distinguish between:

  • Household base load (meter=1) - total home consumption excluding loadpoints
  • Individual loadpoints (meter=lpID+1000) - specific device consumption (e.g., heaters)

The meter ID offset (lpID+1000) provides sufficient separation from the household meter ID (1) and allows for future expansion. Historical data starts accumulating from deployment with no migration needed.

Dependencies

  • Requires base PR Tariffs: add temperature type and OpenMeteo #27780 (Weather Tariff) to be merged first
  • No new external dependencies
  • Uses existing api.Heating feature flag for device identification
  • Adds new api.OutdoorTemperatureSensitive feature flag for selective correction

Copy link
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 4 issues, and left some high level feedback:

  • Access to loadpointEnergy and loadpointSlotStart maps happens from multiple goroutines in updateLoadpoints/updateLoadpointConsumption without synchronization, which risks concurrent map access panics; consider guarding these maps (and their contained state) with a mutex or encapsulating per-loadpoint state in a concurrency-safe structure.
  • In applyTemperatureCorrection, the loop for h := range 24 is invalid in Go and will not compile; it should be replaced with an index loop like for h := 0; h < 24; h++ when building pastTempAvg.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- Access to `loadpointEnergy` and `loadpointSlotStart` maps happens from multiple goroutines in `updateLoadpoints`/`updateLoadpointConsumption` without synchronization, which risks concurrent map access panics; consider guarding these maps (and their contained state) with a mutex or encapsulating per-loadpoint state in a concurrency-safe structure.
- In `applyTemperatureCorrection`, the loop `for h := range 24` is invalid in Go and will not compile; it should be replaced with an index loop like `for h := 0; h < 24; h++` when building `pastTempAvg`.

## Individual Comments

### Comment 1
<location path="core/site_optimizer.go" line_range="658-659" />
<code_context>
+			pastTempCount[h]++
+		}
+	}
+	pastTempAvg := make([]float64, 24)
+	for h := range 24 {
+		if pastTempCount[h] > 0 {
+			pastTempAvg[h] = pastTempSum[h] / float64(pastTempCount[h])
</code_context>
<issue_to_address>
**issue (bug_risk):** The `for h := range 24` loop is invalid Go and will not compile.

Use a standard indexed `for` loop here instead of ranging over an `int`, for example:

```go
pastTempAvg := make([]float64, 24)
for h := 0; h < 24; h++ {
    if pastTempCount[h] > 0 {
        pastTempAvg[h] = pastTempSum[h] / float64(pastTempCount[h])
    }
}
```
</issue_to_address>

### Comment 2
<location path="core/site.go" line_range="927-928" />
<code_context>
 	)

-	for _, lp := range site.loadpoints {
+	for i, lp := range site.loadpoints {
+		lpID := i // capture loop variable for goroutine
 		wg.Go(func() {
 			power := lp.UpdateChargePowerAndCurrents()
</code_context>
<issue_to_address>
**issue (bug_risk):** The goroutine still closes over the loop variable `lp`, which can cause data races and incorrect behavior.

You correctly capture `lpID`, but `lp` is still shared across iterations due to `range` semantics. Capture `lp` in a new local variable before starting the goroutine:

```go
for i, lp := range site.loadpoints {
    lpID := i
    lp := lp // capture lp for goroutine
    wg.Go(func() {
        power := lp.UpdateChargePowerAndCurrents()
        // ...
    })
}
```
This prevents the goroutine from seeing a different loadpoint than intended and avoids subtle concurrency bugs in the update methods.
</issue_to_address>

### Comment 3
<location path="core/site_optimizer.go" line_range="687-688" />
<code_context>
+			continue
+		}
+
+		h := ts.UTC().Hour()
+		tPastAvg := pastTempAvg[h]
+
+		// delta > 0: tomorrow colder than historical average → load increases
</code_context>
<issue_to_address>
**suggestion (bug_risk):** Using a zero default for `tPastAvg` when there is no historical data for a given hour can skew the correction significantly.

When `pastTempCount[h] == 0`, `pastTempAvg[h]` stays at its zero value, which makes `tFuture` look much warmer than "history" and can drive overly strong negative corrections for those hours.

Consider skipping correction or using a safer fallback (e.g. overall past 24h average) when there’s no data:

```go
h := ts.UTC().Hour()
if pastTempCount[h] == 0 {
    continue // or use a fallback average
}
tPastAvg := pastTempAvg[h]
```

This prevents over-correcting when the historical baseline is unknown.

```suggestion
		h := ts.UTC().Hour()
		if pastTempCount[h] == 0 {
			// no historical data for this hour; skip correction (or use a fallback if available)
			continue
		}
		tPastAvg := pastTempAvg[h]
```
</issue_to_address>

### Comment 4
<location path="core/site_optimizer.go" line_range="515" />
<code_context>
 func (site *Site) homeProfile(minLen int) ([]float64, error) {
-	// kWh over last 30 days
-	profile, err := metrics.Profile(now.BeginningOfDay().AddDate(0, 0, -30))
+	from := now.BeginningOfDay().AddDate(0, 0, -7)
+	
+	// kWh average over last 7 days - base load (excludes loadpoints)
</code_context>
<issue_to_address>
**issue (complexity):** Consider refactoring the new home profile and temperature-correction logic into small reusable helpers and pre-indexed data structures to keep the core control flow simple and readable.

The new logic is valid but it does increase complexity in a few focused places. You can reduce it without changing behavior by extracting small helpers and pre-indexing data.

### 1) Repeated “repeat profile until minLen then trim” logic

This pattern appears multiple times in `homeProfile` for both base and heater profiles:

```go
slots := make([]float64, 0, minLen+1)
for len(slots) <= minLen+24*4 { // allow for prorating first day
	slots = append(slots, gt_base[:]...)
}
res := profileSlotsFromNow(slots)
if len(res) < minLen {
	return nil, fmt.Errorf("minimum home profile length %d is less than required %d", len(res), minLen)
}
if len(res) > minLen {
	res = res[:minLen]
}
```

and similarly for `heaterSlots`.

This can be encapsulated in a reusable helper:

```go
func repeatAndTrimProfile(profile []float64, minLen int) ([]float64, error) {
	slots := make([]float64, 0, minLen+1)
	for len(slots) <= minLen+24*4 {
		slots = append(slots, profile...)
	}

	res := profileSlotsFromNow(slots)
	if len(res) < minLen {
		return nil, fmt.Errorf("minimum home profile length %d is less than required %d", len(res), minLen)
	}
	if len(res) > minLen {
		res = res[:minLen]
	}
	return res, nil
}
```

Then `homeProfile` becomes simpler:

```go
gtBaseSlots, err := repeatAndTrimProfile(gt_base, minLen)
if err != nil {
	return nil, err
}

var gtHeaterSlots []float64
if len(gt_heater_raw) > 0 {
	if heater, err := repeatAndTrimProfile(gt_heater_raw, len(gtBaseSlots)); err == nil {
		gtHeaterSlots = heater
	}
}
```

This removes duplicated branching and makes the length-normalization logic easier to reason about.

### 2) Split `homeProfile` into clearer orchestration steps

`homeProfile` is currently doing: read base, read heater, normalize both, correct heater by temperature, merge, convert units. Extracting those as small helpers keeps the control flow readable:

```go
func (site *Site) buildBaseProfile(from time.Time, minLen int) ([]float64, error) {
	base, err := metrics.Profile(from)
	if err != nil {
		return nil, err
	}
	return repeatAndTrimProfile(base, minLen)
}

func (site *Site) buildHeaterProfile(from time.Time, minLen int) []float64 {
	raw := site.extractHeaterProfile(from, time.Now())
	if len(raw) == 0 {
		return nil
	}
	heater, err := repeatAndTrimProfile(raw, minLen)
	if err != nil {
		return nil
	}
	return site.applyTemperatureCorrection(heater)
}
```

`homeProfile` then reduces to an orchestrator:

```go
func (site *Site) homeProfile(minLen int) ([]float64, error) {
	from := now.BeginningOfDay().AddDate(0, 0, -7)

	base, err := site.buildBaseProfile(from, minLen)
	if err != nil {
		return nil, err
	}

	heater := site.buildHeaterProfile(from, len(base))

	final := make([]float64, len(base))
	for i := range base {
		final[i] = base[i]
		if heater != nil && i < len(heater) {
			final[i] += heater[i]
		}
	}

	return lo.Map(final, func(v float64, _ int) float64 { return v * 1e3 }), nil
}
```

Same behavior, but the main function reads as a high-level description of the steps.

### 3) Avoid O(N²) search in `applyTemperatureCorrection`

The inner loop that finds `tFuture` completely dominates the function’s complexity and obscures the main idea:

```go
for i := range profile {
	ts := slotStart.Add(time.Duration(i) * tariff.SlotDuration)

	var tFuture float64
	found := false
	for _, r := range rates {
		if r.Start.Equal(ts) {
			tFuture = r.Value
			found = true
			break
		}
	}
	if !found {
		continue
	}
	// ...
}
```

You can pre-index the forecast into a map and keep the slot loop single-level:

```go
func indexRatesByStart(rates api.Rates) map[time.Time]float64 {
	m := make(map[time.Time]float64, len(rates))
	for _, r := range rates {
		m[r.Start] = r.Value
	}
	return m
}
```

Use it in `applyTemperatureCorrection`:

```go
forecastByTime := indexRatesByStart(rates)

slotStart := currentTime.Truncate(tariff.SlotDuration)
for i := range profile {
	ts := slotStart.Add(time.Duration(i) * tariff.SlotDuration)

	tFuture, ok := forecastByTime[ts]
	if !ok {
		continue
	}

	h := ts.UTC().Hour()
	tPastAvg := pastTempAvg[h]
	delta := tPastAvg - tFuture
	result[i] = profile[i] * (1 + coeff*delta)
}
```

This both improves performance and reduces nesting, making the correction logic easier to follow.

### 4) Separate guard/validation from the core correction

The top of `applyTemperatureCorrection` is mostly guards and configuration checks. Extracting them into a small helper can clarify the “happy path”:

```go
func (site *Site) getWeatherRates() (api.Rates, float64, float64, error) {
	weatherTariff := site.GetTariff(api.TariffUsageTemperature)
	if weatherTariff == nil {
		return nil, 0, 0, fmt.Errorf("no weather tariff")
	}

	rates, err := weatherTariff.Rates()
	if err != nil || len(rates) == 0 {
		return nil, 0, 0, fmt.Errorf("no weather rates")
	}

	threshold := site.HeatingThreshold
	coeff := site.HeatingCoefficient
	if threshold == 0 || coeff == 0 {
		return nil, 0, 0, fmt.Errorf("heating config missing")
	}

	return rates, threshold, coeff, nil
}
```

Then:

```go
func (site *Site) applyTemperatureCorrection(profile []float64) []float64 {
	rates, threshold, coeff, err := site.getWeatherRates()
	if err != nil {
		return profile
	}

	// ... 24h avg, hourly history, forecastByTime, main loop ...
}
```

This keeps `applyTemperatureCorrection` focused on the actual correction math rather than the early-exit conditions.

These refactors maintain all functionality but reduce branching and nesting, making the new behavior easier to understand and maintain.
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@andig
Copy link
Member

andig commented Mar 15, 2026

First thing is to get #23185 in.

@naltatis
Copy link
Member

This contribution does not appear to meet our AI contribution guidelines.

@andig andig marked this pull request as draft March 15, 2026 09:54
daniel309 added a commit to daniel309/evcc that referenced this pull request Mar 15, 2026
- Fix goroutine loop variable capture for loadpoint
- Skip temperature correction when no historical data for hour
- Add debug logging for missing historical temperature data

Addresses review comments from PR evcc-io#28232
@andig
Copy link
Member

andig commented Mar 15, 2026

@naltatis this PR shows general understanding of evcc and has prior PRs. Fine for me.

@daniel309
Copy link
Contributor Author

daniel309 commented Mar 15, 2026

ill add more evidence for local testing and benefits once I have the build green. should be there is a bit.

ai reviews addressed as follows
✅ Comment 2: Fixed goroutine loop variable capture
✅ Comment 3: Fixed zero default temperature handling with debug logging
✅ Comment 4: Implemented map-based rate lookup for clarity
✅ Linter: Fixed all gci formatting issues
❌ Comment 1: Not applicable (Go 1.22+ syntax valid)
❌ Overall: Not applicable (no concurrent map access issue)

@daniel309
Copy link
Contributor Author

daniel309 commented Mar 15, 2026

ok, ready for review.

because Base PR: #27780 (Weather Tariff) is open,

only look at these 3 files below. The rest are the temperature tariff code changes that go away once that PR is merged.

image

ill continue local testing and add evidence from the sqllite db, about migration (using my evcc.db file from latest) and most importantly, about the added precision of household load forecasts.

With this PR optimizer forecasts are finally useable when you have a heatpump device registered in evcc.

Before that they were unuseable because:

  1. heater loadpoints got substracted from profile gt, and
  2. no temperature correction applied.

@daniel309 daniel309 marked this pull request as ready for review March 15, 2026 10:28
Copy link
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 2 issues, and left some high level feedback:

  • In applyTemperatureCorrection, the loop for h := range 24 will not compile in Go; it should be replaced with a standard indexed loop such as for h := 0; h < 24; h++ { ... }.
  • The temperature correction currently relies on an exact time.Time match between slotStart.Add(i*SlotDuration) and ratesByTime keys; to avoid missed corrections due to timezone or truncation mismatches, consider deriving future slots directly from the tariff rates or using a nearest-slot lookup instead of exact equality.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- In applyTemperatureCorrection, the loop `for h := range 24` will not compile in Go; it should be replaced with a standard indexed loop such as `for h := 0; h < 24; h++ { ... }`.
- The temperature correction currently relies on an exact `time.Time` match between `slotStart.Add(i*SlotDuration)` and `ratesByTime` keys; to avoid missed corrections due to timezone or truncation mismatches, consider deriving future slots directly from the tariff rates or using a nearest-slot lookup instead of exact equality.

## Individual Comments

### Comment 1
<location path="core/site_optimizer.go" line_range="659-660" />
<code_context>
+			pastTempCount[h]++
+		}
+	}
+	pastTempAvg := make([]float64, 24)
+	for h := range 24 {
+		if pastTempCount[h] > 0 {
+			pastTempAvg[h] = pastTempSum[h] / float64(pastTempCount[h])
</code_context>
<issue_to_address>
**issue (bug_risk):** The `for h := range 24` loop does not compile and should iterate over a slice or a numeric range explicitly.

`range` requires an array, slice, map, string, or channel. Here you probably want either `for h := range pastTempAvg { ... }` or `for h := 0; h < 24; h++ { ... }`. Given the fixed size, iterating over `pastTempAvg` is likely the most idiomatic option.
</issue_to_address>

### Comment 2
<location path="core/site.go" line_range="96-97" />
<code_context>
 	householdEnergy    *meterEnergy
 	householdSlotStart time.Time

+	// per-loadpoint energy tracking for heating devices
+	loadpointEnergy    map[int]*meterEnergy
+	loadpointSlotStart map[int]time.Time
+
</code_context>
<issue_to_address>
**issue (bug_risk):** Concurrent access to `loadpointEnergy` and `loadpointSlotStart` maps is not synchronized and can cause data races.

These maps are written in `updateLoadpointConsumption`, which runs in goroutines spawned by `updateLoadpoints`. Even if goroutines touch different keys, Go maps are not safe for concurrent writes and can panic (`concurrent map writes`) or introduce data races. Please either switch to a slice indexed by loadpoint ID or protect map access with synchronization (e.g., a shared mutex or a per-loadpoint struct with an embedded mutex).
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@daniel309
Copy link
Contributor Author

daniel309 commented Mar 15, 2026

update: db schema changes removed. see PR description. no more additional loadpoint column.

image image

@andig andig changed the title Feature/Optimizer to take heating into account Energy demand profile: to take heating into account Mar 15, 2026
@andig andig added the heating Heating label Mar 15, 2026
@daniel309
Copy link
Contributor Author

daniel309 commented Mar 15, 2026

@andig I removed any db schema change from this PR, so this is now independent of #23185.

see
image

Just tested compatibility with v0.303. works fine without issues.

also, table "meter" remains lean which is a good thing. adding a varchar added significant volume to the file (20 bytes per each row...)

the offset between meter ids and loadpoint ids is now 1000 to give enough space between the two for all practical numbers of meters and loadpoints.

@daniel309 daniel309 force-pushed the feature/temperature-correction branch from a5764b2 to 16203c8 Compare March 15, 2026 20:50
@daniel309
Copy link
Contributor Author

daniel309 commented Mar 16, 2026

evidence of the entire process working and about the impact on optimizer forecast and corrections.

log messages of a working system:

[site  ] DEBUG 2026/03/16 21:57:28 optimizer: optimizing 105 slots until 2026-03-18 00:00:00 +0100 CET: grid=105, feedIn=585, solar=193, first slot: 2m31s
[site  ] DEBUG 2026/03/16 21:57:28 heater profile: querying 1 heating loadpoint(s)
[site  ] DEBUG 2026/03/16 21:57:28 heater profile: loadpoint 0 has 96 slots of data
[site  ] DEBUG 2026/03/16 21:57:28 heater profile: aggregated 1 heating loadpoint(s) into 96 slots
[site  ] DEBUG 2026/03/16 21:57:28 home profile: extracted heater profile with 96 slots
[site  ] DEBUG 2026/03/16 21:57:28 home profile: attempting temperature correction on heater profile
[site  ] DEBUG 2026/03/16 21:57:28 temperature correction: slot 21:45 (hour 20): forecast=4.3°C, hist_avg=6.3°C, delta=2.0°C, load: 200Wh -> 220Wh (9.9%)
[site  ] DEBUG 2026/03/16 21:57:28 temperature correction: slot 22:00 (hour 21): forecast=4.3°C, hist_avg=6.1°C, delta=1.8°C, load: 182Wh -> 198Wh (8.8%)
[site  ] DEBUG 2026/03/16 21:57:28 temperature correction: slot 22:15 (hour 21): forecast=4.3°C, hist_avg=6.1°C, delta=1.8°C, load: 150Wh -> 163Wh (8.8%)

-> see how the heater profile slots got adjusted based on temperature forecast

here is the result in the optimizer. see the wavy pattern of a modulating heatpump, how de-icing was running around 5am and how warm water was produced starting around noon.

image

now here is the same optimizer profile without the temperature adjustments (you see differences in the range of 10% as per log messages above temperature didnt change that much between profile avg and tomorrow)

[site ] DEBUG 2026/03/16 22:06:43 temperature correction: heatingThreshold or heatingCoefficient not configured, skipping correction

image

and finally, here is the exact same optimizer picture from vanilla evcc v0.303. Completely different household load and honestly not very useable/realistic.

image

#######################

@andig relevant code is in these 3 files. the remaining changes are all from the base PR (temperature tariff)

image

Adds optional temperature correction to household load forecasts for the
optimizer, allowing more accurate predictions when heating/cooling loads
vary with outdoor temperature.

Configuration (optional, in site config):
- heatingThreshold: °C 24h avg above which corrections are disabled (default 12°C)
- heatingCoefficient: fractional load change per °C delta (default 0.05 = 5%)

Algorithm:
1. Gates on past 24h avg temperature vs threshold (heating active if below)
2. For each future slot, compares forecast temp to historical avg at same hour
3. Adjusts load: load *= (1 + coeff * (T_historical_avg - T_forecast))
   - Colder forecast → higher load estimate
   - Warmer forecast → lower load estimate

Requires the Temperature tariff (TariffUsageTemperature) to be configured.

Changes household profile lookback from 30 days to 7 days because:
- 7 days follows household rhythms more closely (weekly patterns)
- Temperature changes too much over 30 days, making older data less relevant
- Separate heater load from base household consumption
- Apply temperature correction only to heating devices (heat pumps, electric heaters)
- Base loads (lighting, appliances) remain unchanged
- Backward compatible: falls back to old behavior if no heating devices

Changes:
- Extended metrics DB to track per-loadpoint consumption
- Added loadpoint energy tracking infrastructure in Site
- Implemented profile extraction and aggregation functions
- Modified homeProfile() to separate, correct, and merge profiles

Addresses feedback on PR evcc-io#27780 that temperature adjustment should only
apply to heating devices, not entire household consumption.
The fallback behavior was incorrectly applying temperature correction
to the entire household profile when no heating devices or heater data
were available. This defeated the purpose of heater profile separation.

Temperature correction should ONLY be applied to heating device loads,
never to the entire household consumption. When heating devices or data
are not available, the profile should be returned uncorrected.

This ensures:
- Systems without heating devices: no temperature correction
- Systems with heating devices but no data: no temperature correction
- Systems with heating devices and data: correction only to heater load

Fixes the logic to align with the design intent of separating base
household loads (lighting, appliances, cooking) from temperature-
dependent heating loads.
Added cautious logging to track heater profile extraction and separation:
- DEBUG: Log when heater profile is extracted with slot count
- DEBUG: Log when no heating devices/data, returning uncorrected profile
- DEBUG: Log when applying temperature correction to heater profile only
- WARN: Log when negative base load detected (heater > total consumption)
- DEBUG: Detailed logging in extractHeaterProfile for each loadpoint

This helps diagnose issues with:
- Heating device detection
- Historical data availability
- Profile separation calculations
- Temperature correction application

Follows existing evcc logging patterns using site.log.DEBUG/WARN.
Changed temperature correction to be fully opt-in by removing default
values for heatingThreshold and heatingCoefficient. The correction
algorithm is now only active when BOTH values are explicitly configured.

Before: Used defaults (12.0°C threshold, 0.05 coefficient)
After: Skips correction if either value is 0 (not configured)

This ensures users must consciously enable and tune the feature for
their specific setup, preventing unexpected behavior with default values
that may not suit all installations.

Configuration required:
  heatingThreshold: 12.0    # Must be set explicitly
  heatingCoefficient: 0.05  # Must be set explicitly
Removed PR_DESCRIPTION.md and TEMPERATURE_CORRECTION_LOGIC_FIX.md.
These are temporary development files not needed in the repository.
Both weather data and profiles are on 15-minute slots, so direct
timestamp matching is simpler and more accurate than searching for
the nearest rate.
Replace magic numbers with named constants:
- MeterTypeHousehold = 1
- MeterTypeLoadpoint = 2

Makes the code more self-documenting and maintainable.
- Changed LoadpointNameHousehold from empty string to '_household'
- Added default:'_household' to loadpoint column in GORM schema
- Ensures automatic backfilling when column is added to existing databases
- More explicit and maintainable than empty string identifier
- Fix goroutine loop variable capture for loadpoint
- Skip temperature correction when no historical data for hour
- Add debug logging for missing historical temperature data

Addresses review comments from PR evcc-io#28232
Replace nested loop with map-based lookup for better code clarity.
Shows intent of 'lookup by timestamp' more explicitly.

Addresses Comment 4 from code review.
@daniel309
Copy link
Contributor Author

daniel309 commented Mar 16, 2026

did a rebase to master. if desired, I can do squashes to group those commits better.

@premultiply premultiply added the enhancement New feature or request label Mar 18, 2026
@premultiply
Copy link
Member

Thank you for this interesting PR!

I have a few conceptual comments:

I think it is necessary to distinguish between the actual types of heat generation used for each heating appliance.
Not every ‘heating system’ automatically has temperature-dependent consumption.
For example, a heat pump can be used in at least three different scenarios:

  • Heating the building only (depending on the outside temperature)
  • Providing hot water only (temporarily) (not, or only to a limited extent, dependent on the outside temperature)
  • Doing both together or alternating between them at times

The same applies to other types of heating, such as electric heating elements.

Unfortunately, there are also many other interfering factors in hybrid configurations where, for example, additional heating is provided by other (fossil fuel) heat generators below certain temperatures, or where the output of an existing solar thermal system is also taken into account.

Furthermore, I would suggest simplifying the configuration of the temperature-dependent load profile by using a load curve defined by two temperature points (min/max).
It may also be necessary to include a cut-off temperature at which the power consumption is assumed to be 0. The simple heating curve setting on Panasonic Aquarea heat pumps could serve as a model.

Perhaps, through a careful selection of configuration options, it would be possible to adequately represent at least the most common hybrid systems.

@daniel309
Copy link
Contributor Author

daniel309 commented Mar 18, 2026

Not every ‘heating system’ automatically has temperature-dependent consumption.

and

Unfortunately, there are also many other interfering factors in hybrid configurations where, for example, additional heating is provided by other (fossil fuel) heat generators below certain temperatures, or where the output of an existing solar thermal system is also taken into account.

and

Furthermore, I would suggest simplifying the configuration of the temperature-dependent load profile by using a load curve defined by two temperature points (min/max).
It may also be necessary to include a cut-off temperature at which the power consumption is assumed to be 0. The simple heating curve setting on Panasonic Aquarea heat pumps could serve as a model.

Agreed. I wanted to rework the configuration parts next. My plan is to move configuration up to individual charger, remove from site. I.e. the items

  heatingThreshold: 12.0    # °C — 24h avg above which corrections are disabled
  heatingCoefficient: 0.05  # fraction — load changes by this fraction per °C delta

will go away in favour of one single (optional) config entry for the "heating" charger and if its load should be scaled according to outdoor temperature or not.

Also, heatingThreshold will go away as part of this excercise, the heater itself decides when to heat or not. When its not heating, load is 0 and I can just scale that (effectively doing nothing).

I like the idea of the model being based on a heatcurve. I can calculate the coefficient from the two points given.

But I am also playing with an alternative model that simply looks at the load (Q) and scales it according to

image

Its more accurate it seems and it also doesnt need any external config, which is always preferable :-)

Ill play a bit more on my system and then decide.

A heater decides by itself when to stop heating, and then its consumption
is 0 or close to 0. It doesn't matter if we continue to scale this load,
it will always be 0 when the heater is off.

Changes:
- Removed HeatingThreshold field from Site struct
- Removed compute24hAverageTemperature function (only used for threshold check)
- Updated applyTemperatureCorrection to always apply correction when heatingCoefficient is configured
- Removed heatingThreshold-related tests
- Updated test to remove past 24h temperature data that was only used for threshold check
…ve feature

- Remove heatingCoefficient site parameter
- Add OutdoorTemperatureSensitive feature flag
- Implement physics-based correction: load × ((21°C - T_forecast) / (21°C - T_past_avg))
- Separate temp-sensitive and non-sensitive heater profiles
- Enable by default for Luxtronik template
@daniel309
Copy link
Contributor Author

daniel309 commented Mar 18, 2026

All changes implemented, including adding the OutdoorTemperatureSensitive feature to the luxtronik.yaml template. Ill let other template maintainers add this flag once merged.

I also updated the PR description to describe the updated approach.

Ill continue to test this PR on my system. The new algorithm looks much better.

Example trace with the latest changes

[site  ] DEBUG 2026/03/19 13:59:49 optimizer: optimizing 137 slots until 2026-03-21 00:00:00 +0100 CET: grid=137, feedIn=617, solar=196, first slot: 10s
[site  ] DEBUG 2026/03/19 13:59:49 heater profile: querying 1 heating loadpoint(s)
[site  ] DEBUG 2026/03/19 13:59:49 heater profile: loadpoint 0 has 96 slots of data (temperature-sensitive: true)
[site  ] DEBUG 2026/03/19 13:59:49 heater profile: aggregated 1 temperature-sensitive heating loadpoint(s) into 96 slots
[site  ] DEBUG 2026/03/19 13:59:49 home profile: extracted temperature-sensitive heater profile with 96 slots
[site  ] DEBUG 2026/03/19 13:59:49 home profile: applying temperature correction
[site  ] DEBUG 2026/03/19 13:59:49 temperature correction: slot 13:45 (hour 12): forecast=13.7°C, hist_avg=9.4°C, factor=0.631, load: 125Wh -> 79Wh (-36.9%)
[site  ] DEBUG 2026/03/19 13:59:49 temperature correction: slot 14:00 (hour 13): forecast=13.8°C, hist_avg=9.1°C, factor=0.607, load: 125Wh -> 76Wh (-39.3%)
[site  ] DEBUG 2026/03/19 13:59:49 temperature correction: slot 14:15 (hour 13): forecast=13.9°C, hist_avg=9.1°C, factor=0.599, load: 125Wh -> 75Wh (-40.1%)

cc: @premultiply @andig

@daniel309
Copy link
Contributor Author

daniel309 commented Mar 19, 2026

Why the Change to a Ratio-based Function for Heating Load Correction?

Function 1 is preferred because it directly models heating physics: heat loss is proportional to temperature difference from room temperature (Q ∝ ΔT). When the baseline is 5°C (ΔT = 16°C) and forecast drops to 0°C (ΔT = 21°C), heating demand increases by the ratio 21/16 = 1.31, which is physically accurate. Function 2's linear 5%/°C assumption underestimates this by treating all temperature changes equally, regardless of the absolute temperature difference from room temperature.

This behavior is modelled as follows:
image

which becomes:
load[i] = load_avg[i] × ((T_room − T_forecast[i]) / (T_room − T_past_avg[h]))

Since heating usually only active below ~15°C average temperature, Function 1's instability issues (which only occur when T_past_avg approaches T_room = 21°C) are irrelevant in practice. The current implementation already includes a safety check (math.Abs(denominator) < 0.5) that prevents edge cases, making Function 1 both physically accurate and operationally safe for heating optimization.

image

The graph shows that in realistic heating scenarios (-5°C to 10°C baselines), Function 1 (blue) provides more accurate load predictions that better match the non-linear relationship between temperature and heating demand, while Function 2 (red) consistently underestimates the impact of temperature changes.

Background:

Function 1. load[i] = load_avg[i] × ((T_room − T_forecast[i]) / (T_room − T_past_avg[h]))
Function 2. load[i] = load_avg[i] × (1 + heatingCoefficient × (T_past_avg[h] − T_forecast[i]))

with

t_room = 21 
heatingCoefficient=0.05
T_past_avg[h] = average temperature at hour-of-day h over the past 7 days 
T_forecast[i] = forecast temperature at the wall-clock time of slot i 
heatingCoefficient = fractional load sensitivity per °C

@florian240483
Copy link

Was solar radiation taken into account in the calculation of the heating energy demand? For houses with large window areas, the heating energy demand in winter can depend on solar radiation.

@daniel309
Copy link
Contributor Author

daniel309 commented Mar 20, 2026

Was solar radiation taken into account in the calculation of the heating energy demand? For houses with large window areas, the heating energy demand in winter can depend on solar radiation.

Solar radiation from windows is not taken into account. Its tough to model that. And I would argue it probably doesnt affect the forecast too much.

This situation happens during daylight, which is exactly when PV produces power as well. So it could be that the correction algorithm overestimates heating load for when the sun shines bright, but it will not matter much in practice as at that same time there is usually plenty of excess PV energy anyways and the optimizer deals with excess in those slots already.

In addition, when heating switches off because of solar, this load change is taken into the historical profile (smoothed by the 7-day avg) and will affect future day forecasts.

If you have such a situation, it would be great if you could try this PR and see how it behaves. I am curious.

I can offer a binary with this PR's changes if that makes testing easier for you @florian240483

@daniel309
Copy link
Contributor Author

daniel309 commented Mar 25, 2026

some more data on how impactful heating load corrections based on forecasts are: we have a temperature drop here in my area for the next days. Look at the min/max temp values and how it varies:

image

Today is where temperature drops are very steep, and here are a few samples of the load corrections applied based on how historical load/temperature differs from forecast

Granted, its an extreme example and one of the joys of late March weather I guess.

[site ] DEBUG 2026/03/25 13:44:32 temperature correction: slot 13:30 (hour 12): forecast=7.3°C, hist_avg=11.6°C, factor=1.464, load: 95Wh -> 139Wh (46.4%)
[site ] DEBUG 2026/03/25 13:44:32 temperature correction: slot 13:45 (hour 12): forecast=6.6°C, hist_avg=11.6°C, factor=1.539, load: 114Wh -> 176Wh (53.9%)
[site ] DEBUG 2026/03/25 13:44:32 temperature correction: slot 14:00 (hour 13): forecast=6.1°C, hist_avg=12.4°C, factor=1.736, load: 151Wh -> 262Wh (73.6%)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backlog Things to do later enhancement New feature or request heating Heating

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants