iBoost: optional demand-forecast-driven smart planning (iboost_forecast)

[psweens]

Summary

Predbat can optionally plan iBoost energy from a 30-minute hot-water demand forecast and the tank's current state of charge, treating the tank as a charge-only store, so heating is scheduled before each draw and stops once the forecast is covered. The feature is off unless iboost_forecast is configured in apps.yaml; existing behaviour is unchanged otherwise, and a regression test enforces that the legacy path produces identical output when no forecast is loaded.

Problem

iboost_max_energy is a per-calendar-day scalar; the smart planner books the cheapest slots without knowing when hot water is drawn or how much energy the tank already holds, so intraday shortfalls are served reactively at uncontrolled rates and full budgets are booked on days the cylinder is already hot (#1424). Users with SoC-reporting tanks rebuild the missing logic as template sensors plus an automation rewriting the budget intraday, which races the replan cycle around midnight.

Configuration

apps.yaml:

iboost_forecast:
  - sensor.hot_water_demand_forecast$results   # same format and parsing as load_forecast; cumulative kWh
iboost_forecast_scaling: 1.0                   # multiplier applied to the forecast, e.g. 0.11 for a series in % of an 11 kWh tank
iboost_tank_soc: sensor.tank_charge            # optional; percent of usable stored energy currently in the tank

Home Assistant entities (created under iboost_enable):

• input_number.predbat_iboost_tank_capacity — usable stored energy, kWh (default 10)
• input_number.predbat_iboost_tank_reserve — minimum stored energy to hold before each draw, kWh (default 0)
• input_number.predbat_iboost_fill_rate_threshold — fill remaining headroom in any slot whose import rate is at or below this value, p/kWh (default −99, disabled). Implemented as a separate commit so it can be dropped in review without touching the rest.

A check on the configuration shape and naming would be appreciated.

Behaviour

When iboost_forecast is configured:

• fetch_iboost_forecast() (modelled on fetch_extra_load_forecast()) reads the configured sensors, scales the cumulative series by iboost_forecast_scaling and converts it into demand per plan interval as the per-minute positive delta (the same reading the load forecast gets), logging a one-line summary. If the sensor is missing, unavailable, stale or yields no data, a warning is logged and the unchanged legacy planner runs for that cycle.
• Initial stored energy is clamp(iboost_tank_soc / 100 × capacity, 0, capacity) when the SoC sensor is configured and readable, else 0 so the whole forecast is provisioned.
• The tank level is maintained at interval resolution as a charge-only store: draws remove energy (clamped at zero), boosts add it (clamped at capacity). Two constraints hold for every interval: the level never falls below the reserve at any draw, and never exceeds the capacity after any boost.
• Selection is earliest-deadline, cheapest-first: the intervals are walked in time order and whenever a draw would breach the reserve, the cheapest eligible earlier interval is booked (respecting iboost_rate_threshold, iboost_rate_threshold_export, the gas thresholds, the element power, the per-calendar-day iboost_max_energy − iboost_today cap and the capacity headroom), preferring intervals adjacent to already-booked ones on price ties so boosts consolidate into longer runs, and repeating until the draw is covered. If no eligible interval remains a warning names the uncovered interval and energy and planning continues.
• The current interval is usually partially elapsed (the plan is rebuilt every few minutes): only its remainder can be booked and its slot is emitted starting at the current minute, mirroring the legacy planner's clamp, so booked energy is always deliverable. Demand already drawn earlier in the current interval is excluded during ingestion since the tank SoC reading reflects it.
• With the fill threshold set, eligible intervals at or below it top the tank up to its capacity headroom, walking in time order so fill energy carries forward; after each fill the level trajectory is re-verified and later planned boosts the fill energy has made redundant are trimmed immediately, so the level stays at or below capacity everywhere and the day cap only ever counts energy that survives. (Within the set of qualifying slots, an earlier fill displaces a later booking regardless of relative price — flagging in case cheapest-first filling is preferred.)
• The forecast planner runs in exactly the modes the smart planner runs in (iBoost enabled, and either both solar and battery modes off or iboost_smart on); when configured it takes over slot selection in those modes, including the time-ordered plan used when iboost_smart is off.
• Slots are emitted in the existing iboost_plan structure (start, end, kwh, average, cost), sorted by time with no duplicate starts, so in_iboost_slot(), the plan card, binary_sensor.predbat_iboost_active and the export-window clash checks are unchanged. average and cost use each slot's own import rate. The level trajectory is logged at debug level and a one-line summary at info level.

The first commit is a separable two-line fix to plan_iboost_smart() this feature depends on: the sliding-window average priced every window from its first interval only, and the duplicate-slot guard tested a stale loop variable so overlapping windows could double-book a slot start. There is no separate PR for it; it is included here as its own commit with its own test.

Scope

Planner only. No change to run_prediction() or the C++ kernel, no parity revision, no binary rebuild; both prediction engines consume the plan through the existing in_iboost_slot() path. Carrying tank state into the prediction loop (so the solar and charging modes respect headroom) is a possible follow-up, deliberately excluded, as is publishing the modelled per-slot tank level (the iBoost plan slots are not currently published as an entity, so there is no attribute to attach it to).

Alternatives considered

External scaffolding: template sensors deriving a remaining-demand estimate, an automation rewriting iboost_max_energy intraday, or setpoint automations on the diverter. A scalar budget cannot express when the energy is needed, so heating still lands in the cheapest slots of the day regardless of draw timing, cannot span a multi-day horizon, and the intraday rewrite races the replan cycle around midnight.

Forecast source

load_forecast sets the precedent for ingesting an external forecast series (Predheat, PredAI). A producer exists: ML Forecast Lab (disclosure: the author is me), an open-source Home Assistant app that benchmarks state-of-the-art ML forecasting backends on the user's own sensor history and publishes calibrated forecasts as HA sensors in the load_forecast attribute format. For Mixergy tanks the demand series comes from the tank's own charge-drop meter, so it represents actual draw rather than heating energy.

Testing

New cases in apps/predbat/tests/test_iboost.py (registered as iboost_forecast, with the bug-fix case under iboost_smart), all on non-flat rate profiles:

• window pricing and duplicate-slot guard fix (fails on the pre-fix planner)
• forecast ingestion: parsing, scaling, negative-delta clamping, multi-sensor summing, and fallbacks for unconfigured/missing/stale sensors, scalar states and data beyond the planning horizon
• energy booked before a draw, not in a cheaper slot after it
• a draw before any cheap slot books the cheapest preceding slot
• a full tank books nothing (Option to end iBoost early and replan, e.g. when hot water cylinder recovered #1424 behaviour)
• reserve held at every draw; capacity never exceeded by a large cheap window; capacity headroom binding across a draw–refill–draw cycle
• calendar-day cap respected alongside the forecast
• import, export and gas threshold eligibility; adjacency consolidation on price ties
• mid-interval replans: the partially elapsed current interval is pro-rated in both the planner and the ingestion
• uncovered demand warns and later draws are still planned; a zero element power books nothing
• fill threshold fills only qualifying slots, the trajectory stays at or below capacity after trimming, and trimmed energy is refunded to the day cap
• regression: with no forecast loaded the legacy planner output is identical before and after a forecast plan has run, and every plan is sorted with positive-length, non-overlapping, duplicate-free slots

Commands run: ./run_pre_commit (all hooks pass) and ./run_all --quick (all tests pass), plus ./run_all --test iboost_smart --test iboost_forecast.

Known limitation: the tariff compare feature raises forecast_minutes to 48h after sensor data is fetched, so with forecast_hours configured below 48 the comparison sees demand only over the shorter horizon; a follow-up could rebuild the forecast grid inside recompute_iboost().

Related

• Option to end iBoost early and replan, e.g. when hot water cylinder recovered #1424 (budget booked on days the cylinder is already hot)
• Time till iboost starts #1692
• The plan_iboost_smart() window-pricing/duplicate-slot fix is included as the first commit of this PR (separable).

[springfall2008]

iboost_smart_min_length silently stops applying once a forecast is configured: the legacy planner enforces a minimum window length (config min 30 / default 30 / max 120 minutes, plan.py:5225) but plan_iboost_forecast never reads it and emits slots as short as 5 minutes (e.g. the PR's own iboost_plan_reserve case books a 20-minute slot). With small forecast draws a user configured for 30-120 minute minimum boosts gets 5-25 minute diverter cycles instead, with no log line or doc noting that min_length stops applying. Either honour min_length by consolidating adjacent booked intervals, or document the changed behaviour in both doc pages.

[psweens]

Taken the documentation option: both doc pages now state that
iboost_smart_min_length does not apply to forecast-driven slots — each slot's length
follows the energy the draw needs at the element power (minimum 5 minutes), with
adjacent-interval consolidation on price ties. I did not implement a hard minimum run
length because padding a run to min_length would book energy the tank model says it does
not need; if you'd rather have min_length enforced as a booking constraint (only book runs
that can reach min_length), I can do that as a follow-up.

[springfall2008]

Cleanup (reuse/simplification): iboost_forecast_day_usage() duplicates the legacy per-calendar-day cap bookkeeping (plan.py:5142-5146: same total_days expression, same day[0]=iboost_today seed) with drift already present: the legacy loop rounds accumulations with dp3(), this helper does not round. Two independent ledgers for the same iboost_max_energy cap mean a fix to one (midnight boundary, dp3 rounding) leaves the other stale, and the rounding drift makes the planners book different totals against an identical budget. Consider one shared day-usage helper for both planners.

[psweens]

Partially unified: both planners now share iboost_plan_total_days() and the forecast
ledger applies the same dp3 accumulation as the legacy ledger, which removes both drifts
named here (total_days expression and rounding). The legacy planner keeps its incremental
ledger inside its own booking loop: recomputing it from the plan list would change the dp3
accumulation order and risk perturbing golden output for no behaviour gain. If you want one
literal helper for both, I'd rather do that in a follow-up with its own golden review.

[springfall2008]

Cleanup (reuse): fetch_iboost_forecast() copies the load_forecast ingestion block of fetch_extra_load_forecast() (fetch.py:2831) nearly line for line (~60 lines): entity_ids unwrap, 569Xlattribute split, get_state_wrapper try/except, dict->array conversion, identical minute_data() kwargs, per-entity warn-and-skip, cross-source summation. Its docstring says 'modelled on fetch_extra_load_forecast()'. A change to shared forecast-format handling must now be made twice; fixing only load_forecast leaves the iBoost path on old behaviour, and a format Predbat learns to read for load stays unsupported for hot-water demand (silently falling back to the legacy planner). Also, the per-minute positive delta below re-implements Fetch.get_from_incrementing(data, index, backwards=False) (fetch.py:752, 'max(data.get(index+1) - data.get(index), 0)'), which fetch.py already uses for exactly this reading on the load forecast - the comment's 'same reading get_from_incrementing gives the load forecast' guarantee is currently enforced only by copy.

[psweens]

Fixed: fetch_cumulative_forecasts() now carries the shared block (entity unwrap,
$attribute split, state fetch with try/except, dict→list conversion and validation, the
minute_data call, per-source warn-and-skip) for both load_forecast and iboost_forecast, and
the per-minute delta is read through get_from_incrementing(..., backwards=False) as you
point out it should be. Two deliberate side effects on the load path: a non-list state is
now skipped with a warning rather than raising, and the fetch-exception log is unified on
"Warn:" (both paths degrade gracefully — load just skips the source). Say the word if you'd
rather keep "Error:" there.

[springfall2008]

Efficiency (worst case untested): the earliest-deadline while-loop recomputes the full O(N) tank trajectory, a fresh O(target) running-max array, an O(N) day-usage pass and an O(target) candidate scan on every dose iteration, and each dose is capped at one interval's element energy. With plan_interval_minutes=5 and a long forecast (~540 intervals) plus a mis-scaled ibost_forecast_scaling that leaves demand nonzero everywhere, this is ~540 targets x many doses x O(540) work = tens of millions of Python steps per plan rebuild, every 5 minutes. All existing tests use 30-min intervals (36 intervals), so the worst case is untested. Cheaper: update levels incrementally per dose and keep the running max as a single rolling value.

[psweens]

Reduced: each dose now does a single fused pass bounded at the target (trajectory and the
running max together) instead of four full-horizon passes, the day ledger is maintained
incrementally per dose, and the fill pass only recomputes state when a fill actually lands.
It is not yet fully incremental across doses — the clamping makes a correct suffix-only
update fiddly, and at the default 30-minute interval the loop is negligible. If 5-minute
interval profiling shows it matters I'll take the incremental-update step as a follow-up.

[springfall2008]

Automated comment from the triage bot.

The comment above claims the region minute_data back-fills beyond the data reads as zero demand, but minute_data tail-fills with newest_state — the value of the earliest sample (utils.py fill loop) — not the last one. So a cumulative series that ends lower than it started (e.g. an internal reset: 6.0 kWh at min 0 … 0.0 reset … 2.0 last) produces a positive delta at the minute right after the last sample: 6.0 − 2.0 = +4.0 kWh phantom demand, which the planner then pre-heats for.

Related, with multiple configured sensors of different horizons: a sensor whose series ends earlier contributes a step drop where its cumulative value vanishes, and the per-minute max(..., 0) clamp on the summed series can swallow a genuine draw from another sensor in that same minute (its delta is absorbed into the negative sum). Computing deltas per sensor before summing, or tail-filling with the last sample for this series, would close both.

[psweens]

Right — the comment was wrong about the back-fill value, and the reset scenario did produce
phantom demand. Fixed by taking per-minute deltas per source and only within that source's
own raw extent: the tail back-fill and the first-sample step are both excluded, and since
deltas are now per source, one source's tail can no longer swallow another's draw in the
summed series. Tests: iboost_fetch_reset (internal reset books no phantom) and the
multi-sensor case.

[springfall2008]

iBoost review: hot-loop cost. Every dose iteration recomputes the full O(N) iboost_tank_trajectory, an O(target) running-max, an O(N) iboost_forecast_day_usage and an O(target) candidate scan; doses are capped at one candidate's element energy per iteration, so worst case is O(N^2 x doses). With plan_interval_minutes=5 and a 45h horizon (~540 intervals) and demand in every interval that is tens of millions of Python steps per plan rebuild, and the plan is rebuilt every 5 minutes - watchdog-restart territory on low-power hosts. All tests run at 30-min intervals (~36 intervals) so this is untested. A booking at index best only changes the trajectory suffix from best onward and one day bucket - keep levels/day_usage incrementally and update them after each boost[best] += amount; the fill pass (line 5367) has the same shape (trajectory + day_usage recomputed even when no fill is applied).

[psweens]

See the inline perf thread: fused single pass per dose bounded at the target, incremental
day ledger, lazy fill-pass recompute; full incrementality deferred with reasoning.