Skip to content

feat(v1): let a join opt out of filling with fill_value=ABSENT - #894

Merged
FabianHofmann merged 1 commit into
feat/arithmetic-conventionfrom
feat/join-fill-absent
Aug 18, 2026
Merged

feat(v1): let a join opt out of filling with fill_value=ABSENT#894
FabianHofmann merged 1 commit into
feat/arithmetic-conventionfrom
feat/join-fill-absent

Conversation

@FabianHofmann

Copy link
Copy Markdown
Collaborator

Closes #890.

Note

The following content was generated by AI.

Changes proposed in this Pull Request

A reindexing join (outer, left, right) fills every position it creates:
the linopy operand contributes the zero expression there, the constant operand
contributes fill_value= (§10). There was no way to say "create the labels,
but leave them absent"
— the zero-fill was unconditional for linopy operands,
which made it the one remaining place where linopy picks a fill on the caller's
behalf.

That is a silent modelling change where it matters. Two variables on
overlapping node sets, joined and constrained:

gen.add(imp, join="outer") >= 10
[a]: +1 gen[a]            ≥ 10.0   # the requirement now falls on gen alone
[b]: +1 gen[b] + 1 imp[b] ≥ 10.0
[c]: +1 imp[c]            ≥ 10.0   # …and on imp alone

This adds linopy.ABSENT as the spelling for the other intent:

gen.add(imp, join="outer", fill_value=linopy.ABSENT)
[a]: None
[b]: +1 gen[b] + 1 imp[b]
[c]: None

Union coordinates, terms only where both sides are defined, and the constraint
drops at a and c under §12 — identical to the reindex-both-sides
workaround, which the tests assert with assert_linequal.

Semantics

Every position the join creates comes out absent, whichever side was missing
there. The operand missing at a created label contributes absence rather than
the operator's identity, and §6 absorption does the rest — so one rule covers
both sides and both kinds of operand, and no new storage state is introduced.

Unlike a numeric fill, ABSENT is accepted for expression operands too, since
absence is something both kinds of operand can carry. It works on .add /
.sub / .mul / .div (and their Variable counterparts) and on
linopy.merge(..., join=…, fill_value=ABSENT). A numeric fill on an
expression operand still raises, with a message that now points at ABSENT.

It requires an explicit join= (without one there are no created positions),
and it requires v1: under legacy it raises, because legacy fills absent slots
with the operator's identity, so there is nothing for the sentinel to mean
there. Legacy behaviour is untouched throughout.

Absence an operand carries in — from mask=, .where(), .shift(),
.reindex() — is unaffected, as before. Only the positions the join creates
are involved.

Alignment with #712

The created slots are stored exactly like carried-in absence (const NaN,
coeffs NaN, vars -1), so nothing downstream needs to know where the absence
came from:

expression isnull()
gen.add(imp, join="outer", fill_value=ABSENT) [True, False, True]
gen.reindex(node=union) + imp.reindex(node=union) [True, False, True]
result * 3 (absence survives arithmetic) [True, False, True]
gen.add(imp, join="outer") (zero-filled) [False, False, False]

ABSENT adds a third origin of absence — join-created, alongside
operation-induced and mask-induced — without adding a third representation, and
it leaves the absent-vs-zero choice with the caller.

Fix found along the way

fill_value=np.nan was silently accepted and produced a broken expression:
const NaN but coeffs still 1.0 with a live variable — the §1/§2 storage
invariant violated, and exactly the #712 failure mode of an absent-looking slot
that still carries a term. It now raises under v1, pointing at the sentinel:

fill_value=NaN is ambiguous: pass linopy.ABSENT to leave the positions the join creates absent, or a number to fill them.

Legacy is unaffected — it fills with the operator's identity after the align,
so the broken state cannot arise there.

Implementation notes
  • AbsentType / ABSENT live in linopy/semantics.py next to the other
    convention helpers, exported as linopy.ABSENT. __neg__ returns the
    sentinel itself so sub's -fill_value works, and join_fill(fill_value, default) maps it to NaN at each fill site.
  • _align_constant fills both sides with NaN under ABSENT — the per-side
    split from fix(v1): fill join-created positions per side; add fill_value= to add/sub/mul/div #887 is what makes this a two-line change rather than a special
    case.
  • merge uses a NaN const fill in the _term concat, so the existing
    absorb_absence pass strips the terms the surviving operand held.
  • The two v1 constant paths call _absorb_join_absence, guarded on the
    sentinel, so the hot path is untouched.
  • Docs: §10 of doc/design/convention.rst, the migration-guide table,
    doc/api.rst, and a release note.
  • Tests extend the v1-marked TestOuterJoinFill class: parametrised over
    add / sub / mul / div for constant and expression operands, the
    constraint-row drop, equivalence with the reindex workaround, isnull()
    through further arithmetic, the merge path, the NaN rejection, and the
    legacy raise.

Full suite: 7992 passed, 648 skipped, under both semantics. ruff clean;
mypy clean apart from a pre-existing unused-ignore in linopy/solvers.py.

Checklist

  • AI-generated content is marked (see AGENTS.md).
  • Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
  • Unit tests for new features were added (if applicable).
  • A note for the release notes doc/release_notes.rst of the upcoming release is included.
  • I consent to the release of this PR's code under the MIT license.

A reindexing join filled every position it created — the linopy operand
with the zero expression, the constant operand with fill_value=. Add
linopy.ABSENT to keep those positions absent instead, on constant and
expression operands alike (linopy.merge included), so absence stays
distinguishable from a genuine zero (#712). A NaN fill now raises (§5).
@FabianHofmann

Copy link
Copy Markdown
Collaborator Author

@FBumann I remember we had a talk about an absent operator in linopy. For the case above it is needed and makes sense, however it remains a niche case and the ABSENT type would be hardly used by users. do you agree with the approach?

@FBumann

FBumann commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator

@FabianHofmann Yes i do. Im not sure if its really needed for real world usage, but I have no objection.

@FabianHofmann
FabianHofmann merged commit 933fa7b into feat/arithmetic-convention Aug 18, 2026
2 checks passed
@FabianHofmann
FabianHofmann deleted the feat/join-fill-absent branch August 18, 2026 07:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants