feat(v1): let a join opt out of filling with fill_value=ABSENT - #894
Merged
Merged
Conversation
A reindexing join filled every position it created — the linopy operand with the zero expression, the constant operand with fill_value=. Add linopy.ABSENT to keep those positions absent instead, on constant and expression operands alike (linopy.merge included), so absence stays distinguishable from a genuine zero (#712). A NaN fill now raises (§5).
Collaborator
Author
|
@FBumann I remember we had a talk about an absent operator in linopy. For the case above it is needed and makes sense, however it remains a niche case and the ABSENT type would be hardly used by users. do you agree with the approach? |
Collaborator
|
@FabianHofmann Yes i do. Im not sure if its really needed for real world usage, but I have no objection. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #890.
Note
The following content was generated by AI.
Changes proposed in this Pull Request
A reindexing join (
outer,left,right) fills every position it creates:the linopy operand contributes the zero expression there, the constant operand
contributes
fill_value=(§10). There was no way to say "create the labels,but leave them absent" — the zero-fill was unconditional for linopy operands,
which made it the one remaining place where linopy picks a fill on the caller's
behalf.
That is a silent modelling change where it matters. Two variables on
overlapping node sets, joined and constrained:
This adds
linopy.ABSENTas the spelling for the other intent:Union coordinates, terms only where both sides are defined, and the constraint
drops at
aandcunder §12 — identical to thereindex-both-sidesworkaround, which the tests assert with
assert_linequal.Semantics
Every position the join creates comes out absent, whichever side was missing
there. The operand missing at a created label contributes absence rather than
the operator's identity, and §6 absorption does the rest — so one rule covers
both sides and both kinds of operand, and no new storage state is introduced.
Unlike a numeric fill,
ABSENTis accepted for expression operands too, sinceabsence is something both kinds of operand can carry. It works on
.add/.sub/.mul/.div(and theirVariablecounterparts) and onlinopy.merge(..., join=…, fill_value=ABSENT). A numeric fill on anexpression operand still raises, with a message that now points at
ABSENT.It requires an explicit
join=(without one there are no created positions),and it requires v1: under legacy it raises, because legacy fills absent slots
with the operator's identity, so there is nothing for the sentinel to mean
there. Legacy behaviour is untouched throughout.
Absence an operand carries in — from
mask=,.where(),.shift(),.reindex()— is unaffected, as before. Only the positions the join createsare involved.
Alignment with #712
The created slots are stored exactly like carried-in absence (
constNaN,coeffsNaN,vars-1), so nothing downstream needs to know where the absencecame from:
isnull()gen.add(imp, join="outer", fill_value=ABSENT)[True, False, True]gen.reindex(node=union) + imp.reindex(node=union)[True, False, True]result * 3(absence survives arithmetic)[True, False, True]gen.add(imp, join="outer")(zero-filled)[False, False, False]ABSENTadds a third origin of absence — join-created, alongsideoperation-induced and mask-induced — without adding a third representation, and
it leaves the absent-vs-zero choice with the caller.
Fix found along the way
fill_value=np.nanwas silently accepted and produced a broken expression:constNaN butcoeffsstill1.0with a live variable — the §1/§2 storageinvariant violated, and exactly the #712 failure mode of an absent-looking slot
that still carries a term. It now raises under v1, pointing at the sentinel:
Legacy is unaffected — it fills with the operator's identity after the align,
so the broken state cannot arise there.
Implementation notes
AbsentType/ABSENTlive inlinopy/semantics.pynext to the otherconvention helpers, exported as
linopy.ABSENT.__neg__returns thesentinel itself so
sub's-fill_valueworks, andjoin_fill(fill_value, default)maps it to NaN at each fill site._align_constantfills both sides with NaN underABSENT— the per-sidesplit from fix(v1): fill join-created positions per side; add fill_value= to add/sub/mul/div #887 is what makes this a two-line change rather than a special
case.
mergeuses a NaNconstfill in the_termconcat, so the existingabsorb_absencepass strips the terms the surviving operand held._absorb_join_absence, guarded on thesentinel, so the hot path is untouched.
doc/design/convention.rst, the migration-guide table,doc/api.rst, and a release note.TestOuterJoinFillclass: parametrised overadd/sub/mul/divfor constant and expression operands, theconstraint-row drop, equivalence with the reindex workaround,
isnull()through further arithmetic, the merge path, the NaN rejection, and the
legacy raise.
Full suite: 7992 passed, 648 skipped, under both semantics.
ruffclean;mypyclean apart from a pre-existing unused-ignore inlinopy/solvers.py.Checklist
AGENTS.md).doc.doc/release_notes.rstof the upcoming release is included.