Repository navigation
dealing with rows #202
Description
The current major blocker for getting rid of TableStatisticalModel wrappers (#32) and the ModelFrame/ModelMatrix structs is that they also keep a mask for the rows that are actually included from the underlying table. It's necessary to keep these around if (possibly among other things) you want to predict back into the original table, since you need to know where the missing values were. So just keeping the FormulaTerm around isn't enough to have a completely stand-alone table-to-matrix transformation which can replace ModelFrame.
A related issue is that missing values in the output are now also introduced by terms themselves, because of the lead/lag support.
So my current thinking is that we ought to give up on trying to never generate missing values in the output (consistent with #153) and set rows to missing where any missing value is encountered in any column of the table, and provide some kind of functionality to compute the missings mask if necessary so that consumers can decide what to do. That leads to a situation where the consumer has to do a bit more fiddly book-keeping and it also means that the terms are not fully stand-alone but I'm not sure I see an alternative at this point (except for some kind of extremely lazy architecture where the terms actually hold views of the underlying table or something like that, I think @nalimilan were talking about this at ZiF...)
Activity
palday commentedon Dec 18, 2020
This would actually fix a number of current issues with handling missings, especially when expanding categorical variables into contrasts.
kleinschmidt commentedon Jan 20, 2021
Just as a way of organizing thoughts about how to approach this. There are (at
least) two approaches to dealing with rows.
modlecolsgenerates each column independently, possibly generating missingswhen they're encountered in the underlying table. The resulting Arrays will
have a "swiss cheese" structure where there are "holes" (missings)
throughout, but other values in the same row might still be present. This is
maybe more straightforward since each term is handled independently, but it
could mess up stuff like schema extraction/contrast coding (if, e.g., there's
a categorical variable with a level that is only present in rows where
another variable is missing and hence may never appear in the generated data;
IIRC we have tests for exactly this case...).
modelcolsattempts to avoid rows in the data where there is at least onemissing, setting the entire row to missing or omitting it altogether. Thisavoids the problem of possibly pathological schemas but it also closes the
door on e.g. creating tools to do missing value imputation on the output. It
might also mess with uses of
lead/lagwhere you want to lead/lag based onthe original rows, not the complete rows.
So it might be necessary to be able to switch between these modes somehow; maybe
adding a
completecasesflag on theschemaandmodelcolsfunctions? Or atthe very least provide a boolean mask of complete cases (or a convenient way to
create one).