feat(dpp)!: skipIfAbsent at any position, true or an array, on every document type (PV14)

book/src/contract-keywords.md

@@ -282,7 +282,7 @@ Expressions:
 | `timeRange.ttl` | seconds, at most one week | Expires the index's entries after their window; on an index-only type, the rows leave this index. | 14 | [The keys](contract-keywords/time-range.md#the-keys) · [internals](drive/time-range-ttl.md#cleanup) |
 | `terminal` | property name or list | On an index-only type, what keys each entry in place of the document id. | 14 | [terminal](contract-keywords/index-only.md#terminal) |
 | `preallocated` | boolean | On an index-only type, creates the index's trees with the referenced document. | 14 | [preallocated](contract-keywords/index-only.md#preallocated) · [internals](drive/index-only-document-types.md#preallocated-index-paths) |
-| `skipIfAbsent` | boolean | On an index-only type, a document without the first property writes no entry. | 14 | [skipIfAbsent](contract-keywords/index-only.md#skipifabsent) · [internals](drive/index-only-document-types.md#conditional-participation-skipifabsent) |
+| `skipIfAbsent` | `true` or property names | A document missing a property of the skip set writes no entry into the index. | 14 | [skipIfAbsent](contract-keywords/indexes.md#skipifabsent) · [internals](drive/index-only-document-types.md#conditional-participation-skipifabsent) |
 
 ### System properties
 

book/src/contract-keywords/index-only.md

@@ -81,8 +81,8 @@ Rules at registration:
 - Every index holds `$ownerId`, as a property or in its terminal.
 - The only system properties an index may list are `$ownerId` and `$createdAt`, and an indexed `$createdAt` must be in `required`.
 - At least one index involves no `$createdAt` and does not set `skipIfAbsent`: the proof index.
-- Every property is in `required`, except the first property of a `skipIfAbsent` index. An object holding an indexed property is required too.
-- Every property appears in at least one index that does not skip, as a property or a terminal component, except the `entryPayload` properties and a skip index's first property.
+- Every property is in `required`, except a skip property of a `skipIfAbsent` index. An object holding an indexed property is required too.
+- Every required property appears in at least one index that does not skip, as a property or a terminal component, except the `entryPayload` properties. Every optional property appears in a skip index without a `timeRange` whose skip set is that property alone.
 - The type cannot also set [`ttl`](ttl.md) or `moderatorAbilities.delete`, and a `refersTo` lookup cannot target it.
 
 ## `entryPayload`
@@ -163,21 +163,34 @@ A referenced document whose agreed value takes more bytes than the referring pro
 
 | | |
 |---|---|
-| **Where** | index of an `indexOnly` type |
-| **Value** | boolean |
+| **Where** | index |
+| **Value** | `true`, or an array of the index's property names |
 | **Default** | `false` |
 | **Since** | protocol version 14 |
 | **On update** | Fixed (10217) |
 
-A document that leaves out the index's first property writes no entry into this index, and its delete looks for none. The index then holds only the documents that carry the property, and its counts and rankings are "among the documents that have it". It is the one way a property of an index-only type can be optional: in the example, a like without a hashtag is not in `byHashtagPost` and pays nothing for it. A present but empty value is not absent and is indexed.
+The keyword is described in [Indexes](indexes.md#skipifabsent): a document that leaves out a property of the index's skip set writes nothing into the index, and its delete looks for nothing there. It is the one way a property of an index-only type can be optional: in the example, a like without a hashtag is not in `byHashtagPost` and pays nothing for it. A skip property may sit below the first position, which is what lets a windowed ranking skip it:
+
+```json
+{
+  "name": "byDayHashtagPost",
+  "properties": [{ "$createdAt": "asc" }, { "hashtag": "asc" }, { "postId": "asc" }],
+  "terminal": "$ownerId",
+  "rangeCountable": true,
+  "rankedCountable": { "at": ["hashtag", "postId"] },
+  "timeRange": { "on": "$createdAt", "range": 86400, "step": 86400, "ttl": 604800 },
+  "skipIfAbsent": true
+}
+```
 
-A query only uses a skip index when it constrains or orders by the index's first property, so a query cannot silently miss the documents that lack it.
+An untagged like still enters the type's other indexes over the same day window, but writes nothing under `hashtag` in it.
 
-Rules at registration:
+What an index-only type adds to the rules of every type:
 
-- Only on an `indexOnly` type.
-- The first property is a top-level property of the type, not a system property, and not listed in `required`.
-- Every index that involves an optional property sets `skipIfAbsent` and lists that property first; an optional property is never a terminal.
+- An index path has no representation for a missing value, so every index holding an optional property skips on it: the skip set is every optional property of the index, and an array must name them all.
+- Each optional property needs a skip index without a `timeRange` whose skip set is that property alone. A document carrying one optional property but missing another skips every index holding both, and its value would otherwise be written nowhere; a windowed index keeps it only until its windows drain, where document queries do not read it.
+- An optional property is never a terminal.
+- At least one index that involves no `$createdAt` does not skip: the proof index.
 
 ## See also
 

book/src/contract-keywords/indexes.md

@@ -2,7 +2,7 @@
 
 A document type's `indices` list says which queries its documents can answer and which values must be unique. Without an index, a query can only address a document by its `$id`. With one, Drive keeps the documents sorted by the index's properties, so a query that fixes those properties reaches the matching documents directly instead of reading the whole type. Every index costs storage and processing on each write, and an index can never be added, removed or changed once the document type exists, so a contract author decides them before the document type is registered.
 
-This chapter covers the four keywords every index uses: `name`, `properties`, `unique` and `nullSearchable`. The other index keywords, for contests, counts, sums, rankings, time windows and index-only types, have their own chapters (see [More index keywords](#more-index-keywords)).
+This chapter covers the keywords every index can use: `name`, `properties`, `unique`, `nullSearchable` and `skipIfAbsent`. The other index keywords, for contests, counts, sums, rankings, time windows and index-only types, have their own chapters (see [More index keywords](#more-index-keywords)).
 
 ## Example
 
@@ -126,7 +126,64 @@ With the default, a document whose indexed properties are all missing is still e
 
 DPNS sets `"nullSearchable": false` on its `records.identity` index, so domains that point at no identity take no room in it.
 
-`nullSearchable: false` is refused on an index with a ranking or a `timeRange`, and on an index of an index-only type.
+`nullSearchable: false` is refused on an index with a ranking or a `timeRange`, on an index of an index-only type, and together with `skipIfAbsent`.
+
+## `skipIfAbsent`
+
+| | |
+|---|---|
+| **Where** | index |
+| **Value** | `true`, or an array of 1 to 10 of the index's property names |
+| **Default** | `false` |
+| **Since** | protocol version 14 |
+| **On update** | Fixed (10217) |
+
+A document that leaves out a property of the index's **skip set** writes nothing into this index, and its delete looks for nothing there. `true` makes the skip set every optional property of the index; an array names it. The skip property can sit anywhere in the index, under a [`timeRange`](time-range.md) window too. The index then holds only the documents carrying every skip property, and its counts and rankings are "among the documents that have them". A present but empty value is not absent and is indexed.
+
+```json
+"post": {
+  "type": "object",
+  "documentsMutable": true,
+  "indices": [
+    {
+      "name": "byHashtagLanguageTime",
+      "properties": [{ "hashtag": "asc" }, { "language": "asc" }, { "$createdAt": "asc" }],
+      "skipIfAbsent": ["hashtag"]
+    },
+    {
+      "name": "byDayHashtag",
+      "properties": [{ "$createdAt": "asc" }, { "hashtag": "asc" }],
+      "rangeCountable": true,
+      "rankedCountable": true,
+      "timeRange": { "on": "$createdAt", "range": 86400, "step": 86400 },
+      "skipIfAbsent": true
+    }
+  ],
+  "properties": {
+    "hashtag": { "type": "string", "maxLength": 61, "position": 0 },
+    "language": { "type": "string", "maxLength": 8, "position": 1 },
+    "text": { "type": "string", "maxLength": 280, "position": 2 }
+  },
+  "required": ["$createdAt", "text"],
+  "additionalProperties": false
+}
+```
+
+An untagged post writes nothing into either index: no entry under `hashtag`, and no day window in `byDayHashtag`, whose ranking of today's hashtags therefore never has to step over an empty hashtag. A tagged post without a language is entered into `byHashtagLanguageTime` under null for `language`, since the array leaves `language` out of the skip set. A replace that adds or drops the hashtag moves the post into or out of both indexes.
+
+A query only uses a skip index when it constrains every skip property, so it cannot silently miss the documents the index skipped. On a stored type the constraint must be one no missing value can meet: an equality or `in` with values that are neither null nor empty (an empty byte array is keyed like null), a range with a non-empty lower bound, `startsWith` with a non-empty prefix, or ranking by the property. Ordering by the property alone, or a range with only an upper bound, does not count: an index that does not skip would return the documents without the property too (they sort first, under null).
+
+Rules at registration:
+
+- The skip set is not empty: an index whose properties are all required could never skip.
+- Each skip property is a top-level property of the type, not a system property, and not listed in `required`.
+- No `rankedCountable` `at` level sits above the index's deepest skip property: it would count only documents carrying that property, and no query could read it without binding the property.
+- The index is not contested, and does not also set `nullSearchable: false` (the skip already leaves out a document with every indexed value missing).
+- On a stored type, a skip property that is a byte array sets `minItems` to at least 1: an empty byte array is keyed like a missing value.
+- On a stored type, a ranking at a skip property's level does not share that level with an index that keeps null for the property: that index would create the null value, and the ranking would show it as a group with no documents.
+- On an [index-only type](index-only.md#skipifabsent) the skip set holds every optional property of the index, and each optional property needs a skip index of its own, without a `timeRange` (a window keeps the value only until the window drains).
+
+Respelling `true` as the array of the same properties, or the other way round, is no change on a contract update.
 
 ## Null handling
 
@@ -138,6 +195,8 @@ A property is null in an index when the document leaves it out. Putting the rule
 | some missing | yes, under null for the missing ones | no |
 | all missing | only if `nullSearchable` is `true` | no |
 
+An index with `skipIfAbsent` leaves out a document missing any of its skip properties, whatever the other values are.
+
 The storage shape of each case is in [Null Handling](../drive/indexes.md#null-handling).
 
 ## Changing indexes