Incremental reparse drops trailer value that continues onto an indented line #96

Description

After certain edits, the tree from an incremental reparse differs from a fresh parse of the same text. Every case I found involves a trailer or BREAKING CHANGE: value that can continue onto a following line starting with a space. A fresh parse makes the continuation part of the value; the incremental parse leaves the trailer without a value and reads the lines as separate message_lines, or reads a # in the value as a comment. For an editor, the highlighting after typing then differs from the highlighting after reopening the file.

It needs at least two edits: the same final edit on a freshly parsed tree gives the correct result.

Reproduction

Self-contained C program (Tree-sitter 0.27.0 runtime; tree-sitter-gitcommit v0.5.0, and also current main at 55a265c):

#include <stdio.h>
#include <string.h>
#include <tree_sitter/api.h>

const TSLanguage *tree_sitter_gitcommit(void);

int main(void)
{
    TSParser *parser = ts_parser_new();
    ts_parser_set_language(parser, tree_sitter_gitcommit());

    /* 1. Parse a message whose last line is a single space */
    const char *v1 = "Subject\n\nBody.\nSigned-off-by: A\n ";
    TSTree *tree = ts_parser_parse_string(parser, NULL, v1, strlen(v1));

    /* 2. Insert "\n" after "Body." (byte 14, row 2, column 5) */
    const char *v2 = "Subject\n\nBody.\n\nSigned-off-by: A\n ";
    ts_tree_edit(tree, &(TSInputEdit){14, 14, 15, {2, 5}, {2, 5}, {3, 0}});
    TSTree *t2 = ts_parser_parse_string(parser, tree, v2, strlen(v2));
    ts_tree_delete(tree);

    /* 3. Append "B" to the last line (byte 34, row 5, column 1) */
    const char *v3 = "Subject\n\nBody.\n\nSigned-off-by: A\n B";
    ts_tree_edit(t2, &(TSInputEdit){34, 34, 35, {5, 1}, {5, 1}, {5, 2}});
    TSTree *t3 = ts_parser_parse_string(parser, t2, v3, strlen(v3));

    TSTree *fresh = ts_parser_parse_string(parser, NULL, v3, strlen(v3));
    printf("incremental: %s\n", ts_node_string(ts_tree_root_node(t3)));
    printf("fresh:       %s\n", ts_node_string(ts_tree_root_node(fresh)));
    return 0;
}

Build (from a tree-sitter checkout and this repository):

cc -std=c11 -I tree-sitter/lib/include -I tree-sitter/lib/src -I tree-sitter-gitcommit/src \
   repro.c tree-sitter/lib/src/lib.c tree-sitter-gitcommit/src/parser.c tree-sitter-gitcommit/src/scanner.c -o repro

Actual

incremental: (source (subject) (message (message_line) (trailer (token)) (message_line) (message_line)))
fresh:       (source (subject) (message (message_line) (trailer (token) (value))))

Expected

Both trees are the same: the trailer has a value (A\n B), as in the fresh parse.

Notes

• Random edit sequences find this about once per 5,000 sequences of ~30 single-character or short-word edits, always around a value that continues onto a line starting with a space (TRAILER_VALUE, BREAKING_CHANGE_VALUE: [^\n\r]+(\r?\n [^\n\r]+)*).
• Since a single edit on a fresh tree does not reproduce it, the cause might be in how Tree-sitter decides whether a previously lexed value token can be reused (its lookahead past the newline), rather than in the grammar itself. I haven't pinned it down; happy to move this to tree-sitter/tree-sitter if you think it belongs there.
• tree-sitter fuzz on the corpus may find more cases.