Indentation-sensitive grammar forgets consumed dedents

[simonwo]

Hi, I'm using Ohm v17.1.0 to build a whitespace-sensitive grammar using the experimental indentation support. I am struggling to get the following MVE grammar to correctly parse the example:

Blocks <: IndentationSensitive {
    Object = Block<ObjectHead, ObjectBody>
    ObjectHead = "key:"
    ObjectBody = ObjectHead "value"
        
    Block<Head, Body> = Head newline indent BlockBody<Head, Body> dedent
    BlockBody<Head, Body> = Block<Head, Body> BlockBody<Head, Body> -- block
                          | Body newline BlockBody<Head, Body>  -- stmt                          
                          | space* -- zero

    newline = "\n" "\r"?    
    space := " " | "\t"
}

And the example:

key:
    key:
        key: value
        key: value
    key: value

The parse fails at line 5 col 5 with Expected a dedent, a space, or "key:", which is weird because it should be able to consume a dedent, or key: if it has just consumed one. After banging my head against the desk for a while, I've stepped through the parsing code and whilst it does consume the dedent, when it returns to the same position to continue the parse it seems to still be expecting one. AFAICS the result of attempting to parse at this position is memoized when it should not be, because the memoized version does not take into account the consumed dedent.

Specifically, if I comment out this line which resets memoization for a match state that has it enabled

ohm/packages/ohm-js/src/pexprs-eval.js

Lines 277 to 279 in 39ccead

	if (state.doNotMemoize) {
	state.doNotMemoize = false;
	} else if (isHeadOfLeftRecursion) {

...the parse works as expected.

However, I'm not really sure why this line is here so not sure if I've found a bug or have just two-wrongs-make-a-righted myself.

I'd be happy to work on and submit a patch for this, but I could do with a positive confirmation that this is a problem, and pointers to any other areas that I might need to change as result. No doubt it's all more complicated than I understand at the moment! Thanks!

[simonwo]

FYI this patch solves the issue for me:
01-keep-memoization-disabled.patch

And this one provides some more debugging information:
02-output-indents-in-trace.patch

[pdubroy]

Thank you both, @simonwo and @ianb. Last year was busy for me and I wasn't able to stay on top of the bug reports and PRs.

This one needs a closer look, which I should be able to do in the coming days. Thanks for taking a crack at this, it's definitely not a trivial fix!

Also I would be curious to hear if you were able to successfully build an indentation-senstive language. The feature was fairly experimental and unfortunately I wasn't able to find the time to continue to focus on it.

[simonwo]

Sounds great, thanks @pdubroy!

I was able to build an indentation-sensitive language, yes, here: https://github.com/simonwo/bigdeal/blob/main/lang/src/bigdeal.ohm

With a similar patch to address ignoring blank lines, it can parse things that look like this: https://github.com/simonwo/bigdeal/blob/main/examples/tic-tac-toe.bigdeal.

Let me know if you'd like me to test any fixes with my own code!

[pdubroy]

@simonwo thanks for the details! I'm glad to hear this worked for you.

And that is a very cool use case. Have you written this up anywhere? I like to share fun uses of Ohm and this is really interesting!