No syntax exception raised in some cases for JDT

[codinuum]

String literals seem to be ignored in some cases.
(tested with build 6584227)

[jrfaller]

Hi! well spotted! Can you attach the test files? Thanks!

[codinuum]

Here they are.
a0.java.txt
a1.java.txt

[jrfaller]

Great!

Ok with a quick gumtree parse a0.java

I get

CompilationUnit [0,114]
    TypeDeclaration [0,113]
        TYPE_DECLARATION_KIND: class [0,5]
        SimpleName: Foo [6,9]
        MethodDeclaration [14,111]
            Modifier: public [14,20]
            Modifier: static [21,27]
            Modifier: final [28,33]
            Modifier: public [49,55]
            Modifier: static [56,62]
            PrimitiveType: void [63,67]
            SimpleName: main [68,72]
            Block [75,111]
                ExpressionStatement [81,107]
                    MethodInvocation [81,106]
                        METHOD_INVOCATION_RECEIVER [81,91]
                            QualifiedName: System.out [81,91]
                        SimpleName: println [92,99]
                        METHOD_INVOCATION_ARGUMENTS [100,105]
                            StringLiteral: "bar" [100,105]

The field declaration is strangely completely ignored... This is strange. I have to look in the jdt generator code.

[jrfaller]

OK by opening the file I realize that there is no return type for the attribute which is a syntax error. If you add the type String to both files it works. I guess the big question is why there is no SyntaxError raised by JDT in this case ;-)

[codinuum]

Confirmed. Thanks!
How about comparing another pair of files with python-treesitter?
a0.py.txt
a1.py.txt
String literals are missing in the parse trees...

[jrfaller]

Working on my side, no?

module [0,113]
    expression_statement [0,9]
        assignment [0,9]
            identifier: s [0,1]
            =: = [2,3]
            string: 'bar' [4,9]
    function_definition [12,112]
        identifier: match [16,21]
        parameters [21,30]
            identifier: command [22,29]
        block [36,112]
            return_statement [36,112]
                parenthesized_expression [43,112]
                    boolean_operator [44,111]
                        comparison_operator [44,74]
                            string: 'push' [44,50]
                            attribute [54,74]
                                identifier: command [54,61]
                                identifier: script_parts [62,74]
                        and: and [75,78]
                        comparison_operator [79,111]
                            string: 'set-upstream' [79,93]
                            attribute [97,111]
                                identifier: command [97,104]
                                identifier: output [105,111]⏎

[codinuum]

Parsing results in the following in my environment...

module [0,113]
    expression_statement [0,9]
        assignment [0,9]
            identifier: s [0,1]
            =: = [2,3]
            string [4,9]
                ": ' [4,5]
                ": ' [8,9]
    function_definition [12,112]
        def: def [12,15]
        identifier: match [16,21]
        parameters [21,30]
            (: ( [21,22]
            identifier: command [22,29]
            ): ) [29,30]
        :: : [30,31]
        block [36,112]
            return_statement [36,112]
                return: return [36,42]
                parenthesized_expression [43,112]
                    (: ( [43,44]
                    boolean_operator [44,111]
                        comparison_operator [44,74]
                            string [44,50]
                                ": ' [44,45]
                                ": ' [49,50]
                            in: in [51,53]
                            attribute [54,74]
                                identifier: command [54,61]
                                .: . [61,62]
                                identifier: script_parts [62,74]
                        and: and [75,78]
                        comparison_operator [79,111]
                            string [79,93]
                                ": " [79,80]
                                ": " [92,93]
                            in: in [94,96]
                            attribute [97,111]
                                identifier: command [97,104]
                                .: . [104,105]
                                identifier: output [105,111]
                    ): ) [111,112]

[tsantalis]

@jrfaller
I can confirm that JDT Parser skips parts of the AST that are syntactically invalid.
Missing type in a field declaration makes it syntactically invalid.
JDT Parser will continue parsing the rest of the file.

[jrfaller]

@tsantalis yes just what I realized. On top of that, neither the flag MALFORMED nor the flag RECOVERED seems to work in these cases. The only thing I get is that I have one "problem" in the problem list when casting the node to a CompilationUnit. I guess it could work better to detect syntax errors but I fear that it will report more than just syntax errors 😢 WDYT?

[codinuum]

@jrfaller Updating tree-sitter-parser resolved the problem.
Thanks a lot!

[tsantalis]

@tsantalis yes just what I realized. On top of that, neither the flag MALFORMED nor the flag RECOVERED seems to work in these cases. The only thing I get is that I have one "problem" in the problem list when casting the node to a CompilationUnit. I guess it could work better to detect syntax errors but I fear that it will report more than just syntax errors 😢 WDYT?

For commit analysis, which cannot be guaranteed to be syntactically valid and compilable, I think what JDT Parser is doing is fine. Instead of failing the whole file, it just skips the syntactically invalid sub-trees.
There should be a way to isolate syntax errors from the problem list. I will look into it.