Syntax tree, scopes and parser behavior

Normative description of what JsParser.Parse produces. A port must reproduce the tree shape, the scope information and the error behavior described here.

Entry point

JsParser.Parse(source, options, cancellationToken) returns ParseResult { Program, Source, Diagnostics, HasErrors }. It never throws for malformed input; every problem is a Diagnostic. ParseOptions:

Positions

Every node has Start and End, UTF-16 offsets into the source. SourceText.GetPosition(offset) gives the 1-based line and column; line terminators are LF, CR, CR LF, U+2028 and U+2029 (ECMA-262 12.3). The --> line of a report is the start of the primary span.

Tree shape

• Node classes live in RecursiveAI.OneEcma.Syntax (Ast.cs is generated by tools/gen-ast.py from a table in that script; edit the table, not the output). Every node has Start, End and Kind. Expressions add IsParenthesized (there is no parenthesized-expression node); statements and declarations derive from Statement.
• One FunctionNode covers declarations, expressions, arrows, methods, getters/setters and generators/async variants, with flags (IsArrow, IsAsync, IsGenerator, IsMethod, ...), Id, Params, Body, Scope, and UsesArguments, UsesThis, HasDirectEval. One ClassNode covers class declarations and expressions.
• Binding and assignment targets are separate pattern nodes (object, array, assignment, rest patterns), produced by reinterpreting the cover grammar (a = [x, y], arrow parameters) with the early errors of ECMA-262 13.15.5 and 15.2.1.
• Optional chains are a ChainExpression wrapper around member and call nodes (ECMA-262 13.3.9).
• Module declarations (import, export) have their own nodes; import(...) is ImportCall, import.meta and new.target are MetaProperty.
• Error recovery inserts ErrorExpression (empty span at the error) where an operand is missing; a tree with errors is for diagnostics, not for execution.
• Operator chains are flat in source but nested in the tree, so a+b+c+... with 100k operands gives a tree 100k deep. Consumers must walk trees iteratively or check a depth counter.

Scopes

Scope analysis runs inside the parser (early errors need it) and leaves Scope objects on the tree: Scope { Kind, Parent, Children, Bindings, ContainsDirectEval, ContainsWith, IsStrict, PrivateNames } and Binding { Name, Kind, Span, FunctionDeclaration }. Kinds: global, module, eval, function, block, catch, for, switch, class, class field initializer and class static block. Redeclaration, let/var clashes, duplicate parameters, labels, private names, break/continue targets, yield/await/arguments/super/new.target context and strict-mode restrictions are reported as early errors in V8's wording (syntax-messages.md).

Annex B: a function declaration in a block (sloppy mode) is hoisted to a var of the same name unless that would conflict (B.3.3). The declaration carries IsAnnexBHoisted and the function scope gets an AnnexBFunctionVar binding. Also accepted: HTML-like comments (B.1.1), legacy octal literals and escapes (sloppy only), __proto__ duplicates rules, for (var x = 1 in y) (sloppy), labelled function declarations (sloppy), if function bodies.

Error recovery and reporting

• The parser reports every syntax error it finds. After an error it inserts the missing token or skips the stuck token (recorded in Diagnostic.Recovery) and resynchronizes at statement boundaries and at the closing bracket.
• Resynchronization: an error inside a parameter list skips to the next , or the closing ) on bracket structure; a broken if/while/for/with head skips to its ) on the same line. The body is then parsed as a body instead of as fallout of the head.
• Cooldown: a new error within 3 tokens of a syntax error, or within 12 tokens on the same line, is dropped as fallout; an end-of-input error is exempt from the same-line window (an unclosed bracket is always worth saying). An early error (not a grammar error) within two lines after a syntax error is dropped as well. After a lexical error (unterminated string, typographic quote) errors on the same and next line are dropped. Deviation: two real mistakes closer than that are reported as one.
• Hints: after parsing, SyntaxHints recognizes common mistakes (Python elif/def/and/or/for ... in range, TypeScript annotations, as casts and interface, HTML entities, typographic quotes, stray closers, a bracket still open at the end of the input) and may move the primary span to the construct that is wrong and add a label and help. The headline is never changed. A hint never changes the source.
• A regular expression literal's flags are validated by the lexer; the pattern body goes to RegExpValidator.

Limits and hostile input

• Recursion is bounded by MaxDepth plus RuntimeHelpers.TryEnsureSufficientExecutionStack; hitting either reports the depth-limit diagnostic ("Maximum nesting depth exceeded") and ends the parse. Tested with 100,000 nested (, [, {, functions, arrows, unary and ternary chains on a 256 KB thread stack.
• Cancellation is polled every 256 tokens.
• Work per error is bounded by a constant window of source text, so a long file full of errors stays linear.

Documented deviations from Node

• Sloppy-mode call expressions are accepted as assignment, update and for-in/of targets (web compatibility, Annex B test262 tests); they are a runtime ReferenceError.
• Errors closer together than the cooldown are merged (above).
• Where Node reports one error, the parser may report more after it; the first error's headline and line are always Node's (SyntaxMessageTests).