antlr4 - 用于简单命令处理器的Lexer和Parser规则(Lexer and Parser rules for a simple command processor)

Question

I am attempting to build a simple command processor for a legacy language.

(我正在尝试为传统语言构建简单的命令处理器。)

I am attempting to work with C# with antlr4 version "ANTLR", "4.6.6")

(我正在尝试使用具有antlr4版本“ ANTLR”，“ 4.6.6”的C＃）)

I am unable to make progress against one scenario, of several.

(我无法在几种情况下取??得进展。)

The following examples shows various sample invocations of the command PKS.

(以下示例显示了命令PKS的各种示例调用。)

PKS

PKS?

PKStext_that_is_a_filename

The scenario that I can not solve is the PKS command followed by filename.

(我无法解决的情况是PKS命令后跟文件名。)

Command:
PKS
(block (line (expr (command PKS)) (eol 
)) <EOF>)
Command:
PKS?
(block (line (expr (command PKS) (query ?)) (eol 
)) <EOF>)
Command:
PKSFILENAME
line 1:0 mismatched input 'PKSFILENAME' expecting COMMAND
(block PKSFILENAME 
)
Command:

what I believe to be the relevant snippet of grammar:

(我认为这是相关的语法片段：)

 block      : line+ EOF;
 line       : (expr eol)+;

 expr       : command file
            | command listOfDouble
            | command query
            | command
            ;

command     : COMMAND
            ;

query       : QUERY;
file        : TEXT ;
eol         : EOL;
listOfDouble: DOUBLE (COMMA DOUBLE)* ;

From the lexer:

(从词法分析器：)

COMMAND     : PKS;
PKS         :'PKS' ;

QUERY       : '?'
            ;

fragment LETTER : [A-Z];
fragment DIGIT  : [0-9];
fragment UNDER  : [_];

TEXT        : (LETTER) (LETTER|DIGIT|UNDER)* ;

  ask by Gregg Swanson translate from so

Answer 1

The main problem here is that your TEXT rule also matches what PKS is supposed to match.

(这里的主要问题是您的TEXT规则也与应该匹配的PKS相匹配。)

And since PKStext_that_is_a_filename can entirely be matched by that TEXT rule it is preferred over the PKS rule, even though it appears first in the grammar (if 2 rules match the same input then the first one wins).

(而且由于PKStext_that_is_a_filename可以完全被该TEXT规则匹配，因此它比PKS规则更可取，即使它首先出现在语法中（如果2个规则匹配相同的输入，则第一个规则获胜）。)

In order to fix that problem you have 2 options:

(为了解决该问题，您有2个选择：)

1. Require whitespace(s) between the keyword (PKS) and the rest of the expression.

   (在关键字（PKS）和表达式的其余部分之间需要空格。)

2. Change the TEXT rule to explicitly exclude "PKS" as valid input.

   (更改TEXT规则以明确排除“ PKS”作为有效输入。)

Option 2 is certainly possible, but will get very messy if you have have more keywords (as they all would have to be excluded).

(选项2当然是可能的，但是如果您有更多的关键字（因为它们都必须被排除），它将变得非常混乱。)

With a whitespace between the keywords and the text the lexer would automatically do that for you.

(在关键字和文本之间使用空格，词法分析器将自动为您完成此操作。)

And let me give you a hint to approach such kind of problems: always check the token list produced by the lexer to see if it generated the tokens you expected.

(并且让我给您提示解决此类问题的方法：始终检查词法分析器生成的令牌列表，以查看其是否生成了您期望的令牌。)

I reworked your grammar a bit, added missing tokens and ran it through my ANTLR4 debugger, which gave me:

(我对您的语法进行了一些重做，添加了缺少的标记，并通过我的ANTLR4调试器运行了它，这给了我：)

Parser error (5, 1): extraneous input 'PKStext_that_is_a_filename' expecting {<EOF>, COMMAND, EOL}

Tokens:
[@0,0:2='PKS',<1>,1:0]
[@1,3:3='
',<8>,1:3]
[@2,4:4='
',<8>,2:0]
[@3,5:7='PKS',<1>,3:0]
[@4,8:8='?',<3>,3:3]
[@5,9:9='
',<8>,3:4]
[@6,10:10='
',<8>,4:0]
[@7,11:36='PKStext_that_is_a_filename',<7>,5:0]
[@8,37:37='
',<8>,5:26]
[@9,38:37='<EOF>',<-1>,6:0]

For this input:

(对于此输入：)

PKS

PKS?

PKStext_that_is_a_filename

Here's the grammar I used:

(这是我使用的语法：)

grammar Example;

start: block;

block: line+ EOF;
line:  expr? eol;

expr: command (file | listOfDouble | query)?;

command: COMMAND;

query:        QUERY;
file:         TEXT;
eol:          EOL;
listOfDouble: DOUBLE (COMMA DOUBLE)*;

COMMAND: PKS;
PKS:     'PKS';

QUERY: '?';

fragment LETTER: [a-zA-Z];
fragment DIGIT:  [0-9];
fragment UNDER:  [_];

COMMA:  ',';
DOUBLE: DIGIT+ (DOT DIGIT*)?;
DOT:    '.';
TEXT:   LETTER (LETTER | DIGIT | UNDER)*;
EOL:    [

];

and the generated visual parse tree:

(以及生成的可视化分析树：)