Permalink to this article – https://ift.tt/VTsyotH
In the previous article , we established a complete semantic model for the DSL. We are still one step away from the actual running of the grammar example of the DSL, which is to extract information (reverse Polish style) based on the syntax tree, assemble the semantic model, and load it. With the semantic model and the individual rule processors instantiated, we can process the data! Below is a panorama of our metrics collection program deployed on ocean buoys:
In this article, we will extract the reverse Polish through the syntax tree and assemble the semantic model according to the above figure, so that our syntax example can really run as expected!
1. Extract Reverse Polish from Syntax Tree and Assemble Semantic Model
Through the explanation of the semantic model above, we know that the connection between the syntax tree and the semantic model includes reverse Polish, windowsRange, result and enumableFunc. The main link is that reverse Polish, and information like windowsRange, result, and enumableFunc are relatively easy to extract.
Next, let’s take a look at how to extract from the syntax tree structure of DSL to Reverse Polish, and complete the extraction of Reverse Polish. Our semantic model assembly work is more than half completed. Well, let’s focus on the DSL syntax tree.
In order to focus on the explanation of the principle, we only implement the extraction of information such as the reverse Polish formula of the syntax tree containing a single rule in the grammar example file . The case of multiple rules in the grammar example file is left as a thought question.
In the verification grammar in the second part of this series , we know that the traversal of the DSL syntax tree by the ANTLR Listener is preorder traversal by default. During such a traversal, we extract variables, literals, unary operators, and binary operators, and organize their order of operations in a reverse Polish form. The extraction and transformation algorithm we adopted is as follows:
- We use two Stacks to complete this conversion, s1 is used to store the ordered reverse Polish; s2 is a temporary stack, used to temporarily store unary and binary operators;
- We perform the fetch operation in the ExitXXX callbacks of all nodes;
- When the node is variable or literal, directly convert the node text to the corresponding type value (such as int, float64 or string), package it as Value, and push it into the s1 stack;
- When the node is a unary operator node, calculate the node depth (level) and push it into the s2 stack together with a semantic.UnaryOperator it represents;
- When the node is a binary operator node, including arithmeticOp, comparisonOp and logicalOp, the depth (level) of the current node is used to compare with the top element of the s2 stack, if it is smaller than the depth (level) of the node in the top of the s2 stack (level), then Pop the node at the top of the s2 stack and push it into the s1 stack; loop this step until the s2 stack is empty or the current node depth is greater than the depth of the top element of the s2 stack, then package the node as semantic.BinaryOperator and push it into the s2 stack;
- In the exit callback of the top-level conditionExpr node (parent node is ruleLine), all elements in the s2 stack are popped out and pushed into the s1 stack in turn; at this time, the s1 stack is a reverse Polish style from the bottom of the stack to the top of the stack.
The following is the specific code implementation. We build a ReversePolishExprListener structure to extract information from the syntax tree for building the semantic model:
// tdat/reverse_polish_expr_listener.go type ReversePolishExprListener struct { *parser.BaseTdatListener ruleID string // for constructing Reverse Polish expression // // infixExpr:($speed<5)and($temperature<2)or(roundDown($sanility)<600) => // // reversePolishExpr: // $speed,5,<,$temperature,2,<,and,$sanility,roundDown,600,<,or // reversePolishExpr []semantic.Value s1 semantic.Stack[*Item] // temp stack for constructing reversePolishExpr, for final result s2 semantic.Stack[*Item] // temp stack for constructing reversePolishExpr, for operator temporarily // for windowsRange low int high int // for enumerableFunc ef string // for result result []string }
For variables and literals, they are directly pushed into the s1 stack. For unary operators, they are directly pushed into the s2 stack. For binary operators, we take the comparison operator (comparisonOp) as an example to see its processing logic:
func (l *ReversePolishExprListener) ExitComparisonOp(c *parser.ComparisonOpContext) { l.handleBinOperator(c.BaseParserRuleContext) } func (l *ReversePolishExprListener) handleBinOperator(c *antlr.BaseParserRuleContext) { v := c.GetText() lvl := getLevel(c) for { lastOp := l.s2.Top() if lastOp == nil { l.s2.Push(&Item{ level: lvl, val: &semantic.BinaryOperator{ Val: v, }, }) return } if lvl > lastOp.level { l.s2.Push(&Item{ level: lvl, val: &semantic.BinaryOperator{ Val: v, }, }) return } l.s1.Push(l.s2.Pop()) } }
Binary operators such as arithmetic operators and logical operators, like comparison operators, directly call handleBinOperator. The logic of handleBinOperator is like the algorithm steps we described earlier. First, compare the level of the node at the top of the s2 stack. If the depth of the node is smaller than the depth (level) of the node at the top of the s2 stack, pop the node at the top of the s2 stack. , and push it into the s1 stack; loop this step until the s2 stack is empty or the current node depth is greater than the depth of the top node of the s2 stack, then the node is packaged as semantic.BinaryOperator and pushed into the s2 stack.
We get the reverse Polish expression we expect based on the s1 stack in the topmost conditionExpr:
func (l *ReversePolishExprListener) ExitConditionExpr(c *parser.ConditionExprContext) { // get the rule index of parent context if i, ok := c.GetParent().(antlr.RuleContext); ok { if i.GetRuleIndex() != parser.TdatParserRULE_ruleLine { // 非最顶层的conditionExpr节点return } } // pop all left in the stack for l.s2.Len() != 0 { l.s1.Push(l.s2.Pop()) } // fill in the reversePolishExpr var vs []semantic.Value for l.s1.Len() != 0 { vs = append(vs, l.s1.Pop().val) } for i := len(vs) - 1; i >= 0; i-- { l.reversePolishExpr = append(l.reversePolishExpr, vs[i]) } }
It is relatively simple to extract other information required to construct a semantic model, such as result, windowsRange, etc. You can directly refer to the source code of the corresponding method of ReversePolishExprListener.
2. Instantiate the Processor and run the syntax example
Time to string together the front end (syntax tree) and back end (semantic model) of the language! To this end, we define a type Processor to assemble the front-end and back-end:
type Processor struct { name string // for ruleid model *semantic.Model }
At the same time, each Processor instance corresponds to a grammar rule. If there are multiple rules, different Processors can be instantiated, and then we can use the Exec method of the Processor instance to process the data:
func (p *Processor) Exec(in []map[string]interface{}) (map[string]interface{}, error) { return p.model.Exec(in) }
Let’s take a look at the main function:
// tdat/main.go func main() { println("input file:", os.Args[1]) input, err := antlr.NewFileStream(os.Args[1]) if err != nil { panic(err) } lexer := parser.NewTdatLexer(input) stream := antlr.NewCommonTokenStream(lexer, 0) p := parser.NewTdatParser(stream) tree := p.Prog() l := NewReversePolishExprListener() antlr.ParseTreeWalkerDefault.Walk(l, tree) processor := &Processor{ name: l.ruleID, model: semantic.NewModel(l.reversePolishExpr, semantic.NewWindowsRange(l.low, l.high), l.ef, l.result), } // r0006: Each { |1,3| ($speed < 50) and (($temperature + 1) < 4) or ((roundDown($salinity) <= 600.0) or (roundUp($ph) > 8.0)) } => (); in := []map[string]interface{}{ { "speed": 30, "temperature": 6, "salinity": 500.0, "ph": 7.0, }, { "speed": 31, "temperature": 7, "salinity": 501.0, "ph": 7.1, }, { "speed": 30, "temperature": 6, "salinity": 498.0, "ph": 6.9, }, } out, err := processor.Exec(in) if err != nil { panic(err) } fmt.Printf("%v\n", out) }
The steps of the main function are roughly: build a syntax tree (p.Prog), extract the information required for the semantic model (ParseTreeWalkerDefault.Walk), then instantiate the Processor, connect the front and back ends, and finally process the input data in through processor.Exec.
Next, we define samples/sample4.t as a syntax example to test main:
// samples/sample4.t r0006: Each { |1,3| ($speed < 50) and (($temperature + 1) < 4) or ((roundDown($salinity) <= 600.0) or (roundUp($ph) > 8.0)) } => ();
Build and execute main:
$make $./tdat samples/sample4.t map[ph:7 salinity:500 speed:30 temperature:6]
We see that the program outputs what we expected!
3. Summary
At this point, the DSL language we built for the meteorologist in “The Day After Tomorrow” and the core of its processing engine have been introduced. The above code can currently only handle only one rule in a source file . The task of extending the processing engine to support placing multiple rules in a source file is left to you as a “job” ^_^.
After this series of four articles, I believe you have a basic understanding of how to design and implement a DSL language based on ANTLR and Go. Now you can design a DSL for your own use or your team’s own use in your field. You are welcome to leave a message at the end of the article to communicate, and we will improve the level of design and implementation of DSL together.
The code covered in this article can be downloaded here – https://ift.tt/GSXCo7s.
“Gopher Tribe” Knowledge Planet aims to create a high-quality Go learning and advanced community! High-quality first published Go technical articles, “three-day” first published reading rights, analysis of the current situation of Go language development twice a year, read the fresh Gopher daily 1 hour in advance every day, online courses, technical columns, book content preview, must answer within six hours Guaranteed to meet all your needs about the Go language ecosystem! In 2022, the Gopher tribe will be fully revised, and will continue to share knowledge, skills and practices in the Go language and Go application fields, and add many forms of interaction. Everyone is welcome to join!
I love texting : Enterprise-level SMS platform customization development expert https://51smspush.com/. smspush : A customized SMS platform that can be deployed within the enterprise, with three-network coverage, not afraid of large concurrent access, and can be customized and expanded; the content of the SMS is determined by you, no longer bound, with rich interfaces, long SMS support, and optional signature. On April 8, 2020, China’s three major telecom operators jointly released the “5G Message White Paper”, and the 51 SMS platform will also be newly upgraded to the “51 Commercial Message Platform” to fully support 5G RCS messages.
The famous cloud hosting service provider DigitalOcean released the latest hosting plan. The entry-level Droplet configuration is upgraded to: 1 core CPU, 1G memory, 25G high-speed SSD, and the price is 5$/month. Friends who need to use DigitalOcean can open this link : https://ift.tt/WF9j54n to open your DO host road.
Gopher Daily Archive Repository – https://ift.tt/NI8p3aJ
my contact information:
- Weibo: https://ift.tt/Xs3e8HZ
- WeChat public account: iamtonybai
- Blog: tonybai.com
- github: https://ift.tt/yfemWk6
- “Gopher Tribe” Planet of Knowledge: https://ift.tt/J28cwPH
Business cooperation methods: writing, publishing books, training, online courses, partnership entrepreneurship, consulting, advertising cooperation.
© 2022, bigwhite . All rights reserved.
This article is reprinted from https://tonybai.com/2022/05/28/an-example-of-implement-dsl-using-antlr-and-go-part4/
This site is for inclusion only, and the copyright belongs to the original author.