Back in 2011, Rob Pike gave a talk on the topic of Lexical Scanning in Golang, where he presented an interesting approach to lexical analysis with coroutines (implemented via goroutines communicating over a channel). Since the author has been pondering the connection between coroutines and state machines in the past, Rob’s talk inspired the author to try approximating this approach in Python using sub-generators.
Since 2011, the author has seen this talk and its technique mentioned many times, both in Golang forums and general programming communities. There’s something in this approach that feels elegant – it’s a problem very well suited for coroutines. However, there was always a small nagging voice in the back of my head doubting the efficiency of the approach.
Since the Author has seen recently found myself playing with lexers again, this seemed like a good opportunity to revisit Rob Pike’s lexer and compare its performance to other approaches, using the same problem and benchmark for fairness.
Rob Pike’s original golang lexer
There is no ads to display, Please add some