Golang : How to tokenize source code with text/scanner package?
It has been a while since I touch the topic on tokenizer.....err... since the day I actually built a compiler back in the 90's in Ada95!
Ok, time to cut the grandma or grandpa story.
This short code example below demonstrate how to tokenize souce code with Golang's text/scanner
package.
Here you go !
package main
import (
"fmt"
"strings"
"text/scanner"
)
func main() {
code := `<?php
echo "Hello World!";
?>`
codeReader := strings.NewReader(code)
fmt.Println(code)
fmt.Println("------------------------------")
fmt.Println("TOKENS : ")
fmt.Println("------------------------------")
var scn scanner.Scanner
scn.Init(codeReader)
tok := scn.Scan()
fmt.Println(scn.TokenText())
for tok != scanner.EOF {
tok = scn.Scan()
fmt.Println(scn.TokenText())
}
}
Play at : http://play.golang.org/p/R9pDAt2jOO
By Adam Ng
IF you gain some knowledge or the information here solved your programming problem. Please consider donating to the less fortunate or some charities that you like. Apart from donation, planting trees, volunteering or reducing your carbon footprint will be great too.
Advertisement
Tutorials
+8.2k Golang : How to control fmt or log print format?
+35.2k Golang : Convert(cast) int64 to string
+18.3k Swift : Convert (cast) Int to int32 or Uint32
+5.3k Golang : Test input string for unicode example
+44.1k Golang : Read tab delimited file with encoding/csv package
+8.8k Golang : Test a slice of integers for odd and even numbers
+42.6k Golang : Use wildcard patterns with filepath.Glob() example
+27.2k Golang : Detect (OS) Operating System
+28.8k Golang : Remove characters from string example
+19.9k Golang : Convert string slice to struct and access with reflect example
+8.9k Golang : Use regular expression to get all upper case or lower case characters example