Golang : How to tokenize source code with text/scanner package?
It has been a while since I touch the topic on tokenizer.....err... since the day I actually built a compiler back in the 90's in Ada95!
Ok, time to cut the grandma or grandpa story.
This short code example below demonstrate how to tokenize souce code with Golang's text/scanner
package.
Here you go !
package main
import (
"fmt"
"strings"
"text/scanner"
)
func main() {
code := `<?php
echo "Hello World!";
?>`
codeReader := strings.NewReader(code)
fmt.Println(code)
fmt.Println("------------------------------")
fmt.Println("TOKENS : ")
fmt.Println("------------------------------")
var scn scanner.Scanner
scn.Init(codeReader)
tok := scn.Scan()
fmt.Println(scn.TokenText())
for tok != scanner.EOF {
tok = scn.Scan()
fmt.Println(scn.TokenText())
}
}
Play at : http://play.golang.org/p/R9pDAt2jOO
By Adam Ng
IF you gain some knowledge or the information here solved your programming problem. Please consider donating to the less fortunate or some charities that you like. Apart from donation, planting trees, volunteering or reducing your carbon footprint will be great too.
Advertisement
Tutorials
+10k Golang : Find age or leap age from date of birth example
+19.1k Golang : Underscore or snake_case to camel case example
+9.8k Golang : Natural string sorting example
+19.3k Golang : Convert PNG transparent background image to JPG or JPEG image
+18.2k Golang : Archive directory with tar and gzip
+6.2k Golang : File system scanning
+19.4k Golang : Clean up null characters from input data
+35k Golang : Convert date or time stamp from string to time.Time type
+11k Golang : Display list of countries and ISO codes
+4.7k PHP : Get client IP address
+16.3k Golang : Aligning strings to right, left and center with fill example
+4.1k Swift : Convert (cast) Float to Int or Int32 value