Golang : Read large file with bufio.Scanner cause token too long error
There are couple of ways to read files with Golang and one of the common way to read in file is with bufio.NewScanner() function. For example :
package main
import (
"bufio"
"fmt"
"os"
)
func main() {
file, err := os.Open("somefile.dat")
if err != nil {
panic(err.Error())
}
defer file.Close()
reader := bufio.NewReader(file)
scanner := bufio.NewScanner(reader)
scanner.Split(bufio.ScanBytes)
for scanner.Scan() {
fmt.Println(scanner.Bytes())
}
if err := scanner.Err(); err != nil {
fmt.Println("error while reading :", err)
}
}
Most of the time, things should be fine and dandy. However, there will be cases when the program attempts to read in a very large file and it will throw out this error message :
"bufio.Scanner: token too long"
This error message was caused by the input token size that exceeded the bufio.MaxScanTokenSize
buffer size. The allocated size is MaxScanTokenSize = 64 * 1024
To read in large files, it is recommended to use bufio.Reader.ReadLine instead.
References :
See also : Golang : Scanf function weird error in Windows
By Adam Ng
IF you gain some knowledge or the information here solved your programming problem. Please consider donating to the less fortunate or some charities that you like. Apart from donation, planting trees, volunteering or reducing your carbon footprint will be great too.
Advertisement
Tutorials
+6.1k PHP : How to handle URI or URL with non-ASCII characters such as Chinese/Japanese/Korean(CJK) ?
+22.6k Golang : Strings to lowercase and uppercase example
+7.9k Golang : Grayscale Image
+22.8k Golang : Gorilla mux routing example
+4.7k Facebook : How to place save to Facebook button on your website
+7.3k Golang : Calculate how many weeks left to go in a given year
+9.8k Golang : Get current, epoch time and display by year, month and day
+17.6k Golang : [json: cannot unmarshal object into Go value of type]
+10.5k Golang : Resolve domain name to IP4 and IP6 addresses.
+8.2k Golang : Count leading or ending zeros(any item of interest) example
+12.2k Golang : Get month name from date example
+6k Golang : Dealing with backquote