Golang : How to handle file size larger than available memory panic issue

Problem:

You are trying to convert create a slice with a size of a very large file. The file size is larger than available memory and it caused your Golang program to panic during runtime. How to overcome this panic issue?

Solution:

It was once thought that 640KB of memory should be enough for all our computing needs, but that is no longer true. However, hitting a memory limit issue will always be there. Hence the solution is:

Get the available memory first before creating the slice.

If the file size is lower than the available memory than proceed as usual. However, if the file size is larger than available memory, chunk the file according to a sensible size and process each chunks without bursting the total available memory.

 package main

 import (
  "fmt"
  "net/http"
  "os"

  "github.com/shirou/gopsutil/mem"
 )

 func main() {

  // memory
  vmStat, err := mem.VirtualMemory()

  if err != nil {
 panic(err)
  }

  availableMemory := vmStat.Available
  //availableMemory := uint64(655360) // uncomment to test low memory situation

  fmt.Println("Available memory : ", availableMemory, " bytes")

  fileToBeUploaded := "somebigfile"

  file, err := os.Open(fileToBeUploaded)

  if err != nil {
 panic(err)
  }

  defer file.Close()

  fileInfo, _ := file.Stat()
  fileSize := uint64(fileInfo.Size())

  fmt.Println("File size : ", fileSize, " bytes")

  // previous tutorials did not check if the fileSize is larger than
  // available memory. However, bear in mind that
  // http.DetectContentType() only needs 512 bytes, not the entire file

  if fileSize < availableMemory {
 fileBytes := make([]byte, fileSize)
 fmt.Println("File converted into []byte")

 fileType := http.DetectContentType(fileBytes)
 fmt.Println("File type : ", fileType)
  } else {
 // see https://golang.org/pkg/net/http/#DetectContentType
 fmt.Println("Not enough memory to read the entire file into []byte!")
 fileBytes := make([]byte, 512) // we only need 512 bytes
 fileType := http.DetectContentType(fileBytes)
 fmt.Println("File type : ", fileType)
  }

  // However, if http.DetectContentType() is not your concern then you will need to
  // split the file to smaller pieces within the available memory and process it
  // without causing panic

  // such as
  // https://www.socketloop.com/tutorials/golang-how-to-split-or-chunking-a-file-to-smaller-pieces
  // https://www.socketloop.com/tutorials/golang-upload-big-file-larger-than-100mb-to-aws-s3-with-multipart-upload

  // calculate the number of parts by dividing up the file size by 5MB
  const fileChunk = 5242880 // 5MB in bytes

  // or
  const fileChunk = availableMemory - 10000 // not wise to use up all availableMemory, tweak around figure to minus!
 }