Golang: Prevent over writing file with md5 hash
This is an additional note for previous tutorial on how to upload/submit file from command line to a server. In the receiving server program, we overwrite the file upon receiving. There are times when we need to check first if the file that we receive from the client is the same file. If the file is same, then we do not need to save the file on server.
Now, do not confuse checking if a file exist is similar to checking if a new file is the same as the old file. Exist != Same.
To check if the new file that our server program just received is the same as the old file, we need to compute the MD5(can be sha1, sha512 or sha256 too) hash and check if any file in the directory has the same MD5 hash. If no, then proceed to overwrite. If yes, ignore the uploaded file or you can save the new file but with incremental filenames. See https://www.socketloop.com/tutorials/golang-increment-string-example
The code example that follows is a modification of a file receiving program from the command line upload file tutorial.
checkSameFile.go
package main
import (
"crypto/md5"
"fmt"
"io"
"io/ioutil"
"net/http"
"os"
)
func receiveHandler(w http.ResponseWriter, r *http.Request) {
// the FormFile function takes in the POST input id file
file, header, err := r.FormFile("fileUploadName")
if err != nil {
fmt.Fprintln(w, err)
return
}
// sanity check on the MIME FileHeader
fmt.Println(header.Header)
defer file.Close()
out, err := os.Create("/tmp/uploadedFile")
if err != nil {
fmt.Fprintf(w, "Unable to create the file for writing. Check your write access privilege")
return
}
defer out.Close()
// write the content from POST to the file
_, err = io.Copy(out, file)
if err != nil {
fmt.Fprintln(w, err)
}
// now we want to check if the new uploaded file is a duplicate of our target file
// at /tmp/keepThisFile. If yes, abort writing to target file
// modified from https://www.socketloop.com/tutorials/golang-find-duplicate-files-with-filepath-walk
oldFile, err := ioutil.ReadFile("/tmp/keepThisfile")
// target file not found
if err != nil {
fmt.Fprintln(w, "Old file not found, proceed to rename /tmp/uploadedFile to /tmp/keepThisFile. \n")
// rename /tmp/uploadedFile to /tmp/keepThisFile
os.Rename("/tmp/uploadedFile", "/tmp/keepThisFile")
fmt.Fprintln(w, "Renamed /tmp/uploadedFile to /tmp/keepThisFile. You can proceed to digest the file for further analysis. \n")
return
} else {
// target file found, get the MD5 checksum of target / old file
oldHexDigest := md5.Sum(oldFile)
fmt.Fprintf(w, "Old file MD5 checksum is %x \n", oldHexDigest)
newFile, err := ioutil.ReadFile("/tmp/uploadedFile")
if err != nil {
fmt.Fprintln(w, err)
}
newHexDigest := md5.Sum(newFile)
fmt.Fprintf(w, "New file MD5 checksum is %x \n", newHexDigest)
if newHexDigest != oldHexDigest {
// rename /tmp/uploadedFile to /tmp/keepThisFile
// don't worry - /tmp/uploadedFile will be deleted in the process
os.Rename("/tmp/uploadedFile", "/tmp/keepThisFile")
fmt.Fprintln(w, "Renamed /tmp/uploadedFile to /tmp/keepThisFile. You can proceed to digest the file for further analysis. \n")
fmt.Fprintf(w, "File [%s] uploaded.", header.Filename)
return
} else {
fmt.Fprintf(w, "%s is a duplicate of previously uploaded file.", header.Filename)
return
}
}
}
func page(w http.ResponseWriter, r *http.Request) {
html := ` <html>
<title>Upload your submission here.</title>
<body>
<h1>This the web way of submitting/uploading file</h1>
<br>
<form action="http://localhost:8888/submit" method="post" enctype="multipart/form-data">
<label for="file">Filename:</label>
<input type="file" name="fileUploadName" id="fileUploadName">
<input type="submit" name="submit" value="Submit">
</form>
</body>
</html>`
w.Write([]byte(fmt.Sprintf(html)))
}
func main() {
http.HandleFunc("/submit", receiveHandler)
http.HandleFunc("/", page)
http.ListenAndServe(":8888", nil)
}
Build checkSameFile.go
and run on the background. Point your browser to http://localhost:8888/
and upload a file to see the output message for the first time.
Next, try uploading the same file, but with a different name. If all goes well, the server detects that it is a duplicate file.
Happy coding!
References:
https://www.socketloop.com/tutorials/golang-find-duplicate-files-with-filepath-walk
https://www.socketloop.com/tutorials/golang-increment-string-example
https://www.socketloop.com/tutorials/golang-generate-md5-checksum-of-a-file
https://socketloop.com/tutorials/golang-check-if-a-file-exist-or-not
See also : Golang : Command line file upload program to server example
By Adam Ng
IF you gain some knowledge or the information here solved your programming problem. Please consider donating to the less fortunate or some charities that you like. Apart from donation, planting trees, volunteering or reducing your carbon footprint will be great too.
Advertisement
Tutorials
+17.7k Golang : How to log each HTTP request to your web server?
+9.2k Golang : Scramble and unscramble text message by randomly replacing words
+11.3k Golang : Display a text file line by line with line number example
+19.7k Golang : Determine if directory is empty with os.File.Readdir() function
+15.4k Golang : Get current time from the Internet time server(ntp) example
+27.3k PHP : Count number of JSON items/objects
+6k Golang : Process non-XML/JSON formatted ASCII text file example
+12.4k Golang : zlib compress file example
+12.2k Elastic Search : Return all records (higher than default 10)
+23.7k Golang : Upload to S3 with official aws-sdk-go package
+19.4k Golang : Set or Add HTTP Request Headers
+12k Golang : calculate elapsed run time