Golang : How to determine if request or crawl is from Google robots
For this tutorial, we will learn how to detect a visit by Google robots or web crawlers and learn how to distinguish fake/spoof user agents from bad actors pretending to be Google.
Taking example from the official guide from Google on how to see which robots Google uses to crawl website. We can see that all of the user agents contain the string "google".
The simplest way to detect Google robots or crawlers should look like this :
package main
import (
"strings"
"fmt"
"net/http"
)
func getUserAgent(w http.ResponseWriter, r *http.Request) {
ua := r.Header.Get("User-Agent")
fmt.Printf("user agent is: %s \n", ua)
w.Write([]byte("user agent is " + ua))
ualow := strings.ToLower(ua)
if strings.Contains(ualow, "google") {
fmt.Println("Visited by Google bot")
} else {
fmt.Println("Visited by some thing else")
}
}
func main() {
http.HandleFunc("/", getUserAgent)
http.ListenAndServe(":8080", nil)
}
Sample output :
These are the results simulated by Google PageSpeed Insights
user agent is: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Safari/537.36
Visited by Google bot
user agent is: Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Version/8.0 Mobile/12F70 Safari/600.1.4
Visited by Google bot
However, the above code is kinda primitive and can be easily spoofed. We need to verify if the Google robot is indeed genuine by following this guideline from Google on how to verify Google robots.
package main
import (
"fmt"
"net"
"net/http"
"os/exec"
"strings"
"os"
)
func getUserAgent(w http.ResponseWriter, r *http.Request) {
ua := r.Header.Get("User-Agent")
fmt.Printf("user agent is: %s \n", ua)
w.Write([]byte("user agent is " + ua))
ualow := strings.ToLower(ua)
if strings.Contains(ualow, "google") {
fmt.Println("Visited by Google bot")
} else {
fmt.Println("Visited by some thing else")
}
// get IP address
ip, _, _ := net.SplitHostPort(r.RemoteAddr)
// capture the output of host command to verify Google robots
// based on https://support.google.com/webmasters/answer/80553
cmd := exec.Command("host", ip)
result, err := cmd.Output() // capture the exec output to variable result
if err != nil {
//fmt.Println(err)
fmt.Printf("Host %s command execution failed \n", ip)
os.Exit(1)
}
fmt.Println("Host reply : ", string(result))
// if result contain the word google, then it is genuine user agent
// else fake
if strings.Contains(strings.ToLower(string(result)), "google") {
fmt.Println(" and the user agent is real. ")
} else {
fmt.Println(" and the user agent is determine to be FAKED after verifying with host command. ")
}
}
func main() {
http.HandleFunc("/", getUserAgent)
http.ListenAndServe(":8080", nil)
}
References :
https://support.google.com/webmasters/answer/1061943
https://www.socketloop.com/tutorials/golang-how-to-check-if-a-string-contains-another-sub-string
By Adam Ng
IF you gain some knowledge or the information here solved your programming problem. Please consider donating to the less fortunate or some charities that you like. Apart from donation, planting trees, volunteering or reducing your carbon footprint will be great too.
Advertisement
Tutorials
+9.7k Golang : Identifying Golang HTTP client request
+28.6k Golang : Get first few and last few characters from string
+6.5k Swift : substringWithRange() function example
+4.7k Golang : Calculate a pip value and distance to target profit example
+14.5k Golang : How to check for empty array string or string?
+8.9k Golang : Apply Histogram Equalization to color images
+5.7k PHP : How to check if an array is empty ?
+13.4k Golang : Check if an integer is negative or positive
+18.2k Golang : Send email with attachment
+14.2k Golang : GUI with Qt and OpenCV to capture image from camera
+11.1k Golang : Concatenate (combine) buffer data example
+18.3k Golang : Display list of time zones with GMT