Golang : How to extract links from web page ?

Golang's third party package goquery provides excellent tools to assist developers in building HTML parser/crawler. In this tutorial, we will learn how to use goquery to extract links from a web page. The code below will scan a part of the web page that is within the <div class=".nav-collapse ..." > and <ul class=".nav.."> and extract the links.

 package main

 import (
  "fmt"
  "github.com/PuerkitoBio/goquery"
  "strings"
 )

 func ScrapeLinks(url string) {

  doc, err := goquery.NewDocument(url)

  if err != nil {
 panic(err)
  }

  // process this part :

  // <div class="nav-collapse collapse navbar-responsive-collapse">
  // <ul class="nav nav-pills">
  // <li>
  // <a href="/references">References</a>
  // </li>
  // <li>
  // <a href="/tutorials">Tutorials</a>
  // </li>
  // </ul>
  // </div>

  doc.Find(".nav-collapse .nav").Each(func(i int, s *goquery.Selection) {
 Title := strings.TrimSpace(s.Find("li").Text()) // https://www.socketloop.com/tutorials/trim-white-spaces-string-golang

 // convert string to array
 Fields := strings.Fields(Title)

 // go deeper by 1 level to get the <a href=""></a>

 doc.Find(".nav-collapse .nav a").Each(func(i int, s *goquery.Selection) {
 Link, _ := s.Attr("href")
 Link = url + Link

 fmt.Printf("Title is [%s] and link is [%s]\n", Fields[i], Link)
 })

  })

 }

 func main() {

  ScrapeLinks("https://socketloop.com")

 }

Output :

Title is [References] and link is [https://socketloop.com/references]

Title is [Tutorials] and link is [https://socketloop.com/tutorials]

Bear in mind that goquery's doc.Find only look for first part of the <div class="nav-collapse collapse navbar-responsive-collapse"> .. i.e .. only

 doc.Find(".nav-collapse")  // will do.

in this tutorial case, because we want to dig 1 level deeper to <ul class="nav..."> we use this instead

 doc.Find(".nav-collapse .nav") 

Reference :

http://godoc.org/github.com/PuerkitoBio/goquery