Go Web Scraping Guide: Colly Crawlers, goquery Parsing and chromedp Rendering Behind Proxies

Go suits data collection unusually well: one statically linked executable can keep thousands of requests in flight while using modest memory. Gophers typically combine three tools. The standard net/http package plus goquery downloads and dissects server-rendered markup, Colly orchestrates whole-site crawls through event handlers, rate rules and link following, and chromedp remote-controls Chrome for pages that reveal their content only after scripts execute. Each of them can be pointed at a proxy, and changing the exit address between requests needs just a handful of lines.

A typical tutorial shows one Colly snippet and calls it a day. This one implements the same tiny book-catalogue collector in three styles, wires an authenticated gateway into every layer, spreads traffic across several exits, keeps parallelism courteous and writes the output to CSV and JSON. It finishes with a lookup table of common failures.

Which Go Package Does What

Sensible order of escalation: let Colly handle site traversal, drop down to net/http with goquery whenever a request needs hand tuning, and keep chromedp in reserve for views whose initial HTML arrives empty. Newcomers can first read our primer on web scraping, which walks through the download, extract and persist loop shared by every language.

Setting Up the Project

Grab a recent Go toolchain and initialise a module:

mkdir shelfscout && cd shelfscout
go mod init example.com/shelfscout
go get github.com/gocolly/colly/v2 github.com/PuerkitoBio/goquery github.com/chromedp/chromedp

Every example hits books.toscrape.com, a fake bookstore published specifically so developers can rehearse crawling on it. Gateway details live in environment variables: EXIT_URL contains the complete http://user:pass@host:port string, and EXIT_HOSTPORT stores just host:port for Chrome.

Step 1. net/http and goquery

In the standard library the gateway is attached to the transport. Hand http.ProxyURL a parsed address and any username and password inside it are forwarded to the gateway for you.

package main

import (
	"fmt"
	"log"
	"net/http"
	"net/url"
	"os"
	"time"

	"github.com/PuerkitoBio/goquery"
)

func newFetcher() *http.Client {
	exit, err := url.Parse(os.Getenv("EXIT_URL"))
	if err != nil {
		log.Fatal(err)
	}
	return &http.Client{
		Timeout: 30 * time.Second,
		Transport: &http.Transport{
			Proxy:               http.ProxyURL(exit),
			MaxIdleConnsPerHost: 4,
		},
	}
}

func main() {
	fetcher := newFetcher()
	req, _ := http.NewRequest("GET", "https://books.toscrape.com/", nil)
	req.Header.Set("User-Agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36")
	req.Header.Set("Accept-Language", "en-US,en;q=0.8")

	resp, err := fetcher.Do(req)
	if err != nil {
		log.Fatal(err)
	}
	defer resp.Body.Close()

	doc, err := goquery.NewDocumentFromReader(resp.Body)
	if err != nil {
		log.Fatal(err)
	}
	doc.Find("article.product_pod").Each(func(_ int, card *goquery.Selection) {
		title, _ := card.Find("h3 a").Attr("title")
		price := card.Find("p.price_color").Text()
		fmt.Printf("%s | %s\n", title, price)
	})
}

Production code should inspect resp.StatusCode first, since goquery will parse an error page just as readily as a real one. The scheme of the gateway address picks the protocol, which means a socks5:// value works with the same transport. If you are not sure which flavour to buy, our breakdown of SOCKS5 and HTTP proxies lays out the differences, and the explainer on HTTP vs HTTPS proxies describes how TLS traffic travels through a CONNECT tunnel.

Step 2. Crawling a Catalogue in Colly

Colly hides the fetch loop behind event handlers. Each element matched by an OnHTML selector triggers your function, a Visit call inside a handler queues the linked page, and a LimitRule caps parallel requests and adds jitter for a given domain.

package main

import (
	"fmt"
	"log"
	"os"
	"time"

	"github.com/gocolly/colly/v2"
)

type Book struct {
	Title string `json:"title"`
	Price string `json:"price"`
	Link  string `json:"link"`
}

func main() {
	var shelf []Book

	crawler := colly.NewCollector(
		colly.AllowedDomains("books.toscrape.com"),
		colly.UserAgent("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36"),
	)
	crawler.SetRequestTimeout(30 * time.Second)

	if err := crawler.SetProxy(os.Getenv("EXIT_URL")); err != nil {
		log.Fatal(err)
	}
	crawler.Limit(&colly.LimitRule{
		DomainGlob:  "*",
		Parallelism: 2,
		RandomDelay: 2 * time.Second,
	})

	crawler.OnHTML("article.product_pod", func(card *colly.HTMLElement) {
		shelf = append(shelf, Book{
			Title: card.ChildAttr("h3 a", "title"),
			Price: card.ChildText("p.price_color"),
			Link:  card.Request.AbsoluteURL(card.ChildAttr("h3 a", "href")),
		})
	})
	crawler.OnHTML("li.next a", func(next *colly.HTMLElement) {
		next.Request.Visit(next.Attr("href"))
	})
	crawler.OnError(func(r *colly.Response, err error) {
		log.Printf("failed %s: %d %v", r.Request.URL, r.StatusCode, err)
	})

	crawler.Visit("https://books.toscrape.com/catalogue/page-1.html")
	fmt.Println("books collected:", len(shelf))
}

Because this collector is synchronous, appending to the slice needs no locking. Switch on colly.Async(true) for throughput and you must guard shared state with a mutex and call crawler.Wait() before touching the results.

Step 3. Spreading Requests Across Several Exits

For multiple gateways, the proxy subpackage of Colly offers a helper that cycles through the addresses you give it, one per request:

import "github.com/gocolly/colly/v2/proxy"

switcher, err := proxy.RoundRobinProxySwitcher(
	os.Getenv("EXIT_URL_A"),
	os.Getenv("EXIT_URL_B"),
	os.Getenv("EXIT_URL_C"),
)
if err != nil {
	log.Fatal(err)
}
crawler.SetProxyFunc(switcher)

In bare net/http the Proxy field of the transport is itself a callback, so selecting an exit per request is up to you, for instance by stepping through a slice with an atomic counter. Many teams avoid client-side rotation altogether: a rotating gateway hands every new connection a fresh address, or pins one address for a sticky session, depending on the username or port. Aim all clients at one rotating proxies endpoint and the swapping happens on the provider side. Our piece on IP rotation explains when to rotate per request and when per session.

A subtlety: Go keeps connections alive and reuses them, so a burst of requests may share one tunnel and therefore one exit address. When each request must leave from a different IP, close idle connections between calls or turn keep-alives off in the transport.

Step 4. Rendering Script-Driven Pages in chromedp

When values appear on screen but not in the fetched HTML, client-side code is injecting them. Inspect the Fetch/XHR entries in DevTools first: replaying a JSON call through net/http costs a fraction of a browser session. Only if nothing like that exists should you start Chrome through chromedp.

The snippet follows the long-standing chromedp style seen in most guides, where Run executes a list of actions and fills variables you pass by pointer. According to the project README, releases from v0.20 onward switched to a generic, iterator-based interface and come with a migration document, so pin an older release in go.mod or translate these calls using that document.

package main

import (
	"context"
	"fmt"
	"log"
	"os"
	"strings"
	"time"

	"github.com/PuerkitoBio/goquery"
	"github.com/chromedp/chromedp"
)

func main() {
	flags := append(chromedp.DefaultExecAllocatorOptions[:],
		chromedp.ProxyServer("http://"+os.Getenv("EXIT_HOSTPORT")),
		chromedp.Flag("blink-settings", "imagesEnabled=false"),
	)
	allocCtx, stopAlloc := chromedp.NewExecAllocator(context.Background(), flags...)
	defer stopAlloc()

	tabCtx, stopTab := chromedp.NewContext(allocCtx)
	defer stopTab()
	tabCtx, stopTimer := context.WithTimeout(tabCtx, 45*time.Second)
	defer stopTimer()

	var markup string
	err := chromedp.Run(tabCtx,
		chromedp.Navigate("https://quotes.toscrape.com/js/"),
		chromedp.WaitVisible("div.quote", chromedp.ByQuery),
		chromedp.OuterHTML("body", &markup, chromedp.ByQuery),
	)
	if err != nil {
		log.Fatal(err)
	}

	page, _ := goquery.NewDocumentFromReader(strings.NewReader(markup))
	page.Find("div.quote span.text").Each(func(_ int, s *goquery.Selection) {
		fmt.Println(s.Text())
	})
}

Chrome discards any username and password embedded in its proxy flag, hence EXIT_HOSTPORT carries only the host and port. Two easy workarounds are adding your machine IP to the provider allowlist or running a local relay without a password that forwards to the authenticated gateway. Responding to the authentication challenge via the DevTools Fetch domain also works in principle, but the exact code varies between chromedp versions, so verify it on yours.

Step 5. Polite Parallelism

Spawning goroutines costs next to nothing, which means the brake has to be yours. A buffered channel doubles as a semaphore and caps how many requests hit one site at once:

gate := make(chan struct{}, 4)
var wg sync.WaitGroup

for _, target := range pages {
	wg.Add(1)
	go func(addr string) {
		defer wg.Done()
		gate <- struct{}{}
		defer func() { <-gate }()

		grab(addr)
		time.Sleep(time.Duration(800+rand.Intn(1200)) * time.Millisecond)
	}(target)
}
wg.Wait()

Four simultaneous requests with jitter make a reasonable opening setting for a modest site. Raise the number only while latency and status codes stay healthy, and when 429 replies appear, pause for the interval named in Retry-After. Pacing, headers and the choice of IP type are covered in more depth in our guide to proxies in web scraping.

Step 6. Writing CSV and JSON Files

func saveShelf(shelf []Book) error {
	sheet, err := os.Create("books.csv")
	if err != nil {
		return err
	}
	defer sheet.Close()
	w := csv.NewWriter(sheet)
	w.Write([]string{"title", "price", "link"})
	for _, b := range shelf {
		w.Write([]string{b.Title, b.Price, b.Link})
	}
	w.Flush()
	if err := w.Error(); err != nil {
		return err
	}

	blob, err := json.MarshalIndent(shelf, "", "  ")
	if err != nil {
		return err
	}
	return os.WriteFile("books.json", blob, 0o644)
}

Field names in the JSON output come from the struct tags on Book. Since csv.Writer holds data in a buffer, calling Flush and then checking Error is mandatory.

Troubleshooting Table

Restrict collection to public pages, review robots.txt and the terms of each site beforehand, and leave out personal data you have no use for.