/blog

🏊 Worker Pool in Go

A simple pattern to get started using worker pools with Golang.

In Go there is one boilerplate that I use more than any other - the worker pool. The worker pool that I like most is the one presented by Go By Example which I modify each time I use.

Since I like to have my own control over the worker pool I write the code each time. Each time it requires implementation of a worker function, and the job and result types. However, to make it easier, I’ve boiled it down to a simple copy and paste with only 5 steps of user input:

// step 1
// specify the number of jobs
var numJobs = 1

// step 2
// specify the job and result type
type job struct {

}
type result struct {
	err error
}

jobs := make(chan jobs, numJobs)
results := make(chan result, numJobs)
runtime.GOMAXPROCS(runtime.NumCPU())
for i := 0; i < runtime.NumCPU(); i++ {
	go func(jobs <-chan job, results chan<- result) {
		for j := range jobs {
			// step 3
			// specify the work for the worker 
			var err error 				
			results <- result{
				err: err,
			}
		}
	}(jobs,results)
}

// step 4
// specify how to push out jobs
for i:=0;i<numJobs;i++{
	jobs <- job{}
}
close(jobs)

// step 5
// do something with results
for i:=0;i<numJobs;i++{
	<-results
}

The main change is that I’ve created a job and a result type which you can populate with anything you want exchanged between the worker and main thread. Also I use runtime to automatically utilize all the porcessors.

Hope that might work for you too!

February 19, 2020

⚱️ Programming to register for pottery