Go goroutines all processed the last item: loop variable capture and go.mod

On a Monday morning our support inbox had 380 near-identical replies from the same customer, a small accounting firm, asking why they had received 380 copies of an overdue-invoice reminder. At the same time, our finance team noticed that none of the other 379 customers with overdue invoices had been reminded at all.

The reminder job is a small Go worker that runs at 07:00, loads overdue invoices and sends one email per invoice through our mail provider. Its log was suspiciously uniform:

2025-01-20T07:00:03Z INFO reminder sent invoice=INV-24-11873 customer=cus_8812 attempt=1
2025-01-20T07:00:03Z INFO reminder sent invoice=INV-24-11873 customer=cus_8812 attempt=1
2025-01-20T07:00:03Z INFO reminder sent invoice=INV-24-11873 customer=cus_8812 attempt=1
...
2025-01-20T07:00:09Z INFO reminder batch done total=380 failed=0

INV-24-11873 was the last row returned by the query, ordered by due date.

Reproducing it

The job had been rewritten the previous Friday to send emails concurrently instead of one at a time. Running it against a staging database with three overdue invoices reproduced the problem on the first try: three emails, all for the third invoice. Adding a time.Sleep before starting each goroutine made it "work", which is a strong hint of a race rather than a logic error.

What we ruled out

• The SQL query. We ran it by hand and it returned 380 distinct invoices.
• The mail provider retrying. Every log line had attempt=1, and the provider's dashboard showed 380 distinct message IDs, so these were 380 separate sends.
• The new code being wrong. The engineer who wrote it had copied the fan-out pattern from our newer notifier service, where the exact same loop has worked for months. That turned out to be the key detail.

Root cause

The notifier module declares go 1.23 in its go.mod. The billing-worker module, created in 2023, still declared go 1.20. Both are built with the same Go 1.23.4 toolchain, but the language version for each package comes from its module's go line. Under pre-1.22 semantics, the range variable inv is a single variable reused by every iteration, so all goroutines captured the same variable and most of them read it after the loop had already assigned the final invoice. The race detector confirmed it immediately:

WARNING: DATA RACE
Read at 0x00c0001a4030 by goroutine 9:
  example.com/billing-worker/reminders.(*Job).Run.func1()
      /src/reminders/job.go:47 +0x44

Previous write at 0x00c0001a4030 by goroutine 7:
  example.com/billing-worker/reminders.(*Job).Run()
      /src/reminders/job.go:44 +0x1d0

What is happening under the hood

Before Go 1.22, a for loop declared its variables once. Each iteration assigned a new value to the same variable. A closure captures variables, not values, so a goroutine started inside the loop reads whatever the variable holds at the moment the goroutine actually runs. Goroutines are usually scheduled after the loop has advanced, often after it has finished.

Go 1.22 changed this: each iteration gets a fresh variable. To avoid silently changing the behaviour of existing programs, the change is gated per module by the go directive in go.mod. Packages in a module that says go 1.22 or later get the new semantics; packages in a module that says go 1.21 or earlier keep the old ones, even when compiled by a newer toolchain. A per-file //go:build go1.22 constraint can also opt a single file in.

So the same source code has two meanings depending on which go.mod it sits under. Copying a snippet between repositories is enough to introduce the bug.

Environment

Minimal reproduction

// go.mod: module example.com/loopdemo
//         go 1.20
package main

import (
	"fmt"
	"sync"
)

func main() {
	ids := []string{"INV-1", "INV-2", "INV-3"}
	var wg sync.WaitGroup
	for _, id := range ids {
		wg.Add(1)
		go func() {
			defer wg.Done()
			fmt.Println("sending", id) // usually prints INV-3 three times
		}()
	}
	wg.Wait()
}

Change the go line to go 1.22, run it again, and it prints each ID once, in some order.

Who is affected

Modules whose go.mod declares go 1.21 or lower, regardless of the installed toolchain. The bug shows up with goroutines, errgroup.Group.Go, defer closures, callbacks stored in a slice, and anything that keeps the address &v of a loop variable beyond the iteration. Three-clause loops such as for i := 0; i < n; i++ are affected the same way as range loops.

The fix

We fixed it twice: once in the code so it is correct under either semantics, and once in go.mod so the module stops living in the past.

The code change passes the value explicitly into the closure's scope:

 	g, ctx := errgroup.WithContext(ctx)
 	g.SetLimit(16)
 	for _, inv := range invoices {
+		inv := inv // per-iteration copy; redundant once go.mod says go 1.22+
 		g.Go(func() error {
 			return j.sendReminder(ctx, inv)
 		})
 	}
 	return g.Wait()

Then we raised the language version for the module:

go mod edit -go=1.23
go mod tidy

 module example.com/billing-worker

-go 1.20
+go 1.23

Raising the go line changes the semantics of every loop in the module, so we did not ship it blind. The Go team published the bisect tool for exactly this migration. It compiles the tests with the new semantics applied to subsets of loops and reports which loop, if any, changes a test result:

go install golang.org/x/tools/cmd/bisect@latest
bisect -compile=loopvar go test ./...

It found no loop whose behaviour changed for the worse, which matches the Go team's experience that the change almost exclusively fixes latent bugs.

Alternatives and trade-offs

1. Pass the value as an argument. go func(inv Invoice) { ... }(inv) is the traditional fix and makes the data flow obvious. It works with any Go version.
2. Only bump go.mod. Simplest, and the inv := inv line becomes redundant (linters such as golangci-lint's copyloopvar flag it). We kept the explicit copy for a release because the file had already been copied between repositories once.
3. Per-file opt-in with a build constraint. Useful in a large module you cannot migrate at once, but it makes semantics differ between files of the same package, which we find confusing.

How to test the fix

A test that asserts each invoice is sent exactly once, run with the race detector:

func TestRunSendsEachInvoiceOnce(t *testing.T) {
	sender := &fakeSender{}
	job := &Job{sender: sender, invoices: fixtureInvoices(50)}

	if err := job.Run(context.Background()); err != nil {
		t.Fatal(err)
	}

	got := sender.SentIDs()
	sort.Strings(got)
	if want := fixtureIDs(50); !slices.Equal(got, want) {
		t.Fatalf("sent %v, want %v", got, want)
	}
}

go test -race -count=20 ./reminders/...

Before the fix this failed on every run and the race detector printed the data race above.

Confirming in production

The next morning's run sent 402 reminders to 402 distinct invoices. We verified that with the provider's API by counting unique invoice IDs in the message metadata, and by checking that the reminder sent log lines had no duplicate invoice= values:

kubectl logs job/reminders-29015820 | grep 'reminder sent' \
  | grep -o 'invoice=[^ ]*' | sort | uniq -d | wc -l

0

We also sent an apology to the affected customer and manual reminders to the 379 who had been skipped.

Regression guard

• CI now runs go vet ./... in full. go test only runs a small subset of vet analyzers, and loopclosure is not one of them. On a module with a pre-1.22 go line, vet reports loop variable inv captured by func literal.
• CI runs go test -race ./... on every pull request.
• A repository-wide check fails if any go.mod declares a go version below 1.22:

for f in $(git ls-files '*go.mod'); do
  v=$(awk '/^go /{print $2}' "$f")
  printf '%s\n%s\n' 1.22 "$v" | sort -V -C || { echo "$f declares go $v"; exit 1; }
done

Security implications

The bug sent one customer's invoice details 380 times to that same customer, so there was no cross-customer disclosure in this instance. The same pattern in a loop that builds per-tenant requests could easily send tenant A's data to tenant B's endpoint, or apply one user's permissions to another's request. The data race itself is also undefined behaviour in Go's memory model for non-word-sized values, so a string or struct read during a write can be torn. Treat any race on request data as a potential confidentiality issue.

Prevention checklist

• Keep the go line in every go.mod at 1.22 or later and review it during toolchain upgrades.
• Run full go vet and go test -race in CI, not just go test.
• When copying concurrent code between repositories, compare the go.mod language versions.
• Use bisect -compile=loopvar before raising the go line on an old module.
• Assert "each item processed exactly once" in tests for any fan-out code.

References

• Go blog, Fixing For Loops in Go 1.22 — https://go.dev/blog/loopvar-preview
• Go 1.22 release notes — https://go.dev/doc/go1.22
• Go wiki, LoopvarExperiment — https://go.dev/wiki/LoopvarExperiment
• Go modules reference, go directive — https://go.dev/ref/mod#go-mod-file-go
• Go documentation, Data Race Detector — https://go.dev/doc/articles/race_detector