Curiosities of the Go testing package

When I learn a new programming language (or any big technical topic), I tend to defer questions about the basics, assuming the answers will be obvious later on. I remember having questions about how go test and the testing package worked, but I hadn't yet covered advanced parts of the language like channels and goroutines, so I figured testing would make sense later.

Unfortunately, that wasn't the case. go test and the testing package have a pretty unique way of doing things. Their implementations are clever but far from obvious, and the answers to my questions weren't clear until I started working on the testing package itself.

How are test functions discovered?

This is the first question I had. A test function accepts a *testing.T parameter and has a name that starts with Test. How are those functions discovered and called? I initially assumed there was some reflection-based system like you might find in a dynamically typed langauge, but the reflect package doesn't have a way to list all the functions in a package.

func TestHowDoesThisGetCalled(t *testing.T) {
  // ???
}

The answer is that go test parses the test source code, then generates a list of functions to call. For each package named on the command line, go test builds a test executable from three packages and their imports:

• An external test package from _test.go files whose package name ends with _test.
• An internal test package from other _test.go files, combined with the library being tested.
• A generated test main package.

The code generation is the magic part. go test uses a template for generating this file. The generated test main file imports the internal and external packages and has static lists of tests, benchmarks, fuzz targets, and examples from both. The main function creates a testing.M object with those lists using testing.MainStart, then calls either the TestMain function or testing.M.Run.

You can see the generated test main package by running go list -test -compiled on a given package. The -compiled flag shows files passed to the Go compiler; it's not on by default since go list may need to run commands like compiling cgo code. The test main package will have the name main and the ImportPath suffix .test. You'll need to use the -json or -f flags to see its source file path, which will be in the build cache.

$ go list -test -compiled -f '{{if eq .Name "main"}}{{.GoFiles}}{{end}}' fmt
[/home/jay/.cache/go-build/7e/
7efecbd24a52650ba294003161ea404795f980135d525445bfb13d4df1e6d97a-d]

If you're trying to replicate go test in another build system, unfortunately this code generation is internal to the go command. Bazel's rules_go has its own utility for generating similar code. Incidentally, this is also where code coverage instrumentation is registered.

How does T.Fatal work?

This is the part I was most curious about early on. T.Fatal doesn't return, but it does execute deferred calls and cleanups. That sounds like a panic, but if you call recover, it returns nil and the stack keeps unwinding. What's up with that?

Reading the docs, we find that T.Fatal and friends call runtime.Goexit which has exactly the behavior described: it terminates the caller's goroutine. It calls deferred functions but it cannot be recovered from.

So this question has a simple answer, but I found it surprising. Goexit is a third way to leave a function, other than returning or panicking. I thought about using this in HTTP handlers because I sometimes forget to return after writing out an error. But the Handler documentation explicitly says that ServeHTTP should return. Perhaps that's for the best. Goexit is unusual and should be avoided in code that anyone else might read.

How about T.Run and T.Parallel?

T.Run seems like it would be straightforward: just adjust some bookkeeping info then call the function, right?. But how does T.Parallel work? How can you enable parallelism with a function call?

The testing package runs each test (and sub-test) in its own goroutine. When you call T.Run, it creates a new T, starts a new goroutine, and calls the sub-test's function there. Meanwhile, T.Run blocks the parent goroutine by receiving on a channel called signal owned by the child T. When the sub-test finishes, it sends a value on signal, unblocking its parent. So tests run sequentially, even though each test is in a separate goroutine.

T.Parallel makes things more complicated. You can't know ahead of time which tests will call T.Parallel, so you need to run tests sequentially first. When a test calls T.Parallel, it sends a value on signal (unblocking its parent) then receives on another channel named barrier owned by its parent T. When a test function returns, if it spawned any parallel subtests, the wrapper code that calls the test function (within its goroutine) closes barrier, unblocking all parallel sub-tests. The calling code then waits for each parallel sub-test to finish by receiving a second value on signal. After all the parallel sub-tests are finished, the calling code runs cleanup functions and sends on signal to its parent.

I hope this explanation is not too murky. This has an important consequence: sub-tests of one test are not run concurrently with sub-tests of another test unless both parent tests call T.Parallel before spawning sub-tests.

If you'd like to read through the code, T.Run and T.Parallel are relatively straightforward. The gnarly part is in tRunner, the wrapper at the top of each test's goroutine. That's the code that actually calls each test function and handles panics and cleanups. You might want to start with the Go 1.13 version of tRunner since it was simpler before cleanup functions were supported.