Deconstructing Go's magical defer
This post is targetted at Go beginners, but can serve as a good refresher to more experienced gophers.
When starting out, defer feels magical, doesn’t it? Guess what: it’s not. In this blog post, we will let go of the magic to have a basic understanding of what is happening under the hood when using the defer statement.
What is defer?
The defer statement allows you to register some code to be executed when the current function returns. One obvious use of this is to close resources:
func somefunc() error {
f, err := os.Open("somefile.txt")
if err != nil {
return fmt.Errorf("could not open file: %w", err)
}
defer f.Close()
// ...do something with f
_ = f
return nil
}
Here, we are opening a file for reading. If there were no errors, it means that we have a file handle in f, and we want to make sure it is closed. To do that, we just defer f.Close() so that we are sure it will get executed when the function returns, and our mind is now free to focus on the rest of the function. If there are multiple return paths, we don’t have to worry about closing the file handle in each of them, we just defer the call and it will be closed no matter what. The next person (or the future you) modifying this function will not need to worry about what to do on returns, it’s already taken care of.
This is a very, very common pattern in Go: from draining http bodies to unlocking mutexes, you see defer everywhere.
defer’s evaluation time
Let’s start with the elephant in the room: evaluation time.
Consider the following:
func somefunc() int {
var birdsPerSquareMeter int
defer fmt.Printf("birdsIndicator=%d\n", birdsPerSquareMeter)
birdsPerSquareMeter = 42
return birdsPerSquareMeter
}
The function above first defines the integer variable birdsPerSquareMeter, then defers printing it, assigns it a value, and returns it. One could think that as the defered statement is, well, deferred to the end of the function, it will print birdsIndicator=42. But it does not, here’s why:
Each time a “defer” statement executes, the function value and parameters to the call are evaluated as usual and saved anew but the actual function is not invoked.
— Go’s Holy Spec 1
Under the hood, deferred function calls are pushed onto a local defer stack. When the defer statement is made, the call stack is prepared, including the function parameters. When the surrounding function is ready to return, deferred functions are popped and executed one by one out of the defer stack, in a last-in-first-out (LIFO) order. This means they are executed in the reverse order they were deferred.
These deferred functions are actually closures executed in the context of the sourrounding function. What does this mean? It means that they have access to the surrounding function’s variables, even if they are not passed as parameters:
func somefunc() int {
var birdsPerSquareMeter int
defer func() {
fmt.Printf("birdsIndicator=%d\n", birdsPerSquareMeter)
}()
birdsPerSquareMeter = 42
return birdsPerSquareMeter
}
Here, we are defering an anonymous function that has access to the birdsPerSquareMeter variable. As birdsPerSquareMeter is not passed as a parameter, it is captured by the closure, and the above will print birdsIndicator=42.
The crux of the matter is that the function call is prepared when the defer statement is made, not when it is executed. Just like a normal function call, if parameters are passed by value, they are copied, and are pushed onto the defer stack. You can think of these as snapshots of the variables at the time of the defer statement. In other words, the execution of the function is deferred, not its evaluation.
defer’s execution time
Now that we know when the function call is prepared, let’s see when it is executed. As we said earlier, deferred functions are executed in a LIFO order. This means that the last function to be deferred will be the first to be executed.
var mu sync.RWMutex
func readstuff() error {
mu.RLock()
defer mu.RUnlock()
f, err := os.Open("somefile.txt")
if err != nil {
return fmt.Errorf("could not open file for reading: %w", err)
}
defer f.Close()
// ...do something with f
_ = f
return nil
}
func writestuff() error {
mu.Lock()
defer mu.Unlock()
f, err := os.OpenFile("somefile.txt", os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
if err != nil {
return fmt.Errorf("could not open file for writing: %w", err)
}
defer f.Close()
_, err = f.WriteString("hello world\n")
if err != nil {
return fmt.Errorf("could not write to file: %w", err)
}
return nil
}
Here, there are two functions: one is reading a file, the other is writing to it. A mutex is used to prevent reading and writing at the same time. The order in which the defer statements are declared is important: we want to make sure the file is closed before the lock is released, or else we might start reading while the file is being written to, or the other way around. This happens naturally, as the defer statements are pushed in the order they are written, and executed in the reverse order.
[…] deferred functions are invoked immediately before the surrounding function returns, in the reverse order they were deferred. That is, if the surrounding function returns through an explicit return statement, deferred functions are executed after any result parameters are set by that return statement but before the function returns to its caller.
— Go’s Holy Spec 1
The defer stack is kind of an ad-hoc mechanism, a post-processing step. After the surrounding function hits a return statement, but before control flow is returned to the caller, deferred functions are executed one by one. This is an interstice we’re not used to think of in other languages. Another property of this is that deferred functions are executed even if the surrounding function panics:
func somefunc() {
defer fmt.Println("this will get printed before the panic")
panic("the earth is flat")
}
func someotherfunc() {
defer fmt.Println("this will get printed before exiting")
os.Exit(1)
}
Another interesting property is that deferred functions not only have access to the surrounding function’s variables, but can also modify them, including named return parameters:
[…] if the deferred function is a function literal and the surrounding function has named result parameters that are in scope within the literal, the deferred function may access and modify the result parameters before they are returned.
— Go’s Holy Spec 1
func somefunc() (err error) {
defer func() {
if err != nil {
err = fmt.Errorf("somefunc failed: %w", err)
}
}()
return errors.New("the earth is flat")
}
Adding this kind of side-effects to deferred functions is generally not a great idea, but it is possible. It generally makes the code harder to understand, adding a layer of indirection that is not obvious at first glance.
Gotcha: deferring in loops
This is a very common gotcha: you inline a function in a loop, and you don’t pay attention to defer statements. You end up with defer statements in the loop, but these won’t get executed until the surrounding function returns, and they start piling up on the defer stack. This can easily derail into a serious memory leak! Just as returns should become breaks or continues, defers should be taken care of.
Panic and recover
Panics can be recovered in a defer. This is generally a bad idea: panics are meant to be used in exceptional situations, and you generally want to let them bubble up to the top of the call stack. But there are legitimate use cases for this, like when using a 3rd party library a bit too keen on panicking. In this case, you can use recover to catch the panic and handle it gracefully:
func somefunc() (err error) {
defer func() {
if r := recover(); r != nil {
err = fmt.Errorf("somefunc panicked: %v", r)
}
}()
stressedfunc(-42)
return nil
}
func stressedfunc(n int) {
if n < 0 {
panic("n should be positive!!1")
}
}
Conclusion
Understanding how defer works is crucial to avoid misusing it. Here I tried to demystify it, but this post is far from complete. I hope I can get time to expand on how the defer stack is made, how it evolved over time, and performance implications. I hope this was helpful, now go have fun!