Background
Although the restrictions on type-safe pointers really make us be able to write safe Go code with ease, they also make some obstacles to write efficient code for some scenarios.
In fact, Go also supports type-unsafe pointers, which are pointers without the restrictions made for safe pointers. Type-unsafe pointers are also called unsafe pointers in Go. Go unsafe pointers are much like C pointers, they are powerful, and also dangerous. For some cases, we can write more efficient code with the help of unsafe pointers. On the other hand, by using unsafe pointers, it is easy to write bad code which is too subtle to detect in time.
If you really desire the code efficient improvements by using unsafe pointers for any reason, you should not only know the above mentioned risks, but also follow the instructions written in the official Go documentation and clearly understand the effect of each unsafe pointer use, so that you can write safe Go code with unsafe pointers.
About the unsafe Standard Package
Go provides a special kind of types for unsafe pointers. We must import the unsafe standard package to use unsafe pointers. The unsafe.Pointer type is defined as
type Pointer *ArbitraryType
Surely, it is not a usual type definition. Here the ArbitraryType just hints that a unsafe.Pointer value can be converted to any safe pointer values in Go (and vice versa). In other words, unsafe.Pointer is like the void* in C language.
Go unsafe pointers mean the types whose underlying types are unsafe.Pointer.
The zero values of unsafe pointers are also represented with the predeclared identifier nil.
Before Go 1.17, the unsafe standard package has already provided three functions.
func Alignof(variable ArbitraryType) uintptr, which is used to get the address alignment of a value. Please notes, the aligns for struct-field values and non-field values of the same type may be different, though for the standard Go compiler, they are always the same. For the gccgo compiler, they may be different.func Offsetof(selector ArbitraryType) uintptr, which is used to get the address offset of a field in a struct value. The offset is relative to the address of the struct value. The return results should be always the same for the same corresponding field of values of the same struct type in the same program.func Sizeof(variable ArbitraryType) uintptr, which is used to get the size of a value (a.k.a., the size of the type of the value). The return results should be always the same for all values of the same type in the same program.
Some Facts in Go We Should Know
Before introducing the valid unsafe pointer use patterns, we need to know some facts in Go.
Fact 1: unsafe pointers are pointers and uintptr values are integers
Each of non-nil safe and unsafe pointers references another value. However uintptr values don’t reference any values, they are just plain integers, though often each of them stores an integer which can be used to represent a memory address.
Go is a language supporting automatic garbage collection. When a Go program is running, Go runtime will check which memory blocks are not used by any value any more and collect the memory allocated for these unused blocks, from time to time. Pointers play an important role in the check process. If a memory block is unreachable from (referenced by) any values still in use, then Go runtime thinks it is an unused value and it can be safely garbage collected.
As uintptr values are integers, they can participate arithmetic operations.
The example in the next subsection shows the differences between pointers and uintptr values.
Fact 2: unused memory blocks may be collected at any time
At run time, the garbage collector may run at an uncertain time, and each garbage collection process may last an uncertain duration. So when a memory block becomes unused, it may be collected at an uncertain time.
For example:
import "unsafe"
// Assume createInt will not be inlined.
func createInt() *int {
return new(int)
}
func foo() {
p0, y, z := createInt(), createInt(), createInt()
var p1 = unsafe.Pointer(y)
var p2 = uintptr(unsafe.Pointer(z))
// At the time, even if the address of the int
// value referenced by z is still stored in p2,
// the int value has already become unused, so
// garbage collector can collect the memory
// allocated for it now. On the other hand, the
// int values referenced by p0 and p1 are still
// in use.
// uintptr can participate arithmetic operations.
p2 += 2; p2--; p2--
*p0 = 1 // okay
*(*int)(p1) = 2 // okay
*(*int)(unsafe.Pointer(p2)) = 3 // dangerous!
}
In the above example, the fact that value p2 is still in use can’t guarantee that the memory block ever hosting the int value referenced by z has not been garbage collected yet. In other words, when *(*int)(unsafe.Pointer(p2)) = 3 is executed, the memory block may be collected, or not. It is dangerous to dereference the address stored in value p2 to an int value, for it is possible that the memory block has been already reallocated for another value (even for another program).
How to Use Unsafe Pointers Correctly?
Pattern 1: convert a *T1 value to unsafe Pointer, then convert the unsafe pointer value to *T2.
As mentioned above, by using the unsafe pointer conversion rules above, we can convert a value of *T1 to type *T2, where T1 and T2 are two arbitrary types. However, we should only do such conversions if the size of T1 is no smaller than T2, and only if the conversions are meaningful.
As a result, we can also achieve the conversions between type T1 and T2 by using this pattern.
One example is the math.Float64bits function, which converts a float64 value to an uint64 value, without changing any bit in the float64 value. The math.Float64frombits function does reverse conversions.
func Float64bits(f float64) uint64 {
return *(*uint64)(unsafe.Pointer(&f))
}
func Float64frombits(b uint64) float64 {
return *(*float64)(unsafe.Pointer(&b))
}
Please note, the return result of the math.Float64bits(aFloat64) function call is different from the result of the explicit conversion uint64(aFloat64).
In the following example, we use this pattern to convert a []MyString slice to type []string, and vice versa. The result slice and the original slice share the underlying elements. Such conversions are impossible through safe ways,
package main
import (
"fmt"
"unsafe"
)
func main() {
type MyString string
ms := []MyString{"C", "C++", "Go"}
fmt.Printf("%s\n", ms) // [C C++ Go]
// ss := ([]string)(ms) // compiling error
ss := *(*[]string)(unsafe.Pointer(&ms))
ss[1] = "Rust"
fmt.Printf("%s\n", ms) // [C Rust Go]
// ms = []MyString(ss) // compiling error
ms = *(*[]MyString)(unsafe.Pointer(&ss))
}
Note: since Go 1.17, we may also use unsafe.Slice((*string)(&ms[0]), len(ms)) to do the conversion.
A practice by using the pattern is to convert a byte slice, which will not be used after the conversion, to a string, as the following code shows. In this conversion, a duplication of the underlying byte sequence is avoided.
func ByteSlice2String(bs []byte) string {
return *(*string)(unsafe.Pointer(&bs))
}
This is the implementation adopted by the String method of the Builder type supported since Go 1.10 in the strings standard package. The size of a byte slice is larger than a string, and their internal structures are similar, so the conversion is valid (for main stream Go compilers). However, despite the implementation may be safely used in standard packages now, it is not recommended to be used in general user code. In user code, we should try to use another implementation provided at the end of the current article.
The converse, converting a string to a byte slice in the similar way, is invalid, for the size of a string is smaller than a byte slice.
func String2ByteSlice(s string) []byte {
return *(*[]byte)(unsafe.Pointer(&s)) // dangerous!
}
In the pattern 6 section below, a valid implementation to do the same job is introduced.
Note: when using the just introduced unsafe way to convert a byte slice to a string, please make sure not to modify the bytes in the byte slice if the result string still survives.
Pattern 2: convert unsafe pointer to uintptr, then use the uintptr value.
This pattern is not very useful. Usually, we print the result uintptr values to check the memory addresses stored in them. However, there are other both safe and less verbose ways to this job. So this pattern is not much useful.
Example:
package main
import "fmt"
import "unsafe"
func main() {
type T struct{a int}
var t T
fmt.Printf("%p\n", &t) // 0xc6233120a8
println(&t) // 0xc6233120a8
fmt.Printf("%x\n", uintptr(unsafe.Pointer(&t))) // c6233120a8
}
Pattern 3: convert unsafe pointer to uintptr, do arithmetic operations with the uintptr value, then convert it back
In this pattern, the result unsafe pointer must continue to point into the original allocated memory block. For example:
package main
import "fmt"
import "unsafe"
type T struct {
x bool
y [3]int16
}
const N = unsafe.Offsetof(T{}.y)
const M = unsafe.Sizeof(T{}.y[0])
func main() {
t := T{y: [3]int16{123, 456, 789}}
p := unsafe.Pointer(&t)
// "uintptr(p)+N+M+M" is the address of t.y[2].
ty2 := (*int16)(unsafe.Pointer(uintptr(p)+N+M+M))
fmt.Println(*ty2) // 789
}
In fact, since Go 1.17, it is more recommended to use the above introduced unsafe.Add function to do such address offset operations.
Please note, in this specified example, the conversion unsafe.Pointer(uintptr(p) + N + M + M) shouldn’t be split into two lines, like the following code shows. Please read the comments in the code for the reason.
func main() {
t := T{y: [3]int16{123, 456, 789}}
p := unsafe.Pointer(&t)
// ty2 := (*int16)(unsafe.Pointer(uintptr(p)+N+M+M))
addr := uintptr(p) + N + M + M
// ... (some other operations)
// Now the t value becomes unused, its memory may be
// garbage collected at this time. So the following
// use of the address of t.y[2] may become invalid
// and dangerous!
// Another potential danger is, if some operations
// make the stack grow or shrink here, then the
// address of t might change, so that the address
// saved in addr will become invalid (fact 3).
ty2 := (*int16)(unsafe.Pointer(addr))
fmt.Println(*ty2)
}
Such bugs are very subtle and hard to detect, which is why the uses of unsafe pointers are dangerous.
The intermediate uintptr value may also participate in &^ bitwise clear operations to do address alignment, as long as the result unsafe pointer and the original one point into the same allocated memory block.
Pattern 4: convert unsafe pointers to uintptr values as arguments of syscall.Syscall calls.
From the explanations for the last pattern, we know that the following function is dangerous.
// Assume this function will not inlined.
func DoSomething(addr uintptr) {
// read or write values at the passed address ...
}
The reason why the above function is dangerous is that the function itself can’t guarantee the memory block at the passed argument address is not garbage collected yet. If the memory block is collected or is reallocated for other values, then the operations made in the function body are dangerous.
However, the prototype of the Syscall function in the syscall standard package is as
func Syscall(trap, a1, a2, a3 uintptr) (r1, r2 uintptr, err Errno)
How does this function guarantee that the memory blocks at the passed addresses a1, a2 and a3 are still not garbage collected yet within the function internal? The function can’t guarantee this. In fact, compilers will make the guarantee. It is the privilege of calls to syscall.Syscall alike functions.
We can think that, compilers will automatically insert some instructions for each of the unsafe pointer arguments who are converted to uintptr, like the third argument in the following syscall.Syscall call, to prevent the memory block referenced by that argument from being garbage collected or moved.
Please note that, before Go 1.15, it was okay the conversion expressions uintptr(anUnsafePointer) act as sub-expressions of the talked arguments. Since Go 1.15, the requirement becomes a bit stricter: the talked arguments must present exactly as the uintptr(anUnsafePointer) form.
The following call is safe:
syscall.Syscall(SYS_READ, uintptr(fd),
uintptr(unsafe.Pointer(p)), uintptr(n))
But the following calls are dangerous:
u := uintptr(unsafe.Pointer(p))
// At this time, the value referenced by p might
// have become unused and been collected already,
// or the address of the value has changed.
syscall.Syscall(SYS_READ, uintptr(fd), u, uintptr(n))
// Arguments must be in the "uintptr(anUnsafePointer)"
// form. In fact, the call was safe before Go 1.15.
// But Go 1.15 changes the rule a bit.
syscall.Syscall(SYS_XXX, uintptr(uintptr(fd)),
uint(uintptr(unsafe.Pointer(p))), uintptr(n))
Again, never use this pattern when calling other functions.
Demo
An example of using the three functions.
package main
import "fmt"
import "unsafe"
func main() {
var x struct {
a int64
b bool
c string
}
const M, N = unsafe.Sizeof(x.c), unsafe.Sizeof(x)
fmt.Println(M, N) // 16 32
fmt.Println(unsafe.Alignof(x.a)) // 8
fmt.Println(unsafe.Alignof(x.b)) // 1
fmt.Println(unsafe.Alignof(x.c)) // 8
fmt.Println(unsafe.Offsetof(x.a)) // 0
fmt.Println(unsafe.Offsetof(x.b)) // 8
fmt.Println(unsafe.Offsetof(x.c)) // 16
}
An example which demonstrates the last note mentioned above.
package main
import "fmt"
import "unsafe"
func main() {
type T struct {
c string
}
type S struct {
b bool
}
var x struct {
a int64
*S
T
}
fmt.Println(unsafe.Offsetof(x.a)) // 0
fmt.Println(unsafe.Offsetof(x.S)) // 8
fmt.Println(unsafe.Offsetof(x.T)) // 16
// This line compiles, for c can be reached
// without implicit pointer indirections.
fmt.Println(unsafe.Offsetof(x.c)) // 16
// This line doesn't compile, for b must be
// reached with the implicit pointer field S.
//fmt.Println(unsafe.Offsetof(x.b)) // error
// This line compiles. However, it prints
// the offset of field b in the value x.S.
fmt.Println(unsafe.Offsetof(x.S.b)) // 0
}
Please note, the print results shown in the comments are for the standard Go compiler version 1.16 on Linux AMD64 architecture.