Permanent link to this article– https://ift.tt/z7MQgqb
In Go language development, thePackage is the basic unit of code organization., is also the basic building unit.The Go compiler builds each package into a target file (.a), which is then linked together by the linker to form the final executable program.
Although the process of building Go packages may seem simple, it actually contains many details that are worth delving into. For example, when we execute the go build command, how does the Go compiler choose which source files to compile? You might answer, “Isn’t it selected by the ARCH and OS identifiers in the filename and the build constraints?” While your answer is not wrong, would you still be able to give a definitive answer if I further ask the following question?
Suppose a Go source file uses the following build constraints:
//go:build unix
package foo
// ... ...
Is this file compiled when executing GOOS=android go build? What if the GOOS=aix go build is executed? And what operating systems are included in “unix”?
Taking this a step further, when a source file has ARCH and OS identifiers in its filename, and build constraints are also used in the file contents, how does the Go compiler prioritize this information?
Even seasoned Go experts may only be able to give vague answers to the above details of file selection involved in the package building process.
In real-world development, we often need to write code specific to different operating systems and architectures, which means flexibility and complexity go hand in hand. go’s build constraints and filename conventions provide flexibility, but they also bring additional complexity. Understanding these rules not only helps optimize the build process, but also avoids potential errors and unnecessary headaches.
In this post, I’m going to discuss theDetails of source file selection during Go package buildsIncluding the roles of ARCH and os identification conventions and build constraints in filenames, as well as the prioritization of the two to deal with the problem. It is hoped that this content will help developers better grasp the build mechanism of the Go language, thus improving development efficiency.
To better illustrate the file selection logic of Go package builds, let’s start with some “appearances” of Go package builds.
Note: In this article, we will use theGo 1.17The new build constraints style is introduced: //go:build, the previous // +build aix darwin dragonfly freebsd js,wasm … style is no longer recommended. If you want to get a full understanding of the old build constraints style to compare it with the new one, I recommend reading myThe Road to Go Language Savvy: Programming Ideas, Methods, and Techniques from Novice to Expert, Book 2。
1. Representation
In a Go project, usually a directory corresponds to a Go package, and under each Go package, there can be multiple Go source files with .go suffix, and these source files can only have a unique package name (except for the test source files). Take the standard library fmt package as an example, and the list of source files under its directory is as follows (in terms ofGo 1.23.0source code as an example):
$ls $GOROOT/src/fmt
doc.go export_test.go print.go stringer_example_test.go
errors.go fmt_test.go scan.go stringer_test.go
errors_test.go format.go scan_test.go
example_test.go gostringer_example_test.go state_test.go
Which of these files end up in the target file (fmt.a) of the fmt package?Thoughtful Go ToolchainProvides us with a way to view it:
$go list -f '' fmt
[doc.go errors.go format.go print.go scan.go]
For fmt packages that are independent of the target ARCH and OS, the selection of their Go source files seems to be a bit simpler. We see that all files except the package test file (xxx_test.go) are compiled into the final fmt package.
Let’s look at another net package that has a high relevance to the target ARCH and OS. Excluding subdirectories, the number of Go source files in this package directory is about 220 or so, but in themacOS/amd64Down through the go list to see the files that end up in the net package target file, there are only about a couple dozen:
$go list -f '' net
[addrselect.go cgo_darwin.go cgo_unix.go cgo_unix_syscall.go conf.go dial.go dnsclient.go dnsclient_unix.go dnsconfig.go dnsconfig_unix.go error_posix.go error_unix.go fd_posix.go fd_unix.go file.go file_unix.go hook.go hook_unix.go hosts.go interface.go interface_bsd.go interface_darwin.go ip.go iprawsock.go iprawsock_posix.go ipsock.go ipsock_posix.go lookup.go lookup_unix.go mac.go mptcpsock_stub.go net.go netcgo_off.go netgo_off.go nss.go parse.go pipe.go port.go port_unix.go rawconn.go rlimit_unix.go sendfile_unix_alt.go sock_bsd.go sock_posix.go sockaddr_posix.go sockopt_bsd.go sockopt_posix.go sockoptip_bsdvar.go sockoptip_posix.go splice_stub.go sys_cloexec.go tcpsock.go tcpsock_posix.go tcpsock_unix.go tcpsockopt_darwin.go tcpsockopt_posix.go udpsock.go udpsock_posix.go unixsock.go unixsock_posix.go unixsock_readmsg_cloexec.go writev_unix.go]
Next, let’s step outside the Go standard library and look at a customized example:
$tree -F buildconstraints/demo1
buildconstraints/demo1
├── foo/
│ ├── f1_android.go
│ ├── f2_linux.go
│ └── f3_darwin.go
└── go.mod
// buildconstraints/demo1/foo/f1_android.go
//go:build linux
package foo
func F1() {
}
// buildconstraints/demo1/foo/f2_linux.go
//go:build android
package foo
func F2() {
}
// buildconstraints/demo1/foo/f3_darwin.go
//go:build android
package foo
func F3() {
}
Build the package buildconstraints/demo1/foo under GOOS=android, which files will be selected, see the output below:
$GOOS=android go list -f '' github.com/bigwhite/demo1/foo
[f1_android.go f2_linux.go]
If the first two examples are easy to understand, the third example is likely to make many developers feel a little “confused”. Don’t worry, the above three examples are just scratching the surface. Next, let’s take a closer look at the file selection mechanism in Go builds.
2. Documentation selection mechanisms
Go package build when selecting the source file mechanism is still quite cumbersome, we need to start from the source code to sort out the main logic, in Go version 1.23, Go package build process source file selection logic code is located in \$GOROOT/src/go/build/build.go, this source file has more than 2k lines, but don’t worry, I’m here to help you put the main call logic for you in the following diagram:
The Import function calls Default.Import to get the package details, which are represented by the build.Package structure:
// $GOROOT/src/go/build/build.go
// A Package describes the Go package found in a directory.
type Package struct {
Dir string // directory containing package sources
Name string // package name
ImportComment string // path in import comment on package statement
Doc string // documentation synopsis
ImportPath string // import path of package ("" if unknown)
Root string // root of Go tree where this package lives
SrcRoot string // package source root directory ("" if unknown)
PkgRoot string // package install root directory ("" if unknown)
PkgTargetRoot string // architecture dependent install root directory ("" if unknown)
BinDir string // command install directory ("" if unknown)
Goroot bool // package found in Go root
PkgObj string // installed .a file
AllTags []string // tags that can influence file selection in this directory
ConflictDir string // this directory shadows Dir in $GOPATH
BinaryOnly bool // cannot be rebuilt from source (has //go:binary-only-package comment)
// Source files
GoFiles []string // .go source files (excluding CgoFiles, TestGoFiles, XTestGoFiles)
... ...
where GoFiles is the list of source files involved in the compilation of the Go package.
Default is the default context information, including the values of several environment variables in the default goenv, such as GOARCH, GOOS, etc., that are required for the build:
// Default is the default Context for builds.
// It uses the GOARCH, GOOS, GOROOT, and GOPATH environment variables
// if set, or else the compiled code's GOARCH, GOOS, and GOROOT.
var Default Context = defaultContext()
Context’s Import method has a lot of lines of code, and for us who want to understand the details of file selection, one of the most important calls is Context’s matchFile method.
The matchFile is exactly that.Methods used to determine whether a Go source file should be selected for inclusion in the final package file. Its internal logic can be divided into two main steps.
The first step isCalling the Context’s goodOSArchFile method on the name of the Go source file determines, the determination of the goodOSArchFile method also has two sub-steps:
- Determine if the OS and ARCH in the name are in the list of OS and ARCH supported by Go
The current OS and ARCH supported by Go are defined in the syslist.go file:
// $GOROOT/src/go/build/syslist.go
// knownArch is the list of past, present, and future known GOARCH values.
// Do not remove from this list, as it is used for filename matching.
var knownArch = map[string]bool{
"386": true,
"amd64": true,
"amd64p32": true,
"arm": true,
"armbe": true,
"arm64": true,
"arm64be": true,
"loong64": true,
"mips": true,
"mipsle": true,
"mips64": true,
"mips64le": true,
"mips64p32": true,
"mips64p32le": true,
"ppc": true,
"ppc64": true,
"ppc64le": true,
"riscv": true,
"riscv64": true,
"s390": true,
"s390x": true,
"sparc": true,
"sparc64": true,
"wasm": true,
}
// knownOS is the list of past, present, and future known GOOS values.
// Do not remove from this list, as it is used for filename matching.
// If you add an entry to this list, look at unixOS, below.
var knownOS = map[string]bool{
"aix": true,
"android": true,
"darwin": true,
"dragonfly": true,
"freebsd": true,
"hurd": true,
"illumos": true,
"ios": true,
"js": true,
"linux": true,
"nacl": true,
"netbsd": true,
"openbsd": true,
"plan9": true,
"solaris": true,
"wasip1": true,
"windows": true,
"zos": true,
}
We can also view it with the following command:
$go tool dist list
aix/ppc64
android/386
android/amd64
android/arm
android/arm64
darwin/amd64
darwin/arm64
dragonfly/amd64
freebsd/386
freebsd/amd64
freebsd/arm
freebsd/arm64
freebsd/riscv64
illumos/amd64
ios/amd64
ios/arm64
js/wasm
linux/386
linux/amd64
linux/arm
linux/arm64
linux/loong64
linux/mips
linux/mips64
linux/mips64le
linux/mipsle
linux/ppc64
linux/ppc64le
linux/riscv64
linux/s390x
netbsd/386
netbsd/amd64
netbsd/arm
netbsd/arm64
openbsd/386
openbsd/amd64
openbsd/arm
openbsd/arm64
openbsd/ppc64
openbsd/riscv64
plan9/386
plan9/amd64
plan9/arm
solaris/amd64
wasip1/wasm
windows/386
windows/amd64
windows/arm
windows/arm64
Note: Go source files like sock_bsd.go and sock_posix.go are actually just plain old Go source files, even though their filenames contain the words posix and bsd. Their filenames themselves do not affect the results of the Go package’s file selection at build time.
- Call matchTag to determine if the OS and ARCH in the name of this Go source file match the OS and ARCH in the current context message
Go supports the following format for source file name composition:
// name_$(GOOS).*
// name_$(GOARCH).*
// name_$(GOOS)_$(GOARCH).*
// name_$(GOOS)_test.*
// name_$(GOARCH)_test.*
// name_$(GOOS)_$(GOARCH)_test.*
But here are three exceptions, viz:
If GOOS=android in the context, then a Go source file with an OS value of linux in the file name is considered a match;
If GOOS=illumos in the context, then a Go source file with an OS value of solaris in the file name is considered a match;
If GOOS=ios in the context, then a Go source file with an OS value of darwin in the file name is considered a match.
There is also a special treatment that the source file can match the value of GOOS in the following context when the OS value in the file name is unix:
// $GOROOT/src/go/build/syslist.go
// unixOS is the set of GOOS values matched by the "unix" build tag.
// This is not used for filename matching.
// This list also appears in cmd/dist/build.go and
// cmd/go/internal/imports/build.go.
var unixOS = map[string]bool{
"aix": true,
"android": true,
"darwin": true,
"dragonfly": true,
"freebsd": true,
"hurd": true,
"illumos": true,
"ios": true,
"linux": true,
"netbsd": true,
"openbsd": true,
"solaris": true,
}
The os listed here are all so-called “Unix-like” operating systems.
If the goodOSArchFile method returns a successful filename match, then theThe second step is to call the Context’s shouldBuild method to determine the build constraints in the Go source fileThe decision process is also done by calling matchTag, so the rules are the same as described above for matchTag. If the match is successful, the source file will be compiled by the Go compiler into the final Go package target file.
Let’s take a look at the customized example in the first section of the article, “Appearances”, to determine why we ended up with that result.
3. Analysis of examples
There are three Go source files in the buildconstraints/demo1/foo package directory:
$tree -F foo
foo
├── f1_android.go
├── f2_linux.go
└── f3_darwin.go
Note: Currently my system isdarwin/amd64, but we used the GOOS=android environment variable. Let’s go through each of the three files, following the main logic of file selection determination that we sorted out in the previous section.
- f1_android.go
First use goodOSArchFile to determine if the file name matches. When GOOS=android, the os in the file name is android, and the file name matches successfully.
Then use shouldBuild to determine if the build constraints in the file match. The file’s constraints are linux, and as mentioned in the three exceptions to the matchTag rule above, when GOOS=android, it is possible to match if the build constraints are linux.
As a result, f1_android.go will appear in the list of final compiled files.
- f2_linux.go
First use goodOSArchFile to determine if the filename matches. When GOOS=android, the os in the filename is linux, linux is obviously in the list of os supported by go, and according to the exception rule of matchTag, when GOOS=android, the os in the filename is linux is matchable.
Then use shouldBuild to determine if the build constraints in the file match. The constraints in the file are android, which is the same as GOOS, and can be matched.
As a result, f2_linux.go will appear in the list of final compiled files.
- f3_darwin.go
First use goodOSArchFile to determine if the filename matches. When GOOS=android, the os in the filename is darwin, and although darwin is in the list of os supported by go, darwin doesn’t match GOOS=android, so in the goodOSArchFile step, f3_darwin.go is “eliminated! “out”! Even if the build constraints in f3_darwin.go are android.
As a result, f3_darwin.go will not appear in the list of final compiled files.
If you add another source file f4_unix.go with the contents:
//go:build android
func F4() {
}
Will this f4_unix.go appear in the final list of package compiled files? This is left as a food for thought, and you’re welcome to leave a comment in the comments section with the results of your thinking.
4. Summary
In the Go language development process, package building is one of the core aspects, and the selection of source files is a complex and critical detail in the build process. This article discusses how the Go compiler selects the source files to be compiled based on the ARCH and OS identifiers in the filenames and the build constraints when executing the go build command.
Through concrete examples, this article shows how different filenames and build constraints affect the final compilation results, and reveals the Go compiler’s priorities for processing this information. Understanding these internal mechanisms can not only help developers optimize the build process, but also effectively avoid potential errors. I hope the analysis in this article can bring you help.
Note: limited space, this article is only for the most complex part of the package compilation file to explore the choice, and like ReleaseTags (eg: go1.21, etc.), cgo, _test.go suffix and other more obvious constraints are not involved, at the same time, for the new version of the build constraints of the operator combinations have not been mentioned, interested in children’s shoes can be referred to!go build constraintsOfficial Documentation Access.
The source code covered in this article can be found athere areDownload.
5. References
- Go build constraints– https://ift.tt/xpjrgf4
- proposal: cmd/go: allow && and || operators and parentheses in build tags– https://ift.tt/DxEPQL3
- Bug-resistant build constraints — Draft Design– https://ift.tt/tNealGo
- cmd/go: continue conversion to bug-resistant //go:build constraints– https://ift.tt/gFYuf0S
- Go 1.17 release notes– https://ift.tt/2HU9W4x
- cmd/go: provide build tags for architecture environment variables– https://ift.tt/rU0zTjH
Gopher Tribe Knowledge PlanetIn 2024 will continue to work on building a high quality Go language learning and communication platform. We will continue to provide high-quality Go technical article debut and reading experience. At the same time, we will also enhance the sharing of code quality and best practices, including how to write concise, readable and testable Go code. In addition, we will also strengthen the communication and interaction among our star friends. You are welcome to ask questions, share tips and discuss techniques. I will be the first time to answer and exchange. I sincerely hope that Gopher tribe can become a harbor for everyone to learn, progress and communicate. Let me meet in Gopher tribe and enjoy the joy of coding! Welcome to join!
Famous cloud hosting service vendor DigitalOcean released its latest hosting plan, the entry-level Droplet configuration is upgraded to: 1 core CPU, 1G RAM, 25G high-speed SSD, price 5$/month. For those who have the need to use DigitalOcean, you can open thislink address: https://ift.tt/NnbhYk4 Kick off your journey to DO hosting.
Gopher Daily(Gopher每日新闻) – https://ift.tt/0wUvVTi
My contact information:
- Microblogging (not available at this time): https://ift.tt/9ka5yNT
- 微博2：https://ift.tt/75dE693
- Blog: tonybai.com
- github: https://ift.tt/Yr780Cf
- Gopher Daily归档 – https://ift.tt/6iaGr9p
- Gopher Daily Feed订阅 – https://ift.tt/a5ZFTdf
Business cooperation methods: writing articles, publishing books, training, online courses, joint ventures, consulting, advertising cooperation.
© 2024, bigwhite. All rights reserved.