Permalink to this article – https://ift.tt/G8VKZN5
In the previous article “Developing a Hello World-level eBPF Program from Scratch Using C Language” , we explained in detail how to develop an eBPF program (including its user mode part) from scratch based on C language and libbpf library. That article is the basis for subsequent articles on eBPF program development, because so far, no matter what language the user mode part of the eBPF program is developed in, the kernel mode part of the eBPF program running in the kernel mode must still be developed in C language. In this way, other programming languages can only spell out how to make the development of the user-mode part of the eBPF program easier, and the Go language is no exception.
In the Go community, the most active Go eBPF package used to develop the eBPF user space is the open source cilium/ebpf of the cilium project. Isovalent, the company behind the cilium project, is also one of the main promoters of eBPF technology in the cloud native field.
In this article, we will talk about the routine of developing eBPF programs based on cilium/ebpf !
1. Explore the cilium/ebpf project example
The cilium/ebpf project draws on the idea of libbpf-boostrap , and builds the user mode part of the eBPF program by means of code generation and bpf program embedding. In order to understand the routine of developing ebpf programs based on cilium/ebpf, let’s first explore the sample code provided by the cilium/ebpf project.
Let’s first download and look at the structure of the ebpf example.
- Download the cilium/ebpf project
$ git clone https://github.com/cilium/ebpf.git Cloning into 'ebpf'... remote: Enumerating objects: 7054, done. remote: Counting objects: 100% (183/183), done. remote: Compressing objects: 100% (112/112), done. remote: Total 7054 (delta 91), reused 124 (delta 69), pack-reused 6871 Receiving objects: 100% (7054/7054), 10.91 MiB | 265.00 KiB/s, done. Resolving deltas: 100% (4871/4871), done.
- Explore the ebpf project sample code structure
The ebpf example is in the examples directory. Let’s take tracepoint_in_c as an example to see its organization:
$tree tracepoint_in_c tracepoint_in_c ├── bpf_bpfeb.go ├── bpf_bpfeb.o ├── bpf_bpfel.go ├── bpf_bpfel.o ├── main.go └── tracepoint.c 0 directories, 6 files
Judging from experience, tracepoint.c here corresponds to the kernel-mode part of the ebpf program, while main.go and bpf_bpfel.go/bpf_bpfeb.go are the user-mode part of the ebpf program. As for bpf_bpfeb.o/bpf_bpfel.o, it should be a certain an intermediate object file. View this intermediate file with readelf -a bpf_bpfeb.o:
$readelf -a bpf_bpfeb.o ELF Header: Magic: 7f 45 4c 46 02 02 01 00 00 00 00 00 00 00 00 00 Class: ELF64 Data: 2's complement, big endian Version: 1 (current) OS/ABI: UNIX - System V ABI Version: 0 Type: REL (Relocatable file) Machine: Linux BPF Version: 0x1 Entry point address: 0x0 Start of program headers: 0 (bytes into file) Start of section headers: 1968 (bytes into file) Flags: 0x0 Size of this header: 64 (bytes) Size of program headers: 0 (bytes) Number of program headers: 0 Size of section headers: 64 (bytes) Number of section headers: 13 Section header string table index: 1 ... ...
We see that this is an elf file (Machine: Linux BPF) with linux bpf bytecode inside.
After reading the relevant documents of cilium/ebpf, I figured out the relationship between these files, and presented it to you with the following diagram:
The source code file of the ebpf program (such as tracepoint.c in the figure) is compiled by bpf2go (a code generation tool provided by cilium/ebpf) (bpf2go calls clang) into ebpf bytecode files bpf_bpfeb.o (big endian) and bpf_bpfel.o (little endian), then bpf2go will generate bpf_bpfeb.go or bpf_bpfel.go based on the ebpf bytecode file, and the bytecode of the ebpf program will be embedded in the two go source files in the form of binary data, with bpf_bpfel.go For example, we can find the following in its code (using the go:embed feature ):
//go:embed bpf_bpfel.o var _BpfBytes []byte
main.go is the main program of the user mode part of the ebpf program. The ebpf program is formed by compiling main.go with one of bpf_bpfeb.go or bpf_bpfel.go.
After an initial exploration of the cilium/ebpf project example, let’s build the ebpf example code.
2. Build ebpf sample code
cilium/ebpf provides a convenient build script, we just need to execute “make -C ..” under ebpf/examples to build the example code.
The make build process will start the build container based on the quay.io/cilium/ebpf-builder image, but the domestic children’s shoes need to make a little modification to the Makefile content as follows, and increase the GOPROXY environment variable, otherwise the go module outside the wall cannot be pulled :
$git diff ../Makefile diff --git a/Makefile b/Makefile index 3a1da88..d7b1712 100644 --- a/Makefile +++ b/Makefile @@ -48,6 +48,7 @@ container-all: ${CONTAINER_ENGINE} run --rm ${CONTAINER_RUN_ARGS} \ -v "${REPODIR}":/ebpf -w /ebpf --env MAKEFLAGS \ --env CFLAGS="-fdebug-prefix-map=/ebpf=." \ + --env GOPROXY="https://goproxy.io" \ --env HOME="/tmp" \ "${IMAGE}:${VERSION}" \ $(MAKE) all
After this, executing the build will successfully get the result we want:
$ cd examples $ make -C .. make: Entering directory '/root/go/src/github.com/cilium/ebpf' docker run --rm --user "0:0" \ -v "/root/go/src/github.com/cilium/ebpf":/ebpf -w /ebpf --env MAKEFLAGS \ --env CFLAGS="-fdebug-prefix-map=/ebpf=." \ --env GOPROXY="https://goproxy.io" \ --env HOME="/tmp" \ "quay.io/cilium/ebpf-builder:1648566014" \ make all make: Entering directory '/ebpf' find . -type f -name "*.c" | xargs clang-format -i go generate ./cmd/bpf2go/test go: downloading golang.org/x/sys v0.0.0-20210906170528-6f6e22806c34 Compiled /ebpf/cmd/bpf2go/test/test_bpfel.o Stripped /ebpf/cmd/bpf2go/test/test_bpfel.o Wrote /ebpf/cmd/bpf2go/test/test_bpfel.go Compiled /ebpf/cmd/bpf2go/test/test_bpfeb.o Stripped /ebpf/cmd/bpf2go/test/test_bpfeb.o Wrote /ebpf/cmd/bpf2go/test/test_bpfeb.go go generate ./internal/sys enum AdjRoomMode enum AttachType enum Cmd enum FunctionId enum HdrStartOff enum LinkType enum MapType enum ProgType enum RetCode enum SkAction enum StackBuildIdStatus enum StatsType enum XdpAction struct BtfInfo ... ... attr ProgRun attr RawTracepointOpen cd examples/ && go generate ./... go: downloading github.com/cilium/ebpf v0.8.2-0.20220424153111-6da9518107a8 go: downloading golang.org/x/sys v0.0.0-20211001092434-39dca1131b70 Compiled /ebpf/examples/cgroup_skb/bpf_bpfel.o Stripped /ebpf/examples/cgroup_skb/bpf_bpfel.o Wrote /ebpf/examples/cgroup_skb/bpf_bpfel.go Compiled /ebpf/examples/cgroup_skb/bpf_bpfeb.o Stripped /ebpf/examples/cgroup_skb/bpf_bpfeb.o Wrote /ebpf/examples/cgroup_skb/bpf_bpfeb.go Compiled /ebpf/examples/fentry/bpf_bpfeb.o Stripped /ebpf/examples/fentry/bpf_bpfeb.o Wrote /ebpf/examples/fentry/bpf_bpfeb.go Compiled /ebpf/examples/fentry/bpf_bpfel.o Stripped /ebpf/examples/fentry/bpf_bpfel.o Wrote /ebpf/examples/fentry/bpf_bpfel.go Compiled /ebpf/examples/kprobe/bpf_bpfel.o Stripped /ebpf/examples/kprobe/bpf_bpfel.o Wrote /ebpf/examples/kprobe/bpf_bpfel.go Stripped /ebpf/examples/uretprobe/bpf_bpfel_x86.o ... ... Wrote /ebpf/examples/uretprobe/bpf_bpfel_x86.go ln -srf testdata/loader-clang-14-el.elf testdata/loader-el.elf ln -srf testdata/loader-clang-14-eb.elf testdata/loader-eb.elf make: Leaving directory '/ebpf' make: Leaving directory '/root/go/src/github.com/cilium/ebpf'
Take the ebpf below uretprobe as an example, let’s run it:
$go run -exec sudo uretprobe/*.go 2022/06/05 18:23:23 Listening for events..
Open a new terminal and execute vi .bashrc in the user’s home directory. In the execution window of the uretprobe program above we can see:
2022/06/05 18:24:34 Listening for events.. 2022/06/05 18:24:42 /bin/bash:readline return value: vi .bashrc
This indicates that the ebpf program under uretprobe is executed as expected.
3. Use cilium/ebpf to develop the user mode part of the Hello World eBPF program above
With a preliminary understanding of the cilium/ebpf sample program, let’s develop the user mode part of the helloworld ebpf program in the previous article “Developing a Hello World-level eBPF Program from Scratch Using C Language” .
Review the C source code of the hello world ebpf program:
// github.com/bigwhite/experiments/tree/master/ebpf-examples/helloworld-go/helloworld.bpf.c #include <linux/bpf.h> #include <bpf/bpf_helpers.h> SEC("tracepoint/syscalls/sys_enter_execve") int bpf_prog(void *ctx) { char msg[] = "Hello, World!"; bpf_printk("invoke bpf_prog: %s\n", msg); return 0; } char LICENSE[] SEC("license") = "Dual BSD/GPL";
When the ebpf program is loaded into the kernel, every time the execve system call is executed, the ebpf program will be called once, and we will see the corresponding log output in /sys/kernel/debug/tracing/trace_pipe.
1. Use bpf2go to convert ebpf core state program to Go code
According to the “routine” we got when we explored the cilium/ebpf sample program, the first thing we need to do next is to convert helloworld.bpf.c into a Go code file. The indispensable tool for this conversion process is cilium The bpf2go tool provided by /ebpf, let’s install the tool first:
$go install github.com/cilium/ebpf/cmd/bpf2go@latest
Next, we can directly use the bpf2go tool to convert helloworld.ebpf.c to the corresponding go source file:
$GOPACKAGE=main bpf2go -cc clang-10 -cflags '-O2 -g -Wall -Werror' -target bpfel,bpfeb bpf helloworld.bpf.c -- -I /home/tonybai/test/ebpf/libbpf/include/uapi -I /usr/local/bpf/include -idirafter /usr/local/include -idirafter /usr/lib/llvm-10/lib/clang/10.0.0/include -idirafter /usr/include/x86_64-linux-gnu -idirafter /usr/include Compiled /home/tonybai/go/src/github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfel.o Stripped /home/tonybai/go/src/github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfel.o Wrote /home/tonybai/go/src/github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfel.go Compiled /home/tonybai/go/src/github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfeb.o Stripped /home/tonybai/go/src/github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfeb.o Wrote /home/tonybai/go/src/github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfeb.go
However, there is a problem here, that is, a series of header file reference paths provided to the clang compiler after the bpf2go command line refer to the Makefile in the article “Developing a Hello World-level eBPF Program from Scratch Using C Language” . If you refer to these header file paths, although the bpf2go conversion can be successful, we need to rely on and install the libbpf library, which is obviously not what we want.
cilium/ebpf provides a headers directory in the examples, this directory contains all the header files needed to develop the userland part of the ebpf program, we use it as our header file reference path. However, to build ebpf based on this headers directory, we need to include the original header file in helloworld.bpf.c by:
#include <linux/bpf.h> #include <bpf/bpf_helpers.h>
Change it to:
#include "common.h"
Next, let’s execute the bpf2go tool to convert:
$GOPACKAGE=main bpf2go -cc clang-10 -cflags '-O2 -g -Wall -Werror' -target bpfel,bpfeb bpf helloworld.bpf.c -- -I /home/tonybai/go/src/github.com/cilium/ebpf/examples/headers Compiled /home/tonybai/go/src/github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfel.o Stripped /home/tonybai/go/src/github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfel.o Wrote /home/tonybai/go/src/github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfel.go Compiled /home/tonybai/go/src/github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfeb.o Stripped /home/tonybai/go/src/github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfeb.o Wrote /home/tonybai/go/src/github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfeb.go
We see that bpf2go successfully generates ebpf bytecode and corresponding Go source files.
2. Build the user mode part of the helloworld ebpf program
The following is the main.go source code of the user mode part of the helloword ebpf program built with reference to the cilium/ebpf example:
// github.com/bigwhite/experiments/ebpf-examples/helloworld-go/main.go package main import ( "log" "os" "os/signal" "syscall" "github.com/cilium/ebpf/link" "github.com/cilium/ebpf/rlimit" ) func main() { stopper := make(chan os.Signal, 1) signal.Notify(stopper, os.Interrupt, syscall.SIGTERM) // Allow the current process to lock memory for eBPF resources. if err := rlimit.RemoveMemlock(); err != nil { log.Fatal(err) } // Load pre-compiled programs and maps into the kernel. objs := bpfObjects{} if err := loadBpfObjects(&objs, nil); err != nil { log.Fatalf("loading objects: %s", err) } defer objs.Close() //SEC("tracepoint/syscalls/sys_enter_execve") // attach to xxx kp, err := link.Tracepoint("syscalls", "sys_enter_execve", objs.BpfProg, nil) if err != nil { log.Fatalf("opening tracepoint: %s", err) } defer kp.Close() log.Printf("Successfully started! Please run \"sudo cat /sys/kernel/debug/tracing/trace_pipe\" to see output of the BPF programs\n") // Wait for a signal and close the perf reader, // which will interrupt rd.Read() and make the program exit. <-stopper log.Println("Received signal, exiting program..") }
We know that an ebpf program has several key components:
- ebpf program data
- map: used for data interaction between user mode and kernel mode
- attachment point
According to the description of the cilium/ebpf architecture , the ebpf package abstracts the first two parts into a data structure bpfObjects:
// github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfel.go // bpfObjects contains all objects after they have been loaded into the kernel. // // It can be passed to loadBpfObjects or ebpf.CollectionSpec.LoadAndAssign. type bpfObjects struct { bpfPrograms bpfMaps }
We see that the main function loads the ebpf program into the kernel through the generated loadBpfObjects function and fills the bpfObjects structure. Once the bpf program is loaded successfully, we can use the fields in the bpfObjects structure to complete the rest of the operations, such as through the link package function Connect the bpf program with the target hanging node (such as the link.Tracepoint function in the text), so that after the connection, the bpf can be called back and executed after the corresponding event occurs.
Compile and execute the helloworld example below:
$go run -exec sudo main.go bpf_bpfel.go [sudo] password for tonybai: 2022/06/05 14:12:40 Successfully started! Please run "sudo cat /sys/kernel/debug/tracing/trace_pipe" to see output of the BPF programs
Then open a new window and execute sudo cat /sys/kernel/debug/tracing/trace_pipe. When execve is called, we can see log output similar to the following:
<...>-551077 [000] .... 6062226.208943: 0: invoke bpf_prog: Hello, World! <...>-551077 [000] .... 6062226.209098: 0: invoke bpf_prog: Hello, World! <...>-551079 [007] .... 6062226.215421: 0: invoke bpf_prog: Hello, World! <...>-551079 [007] .... 6062226.215578: 0: invoke bpf_prog: Hello, World! <...>-554756 [007] .... 6063476.785212: 0: invoke bpf_prog: Hello, World! <...>-554756 [007] .... 6063476.785378: 0: invoke bpf_prog: Hello, World!
3. Use go generate to drive the conversion of bpf2go
In terms of code generation, the Go tool chain natively provides the go generate tool. In the examples of cilium/ebpf, go generate is also used to drive bpf2go to convert bpf programs into Go source files. Here we will also do some transformation.
First, we add a line of go:generate instruction statement above the main function of main.go:
// github.com/bigwhite/experiments/ebpf-examples/helloworld-go/main.go // $BPF_CLANG, $BPF_CFLAGS and $BPF_HEADERS are set by the Makefile. //go:generate bpf2go -cc $BPF_CLANG -cflags $BPF_CFLAGS -target bpfel,bpfeb bpf helloworld.bpf.c -- -I $BPF_HEADERS func main() { stopper := make(chan os.Signal, 1) ... ... }
In this way, when we explicitly execute the go generate statement, go generate will scan the instruction statement and execute the following commands. Several variables are used here, and the variables are defined in the Makefile. Of course, if you don’t want to use Makefile, you can also replace the variable with the corresponding value. Here we use Makefile, the following is the content of Makefile:
// github.com/bigwhite/experiments/ebpf-examples/helloworld-go/Makefile CLANG ?= clang-10 CFLAGS ?= -O2 -g -Wall -Werror LIBEBPF_TOP = /home/tonybai/go/src/github.com/cilium/ebpf EXAMPLES_HEADERS = $(LIBEBPF_TOP)/examples/headers all: generate generate: export BPF_CLANG=$(CLANG) generate: export BPF_CFLAGS=$(CFLAGS) generate: export BPF_HEADERS=$(EXAMPLES_HEADERS) generate: go generate ./...
With this Makefile, we can execute the make command to convert the bpf program by bpf2go:
$make go generate ./... Compiled /home/tonybai/go/src/github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfel.o Stripped /home/tonybai/go/src/github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfel.o Wrote /home/tonybai/go/src/github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfel.go Compiled /home/tonybai/go/src/github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfeb.o Stripped /home/tonybai/go/src/github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfeb.o Wrote /home/tonybai/go/src/github.com/bigwhite/experiments/ebpf-examples/helloworld-go/bpf_bpfeb.go
4. Summary
In this article, we explained how to develop the userland part of ebpf based on the cilium/ebpf package.
ebpf draws on the idea of libbpf, and builds the user mode part of ebpf by generating code and data inline.
ebpf provides the bpf2go tool, which can convert the C source code of bpf into the corresponding go source code.
ebpf abstracts the bpf program into bpfObjects, completes the process of loading the bpf program into the kernel through the generated loadBpfObjects, and then uses packages such as link provided by the ebpf library to associate ebpf and kernel events.
There are still many ways to play the ebpf package. This article is just to lay the foundation. In subsequent articles, we will further study and explain various types of bpf programs.
The code for this article can be downloaded here .
None. References
- Manage and distribute ebpf programs in Go – https://ift.tt/BOP0qHe
- A Pure Go eBPF library – https://ift.tt/hiGjMd1
- cilium ebpf library architecture – https://ift.tt/MQ9s1IO
“Gopher Tribe” Knowledge Planet aims to create a high-quality Go learning and advanced community! High-quality first published Go technical articles, “three-day” first published reading rights, analysis of the current situation of Go language development twice a year, reading the fresh Gopher daily 1 hour in advance every day, online courses, technical columns, book content preview, must answer within 6 hours Guaranteed to meet all your needs about the Go language ecosystem! In 2022, the Gopher tribe will be fully revised, and will continue to share knowledge, skills and practices in the Go language and Go application fields, and add many forms of interaction. Everyone is welcome to join!
I love texting : Enterprise-level SMS platform customization development expert https://51smspush.com/. smspush : A customized SMS platform that can be deployed within the enterprise, with three-network coverage, not afraid of large concurrent access, and can be customized and expanded; the content of the SMS is determined by you, no longer bound, with rich interfaces, long SMS support, and optional signature. On April 8, 2020, China’s three major telecom operators jointly released the “5G Message White Paper”, and the 51 SMS platform will also be newly upgraded to the “51 Commercial Message Platform” to fully support 5G RCS messages.
The famous cloud hosting service provider DigitalOcean released the latest hosting plan. The entry-level Droplet configuration is upgraded to: 1 core CPU, 1G memory, 25G high-speed SSD, and the price is 5$/month. Friends who need to use DigitalOcean can open this link : https://ift.tt/vymC6on to open your DO host road.
Gopher Daily Archive Repository – https://ift.tt/cf3vzBa
my contact information:
- Weibo: https://ift.tt/epr7aWg
- Blog: tonybai.com
- github: https://ift.tt/914GUwV
Business cooperation methods: writing, publishing books, training, online courses, partnership entrepreneurship, consulting, advertising cooperation.
© 2022, bigwhite . All rights reserved.
This article is reprinted from https://tonybai.com/2022/07/19/develop-ebpf-program-in-go/
This site is for inclusion only, and the copyright belongs to the original author.