Is it worth choosing “Internet celebrity language”? Summary of seven years of experience of database companies using Go and Rust

Author | Zhao Yuying

“I will look at this matter very dialectically now, and I can only say that it is not bad, but the so-called mainstream choice at that time may turn our product into a mediocre system.”

Even for a company that starts a business at this moment, it is a very difficult decision to adopt Go and Rust for all products of the company, let alone seven years ago

From 2015 to 2016, Go is less than five years old, and Rust has not released v1.0 version. Not many companies and developers are optimistic about these two languages. How can any company choose to fully adopt these two languages, or use them to write databases? and storage layer code?

If you heard this story seven or eight years ago, the gut feeling was probably that this company would not last long. In fact, this company has not only gone through seven years, but also received three rounds of financing. This company is PingCAP.

Of course, the programming language is just a tool, and it is by no means the most important factor for PingCAP to go today, but the company has indeed gained a lot because of this choice. Through conversations with Huang Dongxu, one of the founders of PingCAP, and two engineers, this article restores the reasons why the startup company initially chose Go and Rust, and how to solve the subsequent problems of the language itself, talent problems, and applicable scenarios for different languages Thinking.

1 The reason behind the choice

“First of all, I am not a believer in a specific programming language or tool, but it is also very necessary to choose a good tool when doing a project.”

—— Huang Dongxu

The first point of selection: development efficiency

Among the founders of the Go language is a god-level figure recognized by programmers all over the world-Kenneth Thompson (Kenneth Thompson), he is the main developer of the UNIX operating system; its other main designer and early implementer Although Rob Pike was not directly involved in the development of the original version of UNIX, he was also the most senior member of the UNIX development group at Bell Labs and was the main implementer of the character encoding UTF-8.

Therefore, the origin of Go determines that the language has extremely high quality. Moreover, the Go language has been open source since the first version, so programmers from all over the world discovered and used it for the first time, and were immediately attracted by its wonderful language features. For example, the Go language is much lighter than threads. Many goroutines can save up to 80% of the time for context switching. Among these followers are Huang Dongxu and another partner Liu Qi.

Long before founding PingCAP, Huang Dongxu and Liu Qi used the Go language to write an open source software called codis to solve the problem of pea pod business in terms of cache expansion at that time. After a deep understanding of the Go language, the two were attracted by the efficiency improvement brought by Go. “The same system may take one month to develop using C++, but it may only take three days to use Go. It is hard not to be tempted by this level of development efficiency improvement.”

In the past, there were not no people who chose an emerging language because of efficiency. For example, Paul Graham, the author of “Hackers and Painters”, used Lisp to write the earliest Web application Viaweb, which was eventually acquired by Yahoo for more than 50 million. USD acquisition. This story also had some influence on the founding team of PingCAP.

The goal of PingCAP is to be a distributed database, which is now known as TiDB. The team did consider C++ at the beginning. After all, most members have a C++ background, and most good databases are developed with C++. It’s easy to go wrong. For PingCAP, which has just started a business, it is obviously difficult for the team to find powerful C++ R&D talents.

“There is another interesting phenomenon here. Few programmers in the industry admit that they are proficient in C++. Even developers with 20 years of experience may say that they are proficient in Go after learning for a month. The complexity is clear at a glance.”

There was a safe choice at the time, and that was Java. During that time, most mainstream distributed systems were written in Java, such as Hadoop, Zookeeper, Cassandra, HBase, etc. But facing the era of cloud computing, the team believes that new ways to build systems that are more agile, more efficient, and more secure are needed.

At that time, there were also many software product selection tools that believed that they could squeeze out hardware performance to the greatest extent. “To be honest, I don’t believe that the products developed must be able to extract the last bit of performance from the hardware. The cost of recruiting programmers with this ability may not be lower than adding a few machines (in the distributed era, this problem can obviously be solved by adding Machine way to solve), and users will not pay more for the squeezed performance,” Huang Dongxu said in the interview.

After a comprehensive balance, Go became the most suitable starting choice for starting a business at that time, and could quickly develop the prototype of TiDB. In September of that year, TiDB was open sourced on GitHub, and the subsequent iterations were also very fast. In the following year, customers tried it in the production environment, and in the following year, TiDB 1.0 GA version was officially released.

The second point of choice: the characteristics of the language itself

Now that the efficiency of Go has been verified, why did the team choose Rust when developing TiKV, TiDB’s storage engine?

TiKV started at the end of 2015. At that time, the team struggled between Pure Go / Go + Cgo / C++11 / Rust. Although the core team of PingCAP has a lot of Go language development experience, and the SQL layer of TiDB has been completely Developed in Go language, the great improvement in development efficiency brought by Go has also benefited the team a lot. But in the selection of the storage layer, the team first ruled out the option of Pure Go. The reason is very simple. The bottom layer has decided to connect to RocksDB. RocksDB itself is a C++ project, and the implementation of Go’s LSM-Tree is mostly not mature enough. There are too few projects that can be compared with RocksDB. If you choose Go, you can only choose to use Cgo to bridge, but the problem of Cgo was also obvious at the time. At the end of 2015, the performance loss of calling Cgo in Go code was relatively large, not in The thread where the goroutine is located directly calls the code of cgo, and for the database, the underlying storage engine library is called very frequently. It is not cost-effective to call the function of RocksDB every time it requires these extra expenses. Of course, some tricks can also be used. Increase the throughput of calls on the Cgo side, such as packaging calls within a period of time into a cgo batch call, increase the overall throughput by increasing the delay of a single request, and smooth the overhead of the cgo call itself, but in this way, Implementation can become very complicated. On the other hand, there is still no way to completely solve the GC problem. The storage layer hopes to use the memory as efficiently as possible. Extensive use of syscall.Mmap or object reuse and some hacky techniques will reduce the overall code readability.

In fact, there is no problem with C++11, and there must be no problem in terms of performance. RocksDB is written in C++11. After struggling for a while, the team carefully evaluated the background of the members and what to do, and finally did not choose C++. The main reasons are:

The core team used to be heavy developers of C++, and basically had the experience of maintaining large-scale C++ projects. Everyone has a bit of shadow… Dangling pointers, memory leaks, and data races are almost unavoidable in the process of growing projects , Of course, you can say that relying on veteran drivers to lead the way, strict Code Review and coding standards can reduce the probability of problems, but once a problem occurs, the cost of debugging is high, the mental burden is heavy, and what should I do if the third-party library does not meet the standards.

There are too many programming paradigms in C++, and the differences are great, and there are many tricks and tricks. Unifying the style also requires additional learning costs, especially the team members are constantly increasing. Not all of them are old C++ drivers. In particular, everyone has become accustomed to the help of GC for so many years, and it is difficult to return to the era of manual memory management.

Lack of modern peripheral tools such as package management and integrated construction, although this may not seem so important, but these automation tools are extremely important for a large-scale project, directly related to everyone’s development efficiency and project iteration speed. Moreover, the third-party libraries of C++ are uneven, and many wheels have to be built by ourselves.

Rust has released 1.0 at the end of 2015. Rust has several features that are very attractive to the team:

• memory safety

• High performance (thanks to the excellent capabilities of llvm, the runtime is actually almost indistinguishable from C++), affinity with C/C++ packages

• Powerful package management and build tool Cargo

• more modern syntax

• Almost the same debugging and tuning experience as C++, familiar tools such as perf can be reused directly

• FFI, can link and call RocksDB’s C API without loss

On the one hand, the team puts safety first. Although the problems of C++ memory management and data race avoidance can be solved by old drivers, there is still no strong constraint at the compiler level to kill the problem in the cradle. Thoroughly, Rust solves this problem nicely. On the other hand, Rust is a very modern programming language. The modern type system, pattern matching, powerful macros, traits, etc. will greatly improve development efficiency after familiarization.

In the end, Rust did not disappoint the team. It took a team of four or five people about four months to develop the first version of TiKV from scratch. Development started on January 1, 2016, and it was open sourced on April 1. In October of the same year, TiKV was used in the production environment for the first time, and TiDB had not even released the beta version at that time. The development of TiKV is very fast, the released versions are all stable, and the production efficiency is much higher than that of C++.

The third key point of selection: talent

Although not many developers used Rust back then, the early explorers were very capable. This group of people has a strong love for programming itself, which is a very valuable talent for start-up companies (the initial stage PingCAP’s subsequent development has also verified this point. Brian Anderson, who used to be a member of the Rust core team, chose to join the company and participate in the research and development of TiKV in 2018. Before 2018, this case was Very few. The reason why Brian is willing to join, in addition to personal factors, is inseparable from PingCAP’s continuous efforts in open source and community. In 2017, Brian was invited to attend the first domestic Rust Meetup held by PingCAP. In 2019, nrc, the veteran of the Rust Core Team, also joined PingCAP.

“To this day, I still think this is a good idea for a start-up company to attract talents. Otherwise, what a start-up company will use to compete with Internet giants can only be a better idea and tool. We have not done any global branding At that time, I relied on this business card (referring to fully embracing the Rust ecology) to attract talents to join our community and company. But many times, the problem of many domestic companies is that they dare not think about why they can attract such a big cow to join , but everything must be tried first.”

2 Problems with choosing Rust

As mentioned at the beginning, Huang Dongxu now thinks that the original decision was “good or not bad”. Although the development efficiency has been improved, he has also suffered from the problems caused by the immature language at that time.

“A lot of times, people will first understand one thing through advertisements. Our original view of Rust is memory safety, good performance, no GC efficiency must be higher, but it is not the case. If your code is very frustrating, why do you think Allocating memory manually yourself is better than GC.”

At first, the team basically used Rust as Java and C++, and the performance did not improve significantly. It was not until more professional talents joined that the entire code was reversed to a better path.

Also, Rust takes longer to compile. “At that time, our internal Rust programmers often joked that there were only 24 compilation opportunities a day, and one use was one less.” The team did a lot of work to reduce the compilation time of Rust, participated in and contributed to rust-gRPC, Raft library, open-tracing and other projects, and produced a large number of related blog posts.

Although solving these problems took up a lot of the team’s time, the team also benefited from it. “We implemented all the tutorials of the Talent Plan in Rust. Although this is a bit far from our homepage’s main business, it is extremely important for subsequent recruitment and training. This is the first time we have received rave reviews around the world.”

3 How to judge whether to choose or switch between Rust and GO?

Many enterprises will choose to implement commercial versions based on some open source products at the beginning of their business. In this case, the programming language is already determined, so there is no need to get too entangled. If you develop a product from scratch, you can consider the following aspects:

1. If the company has already done a good job of developing specifications on C/C++ and Java, it is not necessary to consider switching to Rust. Rust is equivalent to its own strict security restrictions, so that programmers cannot write code with security risks in most cases, and the design of the language itself helps to avoid some conventional problems.

2. If it is a basic software type enterprise, the level of relevant practitioners is still trustworthy, and generally they will not make too many low-level mistakes. At this time, the benefits of Rust are mainly reflected in the writing of the core components or functions of the database. For the remaining 90%, efficiency may be more important than safety.

3. Scenarios that do not require Rust include write drivers, such as the operating system kernel, etc., which are extremely efficient; SSL encryption or a key link inside the product, such as the rendering engine in the browser, which are CPU-intensive and have high security requirements scene. For non-essential scenarios, it is best to choose a programming language with a relatively large community, such as Java, which is relatively easy to recruit.

4. Consider language backward compatibility. The Go language has done this very well so far, and has also added generic types in response to comments from community users.

5. community style. The Rust community is very open, which brings a lot of benefits to the language, but it may also bring some side effects, such as some splits, which may be something that the Rust community will consider in the future, but the atmosphere in the community is relatively active , where you can find answers to your questions or like-minded friends.

   “The latest complexity added to Rust in recent years is mainly Pin asynchronous programming, but the addition of the two does solve some problems. No matter which language it is, it will face various choices. This is the result of the balance of different goals.”

6. Code implements logic. The Rust language is different from Go, Java, etc. in design, and it avoids some problems. Programs written in Rust can focus on optimizing the program logic itself, such as making the program more adaptable to the operating system, reducing thread switching, and so on.

Rust has now been verified by more and more companies and developers, but there are still new niche languages ​​emerging, just like the new Rust that year, how should companies judge?

4
The new language Zig is also here, what do you think?

Zig is a new and niche language. Version 1.0 has not been released yet. Its competitor is C language. Note that it is not C++, but the most basic C language, but it also absorbs some features of Rust and other languages.

In the middle of last year, Zig attracted the attention of some developers because of the use of Uber. Uber uses Zig to compile its C/C++ code. Now, Uber only uses bazel-zig-cc in the Go Monorepo (according to its internal engineers, the Go Monorepo is bigger than the Linux kernel and has thousands of engineers developing and maintaining it), but plans to promote zig cc as much as possible to other languages ​​that require a C/C++ toolchain.

Uber engineers said that compared with other toolchains, the main advantage of the C/C++ toolchain provided by zig-cc is that the glibc version can be configured to cross-compile with macOS ( for details, please read : https://www.infoq.cn/ article/q016NWR7OjHvOJ3Rkzc0).

Zig today is similar to but different from Rust seven or eight years ago. Back then, Rust was backed by a group of Molliza engineers. Its philosophy, quality, and ability to solve practical problems are trustworthy. Although Zig now has foundation support, But the actual capacity is still open to question.

The interviewed PingCAP engineers took a wait-and-see attitude towards the development of the language, and suggested that when companies choose such emerging niche languages, it is best to find the reason for “it must be it”. If they cannot find it, it proves this. There is no need to risk adopting a new language for this matter. If you have an idea, it is best to communicate face-to-face with the founding team and get some judgments from it.

5 Conclusion

Since the birth of ChatGPT, we have been amazed countless times by the productivity of AI, which can generate code based on simple input and easily convert these codes to other programming languages. In the future, the working mode of developers may undergo earth-shaking changes. The most important thing is our ideas and ideas for building all this, not the tools themselves. From this perspective, the debate about the pros and cons of programming languages ​​becomes less important.

Of course, if you’re a believer in a certain language/tool, setting out to take it to new heights makes more sense than arguing.

The text and pictures in this article are from InfoQ