What is CentOS Stream called the future of CentOS?

In this issue of “A Date with Geeks” , we invited Zhang Jiaju, chief architect of Red Hat, to talk to us about the “stop” event and what the CentOS Stream, called the future of CentOS, is.

InfoQ: Why did Red Hat decide to end the CentOS 8 lifecycle at that time?

Zhang Jiaju: The suspension of CentOS 8 is actually equivalent to changing the name of CentOS – CentOS Stream, which is similar to the fact that many open source projects often change their names during operation. For example, the community edition origin of Red Hat OpenShift is now called OKD. In fact, CentOS Stream inherits the mantle of CentOS and is homologous to CentOS. We think this is a normal version iteration, or an iteration of a community project, and we didn’t expect to have such a big response in the market.

InfoQ: Why didn’t Red Hat respond when the decision was announced?

Zhang Jiaju: Before 2000, everyone’s impression of Red Hat was still selling software installation CDs. Around 2000, Red Hat started doing enterprise Linux, and it has been 20 years since then. Among them, we found that the main demands of enterprises are stability, such as the need for products with a long life cycle. Relatively closed source, the reason why open source has so many innovations is largely because the burden is small, and innovative ideas can be quickly added to achieve a more perfect realization, which is also the design of registers in Microsoft and Intel X86 architecture. The perfect reason is that the historical burden is heavy, and compatibility needs to be guaranteed when updating.

Before Red Hat proposed to do enterprise-level Linux, few people used Linux directly for enterprise production, and enterprises still had concerns about security and stability. Red Hat proposes to send Linux into the production environment of the enterprise. The main method is to backport. Based on a certain version of the community, a branch is pulled out, and the branch is continuously optimized and engineered to improve. If there are some innovative ideas, add it, similar SUSE and other enterprise-level Linux are basically the same way of thinking. This makes our development model similar to waterfall development for a long time, and we constantly stabilize the entire version during this process, which is a traditional development method.

Around 2017, we think this traditional development method is a bit out of date, because the iteration cycle of this method is relatively slow, and now we pay attention to DevOps and microservices, and the development cycle is getting shorter and shorter, and we are also thinking about the operating system kernel including the software on it. Is it possible to improve the speed of development? We added CI/CD to the development process, and performed automated tests (Nightly Builds) on the submitted use cases of the day every night, and released a new version the next day. It’s called Stream.

Around 2018, the prototype of CentOS Stream already existed, but this streaming development model was not announced in the CentOS community. With the incubation of the past few years and comprehensive consideration of various factors, we believe that CentOS Stream can release the Linux version more quickly and stably, so compared to us upgrading CentOS to CentOS Stream.

In our opinion, the stability of CentOS Stream is not inferior to that of CentOS. Many people have doubts about this, thinking that this is for guinea pigs. In fact, Fedora is the experimental field of RHEL. From normal logic, we also do not need two experimental fields.

InfoQ: Does non-waterfall development affect version stability? What is CentOS Stream’s place in the enterprise Linux ecosystem?

Zhang Jiaju: The kernel really pays attention to stability. In the early years, the number of patch contributors to the Linux kernel was relatively small, and relatively speaking, enterprise-level Linux system updates were not too fast. Nowadays, more and more developers join the Linux kernel contribution team. In order to make the new features enter the mainstream version faster, we need to adopt a faster way to develop, otherwise it will be difficult to keep up with this rhythm, which is very important for Red Hat It’s a tricky thing to do, we need to optimize the process to make it faster, and this new development model has been gradually adopted since RHEL 8. Whether this method is necessarily better than traditional waterfall development is a big question. topic, we believe that stability can be achieved through better processes and a higher degree of automation.

Regardless of whether CentOS Stream is launched or not, the development method of the operating system kernel is already the way of CI/CD. Red Hat’s RHEL 8 and 9 are done in this way, and the stability does not need to be too worried.

As for the position of CentOS Stream in the Linux ecosystem, in general, Fedora is the mid-upstream, RHEL is the downstream, and CentOS Stream is the mid-stream. In fact, Fedora comes entirely from the community. Red Hat does more packaging and simple testing, and does less work on its stability and reliability. It follows a rolling update method and releases a new version every six months. , the basic compatibility between the new version and the old version is maintained, and some bad points in the old version may be discarded, which is precisely unacceptable for enterprise-level development.

RHEL is based on a specific version of Fedora to pull the branch, and gradually make enhancements on this version to ensure the compatibility between the old and new versions, and to ensure the stability of the final version. CentOS Stream corresponds to the RHEL version, and its Git submission records are exactly the same. Both pass the same build process and the same test cases. To put it simply, a new version of CentOS Stream will be released only if all the test cases are passed. These test cases may or may not overlap with RHEL, but we believe that only by passing all the test cases is stable, and we can proceed to the next step. RHEL The same is true, the two remain the same in terms of stability.

So, since the two are consistent, why distinguish between midstream and downstream? All changes made in RHEL will be advanced into CentOS Stream, which is convenient for community ecological partners to build together, and allows all developers to enjoy the latest version as soon as possible.

InfoQ: What is the correspondence between CentOS Stream and RHEL versions?

Zhang Jiaju: CentOS Stream 8 and RHEL 8 are corresponding, but RHEL may also have 8.1, 8.2, 8.3… The difference is that CentOS Stream always only corresponds to the latest stable version of RHEL. Generally speaking, our update rhythm is to update a small version every six months. Assuming that the latest stable version of RHEL is 8.3, then CentOS Stream must correspond to this version.

Many traditional enterprises may feel that the new version is not the most stable based on the cognition of closed source software, but in fact CentOS Stream is the latest and most stable version. Another problem is that for enterprises that have deployed CentOS Stream in the production environment, it is too frequent to update once every six months. What should I do?

Most enterprises will continue to use the version they chose at the beginning of the launch, unless there is a security problem before thinking of replacing it. Even in this mode, it is also feasible to use CentOS Stream, and enterprises can choose to update the version when needed.

As for the difference between the two, RHEL will provide long system update support for a single version for up to two years to ensure the safety and reliability of the old version. Although the new version has better support in terms of security, stability and some new features, the enterprise is not a one-person enterprise. Once a problem occurs due to the update, the result is likely to be irreversible, so many enterprises will choose a conservative solution in this regard.

InfoQ: Is compatibility guaranteed?

Zhang Jiaju: Compatibility is definitely guaranteed, but because the user’s application is closed source, we cannot determine which modules in the kernel are called. A large version range, such as 8, is guaranteed to be compatible. Yes, Red Hat has also done a lot of engineering work, so that the community can move forward without any burden, and Red Hat provides support behind it for up to 13 years.

InfoQ: It is understood that Facebook, Intel and other manufacturers have already participated in the community building of CentOS Stream. What are they mainly doing?

Zhang Jiaju: Intel has always embraced open source projects, especially those related to the Linux kernel. The overall code contribution exceeds that of Red Hat. Linux is mainly based on the X86 architecture, and Intel has made important contributions. Facebook is mainly to do some specialized services to meet its own business development, and to give back to the community.

For Linux distribution manufacturers, what we do is more to meet the demands of the public, while Internet manufacturers have higher requirements for the kernel based on their own business characteristics, and often change the upper-layer application to achieve the kernel and upper-layer applications. Collaborate efficiently, or improve efficiency by designing some proprietary hardware. After the release of CentOS Stream, Facebook made the switch immediately and gradually opened up some of its capabilities.

InfoQ: How do former CentOS users switch to CentOS Stream?

Zhang Jiaju: Simply put, you can migrate to CentOS Stream through two commands. However, different companies have different definitions of production environment, quasi-production environment, and R&D test environment. Switching needs to be judged based on actual business scenarios.

These are the two core commands, but it is definitely not so simple for enterprises to migrate. Data backup is the most basic, and you may need to roll back if necessary. Red Hat has also published very detailed migration steps, and you can check them yourself if you need them. And in theory, he shouldn’t be different, because this thing itself is the same, how can it be different? Even if you say define parameters, this is not a compilation system, compilation parameters may change, but will there really be changes? It shouldn’t be much, because your code is all the same, there should be no difference, essentially no difference to a software update, and no more risk introduced.

From an objective point of view, CentOS Stream can run stably in many scenarios. If you encounter problems during the switching process, you can still retrieve the answer in Red Hat’s knowledge base. As long as it is a Patch that appears in CentOS Stream, it will definitely be available in RHEL. If it is a system bug, you can submit a bug report, and we will solve the problem in a targeted manner.

InfoQ: What are the developers, project management committees, and SIG groups in the CentOS community mainly doing?

Zhang Jiaju: For this open source project, the community is not that complicated. It is mainly composed of two governance bodies: the board of directors and the SIG group. The board of directors is basically an independent director like Red Hat. Technical decisions are made through SIG groups, and Red Hat engineers will also participate in discussions and communicate with everyone in the community.

If developers are interested in the discussions of these SIG groups, they can learn about the distribution of existing SIG groups through CentOS.org, some for new technologies, some for specific industries, such as the automotive industry, and some SIG groups on how to communicate with the community. If you have better ideas, you are welcome to submit an application.

InfoQ: Why isn’t CentOS Stream compiled directly from Red Hat’s Enterprise Edition?

Zhang Jiaju: Whether it is Red Hat Enterprise Edition or CentOS Stream, the code and the test cases that have been passed are the same. It is nothing more than that the final package is signed with the Red Hat enterprise product. The content can be seen in CentOS Stream. If it is reversed, users will inevitably feel that Red Hat has left a part of it.

InfoQ: As a developer, how do you choose which platform to develop on at Red Hat? Which platform to deploy on?

Zhang Jiaju: At the development level, many enthusiasts may be accustomed to using Linux Destop; desktop development can choose Fedora, all the packages in it are the latest; in most cases, server-side development, CentOS Stream is a good choice, because it is not like Fedora So radical, the package is also very complete, and it has the stability of RHEL, and the update is fast enough, especially in versions after 8, libraries in different languages, including the images required for containerized application development, exist, which is very worry-free choose. Of course, these are all free. If you need some services from Red Hat, you can also choose our Enterprise Edition.

InfoQ: What are the plans for future maintenance and continuous updates of CentOS Stream?

Zhang Jiaju: CentOS Stream is a long-term maintenance and update project. After all, RHEL also depends on it, so CentOS Stream will always exist, and you don’t need to worry about life cycle related issues.

The text and pictures in this article are from InfoQ