A guide to useful development tools summarized after leaving Google

After leaving Google, it’s hard to enjoy these great development tools.

Blogger Beyang Liu, who had a brief stint at Google years ago, was impressed with Google’s internal tools, albeit for a short time. In his view, Google’s in-house development tools are in many ways the best in the world. Not only is Google adept at scaling its own software systems, it has also been leading the pack when it comes to figuring out how to efficiently build software at scale. Google deals with the real-world challenges of massive code bases, code discoverability, organizational knowledge sharing, and multi-service deployments with a level of sophistication that most other companies can’t match.

But on the other hand, Google’s internal tools are actually few in number, and almost all of them are tightly coupled with Google’s internal environment. So after leaving Google, it is difficult to enjoy these development tools. In addition, while working at Google, everyone has become accustomed to the internal working methods of Google. After leaving, some people will feel that the difficulty of programming has suddenly increased. In fact, this is also related to the loss of familiar tools.

Based on the experience of himself and other former Google employees, Beyang Liu summed up several experiences on the use of development tools, and wrote this pragmatic development tool guide outside of Google to help developers design the conventional path of development tools, and finally for themselves Explore the most efficient tools possible with the new team.

software development life cycle

Leaving Google to join another company can be somewhat frustrating because not many businesses can match Google’s productivity. But what exactly is the difference? First, we should consider what we are doing every day and then identify the source of this frustration.

One thing is for sure, the general form of the software development life cycle is similar whether you work at Google or not:

1. Think of a feature you want to build, or a bug that needs to be fixed.

2. Read a bunch of code, design docs, and ask colleagues questions. In this way, we build an understanding of the problem and see how the solution fits roughly into the existing system.

3. Start writing code. The first priority of development is to make what you write run. During this time, we may look for documentation multiple times or refer to more legacy code.

4. The workable results have come out, but they cannot be used directly. The code result fails some tests, so it needs to be fixed first. After more testing, complete code refactoring, and lowering the understanding threshold of code design, the results can then be released to the branch. At this point, we can wait for the CI to run and introduce some additional fixes and small changes.

5. Submit patches for review. Colleagues will of course comment and we will adjust accordingly. This process may be repeated several times before the change is actually approved.

6. Merge patches and actually deploy.

7. Existing monitoring systems will determine if the new patch will cause production issues. If our patch causes an outage, it is our natural responsibility to fix it. Throughout the various stages of the process, there is usually a tool that anchors the actual developer experience. Specific tools shape our work cycle and have a huge impact on productivity.

To increase productivity, it’s best to find better tools at each step. Here is a useful GitHub repo that can identify the most similar external tools for almost every tool inside Google: https://github.com/jhuangtw/xg2xg .

This list is very comprehensive, but it’s really too much. So where do we start?

Month 1: Don’t rush to introduce new tools, get familiar with existing tools first

After leaving Google, in the first month of joining the new company, don’t rush to make changes, listen and learn more.

As new members of the team, you don’t yet have enough influence or authority to change the various tools the team uses. In addition, we also lacked practical knowledge, such as not knowing how the new team works, why it was chosen to work this way, and why it is necessary to use the current toolset.

If you simply copy over Google’s internal tools, it doesn’t necessarily work well with new teams. So, you need to figure out which tools are suitable for the new team and which are not.

Start with Code Search

You can start with a code search first. In fact, when a programmer leaves Google, what he misses most is often the code search tool.

You can try out different code search engines yourself, verify how well they work, and recommend them to colleagues when you know they work. Don’t be in a hurry to recommend tools you are unfamiliar with to colleagues or submit them to decision makers for review.

At the same time, you don’t have to change other people’s original work habits, after all, new teams often don’t use code search tools. Even if they are already in use, there are two cases: the worse case is that the tools they choose are ineffective and the frequency of use is not high; the better case is that the tools they choose are already very good and don’t need to worry about it anymore.

If you’re at a new company with multiple development teams, there may be more code to deal with, and it’s certainly necessary to understand this code as a new employee. And even in small but sophisticated startups, it’s possible to introduce a lot of open source code through dependencies. Only after digging into the code can we move forward with building new features or tracking critical bugs.

Today, almost every developer has to deal with the sheer size of the code, so if the code search tool can’t keep up, it will definitely slow down your development speed greatly.

When evaluating code search engines, we need to consider the following key points:

• Query language: Regular expressions are the key. We need to ensure that the code search query language is expressive and simple to use. Literal searches should be more intuitive and provide more advanced pattern matching capabilities.

• Scale: Make sure the code search engine is scalable to match the size of your codebase. If your codebase exceeds several gigabytes, it is very important that the code search engine supports triple indexing ( https://swtch.com/~rsc/regexp/regexp4.html ), which is what we do in large scales in a conventional way. The only way to implement expression matching on the codebase.

• Code Browsing: As a Google Code Search user, we all know that search itself is only half the story. After clicking a result, we would of course want to be able to keep clicking, jump to the corresponding definition and find the referenced section, and the whole process should be as easy as checking out the code and setting up the development environment in the IDE. Without a great code browsing experience as a foundation, we would have to constantly switch back and forth between editors and code search engines.

• Permissions: If your company enforces codebase permissions, you should consider whether code search engines support these permissions.

• Overall cost: Consider both the price of the code search engine and the maintenance overhead of keeping the tool running. The following are our usual code search engines:

• OpenGrok : A fairly old but still living code search engine, now maintained by Oracle.

• Hound : A code search engine created and open sourced by Etsy engineers.

• Livegrep : A code search engine created by Stripe company Nelson Elhage. And, of course, Sourcegraph, which we developed ourselves.

those useful monitoring tools

Another important early goal is monitoring. Every engineer needs to deal with production issues in a specific situation. Note that the production environment is completely different from the development environment, and it is impossible to set breakpoints or add printfs in production and expect to see results in seconds. In addition, updates to the production environment are associated with extremely high costs: computing resources, developer time, and possibly a poor experience for customers.

Deployment thinking has changed considerably over the past 5 to 10 years. A series of new things, such as microservices, Kubernetes, and cloud migration, all mark a major shift in the way enterprise software is deployed. Many companies are adopting these new paradigms and technologies without updating their monitoring infrastructure, making it difficult to debug in new production environments.

Fortunately, some great open source tools and vendors have emerged in recent years that have worked to greatly improve monitoring and observability tool options outside of Google.

• Prometheus is a time series metrics tracker and visualizer similar to Borgmon. It helps you instrument your application, continuously tracking metrics like CPU utilization, error rates, and 90th percentile latency over time.

• Grafana is a dashboard tool similar to Viceroy. People often pair Grafana with Prometheus to build a one-page view of a set of key metrics against which to understand the overall health of the application.

• Google pioneered the use of Dapper, an important distributed tracing tool for multi-service architectures. One of the founders of Dapper, Ben Sigelman, went on to create Lightstep. Distributed tracing is now a feature in numerous monitoring systems, including paid products like Honeycomb and Sentry , as well as open-source projects like Jaeger, developed by engineers at Uber. Monitoring is a bit more complicated than code searching because it has to be integrated into production. This often involves changing the deployment environment, so it must be coordinated and coordinated with the team responsible for controlling the deployment environment. Also, it may involve adding instrumentation code, which means submitting patches to the teams that hold the instrumented code.

But in general, the introduction of new tools does not require changing the original work habits of other colleagues, so it is not too difficult. People are free to use or not use the new tool, so there isn’t too much pressure to promote it up front.

After the first battle: code review

The introduction of code search and monitoring does not affect the existing workflow of any team member. But code review tools are different, and it can actually affect the way colleagues work.

If you’ve worked at Google for a while, you may feel a little weird about the way code reviews are done outside of Google. GitHub PRs are the most common code review tool, but Googlers still find quite a few problems:

• Changes that have occurred since the last round of review are not easy to see, sometimes not at all. Simple paths only allow us to see diffs for parts that have not yet been completed.

• Stacked CRs are not supported.

• The change page brings together all the differences in all files, and it’s hard to figure out which specific part we’re dealing with.

• GitHub PRs are not designed with a clear review path. Without adding additional third-party integrations, the entire review process can feel very loose. And even with third-party integrations, fine-grained review and sign-off policies are still lacking.

• For some languages, GitHub PR’s support for fuzzy jumping to a definition or finding a reference is so poor that it’s not at all comparable to Google’s internal Critique. Outside of Google, the closest tool we can find to Critique is Gerrit. Gerrit was originally a fork of Rietveld, itself an open-source fork of Mondrian, Google’s original code review tool. Therefore, it is this bloodline that makes people feel the idea of ​​code review in Google’s eyes in Gerrit.

Phabricator is also a good code review tool, better than GitHub PR for ex-Googlers anyway. Phabricator started out as Facebook’s internal code review tool and was subsequently open-sourced. It’s run by Phacility, which provides managed instances and support, taking the pressure off customers who don’t want to maintain their own instances.

Another tool to watch is Reviewable, developed by former Google employee Piotr Kaminski. Unlike Gerrit or Phabriactor, Reviewable only supports cloud environments, but in exchange is the closest to Google’s internal code review experience.

When explaining the benefits of Gerrit, Phabricator, or Reviewable to the rest of the team, be sure to pay attention to how you feel about the original code review tool. Switching from GitHub PRs to tools like Gerrit can effectively solve the following common pain points:

• Gerrit’s ability to define a signature mechanism takes the structure of the review process to the next level. If your team is growing and you want to enforce stricter vetting policies within your organization, Gerrit is definitely worth the money.

• Gerrit simplifies reviewing large-scale differences, allowing us to go file-by-file, review changes since the last review round, and stack CRs, resulting in faster, more thorough reviews. Gerrit, Phabricator, and Reviewable can all move you one step closer to the regular review process within Google, but there’s still an important missing piece: code intelligence. An extension to the Sourcegraph browser that provides tooltips, jumps to definitions, cross-references, and more during code reviews, it works with GitHub PRs, GitLab MR, Phabricator, and Bitbucket Server, with support for Gerrit in development.

It’s time to take the final step

The hardest part of the software development life cycle is often the CI and build systems. This is because to understand the entire build, one has to look at each part of the entire codebase in a very specific way. As time goes by, everyone wants to speed up the build, so the build code has accumulated more and more adjustments and optimizations, resulting in a lot of manpower investment may not be able to achieve painless updates.

In short, the build system is the last guard boss, so don’t try it lightly until you have completed the previous accumulation of “killing monsters and upgrading”. But it is always tempting us, after all, compared to the existing tools, Blaze is too powerful. Google even open sourced Blaze under the Bazel name. But Bazel is not Blaze after all, it lacks a large-scale distributed build cluster, and it is not running inside Google after all.

So let’s admit it, Bazel is by no means a panacea. When it was first released, many open source projects in the Go community switched to it to support the standard Go build tools. But within a year, everyone chose to give up due to the fact that it was too complicated to use, many members of the Go community were not familiar with it, and Bazel seemed to be slower to build. Since then, Bazel’s support for Go has improved significantly, but it is recommended that you do a serious evaluation before using it.

In order to perform this rigorous assessment, we first had to put other development tools in place: including a good code search, making sure to really dig into the build scripts of the various parts of the codebase to understand what they do and what their dependencies are. You also need great code review tools to ensure that changes to the build system are supported and coordinated across different engineering teams.

Once you’re ready, be clear. Besides Bazel, there are many other build tools that enable scalable builds in large codebases. Specifically include:

• Buck , via Facebook.

• Gradle , very popular in the Java world.

• Pants , designed by ex-Googlers for Twitter and Foursquare.

• Please , a new build tool created by ex-Googlers inspired by Blaze.

• There’s also YourBase, which isn’t a build tool, but a CI service developed by ex-Googler Yves Junqueira, but it does provide a super-fast and scalable build experience, completely independent of the actual underlying build tool. Google has a corporate culture unlike most other companies, including a strong arsenal of tools that prioritize developer experience. Former Googlers have experienced these state-of-the-art development tools, so they will never forget the experience of making the most of their talents and abilities.

One of the big competitive advantages people have after leaving Google is to use these experiences to infuse a great development experience into their new organization, thereby increasing their productivity and that of other team members. With the above-mentioned best practices of large-scale software development, you can completely transform Google’s extremely high engineering organization coordination ability into the underlying strength of the new company.

The process of building large-scale software is quite complex. Experienced friends know that better software cannot be obtained by manpower alone, and we also need better tools. Just as great software can be a productivity amplifier for end users, great development tools are also a productivity amplifier for software developers. So if you want to be the owner of a new company, you can use the expertise of your ex-Googlers to bring world-class developer tools to this new world.

Original link:

https://about.sourcegraph.com/blog/ex-googler-guide-dev-tools

The text and pictures in this article are from InfoQ