Thoughtworks Xu Hao: Are programmers engaged in technology or engineering?

This article is compiled by Geek Time from Thoughtworks global technology strategy consultant and China CTO Xu Hao’s speech in the live broadcast “Are programmers engaged in technology or engineering? 》

Author｜Xu Hao

Editor｜Li Chenyang

In our software industry, many people are very concerned about their own technical level: there is a new version of the Java language, but I don’t know; there is Kotlin in Java, but I don’t know how to use it; the version of the new framework on JavaScript is updated very quickly, and I follow not on; wait.

Their presence often causes us some anxiety. Faced with these new technologies, what should we do? Are you going to be eliminated if you study? This is a common confusion. In fact, the core question behind this confusion is: Are programmers engaged in technology or engineering? This is a question that we must first think and answer when we think about our own career orientation.

It should be known that technical ability and engineering ability are equally important, but engineering ability is what we have neglected and lacked for a long time. For example, I rarely hear people say that their engineering ability to do software is a little weak:

I don’t know how to do software release and deployment;

I don’t know how to setup an efficient deployment pipeline in a team of 2-3 people;

I don’t know how to test software in certain environments.

And this is precisely an important issue facing our industry right now. We blindly think that as long as the technical level is good, career development can be good. But in fact, what should really be mentioned, or closely related to career development, is our engineering capabilities first.

When the engineering level is improved, even if the technical level is very limited, it can be well exerted within the scope of engineering capabilities. So what exactly is technical capability and what is engineering capability?

Technical ability or engineering ability?

Here are two examples where you can feel the stark difference.

An example would be quizzes on LeetCode. It is quite popular in the industry to do this recently. Constantly brushing the questions can make you more familiar with data structure algorithms and more proficient in using language to solve some small technical problems.

Another example is writing CRUD in a project team. If you judge from the perspective of difficulty, I think you will definitely feel that brushing the questions requires more technology, and the technical content of writing CRUD in the project is not high.

That’s right, because they require completely different abilities. You try to recall that when you are writing questions on LeetCode, you usually hope that you can write the code as quickly as possible and in one go within the time limit. That is to say, the code is completed as soon as it is written, and it is rarely necessary to carry out secondary development or repeated modification on this basis. To write the main, and instead of auxiliary.

On the contrary, when the project team is doing CRUD, technically, we are doing CRUD. But at the same time, also need to understand “why do I do CRUD”. This involves how to understand the business context and business logic.

With the development of the business, there is rarely a one-off situation where the code is written and completed. Therefore, when the project team is doing CRUD, writing code may only occupy 5% of the entire code life cycle. The remaining 95% are based on changes in requirements and functional adjustments, and are continuously iterated on the basis of CRUD.

It can be said that no matter how strong the technical ability obtained from brushing the questions, once put into practical application in the project team, the efficiency can be increased by at most 10 times. Assuming that the original is 5%, an increase of 10 times is 0.5%. Even so, there are still 95% that need to be repeatedly modified and adjusted on the basis of the code. And these abilities cannot be learned from brushing questions.

Therefore, it is found that doing CRUD in the project team actually requires our engineering ability, while brushing questions on LeetCode requires only technical ability.

Let’s take a more extreme case. Suppose there is a situation in the project where “needs will not be done”. At this time, you are faced with two choices: one is to ask others for help immediately; the other is to die on your own first, and then find someone to help you if you can’t do it. I see most people in the barrage choose the latter.

Let me start with my conclusion. If in my team, I hope you can choose the former. In fact, whether I’m TL or the rest of the team, I wish you could choose the former. Choosing the former means that your technical ability may be weak and you cannot complete the requirements, but you have very strong engineering ability.

Because the first step to strong engineering ability is not to be a stumbling block for others.

From an engineering perspective, ensuring that projects proceed with low risk is a very high capability priority. If there is a need that can’t be implemented, you can say “I can’t do this place, who can help me?” This is a very good engineering practice. Don’t choose to hold back for a while because you don’t want others to know that you can’t. Doing so will only waste the team’s time and put the team at risk.

Of course, you may also say that asking for help from others is not a waste of other people’s time? What I’m saying is: it’s not up to you. As the TL or PM of the project, he will unify the arrangement of the whole article to see if you can continue to hold back or let others do it. If others can’t, you may need to ask for an external brain.

From the perspective of project risk, we need to reduce project risk as much as possible by passing real information and build a trust relationship among the team. If you say you can do it, you can really do it, and there will be no situation where “you say you can do it, and others think you can’t do it.”

From the perspective of personal engineering ability, building a trust relationship and not becoming a stumbling block for others is the most important quality of programmers. But in practical work, we rarely consider such a problem. Of course, we’re using an extreme example to help you understand how extreme we can go when we emphasize technical and engineering capabilities.

At this point, I can summarize the engineering capabilities as: the ability to output stably for a long time and continuously improve the level in a team collaboration environment. Next, I will expand on it.

What is engineering capability?

As mentioned earlier, the premise of engineering ability is in a team collaboration environment, which means that the project is not carried out by you alone.

When we taught ourselves to code in school, we always assumed that I needed to do it all by myself. Assuming that you are the strongest TL in the project and the person in charge of the overall project, then you really need such a person to come to the bottom. But in actual work, most people work in a collaborative environment, so we are required to have “collaboration” behavior. In other words, the engineering ability asks how we can become a better Team Player.

The second requirement for engineering capability is long-term stability. This is different from what we do on LeetCode. In actual work, it is not so important to write the code at a fast speed, after all, it needs to be modified in the future. So more importantly, we want to write better code, better readability, easier to understand.

There is no need to output a certain amount of code in a short period of time, say 700 lines of code in 1 hour. What is more needed is to maintain the output level of 200 lines per day, which is good enough. In the long run, as requirements change, if we can maintain this output rate all the time, this is the level of stable output we really hope programmers can produce.

The last is to continue to improve the level. Collaborative ability pays more attention to long-term continuous output, continuous learning and continuous improvement, and it even directly determines whether we can have good engineering ability.

So on this premise, I want to emphasize that TDD (Test-Driven Development) is currently the most engineering-efficient development process. Next, let’s take a look at how TDD helps us achieve stable output and continuous improvement in an environment of teamwork.

Most Engineering Effective?

First of all, in the environment of team collaboration, when doing TDD, it is necessary to dismantle the requirements first. In other words, when the task split is done, it actually shows everyone on the team how we understand the requirement and how we plan to implement it.

This method of showing is not as simple as “it takes three steps to put an elephant in the refrigerator”. Instead, you need to understand each step in place and effectively translate the task into one or more tests. If it can be converted into a test, it means that the requirements are really understood. On the contrary, it shows that the understanding is not in place.

In fact, not only did I understand the requirement, but I also proved to others that I did understand the requirement, supported by testing. This is the basic point to prove that you are a reliable employee in teamwork.

The second is long-term stable output. The reason why TDD can guarantee long-term stable output is that we need to write tests and code together. Why do you say that?

I guess you’ve heard “tests are documents”, which is not entirely true. Tests are not documents by nature, they are just records of how we implement the software. When the implementation process is processed and organized, it can become a truly effective document. So testing provides a lot of techniques that are closer to the actual implementation than we write comments in the code or code design in the documentation.

The automated execution method of TDD can not only help us verify the understanding of the requirements at that time and what kind of deviations have occurred. It is also easy to help us trace how and why the bug occurred. On this basis, our output is long-term stable. These are the two main purposes of testing I’m emphasizing: finding bugs and locating bugs.

You must know that the reason why our production efficiency is getting slower and slower, and the reason why we dare not change the ancestral code, is because once the modification is wrong, it is impossible to locate what went wrong.

Therefore, in the process of constructing software, it is necessary to include a detection technology to help us locate where the error is. The test generated by the task list in TDD is not only a record of our software implementation process, but also helps us find problems in the software and the location of the problems.

The third requirement is to continuously improve our level. On the one hand, in the process of developing with TDD, we will have a clearer and clearer understanding of the requirements and architecture, and the requirements and architecture will also be directly reflected in the task list. In fact, in the task list, our ability itself increases as more and more clear tasks are produced, tasks that are easier to convert into tests.

On the other hand, in the process of writing tests, for the same type of task, not only can the efficiency of implementing it become higher and higher, but this efficiency can also be measured. For example, when I implemented a similar task before, it took a day. But with the development of time, as the technology becomes more and more proficient and the cognition becomes higher and higher, the time required becomes shorter and shorter. Because TDD can help us frame a large measurement range.

The overall workflow of TDD

With these clear, let’s further explore the workflow of TDD. As shown in the picture below, it is a picture I mainly use in the course. At first glance, there may be a big deviation from what you understand about TDD (Note: The picture is from the Geek Time column “Xu Hao · TDD Project Practice 70 Lectures”).

Let’s first look at the complete workflow of TDD. For example, if there is a set of requirements, we need to decompose the requirements into function points. From a user perceptible point of view, or when decomposed from large functional blocks, these functional points can be directly translated into tests. However, it is more troublesome to directly convert the function point into a test, so I need to further divide the context within the function point. We call it functional context.

Where does the division of functional context come from? The easiest way to do this is from the architecture, because the architecture is how many kinds of components exist in the system and how the components interact with each other. For example, in the context of the MVC framework, there are three components: Model, View and Controller. If I want to implement a user login function, in terms of function points, there are two major function points: the user can log in after entering the correct password, and the user can not log in if the user enters the wrong password.

When we want to implement the first major function point, we need to make a user login View, a Controller and a Model representing the user. Then in the context of entering the correct password to log in, the different components can be decomposed into smaller functional task items.

For example, when a function is correctly logged in on the View menu, there is a prompt “My current login interface can display the username and password”, and the second item is “When the username and password are entered, the original code should be overwritten. “, and so on, there will be some specific task items.

Many people have a big doubt when learning TDD: they don’t know where the test comes from. Because when we read Kent Beck’s book, we feel that he is imaginative, as if he writes a lot of code at will. There are actually many considerations behind this.

His consideration is to decompose the requirements into functional points first, and then decompose the functional points into functional contexts along our understanding of the architecture. Then it is decomposed into specific task items from the context, and the test items are written by the task items. This is the more complete TDD process, and that’s where the tests come from.

Same problem with refactoring. A concept that is easy to confuse with it is rewriting, that is, after getting the code, rewrite it directly on it. But refactoring is a strict definition. Refactoring is talking about the fact that I want it to function the same and the structure to be better. That is, the code function itself remains unchanged, but the code structure needs to be better. What does this mean? It means that refactoring adjusts the architecture.

Looking at the picture carefully, how does the red/green/refactoring cycle loop? To put it simply, requirements are decomposed into function points, and the relationship between architecture and components is mapped into function contexts at function points, and is decomposed into task items in the functional context, and task items are then converted into tests. Then pass the failure of one or more tests until the test is passed. After refactoring, we can reorganize the components and architecture in the system. The reorganized components and architecture will affect the re-decomposition of functional contexts.

In other words, when you do a refactoring in TDD, it is very likely that the next time you use it for development, the disassembly of the task will change, so it can form a complete cycle.

Analysis here, there is a very interesting discovery. TDD is not actually a coding technique. Because it doesn’t drive us to implement all the functions in the task item, it really drives the architecture. This is exactly what we’re talking about as a coding architect, a true doer rather than a PPT architect.

So why is TDD an engineered development process? Be aware that the architecture (the relationship between components) needs to be shared across teams. In other words, we can observe whether an architect is doing a good job.

I often try something like this when I’m doing counseling. I would call the development engineers on the team into separate rooms and let them de-task the features that will be iterated on. In the end, it was found that the five people decomposed into five different looks. In effect it means that the architects on this team are not doing any work.

Architecture is the strangest kind of output in the entire software development process, it has no value in itself but only guides the work of others. We can only write the right architecture and the right code if we understand what the architecture is. When we say that the architecture was born, if it only exists on paper, it doesn’t make any difference. Or it could be an architectural vision that can be easily broken. Architecture only works when it gets into everyone’s head, directing engineers to do specific work.

If everyone splits the functional points and functional contexts according to the architectural components, it means that the architectural vision has been well planned, and the rest will become simpler.

Therefore, the failure to implement TDD in the team has never been caused by people not writing tests first and not entering the red/green cycle. It is because the architectural vision is not effectively maintained, and it is not known in what way to decompose the requirements.

It can be said that when doing any software development, understanding the requirements and understanding the architecture is the premise and starting point for us to start. In this form, TDD tells us that “I really understand the need” must be shown to others in a way that can be consumed and can be seen and touched. And I really understand the requirements, because I can decompose the requirements into function points. I really understand architecture because I can contextualize architecture within function points.

If this is done, then in subsequent software development, 70% of the problems may not exist. TDD only provides efficiency.

But on the contrary, in our software industry, everyone pays great attention to their own technical level, and does not emphasize the ability of engineering practice. Therefore, our industry generally lacks these two capabilities:

Give me a requirement, I can properly decompose it into corresponding function points;

Give me an architectural vision and I can slice functional points into corresponding functional contexts.

How to get these abilities? The easiest way is to practice! You can think of TDD as a training tool. When you emphasize TDD in every practice, you will eventually become a better programmer, because you have been tempering the core ability of the programmer’s engineering level.

Therefore, through TDD, even if we do not consider the automated testing part at all, and do not consider the advantages brought by automated testing at all, but only emphasize the understanding of the requirements from the requirements decomposition and architectural vision, and turn them into acceptable tasks, we can be in the software. The development efficiency has been greatly improved and improved. That’s why I say that TDD is still the most engineering-efficient development process out there.

However, TDD is also by far the most difficult engineering method to master.

hardest to master

For TDD, it is not only difficult to learn individually, but also difficult to implement in a team. Take my study experience as an example. I have participated in various algorithm competitions since the fourth grade of elementary school. It can be said that I have no lack of technical skills, and I do not think TDD will bring me any additional benefits. However, when I first got into TDD in 2002, I quickly discovered that it really helped me implement engineered code more efficiently and with more confidence.

While teaching myself TDD, I read Kent Beck’s book many times. When I saw it for the 30th time, I suddenly realized that the book has a very small list of what to do next. While there is a chapter in the book that talks about task lists, for a long time I didn’t directly associate task lists with tests.

Later, I thought back and found that TDD first generated a task list, and I can further decompose it into tasks that can be tested on the task list. I spent a long time thinking about this. Similarly, this is also a big difficulty that many students encounter when learning TDD.

Another struggle is that programmers who develop with TDD have more or less their own style and habits, and others’ habits and methods don’t quite apply to us.

TDD can be seen as a programming idiom or programming method. It’s like everyone is running, but everyone’s arm swings and leg lifts are not the same. The same goes for TDD. So when I go to actually use TDD, Kent Beck’s personal methods, like trigonometry, don’t feel like they’re necessary.

So I forced myself to use TDD to write all the programs. Not only have I written application projects with TDD, but I have also written compilers. After more than a year of training, I feel that I have almost mastered TDD. So it is indeed a very difficult development method to master.

For nearly two decades, I’ve tried to lower the barriers to learning TDD by introducing some practice. At the end of the course, I’ll also describe how to use some tools to help us implement TDD more effectively in our teams if we combine architecture, requirements decomposition, and testing strategies.

If you had asked me what the TDD process was ten years ago, the diagram would have been completely different from this one. There was no architectural vision in the diagram at that time. So this class is a summary of my latest attempt. I hope you can provide me with some suggestions, which is also the purpose of this class.

Summary

When thinking about career development, I recommend not focusing solely on technical ability. For example, does AI need to be chased? Unless you want to aspire to be an AI engineer, the more important question to ask is: When will AI be engineered? When AI is engineered, in what form will it relate to software engineering? When AI enters the software industry, how will our approach and style of doing things change?

It should be known that when planning your career, technical ability and engineering ability are equally important, but engineering ability is what we have long neglected and lacked. Fundamentally speaking, it is not only to pay attention to one’s own technical ability, but also to pay attention to one’s own engineering ability. Then TDD is the most efficient engineering development method in my opinion. Of course, it is also difficult to master.

The text and pictures in this article are from Sina.com