Permalink to this article – https://ift.tt/pNDa0qx
1. Means of Code Quality Assurance
Since the birth of the world’s first high-level computer programming language Fortran in the 1950s, the programming industry has gone through nearly 70 years. Although it has been a lot of years, it is undeniable that software production is still not as standardized as hardware. For the same small function, there are N ways for N programmers to realize it .
So how to ensure that the quality of the software produced meets our requirements? Programmers in different fields are making efforts, such as: making compilers to make compilers more strict, and striving to completely eliminate memory safety problems (such as Rust ); making tool chains for programmers to provide programmers with various built-in languages. Tools such as unit testing, integration testing, interface testing, and fuzzing testing (such as the Go tool chain) make it easier for programmers to conduct comprehensive tests on the code they write in order to find out more potential problems in the code …
Of course, there is another subjective code quality assurance method that is still the mainstream, and it is peer code review (code review, cr) .
There are two main methods of code review. One is that everyone sits in a conference room and “speaks” about a certain piece of code of someone; the other is to use tools like gerrit to review other people’s comments online. The code of a certain submission or the code of a PR is “commented”.
However, no matter what, at the beginning, everyone will see the code structure design from the syntax level to the business logic level in detail, but the drawbacks of doing so are also obvious, that is , inefficiency and lack of focus .
So people thought about whether they can use tools to find grammar-level problems as much as possible, so that human experts can focus on code structure design and business logic problems during code review. After the division of labor is clear, the efficiency will naturally improve (as shown below) :
Note: At present, most toolchains can only automatically help programmers solve problems at the syntax level. In the future, with the increasingly powerful tools, the tools can continuously upgrade the level of attention, and gradually evolve to have the ability to discover code structure design problems and even business-level logic problems.
So there is a code repository tool like reviewdog that can call various linter tools to automatically scan the code and automatically submit problems in the form of comments.
Many friends here will ask, even if the tools focus on the grammar level, why use tools like reviewdog, git’s pre-commit hooks, git server hooks, and tools such as Make can also check the code. grammatical issues in, are they no longer fragrant?
Here’s a quick look at the “problems” with these approaches (we assume everyone is already using git as a code versioning tool):
- git pre-commit-hook
The git pre-commit hook is a client-side git hook. It is a hook placed in the .git/hooks directory of the developer’s local code copy. When the developer executes git commit locally, it will be invoked for execution. The problem with pre-commot hooks is that we cannot manage and maintain the script content of pre-commit hooks in a unified manner in the central code repository. This is more suitable for developers to deploy in their own development environment according to their own preferences and code literacy.
In addition, some codes are not necessarily submitted on the developer’s own development machine. After changing the environment, the pre-commit hook will no longer take effect.
- Use tools such as Make to do local inspections
Using the make tool, we can perform various static checks such as lint on the code before building the code locally, but like pre-commit-hook, although the Makefile can submit the code repository, the real tool for checking the code is still in the developer. Locally, it is difficult to uniformly manage and maintain tool versions and set inspection rules, which may lead to inconsistencies between different developer environments. In the same situation, some codes are not necessarily submitted on the developer’s own development machine. After changing the environment, the code checking tool that the Make tool depends on may not exist, and the checking process cannot be effectively implemented.
- git server hooks
git supports server hooks , and gitlab has also supported server hooks since version 12.8 (replacing the previous custom hooks).
The Git server supports the following hooks:
- pre-receive
- post-receive
- update
I haven’t studied whether these server hooks can meet our functional requirements, but it is determined by the deployment characteristics of git server hooks. It is not suitable because it needs to be executed on the gitlab server, which means that we need All static code inspection tools must be deployed and configured in the same environment as the gitlab server, which is too coupled, making it inconvenient for us to manage and maintain these static code inspection tools.
And tools like reviewdog will integrate with ci tools (such as gitlab-ci) and run on slave/worker/runner machines, and the environment on these machines can be easily customized and managed uniformly.
Well, now enter the reviewdog time!
Note: Let’s take the code warehouse as gitlab as an example. I have done a small survey. At present, gitlab is basically used in enterprises to build private git warehouses, except for those big factories that implement the code warehouse platform.
2. What is reviewdog
What kind of tool is reviewdog? Let’s take a look at the diagram below:
We see that this is a ci execution flow chart based on gitlab. In this process, reviewdog runs on the gitlab-runner node, which is the node responsible for actually executing the ci job. Whenever a developer executes a git push to synchronize the commit to the code repository, a ci job will be triggered. On the gitlab-runner node that hosts the ci job, reviewdog will be invoked, and it will do three things:
- Call the static code inspection tool to check the latest pulled code;
- Compare the code inspection result (which line has a problem) with the commit diff result, and get the intersection (that is, the lines of code changed (add and update) in the commit diff are consistent with the lines of the code inspection result, put them in the intersection) ;
- Post the code inspection result information in the intersection to the gitlab repository in the form of a gitlab commit comment
In this way, developers can see these comments through the commit page, and respond to these comments, if necessary, to fix these problems.
We see that the biggest difference between reviewdog and other tools is that it can find the intersection between the commit diff and lint results, interact with the code repository, and put the results of these intersections into the commit page in the form of comments, just like peers During code review, peers add comments directly on your commit page .
However, the current version of reviewdog does not support checking and submitting comments directly on gitlab-push-commit. It may be that such scenarios are rare, because open source projects currently use more pr (pull request)-based workflows, so reviewdog has built-in Code review for workflows such as github-pr-check, github-pr-review, gitlab-mr-commit, etc. And based on gitlab-push-commit we use may be rare (of course, we use this internally in a specific context).
So how to make reviewdog support gitlab-push-commit, that is, to perform static code inspection on the commit in the push action and put the result into the commit page in the form of a comment? We can only fork the reviewdog project , and add support for the gitlab-push-commit mode to the project after the fork .
3. Modify reviewdog to support gitlab-push-commit mode
reviewdog is a command-line tool, usually one-time execution, so its code structure is relatively clear. We can simply figure out how to add support for the gitlab-push-commit mode around the several reporter modes it supports.
The meaning of gitlab-push-commit mode is explained here. First of all, this mode is suitable for ci jobs triggered when developers push code to gitlab through git push. In the ci job, reviewdog will run the configured static code analysis tool (such as golangci-lint, etc.) to scan the latest code and get the problem set; then get the latest commit sha value (CI_COMMIT_SHA) and the latest before push The sha value of the commit (CI_COMMIT_BEFORE_SHA), and compare the diff between the two versions. Finally, the “intersection” between the problem set and the diff set is found through the file name and line number, and the result is submitted to the latest commit page of this push in the form of a comment through the gitlab client api.
At present, there is still a “flaw” in this mode, that is, if there are multiple commits in a push, the gitlab-push-commit mode will not do diff and comment for each commit, but only use the latest commit and push in the push. Compare with the latest commit before.
After defining and clearing the meaning of the gitlab-push-commit mode, we can add support for this mode to reviewdog “according to the gourd and drawing a scoop”!
In main.go, we mainly add a reporter case branch to the run function:
// https://github.com/bigwhite/reviewdog/blob/master/cmd/reviewdog/main.go func run(r io.Reader, w io.Writer, opt *option) error { ... ... case "gitlab-push-commit": build, cli, err := gitlabBuildWithClient(opt.reporter) if err != nil { return err } log.Printf("reviewdog: [gitlab-push-commit-report] gitlabBuildWithClient ok\n") gc, err := gitlabservice.NewGitLabPushCommitsCommenter(cli, build.Owner, build.Repo, build.SHA) if err != nil { return err } log.Printf("reviewdog: [gitlab-push-commit-report] NewGitLabPushCommitsCommenter ok\n") cs = reviewdog.MultiCommentService(gc, cs) ds, err = gitlabservice.NewGitLabPushCommitsDiff(cli, build.Owner, build.Repo, build.SHA, build.BeforeSHA) if err != nil { return err } log.Printf("reviewdog: [gitlab-push-commit-report] NewGitLabPushCommitsDiff ok\n") ... ... }
In this case, we mainly prepare gitlab client objects, PushCommitsCommenter objects (located in service/gitlab/gitlab_push_commits.go), and PushCommitsDiff objects (located in service/gitlab/gitlab_push_commits_diff.go) for the following project.Run or reviewdog.Run methods )Wait.
gitlab_push_commits.go and gitlab_push_commits_diff.go are two newly added go source files, which are also rewritten with reference to gitlab_mr_commit.go and gitlab_mr_diff.go in the same directory. The specific code is not listed here, you can read it yourself if you are interested.
4. Deploy gitlab-runner to verify the new version of reviewdog
Let’s verify the reviewdog after the above transformation.
1. Install gitlab-runner
We first build an experimental project on gitlab, and then configure ci for the project. If your gitlab has not registered gitlab-runner, you can install and register the runner node as follows (it can be established under the top-level group, so that the runner can be shared within the group: settings => CI/CD => Runners => Show runner installation instructions have detailed command instructions for deploying the runner):
//假设我们有一个ubuntu 20.04的主机，我们可以按下面命令安装和注册一个gitlab-runner： sudo curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64 # Give it permissions to execute sudo chmod +x /usr/local/bin/gitlab-runner # Create a GitLab CI user sudo useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash # Install and run as service sudo gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner sudo gitlab-runner start # 注册该runner sudo gitlab-runner register --url http://{gitlab-server-ip-addr}/ --registration-token {registration token}
The above command will create a runner’s own configuration file under /etc/gitlab-runner: config.toml:
// /etc/gitlab-runner/config.toml concurrent = 1 check_interval = 0 [session_server] session_timeout = 1800 [[runners]] name = "runner for ard group" url = "http://gitlab_ip_addr/" id = 1 token = "{registration token}" token_obtained_at = 2022-09-01T11:03:43Z token_expires_at = 0001-01-01T00:00:00Z executor = "shell" shell = "bash" environment = ["PATH=/home/tonybai/.bin/go1.18/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"] [runners.custom_build_dir] [runners.cache] [runners.cache.s3] [runners.cache.gcs] [runners.cache.azure]
Here I chose the shell executor, which executes the commands in the ci job based on the host shell. The environment under runners can set the environment variables of the shell, and the settings here will override the environment variable values under the corresponding account (such as gitlab-runner).
After the successful deployment of gitlab-runner, we can see the following available runners under the runners of the group:
Note: When creating the runner, I set two tags for the runner: ard and ci.
Note: Make sure that the command executed by the runner can be found under the PATH of the host.
2. Create a personal access token
reviewdog needs to access the gitlab repository through the gitlab client API to obtain information and submit comments, which requires us to provide an access token for the command executed by the runner.
Gitlab has a variety of access tokens, such as: personal access token, project access token, etc. We create a personal access token, and I have also tested the project access token. Using the project access token , the comment can be successfully submitted, but the notify mail cannot be sent out nine times out of ten.
The access token should be saved because it is only displayed once.
We configure the personal access token into the variable of the experimental project (Settings => CI/CD => variables), the key of the variable is REVIEWDOG_GITLAB_API_TOKEN, and the value is the token just created.
Each subsequent CI job execution, the variable will take effect on the job as a predefined environment variable. Our reviewdog can use this token to access gitlab.
3. Configure the ci pipeline of the experimental project
We can configure the ci pipeline of the experimental project in the form of code. We create a .gitlab-ci.yml file in the project root directory, and its content is as follows:
// .gitlab-ci.yml build-job: tags: - ard stage: build script: - export CI_REPO_OWNER=ard/incubators - export CI_REPO_NAME=learn-gitlab - reviewdog -reporter=gitlab-push-commit only: - master - pushes
For the specific field meanings of .gitlab-ci.yml, please refer to the gitlab documentation. There are a few things worth noting in this configuration:
- Use tags to associate runner (here, use the ard tag);
- The script part is the list of specific commands executed by the job. Here, first set two environment variables, CI_REPO_OWNER and CI_REPO_NAME, for reviewdog use; then execute reviewdog;
- The only section describes that the ci job is triggered only for the push event of the master branch.
4. Configure.reviewdog.yml
Finally, let’s configure the reviewdog configuration file suitable for the experimental project. We also create a .reviewdog.yml file in the project root directory with the following contents:
runner: golangci: cmd: golangci-lint run --max-same-issues=0 --out-format=line-number ./... errorformat: - '%E%f:%l:%c: %m' - '%E%f:%l: %m' - '%C%.%#' level: warning
Here we see that we use golangci-lint, a static checking tool, to check the code of the experimental project. The meaning of –max-same-issues=0 here is to not limit the number of identical bugs. As for the specific format of .reviewdog.yml, the .reviewdog.yml of the reviewdog project itself is of great reference value, and you can study it carefully when you need it.
5. Push the code and verify the execution result of reviewdog
We can deliberately write some problematic code in the code, and these problems must be scanned by the golangci-lint tool, such as:
package main type Foo struct { A int B string C bool } func Demo1() error { return nil } func Demo2() error { return nil } func Demo3() error { return nil } func main() { f := &Foo{1, "tony", false} _ = f Demo2() Demo1() Demo3() }
There is no error handling for Demo function calls here, and errcheck in golangci-lint can detect this problem. Submit and push these codes to the repository. After a while, we will receive the notify mail, open the commit page, and you will see the following commit comments:
Seeing this result shows that reviewdog is working as expected!
V. Summary
This article describes how to perform static code inspection on push commits based on reviewdog and submit comments in commits like a “peer”.
The purpose of this is to improve the efficiency of code review through tools, while also keeping the lower limit of code quality.
As mentioned at the beginning of this article, with the enhancement of the capabilities of inspection tools, such a solution based on reviewdog’s automatic code inspection can continue to improve in terms of ensuring code quality.
Go has open sourced tool chains such as go/ast, and capable children’s shoes can develop inspection tools with “specific purpose” based on go/ast and integrate them into reviewdog, which will make inspection more targeted and effective.
The source code involved in this article can be downloaded here – https://ift.tt/swtGpZo
“Gopher Tribe” Knowledge Planet aims to create a high-quality Go learning and advanced community! High-quality first published Go technical articles, “three-day” first published reading rights, analysis of the current situation of Go language development twice a year, reading the fresh Gopher daily 1 hour in advance every day, online courses, technical columns, book content preview, must answer within 6 hours Guaranteed to meet all your needs about the Go language ecosystem! In 2022, the Gopher tribe will be fully revised, and will continue to share knowledge, skills and practices in the Go language and Go application fields, and add many forms of interaction. Everyone is welcome to join!
I love texting : Enterprise-level SMS platform customization development expert https://51smspush.com/. smspush : A customized SMS platform that can be deployed within the enterprise, with three-network coverage, not afraid of large concurrent access, and can be customized and expanded; the content of the SMS is determined by you, no longer bound, with rich interfaces, long SMS support, and optional signature. On April 8, 2020, China’s three major telecom operators jointly released the “5G Message White Paper”, and the 51 SMS platform will also be newly upgraded to the “51 Commercial Message Platform” to fully support 5G RCS messages.
The famous cloud hosting service provider DigitalOcean released the latest hosting plan. The entry-level Droplet configuration is upgraded to: 1 core CPU, 1G memory, 25G high-speed SSD, and the price is 5$/month. Friends who need to use DigitalOcean can open this link : https://ift.tt/BCq9JOQ to open your DO host road.
Gopher Daily Archive Repository – https://ift.tt/ERPU8ZB
my contact information:
- Weibo: https://ift.tt/t6weJPX
- Blog: tonybai.com
- github: https://ift.tt/5pekaVh
Business cooperation methods: writing, publishing books, training, online courses, partnership entrepreneurship, consulting, advertising cooperation.
© 2022, bigwhite . All rights reserved.
This article is reprinted from https://tonybai.com/2022/09/08/make-reviewdog-support-gitlab-push-commit-to-preserve-the-code-quality-floor/
This site is for inclusion only, and the copyright belongs to the original author.