Small factory internal private Go module pulling scheme 3

04/03/2023 /

Permalink to this article – https://ift.tt/0xHmYW6

1. Dependence

In the past two years, our Go team has built an internal pull private warehouse completely according to the plans in the previously written “Small Factory Internal Private Go Module Pulling Plan” and “Small Factory Internal Private Go Module Pulling Plan (Continued)” The infrastructure is generally good, and there are no major problems at present.

The only troublesome thing, as mentioned in “Private Go Module Pulling Scheme Inside Small Factory” , when adding some repos used as private dependency packages, the vanity.yml of govanityurls needs to be updated manually or automatically through tools. Developers certainly don’t like to maintain such a set of facilities.

At the beginning of the month, a colleague’s host found that the private module could not be pulled through the internal GOPROXY server. Although it was later proved that this was a problem at the network level, it also triggered my thinking. Is there a supplementary solution for pulling private modules outside the unified proxy ? Just a few days ago, a child shoe in the group shared the method that Rust directly uses a repo on the internal self-built gitlab as a dependency, and only needs to do a simple configuration in cargo.toml:

 foo-rs = {git = "http://192.168.10.10/ard/foo-rs", branch = "master"}

Can the current mechanism based on the go module support a relatively elegant solution like Rust? Based on the cognition of go.mod configuration and go get at that time, I didn’t think about it for a while :(. But I also roughly drew a frame for such a plan in my heart:

  • Based on existing go.mod syntax
  • Minimal changes
  • Use go.mod instead of go.work, so that it can be submitted to the code base for version management, and all team members can use it

I thought of doing it based on go mod replace. Of course, I need to make some extensions to replace, so I submitted a proposal to the official go project!

2. Proposal

The core of the proposal is to extend the replace syntax of go mod, so that the target of replace supports a remote vcs warehouse. The following is an example:

 module github.com/bigwhite/demo go 1.20 require ( mycompany.com/go/common v1.1.0 ) replace mycompany.com/go/common v1.1.0 => 192.168.10.159/ard/go/common v1.1.0 //或replace mycompany.com/go/common => 192.168.10.159/ard/go/common //或replace mycompany.com/go/common => 192.168.10.159/ard/go/common v1.1.0 //或replace mycompany.com/go/common v1.1.0 => 192.168.10.159/ard/go/common

In this way, we can not only keep the cannoical import paths we customized for private modules (such as mycompany.com/go/common), but also conveniently pull private modules based on the self-built vcs server.

3. Reverse

My proposal was closed within two hours after it was proposed. I went to check the details. seankhliao replied: Go now supports this usage , and gave an example:

 192.168.10.159/ard/go/common.git

I’m not sure if seankhliao fully understands my proposal, but his reply still makes me wonder if I’m missing something. So I went to re-learn the reference of go module and the reference of go cmd , and then formed a plan to be confirmed in my mind.

The syntax of the replace directive in current go.mod is as follows:

 replace module-path [module-version] => replacement-path [replacement-version]

The replacement-path [replacement-version] constitutes the target part, and currently supports two targets:

One is module path, such as:

 replace example.com/othermodule => example.com/othermodule v1.2.3

The other is a path in the local file system:

 replace example.com/othermodule => ../othermodule

It should be noted that when the replacement-path uses the module path, the replacement-version must be included. The following example will cause an error to be reported when go compiles or runs the command:

 replace example.com/othermodule v1.2.3 => example.com/othermodule

I used to think that when the replacement-path uses the module path, the module path must be the repo address with a domain name. According to the example of seankhliao, this piece seems to be something like: “192.168.10.159/ard/go/ common.git”, if this is the case, then even without using a unified internal go proxy, we can directly pull the private module from the internal self-built vcs server. Let’s verify this solution below.

4. Protocol Confirmation Trials

The following is the topology of the test environment:

This topology is completely different from the solution with unified go proxy:

  • For external public modules, we use external public go proxy (such as: goproxy.io or goproxy.cn, etc.) to pull;
  • For the private module hosted on the internal vcs server, we pull it directly;
  • For the private module (using private repo) hosted on github, we also pull it directly.

Obviously our new solution needs to solve the latter two situations.

In order to illustrate the new solution more intuitively, we assume that one of our go applications depends on three private packages, and their situations are:

  • privatemodule1

The repo is placed on the internal gitlab, and its custom cannoical import path is: mycompany.com/go/privatemodule1, and the actual address is https://ift.tt/MzR0uHl

 $tree -L 1 -F privatemodule1 privatemodule1 ├── go.mod ├── privatemodule1.go └── README.md $cat privatemodule1/go.mod module mycompany.com/go/privatemodule1 go 1.19 $cat privatemodule1/privatemodule1.go package privatemodule1 import "fmt" func F() { fmt.Println("invoke F of mycompany.com/go/privatemodule1") }
  • privatemodule2

The repo is placed in the private repo of github, its custom cannoical import path is: mycompany.com/go/privatemodule2, and the actual address is https://ift.tt/pZL8GA6

 $tree -L 1 -F privatemodule2 privatemodule2 ├── go.mod ├── privatemodule2.go └── README.md $cat privatemodule2/go.mod module mycompany.com/go/privatemodule2 go 1.19 $cat privatemodule2/privatemodule2.go package privatemodule2 import "fmt" func F() { fmt.Println("invoke F of mycompany.com/go/privatemodule2") }
  • privatemodule3

The repo is placed in the private repo of github, such as: github.com/bigwhite/privatemodule3, but there is no custom cannoical import path.

 $tree -L 1 -F privatemodule3 privatemodule3 ├── go.mod ├── privatemodule3.go └── README.md $cat privatemodule3/go.mod module github.com/bigwhite/privatemodule3 go 1.19 $cat privatemodule3/privatemodule3.go package privatemodule3 import "fmt" func F() { fmt.Println("invoke F of github.com/bigwhite/privatemodule3") }

These three situations should cover most of the private module dependencies in daily Go development. Let’s take a look at how to obtain these three types of private modules. Let’s start with the simplest private module3.

1) Pull github.com/bigwhite/privatemodule3

Let’s first build a go app that depends on privatemodule3:

 $cat go.mod module app go 1.19 $cat app.go import ( "github.com/bigwhite/privatemodule3" ) func main() { privatemodule3.F() }

At this time, the settings of GOPROXY and GOPRIVATE are:

 $echo $GOPROXY https://goproxy.io|direct $echo $GOPRIVATE github.com/bigwhite/privatemodule3

This ensures that the go toolchain can pull privatemodule3 through a direct connection.

When we try to use the go mod tidy command to pull privatemodule3, you may encounter the following error:

 $go mod tidy go: finding module for package github.com/bigwhite/privatemodule3 app imports github.com/bigwhite/privatemodule3: module github.com/bigwhite/privatemodule3: git ls-remote -q origin in /home/tonybai/go/pkg/mod/cache/vcs/2caadc923a575b0b63719d0d8b47b67a3559b4dbae40951b750f317880784ada: exit status 128: fatal: unable to access 'https://github.com/bigwhite/privatemodule3/': GnuTLS recv error (-54): Error in the pull function.

This is because go get uses https to pull repo by default . If you do not configure .netrc to access github.com or convert https requests to git+ssh requests, then even if you configure SSH key under github’s personal profile, you will still encounter the above error !

There are two solutions:

  • If you have configured SSH key in github personal profile, then you can replace https request with git+ssh request through .gitconfig

The configuration method is to add the following content in ~/.gitconfig:

 [url "[email protected]:"] insteadOf = https://github.com/
  • If you want to use a personal access token when operating the github repository, then you can configure ~/.netrc to authenticate go get https requests through github

The configuration method is to add the following content in ~/.netrc:

 machine github.com login user password your_personal_access_token

You can choose one of the above two methods. No matter which method is used, after the configuration is ok, and then execute go mod tidy, you will successfully pull the private module on github.com, just like the output result in the following example:

 $go mod tidy go: finding module for package github.com/bigwhite/privatemodule3 go: downloading github.com/bigwhite/privatemodule3 v0.0.0-20230227061700-3762215e798f go: found github.com/bigwhite/privatemodule3 in github.com/bigwhite/privatemodule3 v0.0.0-20230227061700-3762215e798f $go run app.go invoke F of github.com/bigwhite/privatemodule3

In the following example, we will use .gitconfig to replace the https request with git+ssh for the private module on github.com, and I won’t go into details later.

Note: When accessing github.com through https requests in China, the connection rate is low. The git+ssh method can generally pull successfully.

2) Pull the private module located on github.com: mycompany.com/go/privatemodule2

Next, let’s pull the private module located on github.com: privatemodule2. Unlike the first case, this time privatemodule2 has its own cannoical import path, namely mycompany.com/go/privatemodule2. Let’s take a look at the changes in app.go:

 // app.go package main import ( "github.com/bigwhite/privatemodule3" "mycompany.com/go/privatemodule2" ) func main() { privatemodule3.F() privatemodule2.F() }

We add mycompany.com/go and privatemodule2 to GOPRIVATE:

 $echo $GOPRIVATE github.com/bigwhite/privatemodule3,mycompany.com/go,github.com/bigwhite/privatemodule2

At this time, since the domain name mycompany.com does not exist (assuming it does not exist), an error similar to the following will inevitably appear when you execute go mod tidy to pull privatemodule2:

 $go mod tidy go: finding module for package mycompany.com/go/privatemodule2 app imports mycompany.com/go/privatemodule2: cannot find module providing package mycompany.com/go/privatemodule2: unrecognized import path "mycompany.com/go/privatemodule2": https fetch: Get "https://mycompany.com/go/privatemodule2?go-get=1": dial tcp 52.5.196.34:443: i/o timeout

Our scenario is to use the replace directive to replace mycompany.com/go/privatemodule2 with the private repo: github.com/bigwhite/privatemodule2:

 //go.mod module app go 1.19 require ( github.com/bigwhite/privatemodule3 v0.0.0-20230227061700-3762215e798f mycompany.com/go/privatemodule2 v1.0.0 ) replace mycompany.com/go/privatemodule2 v1.0.0 => github.com/bigwhite/privatemodule2 v0.0.0-20230227061454-a2de3aaa7b27

As mentioned earlier, if the target of replace uses the module path, it must carry the replacement version, so where does v0.0.0-20230227061454-a2de3aaa7b27 come from? This is indeed an annoying thing , but we can get it through go list:

 $go list -m github.com/bigwhite/privatemodule2@latest github.com/bigwhite/privatemodule2 v0.0.0-20230227061454-a2de3aaa7b27

Note: If privatemodule2 has a tag in the future, then we don’t need to use the pseudo version number as the replacement version. In addition, the v1.0.0 used by privatemodule2 in require here is a virtual version number, just to meet the grammatical requirements of go.mod, and the real version is the replacement version.

What follows is as expected:

 $go mod tidy go: downloading github.com/bigwhite/privatemodule2 v0.0.0-20230227061454-a2de3aaa7b27 $go run app.go invoke F of github.com/bigwhite/privatemodule3 invoke F of mycompany.com/go/privatemodule2

3) Pull the private module located on the internal gitlab: mycompany.com/go/privatemodule1

Finally, let’s pull the private module located on the internal gitlab: privatemodule1. Same as the second case, this time privatemodule1 also has its own cannoical import path, namely mycompany.com/go/privatemodule1. Let’s take a look at the changes in app.go:

 // app.go package main import ( "github.com/bigwhite/privatemodule3" "mycompany.com/go/privatemodule2" "mycompany.com/go/privatemodule1" ) func main() { privatemodule3.F() privatemodule2.F() privatemodule1.F() }

For the internal gitlab vcs server, we can simply use the method of configuring personal access token in .netrc to use https requests, see the configuration method above.

go.mod becomes:

 module app go 1.19 require ( github.com/bigwhite/privatemodule3 v0.0.0-20230227061700-3762215e798f mycompany.com/go/privatemodule1 v1.0.0 mycompany.com/go/privatemodule2 v1.0.0 ) replace ( mycompany.com/go/privatemodule1 v1.0.0 => 10.10.30.30/ard/incubators/privatemodule1.git v0.0.0-20230227061032-c4a6ea813d1a mycompany.com/go/privatemodule2 v1.0.0 => github.com/bigwhite/privatemodule2 v0.0.0-20230227061454-a2de3aaa7b27 )

We need to add 10.10.30.30 to GOPRIVATE, which can improve the acquisition efficiency (otherwise go get will try to get it through the go proxy server first):

 $echo $GOPRIVATE github.com/bigwhite/privatemodule3,mycompany.com/go,10.10.30.30,github.com/bigwhite/privatemodule2

Here we also need to clarify how to obtain the pseudo-version number (v0.0.0-20230227061032-c4a6ea813d1a) of privatemodule1:

 $go list -m 10.10.30.30/ard/incubators/privatemodule1.git@latest 10.10.30.30/ard/incubators/privatemodule1.git v0.0.0-20230227061032-c4a6ea813d1a

Note: If your gitlab server does not have https enabled, you need to set export GOINSECURE=10.10.30.30.

The next thing is also as expected:

 $go mod tidy go: downloading 10.10.30.30/ard/incubators/privatemodule1.git v0.0.0-20230227061032-c4a6ea813d1a $go run app.go invoke F of github.com/bigwhite/privatemodule3 invoke F of mycompany.com/go/privatemodule2 invoke F of mycompany.com/go/privatemodule1

5. Summary

To sum up, based on the current go.mod syntax, we can pull private modules in various situations without using a unified internal go proxy service. However, from the perspective of the whole process, this solution is still not perfect, mainly because the replacement part uses the module path, which requires a replacement version, and the way to obtain the replacement version is cumbersome, especially if there is no target repo. In the case of tag.

However, this solution can be used as a supplementary solution to the unified go proxy service solution.

Go official will also continue to improve the support for private module pull, there are currently two issues that can continue to be tracked:

  • proposal: cmd/go: allow GOPRIVATE to provide source repository URI – https://ift.tt/afrMcVE
  • proposal: cmd/go: extend syntax go.mod to allow overriding fetch protocol – https://ift.tt/0A187Cu

The code involved in this article can be downloaded here – https://ift.tt/uwqRz3H

6. References

  • https://ift.tt/qKQpteO
  • https://ift.tt/dAkwQRs
  • https://ift.tt/WvHywcU

“Gopher Tribe” knowledge planet aims to create a boutique Go learning and advanced community! High-quality first release of Go technical articles, “three days” first reading right, analysis of the development status of Go language twice a year, fresh Gopher daily 1 hour in advance every day, online courses, technical columns, book content preview, must answer within six hours Guaranteed to meet all your needs about the Go language ecology! In 2023, the Gopher tribe will further focus on how to write elegant, authentic, readable, and testable Go code, pay attention to code quality and deeply understand Go core technology, and continue to strengthen interaction with star friends. Everyone is welcome to join!

img{512x368}

img{512x368}

img{512x368}

img{512x368}

The well-known cloud hosting service provider DigitalOcean released the latest hosting plan. The entry-level Droplet configuration is upgraded to: 1 core CPU, 1G memory, 25G high-speed SSD, and the price is 5$/month. Friends who need to use DigitalOcean, you can open this link address : https://ift.tt/rp4dPH9 to start your DO hosting road.

Gopher Daily archive repository – https://ift.tt/wOyh26b

my contact information:

  • Weibo (temporarily unavailable): https://ift.tt/EtZGYrB
  • Weibo 2: https://ift.tt/V63a08i
  • Blog: tonybai.com
  • github: https://ift.tt/8aHKvRm

Business cooperation methods: writing, publishing, training, online courses, partnerships, consulting, advertising cooperation.

© 2023, bigwhite . All rights reserved.

This article is transferred from https://tonybai.com/2023/03/03/the-approach-to-go-get-private-go-module-in-house-part3/
This site is only for collection, and the copyright belongs to the original author.