Permalink to this article – https://ift.tt/m0r1zWO
As a small factory, our infrastructure is not complete enough. The project manager informed us that our system will be in the second-to-last stage environment and production environment in the near future. Therefore, considering the deployment efficiency of operation and maintenance personnel, we urgently developed a One-click installation script generation tool, so that operation and maintenance personnel can use this tool to generate one-click installation scripts in combination with the actual target environment. The principle of this tool is very simple, as shown in the following diagram:
As can be seen from the above figure, our tool customizes the final configuration and installation script based on the template, among which:
- Below templates/conf is the service configuration;
- Below templates/manifests is the k8s yaml script of the service;
- The custom/configure file stores customized configuration data for the service configuration under templates/conf;
- The custom/manifests file stores customized configuration data for k8s yaml under templates/manifests;
- templates/install.sh is the installation script.
The two files in the custom directory that store custom configuration are closely related to the target environment .
When it comes to templates, Gophers first think of Go text/template technology , which uses template syntax to write template configuration files in the templates directory above. However, based on text/template, we need to identify all the variables that need to be customized in advance, which is a bit large and not flexible enough.
So what other technical solutions can we use? I finally chose the scheme of yaml file merging (including overwriting and appending) , the scheme diagram is as follows:
This example covers both overwriting and (appending) merging, so let’s look at overwriting first.
- The configuration in custom/manifests.yml overrides the configuration in templates/manifests/*.yaml
Take templates/manifests/a.yml as an example. The default value of metadata.name in this template is default, but the operation and maintenance personnel have customized the custom/manifests.yml file according to the target environment. In this file, the a.yml file name is used as the key value, and then the full path of the configuration item to be overwritten is configured into this file (the full path here is metadata.name):
a.yml： metadata: name: foo
The modified value foo of the namespace name in the custom/manifests.yml file will override the default in the original template, which will be reflected in the final xx_install/manifests/a.yml:
// a.yml apiVersion: v1 kind: Namespace metadata: name: foo
- The configuration in custom/manifests.yml is appended to the templates/manifests/*.yaml configuration
For the new configuration that is not in the original template file but added in custom, it will be appended to the final generated configuration file, take b.yml as an example. The contents of b.yml in the original template directory are as follows:
// templates/manifests/b.yml log: type: file level: 0 compress: true
There are only three sub-configuration items under the log: type, level and compress.
The operation and maintenance has added several other configurations to the log in custom/manifests.yml, such as access_log, error_log, etc.:
// custom/manifests.yml b.yml: log: level: 1 compress: false access_log: "access.log" error_log: "error.log" max_age: 3 maxbackups: 7 maxsize: 100
In this way, except that level and compress will overwrite the values in the original template, the other newly added configurations will be appended to the generated xx_install/manifests/b.yml and will be reflected:
// b.yml log: type: file level: 1 compress: false access_log: "access.log" error_log: "error.log" max_age: 3 maxbackups: 7 maxsize: 100
All right! The plan is determined, how to realize the merging of yaml files ? The yaml package in the Go community is the most famous https://ift.tt/bfS5xFo import paths (gopkg.in/yaml.v2 or gopkg.in/yaml.v3), this package implements the YAML 1.2 specification , which can facilitate the implementation of Yaml marshal and unmarshal between go struct. However, the interfaces provided by the yaml package are relatively rudimentary. In order to realize the merging of yaml files, you need to do a lot of extra work yourself, which may not be allowed in time. Are there any ready-made tools available? The answer is yes, it is the famous viper in the Go community!
viper is an open source Go application configuration framework developed by gohugo author and former Go language project team product manager Steve Francia . Viper not only supports command line parameters to pass in configuration, but also supports obtaining configuration from various types of configuration files, environment variables, remote configuration systems (etcd, etc.). In addition, viper also supports merging of configuration files and writing operations to configuration files.
Can we directly use viper’s Merge series of operations? The answer is no! why? Because this is related to our design above. We put the configuration related to the environment into the file custom/manifests.yml, which will cause the configuration data in custom/manifests.yml to appear in each final generated templates/xx.yml in the configuration file.
Then we will implement a set of merge (overwrite and append) operations by ourselves!
Let’s first look at the main function that drives merge:
// github.com/bigwhite/experiments/tree/master/yml-merge-using-viper/main.go var ( sourceDir string dstDir string ) func init() { flag.StringVar(&sourceDir, "s", "./", "template directory path") flag.StringVar(&dstDir, "d", "./k8s-install", "the target directory path in which the generated files are put") } func main() { var err error flag.Parse() // create target directory if not exist err = os.MkdirAll(dstDir+"/conf", 0775) if err != nil { fmt.Printf("create %s error: %s\n", dstDir+"/conf", err) return } err = os.MkdirAll(dstDir+"/manifests", 0775) if err != nil { fmt.Printf("create %s error: %s\n", dstDir+"/manifests", err) return } // override manifests files with same config item in custom/manifests.yml, // store the final result to the target directory err = mergeManifestsFiles() if err != nil { fmt.Printf("override and generate manifests files error: %s\n", err) return } fmt.Printf("override and generate manifests files ok\n") }
We see that the main package uses the standard library flag package to create two command-line parameters -s and -d, which represent the source path for templates/custom and the target path for storing generated files, respectively. After entering the main function, we first create manifests and conf directories in the target path to store related configuration files respectively (in this example, no files are generated in the conf directory), and then the main function calls mergeManifestsFiles to the templates/manifests in the source path. yml file in custom/manifests.yml to merge:
// github.com/bigwhite/experiments/tree/master/yml-merge-using-viper/main.go var ( manifestFiles = []string{ "a.yml", "b.yml", } ) func mergeManifestsFiles() error { for _, file := range manifestFiles { // check whether the file exist srcFile := sourceDir + "/templates/manifests/" + file _, err := os.Stat(srcFile) if os.IsNotExist(err) { fmt.Printf("%s not exist, ignore it\n", srcFile) continue } err = mergeConfig("yml", sourceDir+"/templates/manifests", strings.TrimSuffix(file, ".yml"), sourceDir+"/custom", "manifests", dstDir+"/manifests/"+file) if err != nil { fmt.Println("mergeConfig error: ", err) return err } fmt.Printf("mergeConfig %s ok\n", file) } return nil }
We see that mergeManifestsFiles traverses the template files and calls the function mergeConfig that actually does the yml file merge once for each file:
// github.com/bigwhite/experiments/tree/master/yml-merge-using-viper/main.go func mergeConfig(configType, srcPath, srcFile, overridePath, overrideFile, target string) error { v1 := viper.New() v1.SetConfigType(configType) // eg "yml" v1.AddConfigPath(srcPath) // file directory v1.SetConfigName(srcFile) // filename(without postfix) err := v1.ReadInConfig() if err != nil { return err } v2 := viper.New() v2.SetConfigType(configType) v2.AddConfigPath(overridePath) v2.SetConfigName(overrideFile) err = v2.ReadInConfig() if err != nil { return err } overrideKeys := v2.AllKeys() // override special keys prefixKey := srcFile + "." + configType + "." // eg "a.yml." for _, key := range overrideKeys { if !strings.HasPrefix(key, prefixKey) { continue } stripKey := strings.TrimPrefix(key, prefixKey) val := v2.Get(key) v1.Set(stripKey, val) } // write the final result after overriding return v1.WriteConfigAs(target) }
We see that the mergeConfig function creates two viper instances (viper.New()) for the files under templates/manifests and the manifests.yml file under custom and loads their respective configuration data. Then traverse the key in manifests.yml under custom, set the value of the configuration item that meets the requirements to the viper instance representing the file under templates/manifests, and finally we write the merged viper instance data to the target file.
Compile and run the build tool:
$make go build $./generator mergeConfig a.yml ok mergeConfig b.yml ok override and generate manifests files ok
In the case of default command line parameters, the file is generated in the k8s-install path, let’s take a look at the generated file:
$cat k8s-install/manifests/a.yml apiversion: v1 kind: Namespace metadata: name: foo $cat k8s-install/manifests/b.yml log: access_log: access.log compress: false error_log: error.log level: 1 max_age: 3 maxbackups: 7 maxsize: 100 type: file
We see that the result of merge is consistent with what we expected (the field order is inconsistent, it is related to the use of go map when viper stores key-value internally, and the traversal order of go map is random).
However, careful friends may find a problem: that is, the original apiVersion in a.yml becomes a lowercase apiversion in the result file, which will cause a.yml to fail to verify when it is submitted to k8s!
Why is this so? The official instructions given by viper are as follows (machine flip):
Viper合并了来自不同来源的配置，其中许多配置是不区分大小写的，或者使用与其他来源不同的大小写（例如，env vars）。为了在使用多个资源时提供最佳体验，我们决定让所有按键不区分大小写。已经有一些人试图实现大小写敏感，但不幸的是，这不是那么简单的事情。我们可能会在Viper v2中试着实现它。。。。
Well, since the official said that it may be supported in v2, but v2 is still far away, let’s use the fork version of viper to solve this problem! The developer lnashier once forked a viper code because of this capitalization problem and fixed this problem. Although it is relatively old (and may not be completely changed), it can meet our requirements! Let’s try replacing spf13/viper with lnashier/viper and rebuild and execute the generator:
$go mod tidy go: finding module for package github.com/lnashier/viper go: found github.com/lnashier/viper in github.com/lnashier/viper v0.0.0-20180730210402-cc7336125d12 $make clean rm -fr generator k8s-install $make go build $./generator mergeConfig a.yml ok mergeConfig b.yml ok override and generate manifests files ok $cat k8s-install/manifests/a.yml apiVersion: v1 kind: Namespace metadata: name: foo $cat k8s-install/manifests/b.yml log: access_log: access.log compress: false error_log: error.log level: 1 max_age: 3 maxbackups: 7 maxsize: 100 type: file
We saw that after changing to lnashier/viper, the apiVersion key in a.yml was no longer changed to lowercase.
This tool is basically ready to use. But is there no problem with this tool? Sadly not! The wrong file is generated when the generator is faced with the following two forms of configuration files:
//c.yml apiVersion: v1 data: .dockerconfigjson: xxxxyyyyyzzz== kind: Secret type: kubernetes.io/dockerconfigjson metadata: name: mysecret namespace: foo
and
//d.yml apiVersion: v1 kind: ConfigMap metadata: name: nginx-conf namespace: foo data: my-nginx.conf: | server { listen 80; client_body_timeout 60000; client_max_body_size 1024m; send_timeout 60000; proxy_headers_hash_bucket_size 1024; proxy_headers_hash_max_size 4096; proxy_read_timeout 60000; location /dashboard { proxy_pass http://localhost:8081; } }
These two problems are more difficult, and lnashier/viper can’t solve it. I can only fork lnashier/viper to bigwhite/viper to solve this problem myself, and the configuration form like d.yml is very specialized and not universal, so bigwhite/viper is not universal, so I won’t go into details here Now, interested friends can read the code (commit diff) by themselves to see how to solve the above problems.
The code involved in this article can be downloaded from here .
postscript:
kustomize is the official tool of k8s, it allows you to customize the YAML file for various purposes based on the k8s resource template YAML file (similar to the files in the templates/manifests directory of this article) combined with kustomization.yaml (similar to custom/manifests.yaml), the original The YAML does not make any changes.
However, its goal is only the yaml file related to k8s, which may not be able to do anything for our business service configuration.
CUE is a powerful declarative configuration language that has become popular over the past two years. It was created by Marcel van Lohuizen , a former Go core team member who co-founded the Borg Configuration Language (BCL) – used at Google to deploy all applications language of the program. CUE is the culmination of Google’s years of experience writing configuration languages, designed to improve the developer experience while avoiding some pitfalls. It is a superset of JSON with additional features. The new entrepreneurial project dagger of Solomon Hykes, the father of Docker, uses CUE extensively, and Kubevela , the enterprise cloud native application management platform promoted by Alibaba, is also a heavy user of CUE.
As for how to use CUE to replace my above-mentioned solution, further research is required.
“Gopher Tribe” Knowledge Planet aims to create a high-quality Go learning and advanced community! High-quality first published Go technical articles, “three-day” first published reading rights, analysis of the current situation of Go language development twice a year, reading the fresh Gopher daily 1 hour in advance every day, online courses, technical columns, book content preview, must answer within 6 hours Guaranteed to meet all your needs about the Go language ecosystem! In 2022, the Gopher tribe will be fully revised, and will continue to share knowledge, skills and practices in the Go language and Go application fields, and add many forms of interaction. Everyone is welcome to join!
I love texting : Enterprise-level SMS platform customization development expert https://51smspush.com/. smspush : A customized SMS platform that can be deployed within the enterprise, with three-network coverage, not afraid of large concurrent access, and can be customized and expanded; the content of the SMS is determined by you, no longer bound, with rich interfaces, long SMS support, and optional signature. On April 8, 2020, China’s three major telecom operators jointly released the “5G Message White Paper”, and the 51 SMS platform will also be newly upgraded to the “51 Commercial Message Platform” to fully support 5G RCS messages.
The famous cloud hosting service provider DigitalOcean released the latest hosting plan. The entry-level Droplet configuration is upgraded to: 1 core CPU, 1G memory, 25G high-speed SSD, and the price is 5$/month. Friends who need to use DigitalOcean can open this link : https://ift.tt/IJE04Wh to open your DO host road.
Gopher Daily Archive Repository – https://ift.tt/8CVB9O2
my contact information:
- Weibo: https://ift.tt/eg3pcoE
- Blog: tonybai.com
- github: https://ift.tt/uwJ9Zsy
Business cooperation methods: writing, publishing books, training, online courses, partnership entrepreneurship, consulting, advertising cooperation.
© 2022, bigwhite . All rights reserved.
This article is reprinted from https://tonybai.com/2022/09/20/use-viper-to-do-merge-of-yml-configuration-files/
This site is for inclusion only, and the copyright belongs to the original author.