Implementation of the program Hot reload config

Leave a Comment / 03/08/2022 /

Original link: https://www.kawabangga.com/posts/4756

Sometimes some processes must be avoided and the restart cost is too high. For example, if there are some long-connected processes, restarting will disconnect the connection, and then all clients need to be reconnected, or a lot of content has been cached in the memory of the process.

But there are some configurations, and we want to modify the configuration of the process without restarting the process.

I talked aboutdesigning configuration items like designing UI. One way is to make a configuration center, and then if there is a configuration modification, push a request to the application, so that when the program receives the configuration update, make a callback to perform configuration changes.

This solution is very good, but it depends on some other components, a centralized configuration center, and SDK dependencies in the program to receive and parse configuration updates.

Originally, the implementation method that does not depend on other components is discussed, that is, updating the file without restarting the process to make the file take effect.

File Watch

This is a more natural way to do it, and it’s the worst way to do it.

Its principle is that after the process starts, a thread is opened to be responsible for all changes in the watch file, and once a change is found, the callback of the configuration update is executed.

The bad part about this implementation is that:

  1. The APIs for file changes in different systems are different, such as Linux’s inotify , Mac/BSD’s kqueue , and Windows’ ReadDirectoryChangesW . In order to be compatible with the APIs of these different systems, it is generally necessary to introduce a professional SDK to encapsulate, such as watchdog ;
  2. Occupy the thread of the system, although this may also be made asynchronous ;
  3. Occupying the fd of the system, this does not seem serious, but sometimes when the fd of the system is used up, we want to use hotreload to modify a parameter to control the behavior of the program, which is troublesome;

The advantage is that after importing an SDK, you don’t need to write a lot of code beforehand.

reload every time

This method is that every time a configuration item is used, global variables are not used, and the configuration file is parsed every time. In this way, there is no problem of hot reload, because each use is a reload, and all the configurations used are the latest.

At first glance, many people think this implementation is very stupid, but if you think about it carefully, many configurations that require hot reload are not very high read and write.

For example , as a personal ssh server, l obbbyboy takes a few ms to load the latest ssh key configuration every time a connection comes in, which is completely acceptable.

When loading scripts and files in prometheus-http-sd that I wrote recently, it is also loaded instantly for each request. Even so, the request can be completed within 5ms.

This saves a lot of work:

  1. There is no need to write a callback for updating the configuration, and the code of the load is directly reused
  2. No need to introduce additional dependencies
  3. Very simple, everyone can understand

There is a problem with these two methods, that is, once the file is changed, it will take effect immediately, there is no time for you to verify whether the file is correct, and this is a one-way fire and forget operation, you do not know your own actions when operating Right or not, it must be checked through the bypass, such as looking at the log or monitoring. Although there can be other work around, such as putting it in other locations for verification.

SIGHUP

This is the best solution. The program registers a signal handler and reloads config when it receives a SIGHUP signal .

The benefits of this are:

  1. Reload is a clear operator’s intention, not just reload after file modification, there is no ambiguity in semantics
  2. Low cost to implement

Also, we can generally control the behavior of reload. For example, when using the systemctl reload service , we can customize the reload command:

  1. First check whether the configuration file is legal, if not, give up reload;
  2. Send a reload signal to the program;

The following is an example, ExecReload can be defined in systemd’s unit file, which defines what to do when systemctl reload serivce is executed.

 ExecReload=/usr/local/bin/promtool check config /etc/prom-conf.yaml ; /bin/kill -HUP $MAINPID

Here are two tricks:

  1. Systemd will give you a $MAINPID for you to use, that is, the PID of the main process, you can send a signal directly here;
  2. Use ; You can define multiple commands, one will execute the next one if it starts successfully. Note that this is not the shell’s ; ;

In this way, when we reload, if the first check fails, the return code of systemctl reload is not 0, and we know that reload failed.

During operation, we can directly see the action error in the operation result of Ansible, instead of needing to ssh to the machine to check whether the execution result is correct.

The post Hot reload config implementation of the program first appeared on Kawabanga! .

This article is reprinted from: https://www.kawabangga.com/posts/4756
This site is for inclusion only, and the copyright belongs to the original author.

Leave a Comment