Original link: https://blog.othree.net/log/2022/12/03/fix-my-archlinux/
I said on twitter before that I accidentally broke the host of my blog. At that time, after updating with pacman, some errors occurred. I quickly re-run pacman -Syu and then there were errors. At that time, I wanted to I said it was because I hadn’t updated for a long time, and then there was a dependency error that caused the system to almost rot, and I wanted to give up at one point, but because the instances related to the website are still running, so I just want to say temporarily put it aside, and get the data when I have time I went out to rebuild the system, and then went to Singapore in mid-November. This trip was very tiring to take care of the children, and then I received emergency maintenance from Linode on the night I came back, and my Linode host has been restarted. , Of course, the website is also dead, it is really a bolt from the blue, but I am too tired and I can only leave it alone.
After a few days, I finally had the strength to look at the problem. My situation at the time was that I couldn’t use pacman, and then I found out that curl would die. When curl died, many things would hang together, such as git and wget They were all hung up, and as a result, the tools and methods I could use became very few. Anyway, let’s take a look at the error message first:
/usr/lib/libcurl.so.4: undefined symbol: BrotliDecoderCreateInstance
It can be seen that the basic problem is that there is an error in dynamically linking the Brotli library. I still remember that when I first installed the machine, Archlinux did not have an official Brotli package, so I compiled a version for nginx, and my nginx is also my own Compiled, I didn’t expect that Archlinux already has a formal Brotli package unknowingly, and curl still depends on it.
Then I started to try and find a way to reinstall the Brotli package. Although curl can’t work, I can still use scp to upload the file, but even if I upload it, pacman still can’t run it at all, even if I just want him to install the local one. file instead of connecting to the network, and then I also went to see the header file of the brotli package in /use/local/ , checked the content, and found that there is really no BrotliDecoderCreateInstance , but this symbol exists in the Brotli repo, and it has been It has existed for four years, so obviously, the version installed in my system is very problematic. Although I have determined where the problem is, there is still no solution. Reinstalling the package requires pacman, but pacman needs to repair brotli to start, and it is stuck in a deadlock To make matters worse, I searched the internet and couldn’t find anyone with the same problem.
Then I started to study what to do if pacman hangs up. After searching for a long time, I finally found a pacman-static tool. It is a pre-compiled and statically linked pacman executable file. After grabbing it, I found that it is really usable. Really I was moved and cried bitterly, and then I immediately executed pacman -Syu , everything ran smoothly, I was touched by QQ, and then I executed curl to check if it was fixed, but I saw the same, familiar error message again…
At this time, I was puzzled. I used pacman to see that the installed package version was indeed new. I went to unpack the package to see that it was new, but I went to the /usr/local/ of the system to see the files inside, but it was old. Yes, it’s the same situation after reinstalling many times. After a long time, I suddenly realized that the things under /usr/local/ were actually compiled and installed manually by us, and they also have higher priority in the path. , and as I mentioned earlier, I manually compiled and installed the Brotli suite a long time ago. Finally, the truth is revealed. I manually installed it five years ago, so this version does not have BrotliDecoderCreateInstance , and Archlinux used it four years ago. Version, so other things in the system that need Brotli will hang because of this. The solution is to cut off all the manual installations.
However, cutting down the Brotli I manually compiled also caused my nginx to fail to restart, because in the configuration file it is the package that I need to manually compile and install. The solution is very simple, just change the required module path to pacman Installed the package, and then my nginx can be started, but my blog is still dead, the non-blog part is alive, I thought it was a php-fpm problem at first, but I found a permission problem when I read the error message , I went to change the socket file permission to 666, and then the website still couldn’t get up. I studied for a long time and wanted to see the error message of PHP, but I couldn’t see anything. I also checked the nginx error log, and journalctl also saw that there was no error. , thought php-fpm was dead, and used
<? echo phpinfo(); ?>
Then I opened the browser and saw the original code and returned it directly. After a while, I thought of changing to <?php , but after changing it, I found that everything was normal. Finally, I found out that the problem was because when I used pacman to update the entire system, I put PHP After upgrading from 7 to 8, there is a place in my code that if the version number is wrong, an error message will be returned, but I did not write the error message to the log, and as a result, I have not been able to find the problem.
After the PHP problem is solved, my blog is back online, but in fact, my mariadb is still dead at this time, and I can see the log through journalctl
sudo journalctl -xeu mariadb.service
There is a line that says:
Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
At first I wanted to say that the installation failed, but later I searched and found another line:
InnoDB: Upgrade after a crash is not supported. The redo log was created with MariaDB 10.4.8
It means that if you update mariadb after your DB crashes and is not closed normally, you will have this error, but I didn’t care at first, because I thought I only had upgrade, but I don’t know when it crashed, but later In retrospect, it should be that when brotli is rotten, mariadb cannot start. In short, the solution to this problem is to return to the old 10.4.8 and restart the DB, so I studied how to install the old version of the suite on Archlinux. Basically, pacman cannot specify the version. There are two ways to install the old version. One is through the pacman cache in the system. However, I have cleared it when I repaired it before, so I can only install it from the Arch Linux Package Archive . Download the package tar file of a specific version, install it with pacman -U , and then because there are dependencies, you need to grab all the required packages and install them together:
pacman -U mariadb-10.4.8-2-x86_64.pkg.tar.xz \ mariadb-clients-10.4.8-2-x86_64.pkg.tar.xz \ mariadb-libs-10.4.8-2-x86_64.pkg.tar.xz
Anyway, if the startup fails, there will be a message prompting you to check the log. As a result, after installing all three of mariadb, there is still a problem. It says that the openssl 1.1 file cannot be found, so I also grab it and install it manually:
pacman -S openssl-1.1
At this point, it is finally fixed, and the next step is to containerize my blog system when I have time. There are too many things that are not easy to install.
PS. Posting this article means that it has really been fixed.
This article is transferred from: https://blog.othree.net/log/2022/12/03/fix-my-archlinux/
This site is only for collection, and the copyright belongs to the original author.