Fix CentOS ‘org.freedesktop.login1’ timed out

Leave a Comment / 06/06/2022 /

Original link: https://blog.phoenixlzx.com/2022/06/06/rescue-broken-centos-systemd-timeout/

It’s time to help people repair the system.

According to the description of the situation is “stuck at ‘Starting switch root'”, the preliminary judgment is that the initramfs is broken. After inquiring, sure enough, a guy who loves to update when he has nothing to do was pressed by the mentally retarded remote hand in the data center during the update ¯(ツ)/¯

Since it can’t be started, enter Rescue. Although the CentOS system is not very good, at least the original ISO provides the option of Rescue, and it can also automatically find the root partition to mount. I thought it was a trivial matter, but when I was preparing to chroot, the sentence “chroot: cannot run command `/bin/sh’: Exec format error” suddenly popped up, and I was instantly heartbroken $#%#$^$#@#%#

In general, the error “Exec format error” indicates a binary format error, such as executing a 64-bit binary on a 32-bit system. But this is impossible. First of all, it is also the x86_64 architecture, the installed system and the ISO system are also the same version, and the disk does not report an error. Why does this problem occur?

The possibility of the file being written bad due to the power failure during the update is not impossible, but the strange thing is that the same error is reported when executing /bin/bash . It’s so bad that I’m afraid that glibc is not hanging…? But in a word, the first step is to be able to advance the system to see what is going on, so I am hesitant and directly overwrite the corresponding directory on the disk with /usr of the Live CD.

 1
 
 ~> cp -r /usr /m nt /sysimage/

Then chroot again, and you can successfully enter the system, and the yum command can also be used. Since the power outage is during the update process, you should try to repair the incomplete update first.

 1
 2
 3
 4
 
 ~> yum-complete-transaction
 Loaded plugins: fastestmirror, langpacks
 Loading mirror speeds from cached hostfile
 No unfinished transactions left.

Ahh.

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 twenty one
 twenty two
 twenty three
 twenty four
 25
 26
 27
 28
 29
 30
 31
 
 ~> yum update
 --> Finished Dependency Resolution
 Error: Multilib version problems found. This often means that the root
 cause is something else and multilib version checking is just
 pointing out that there is a problem. Eg.:

 1. You have an upgrade for libselinux which is missing some
 dependency that another package requires. Yum is trying to
 solve this by installing an older version of glibc of the
 different architecture. If you exclude the bad architecture
 yum will tell you what the root cause is (which package
 requires what). You can try redoing the upgrade with
 --exclude glibc.otherarch ... this should give you an error
 message showing the root cause of the problem.

 2. You have multiple architectures of glibc installed, but
 yum can only see an upgrade for one of those arcitectures.
 If you don't want/need both architectures anymore then you
 can remove the one with the missing update and everything
 will work.

 3. You have duplicate versions of glibc installed already.
 You can use "yum check" to get yum show these errors.

 ...you can also use --setopt=protected_multilib=false to remove
 this checking, however this is almost never the correct thing to
 do as something else is very likely to go wrong (often causing
 much more problems).

 Protected multilib versions: glibc..... != glibc.....
 kbd...x86_64 != kbd...x86_64

…It really is.

Further understanding that this system does not install 32-bit packages, so where does this 32-bit glibc come from? yum check to no avail, not to mention a weird kbd package that doesn’t even have anything to do with 32-bit. It seems that the bad is not light, but since it is determined that there is no 32-bit package in the system, then you can continue to the end.

 1
 2
 3
 4
 5
 6
 7
 
 # complete all unupdated packages
 ~> yum update --setopt=protected_multilib= false
 # Prepare for reinstalling the system
 ~> mkdir /root/tmp
 ~> cd /root/tmp
 # Reinstall all packages
 ~> yum reinstall * --setopt=protected_multilib= false

One operation is as fierce as a tiger. When reinstalling all the packages for the first time, an error will be reported in ldconfig. Some libraries are empty files, and the error will not be seen after the second complete reinstallation. It seems feasible, restart decisively, and see the login prompt after a bunch of OK.

 1
 2
 3
 4
 5
 6
 7
 8
 
 localhost login : root

 Login incorrect

 Login incorrect


 Login incorrect

hmmmm? ? ? I haven’t entered my password yet? ? ?

An error is reported without entering the password, at least it is not a problem with PAM. What about before that? It should be systemd-logind. Fortunately, the network can be started normally, and SSH can log into the system normally. If the network or SSH does not work, then you have to enter the rescue again.

Many underlying services that should have been in the system have not been started, such as systemd-logind. I want to check the status of this product, but the result is an error:

 1
 
 Failed to activate service 'org.freedesktop.systemd1' : timed out

Huh? systemd itself did not start normally, look at the log:

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 
 ...
 localhost dbus-daemon[1166]: Encountered error ' Error in file /etc/dbus-1/system.d /org.freedesktop.hostname1.conf , line 1, column 0: no element found
 ' while parsing ' /etc/dbus-1/system.d/org.freedesktop.hostname1.conf '
 localhost dbus-daemon[1166]: Encountered error ' Error in file /etc/dbus-1/system.d /org.freedesktop.import1.conf , line 1, column 0: no element found
 ' while parsing ' /etc/dbus-1/system.d/org.freedesktop.import1.conf '
 localhost dbus-daemon[1166]: Encountered error ' Error in file /etc/dbus-1/system.d /org.freedesktop.locale1.conf , line 1, column 0: no element found
 ' while parsing ' /etc/dbus-1/system.d/org.freedesktop.locale1.conf '
 localhost dbus-daemon[1166]: Encountered error ' Error in file /etc/dbus-1/system.d /org.freedesktop.login1.conf , line 1, column 0: no element found
 ' while parsing ' /etc/dbus-1/system.d/org.freedesktop.login1.conf '
 localhost dbus-daemon[1166]: Encountered error ' Error in file /etc/dbus-1/system.d /org.freedesktop.machine1.conf , line 1, column 0: no element found
 ' while parsing ' /etc/dbus-1/system.d/org.freedesktop.machine1.conf '
 localhost dbus-daemon[1166]: Encountered error ' Error in file /etc/dbus-1/system.d /org.freedesktop.systemd1.conf , line 1, column 0: no element found
 ' while parsing ' /etc/dbus-1/system.d/org.freedesktop.systemd1.conf '
 localhost dbus-daemon[1166]: Encountered error ' Error in file /etc/dbus-1/system.d /org.freedesktop.timedate1.conf , line 1, column 0: no element found
 ' while parsing ' /etc/dbus-1/system.d/org.freedesktop.timedate1.conf '
 ...

Not lightly ill. These file contents are gone, and repackaging doesn’t overwrite them. Had to override it manually.

 1
 2
 3
 4
 5
 6
 7
 
 # Install necessary tools
 ~> yum install yum-utils rpm2cpio
 # Get and unpack rpm
 ~> cd /root/tmp
 ~> yumdownloader dbus systemd
 ~> rpm2cpio dbus- 1.10.24-15.el 7.x86_64.rpm | cpio -idmv
 ~> rpm2cpio systemd- 219-78.el 7_9.5.x 86_64.rpm | cpio -idmv

So get the contents of these packages. Manually overwrite all files in the etc/dbus-1 directory to the file system, and then execute kill 1

 1
 2
 3
 4
 5
 6
 7
 
 localhost systemd[1]: Received SIGTERM from PID 4332 (bash).
 localhost systemd[1]: Reexecuting.
 localhost systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ
 localhost systemd[1]: Detected architecture x86-64.
 localhost dbus[1166]: [system] Successfully activated service 'org.freedesktop.systemd1'
 localhost systemd[1]: Started Login Service.
 localhost systemd-logind[4723]: New seat seat0.

Check it carefully, all kinds of services are starting normally!

(๑•̀ㅂ•́)و✧

Postscript: Don’t worry about updating the production system.

Postscript 2: There is no reliable remote data center in Western countries.

This article is reprinted from: https://blog.phoenixlzx.com/2022/06/06/rescue-broken-centos-systemd-timeout/
This site is for inclusion only, and the copyright belongs to the original author.

Leave a Comment