(Filed the issues as https://bugzilla.redhat.com/show_bug.cgi?id=2280584, since this update is already in stable)
When I try to upgrade to this build in my test Rawhide VMs, it manages to bring down the user dbus instance during the upgrade:
# rpm -q systemd
systemd-255.5-1.fc41.x86_64
# journalctl --cursor-file=/tmp/cursor -n0 -q
# dnf upgrade systemd
Updating and loading repositories:
Beaker Client - Fedorarawhide 100% | 25.1 KiB/s | 4.6 KiB | 00m00s
Copr repo for qa-tools owned by lpol 100% | 1.5 MiB/s | 68.9 KiB | 00m00s
Beaker harness 100% | 200.8 KiB/s | 9.0 KiB | 00m00s
Fedora rawhide openh264 (From Cisco) - x86_64 100% | 2.9 KiB/s | 4.8 KiB | 00m02s
Fedora - Rawhide - Developmental packages for the next Fedora release 100% | 8.7 MiB/s | 20.6 MiB | 00m02s
Repositories loaded.
Package Arch Version Repository Size
Upgrading:
systemd x86_64 256~rc1^20240509git1781de1-2.fc41 rawhide 15.7 MiB
replacing systemd x86_64 255.5-1.fc41 updates 14.6 MiB
systemd-libs x86_64 256~rc1^20240509git1781de1-2.fc41 rawhide 2.0 MiB
replacing systemd-libs x86_64 255.5-1.fc41 updates 1.9 MiB
systemd-networkd x86_64 256~rc1^20240509git1781de1-2.fc41 rawhide 2.1 MiB
replacing systemd-networkd x86_64 255.5-1.fc41 updates 2.0 MiB
systemd-pam x86_64 256~rc1^20240509git1781de1-2.fc41 rawhide 1.1 MiB
replacing systemd-pam x86_64 255.5-1.fc41 updates 1.0 MiB
systemd-resolved x86_64 256~rc1^20240509git1781de1-2.fc41 rawhide 667.1 KiB
replacing systemd-resolved x86_64 255.5-1.fc41 updates 638.4 KiB
systemd-udev x86_64 256~rc1^20240509git1781de1-2.fc41 rawhide 11.7 MiB
replacing systemd-udev x86_64 255.5-1.fc41 updates 11.5 MiB
Transaction Summary:
Upgrading: 6 packages
Replacing: 6 packages
...
# journalctl -o short-monotonic --no-hostname --cursor-file=/tmp/cursor
...
[65859.779939] systemd[1]: Reexecuting requested from client PID 2348 ('systemctl') (unit session-4.scope)...
[65859.783141] systemd[1]: Reexecuting.
[65859.844531] audit: BPF prog-id=29 op=UNLOAD
[65859.875868] systemd[1]: systemd 256~rc1^20240509git1781de1-2.fc41 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPT>
[65859.875932] systemd[1]: Detected virtualization kvm.
[65859.875945] systemd[1]: Detected architecture x86-64.
...
[65860.682906] systemd[1973]: Created slice session.slice - User Core Session Slice.
[65860.685684] systemd[2417]: /usr/lib/systemd/systemd-executor (deleted): error while loading shared libraries: libsystemd-core-255.5-1.fc41.so: cannot open shared object file: No such file or directory
[65860.689210] systemd[1973]: Starting dbus-broker.service - D-Bus User Message Bus...
[65860.689476] systemd[1973]: dbus-broker.service: Main process exited, code=exited, status=127/n/a
[65860.689670] systemd[1973]: dbus-broker.service: Failed with result 'exit-code'.
[65860.689907] systemd[1973]: Failed to start dbus-broker.service - D-Bus User Message Bus.
[65860.691601] systemd[2418]: /usr/lib/systemd/systemd-executor (deleted): error while loading shared libraries: libsystemd-core-255.5-1.fc41.so: cannot open shared object file: No such file or directory
[65860.691889] systemd[1973]: Starting dbus-broker.service - D-Bus User Message Bus...
[65860.692133] systemd[1973]: dbus-broker.service: Main process exited, code=exited, status=127/n/a
[65860.692245] systemd[1973]: dbus-broker.service: Failed with result 'exit-code'.
[65860.692546] systemd[1973]: Failed to start dbus-broker.service - D-Bus User Message Bus.
[65860.693735] systemd[2419]: /usr/lib/systemd/systemd-executor (deleted): error while loading shared libraries: libsystemd-core-255.5-1.fc41.so: cannot open shared object file: No such file or directory
[65860.693956] systemd[1973]: Starting dbus-broker.service - D-Bus User Message Bus...
[65860.694072] systemd[1973]: dbus-broker.service: Main process exited, code=exited, status=127/n/a
[65860.694156] systemd[1973]: dbus-broker.service: Failed with result 'exit-code'.
[65860.694394] systemd[1973]: Failed to start dbus-broker.service - D-Bus User Message Bus.
[65860.695430] systemd[2420]: /usr/lib/systemd/systemd-executor (deleted): error while loading shared libraries: libsystemd-core-255.5-1.fc41.so: cannot open shared object file: No such file or directory
[65860.695636] systemd[1973]: Starting dbus-broker.service - D-Bus User Message Bus...
[65860.695697] systemd[1973]: dbus-broker.service: Main process exited, code=exited, status=127/n/a
[65860.695793] systemd[1973]: dbus-broker.service: Failed with result 'exit-code'.
[65860.696054] systemd[1973]: Failed to start dbus-broker.service - D-Bus User Message Bus.
[65860.697080] systemd[2421]: /usr/lib/systemd/systemd-executor (deleted): error while loading shared libraries: libsystemd-core-255.5-1.fc41.so: cannot open shared object file: No such file or directory
[65860.696963] systemd[1973]: Starting dbus-broker.service - D-Bus User Message Bus...
[65860.697263] systemd[1973]: dbus-broker.service: Main process exited, code=exited, status=127/n/a
[65860.697344] systemd[1973]: dbus-broker.service: Failed with result 'exit-code'.
[65860.697507] systemd[1973]: Failed to start dbus-broker.service - D-Bus User Message Bus.
[65860.697701] systemd[1973]: dbus-broker.service: Start request repeated too quickly.
[65860.697720] systemd[1973]: dbus-broker.service: Failed with result 'exit-code'.
[65860.697751] systemd[1973]: Failed to start dbus-broker.service - D-Bus User Message Bus.
[65860.697810] systemd[1973]: dbus.socket: Failed with result 'service-start-limit-hit'.
# systemctl --user status dbus-broker
Warning: The unit file, source configuration file or drop-ins of dbus-broker.service changed on disk. Run 'systemctl --user daemon-reload' to reload units.
× dbus-broker.service - D-Bus User Message Bus
Loaded: loaded (/usr/lib/systemd/user/dbus-broker.service; enabled; preset: enabled)
Drop-In: /usr/lib/systemd/user/service.d
└─10-timeout-abort.conf
Active: failed (Result: exit-code) since Wed 2024-05-15 05:33:25 EDT; 2min 13s ago
Invocation: 7a968b77b67b4520aa847c56d90c2de9
TriggeredBy: × dbus.socket
Docs: man:dbus-broker-launch(1)
Process: 2421 ExecStart=/usr/bin/dbus-broker-launch --scope user (code=exited, status=127)
Main PID: 2421 (code=exited, status=127)
CPU: 0
May 15 05:33:25 xxx systemd[1973]: Starting dbus-broker.service - D-Bus User Message Bus...
May 15 05:33:25 xxx systemd[1973]: dbus-broker.service: Main process exited, code=exited, status=127/n/a
May 15 05:33:25 xxx systemd[1973]: dbus-broker.service: Failed with result 'exit-code'.
May 15 05:33:25 xxx systemd[1973]: Failed to start dbus-broker.service - D-Bus User Message Bus.
May 15 05:33:25 xxx systemd[1973]: dbus-broker.service: Start request repeated too quickly.
May 15 05:33:25 xxx systemd[1973]: dbus-broker.service: Failed with result 'exit-code'.
May 15 05:33:25 xxx systemd[1973]: Failed to start dbus-broker.service - D-Bus User Message Bus.
I think bind mounting the whole host's /run
into the chroot is wrong exactly for this reason, but I'm pretty sure there are reasons behind it (which I'm not familiar with), so I can't make any statements about it.
The quickest "fix" could be temporarily building systemd without the new ssh-generator stuff (-Dsshconfdir=no
and probably removing the systemd-ssh-generator
after build, since it seems to be always built, ugh), and then figuring out if it would be faster to deal with the issue on anaconda's side or our side (the preset thingy; this would be great to have in general anyway). IIRC Zbyszek should be back tomorrow, so I'll leave the decision to him.
I was wading through anaconda logs and came across this:
DEBUG:blivet: DirectoryDevice._set_format: /run ; type: bind ; current: None ;
DEBUG:blivet:updated /run size to 960 B (960 B)
Which, if it indeed means what I think it does, would explain where this severed link comes from. Meaning - if /run
from the host is bind-mounted into the new chroot, the error is quite easy to reproduce, because the symlink generated by systemd-ssh-generator
(/run/systemd/generator/sshd-vsock@.service
) becomes severed in the chroot, if sshd
is not part of the initial transaction:
# rpm -q systemd
systemd-256~rc1-99.fc41.x86_64
## Current selinux-policy prevents systemd-ssh-generator from working properly, hence the setenforce & daemon-reexec
# setenforce 0
# systemctl daemon-reexec
# ls -l /run/systemd/generator/sshd-vsock@.service
lrwxrwxrwx. 1 root root 37 Apr 28 16:00 /run/systemd/generator/sshd-vsock@.service -> /usr/lib/systemd/system/sshd@.service
# mkdir -p minimal/{proc,run,sys}
# mount -t proc proc minimal/proc
# mount -t sysfs sysfs minimal/sys
# mount --bind /run minimal/run/
# dnf --installroot $PWD/minimal --nogpgcheck --releasever rawhide -y install @core |& tee log.txt
...
Creating group 'systemd-journal' with GID 190.
Creating group 'systemd-network' with GID 192.
Creating user 'systemd-network' (systemd Network Management) with UID 192 and GID 192.
Creating group 'systemd-oom' with GID 999.
Creating user 'systemd-oom' (systemd Userspace OOM Killer) with UID 999 and GID 999.
Creating group 'systemd-resolve' with GID 193.
Creating user 'systemd-resolve' (systemd Resolver) with UID 193 and GID 193.
-rw-r--r--. 1 root root 547 Apr 16 00:00 dbus-broker.service
lrwxrwxrwx. 1 root root 25 Apr 28 00:00 dbus-org.freedesktop.hostname1.service -> systemd-hostnamed.service
lrwxrwxrwx. 1 root root 23 Apr 28 00:00 dbus-org.freedesktop.locale1.service -> systemd-localed.service
lrwxrwxrwx. 1 root root 22 Apr 28 00:00 dbus-org.freedesktop.login1.service -> systemd-logind.service
lrwxrwxrwx. 1 root root 25 Apr 28 00:00 dbus-org.freedesktop.timedate1.service -> systemd-timedated.service
-rw-r--r--. 1 root root 137 Jan 24 00:00 dbus.socket
drwxr-xr-x. 1 root root 0 Apr 28 00:00 dbus.target.wants
total 0
-- systemctl preset-all BEGIN --
Preset files say disable session-1.scope.
Preset files say disable -.mount.
Preset files say disable boot.mount.
Preset files say disable home.mount.
Preset files say disable var.mount.
Preset files say disable boot-efi.mount.
Cannot find unit sshd@.service.
Failed to preset unit sshd-vsock@.service: Link has been severed
-- systemctl preset-all END--
-- systemctl --global preset-all BEGIN --
Created symlink '/etc/systemd/user/sockets.target.wants/dbus.socket' → '/usr/lib/systemd/user/dbus.socket'.
Created symlink '/etc/systemd/user/dbus.service' → '/usr/lib/systemd/user/dbus-broker.service'.
Created symlink '/etc/systemd/user/timers.target.wants/grub-boot-success.timer' → '/usr/lib/systemd/user/grub-boot-success.timer'.
Created symlink '/etc/systemd/user/timers.target.wants/systemd-tmpfiles-clean.timer' → '/usr/lib/systemd/user/systemd-tmpfiles-clean.timer'.
Created symlink '/etc/systemd/user/basic.target.wants/systemd-tmpfiles-setup.service' → '/usr/lib/systemd/user/systemd-tmpfiles-setup.service'.
-- systemctl --global preset-all END--
...
# stat minimal/etc/systemd/system/dbus.service
stat: cannot statx 'minimal/etc/systemd/system/dbus.service': No such file or directory
I'm not really familiar with how anaconda does the actual installation, so please take this with a grain of salt, but it would explain what's going wrong in the netinstall ISO.
Ugh, the link should've been https://github.com/systemd/systemd/commit/0e3220684c6184a2f70396d991200ae207a25377.
The sshd-vsock@.service
unit was introduced in systemd v256 (https://github.com/systemd/systemd/commit/631cdd59e11b5a3147aa2cf120e9c4a24566cd01).
Also, this upstream issues is quite relevant here: https://github.com/systemd/systemd/issues/21224
I rebuilt the RPM from this update with some more logging* and the systemctl preset-all
command, that should create the dbus.service symlink (among others), seem to choke on the systemd-vsock@.service
unit:
INFO:anaconda.modules.payloads.payload.dnf.transaction_progress:Configuring (running scriptlet for): systemd-256~rc1-99.fc41.x86_64 1714302944 ca3c023ab30f90a3da9987d3b60e9cbb16b91b65fa6bdea00173c18332a2d626
INFO:dnf.rpm:Creating group 'input' with GID 104.
Creating group 'kvm' with GID 36.
Creating group 'render' with GID 105.
Creating group 'sgx' with GID 106.
Creating group 'systemd-journal' with GID 190.
Creating group 'systemd-network' with GID 192.
Creating user 'systemd-network' (systemd Network Management) with UID 192 and GID 192.
Creating group 'systemd-oom' with GID 999.
Creating user 'systemd-oom' (systemd Userspace OOM Killer) with UID 999 and GID 999.
Creating group 'systemd-resolve' with GID 193.
Creating user 'systemd-resolve' (systemd Resolver) with UID 193 and GID 193.
lrwxrwxrwx. 1 root root 25 Apr 28 00:00 dbus-org.freedesktop.hostname1.service -> systemd-hostnamed.service
lrwxrwxrwx. 1 root root 23 Apr 28 00:00 dbus-org.freedesktop.locale1.service -> systemd-localed.service
lrwxrwxrwx. 1 root root 22 Apr 28 00:00 dbus-org.freedesktop.login1.service -> systemd-logind.service
lrwxrwxrwx. 1 root root 25 Apr 28 00:00 dbus-org.freedesktop.timedate1.service -> systemd-timedated.service
Failed to preset unit sshd-vsock@.service: Link has been severed
Created symlink '/etc/systemd/user/sockets.target.wants/dbus.socket' → '/usr/lib/systemd/user/dbus.socket'.
Created symlink '/etc/systemd/user/dbus.service' → '/usr/lib/systemd/user/dbus-broker.service'.
Created symlink '/etc/systemd/user/timers.target.wants/grub-boot-success.timer' → '/usr/lib/systemd/user/grub-boot-success.timer'.
Created symlink '/etc/systemd/user/timers.target.wants/systemd-tmpfiles-clean.timer' → '/usr/lib/systemd/user/systemd-tmpfiles-clean.timer'.
Created symlink '/etc/systemd/user/basic.target.wants/systemd-tmpfiles-setup.service' → '/usr/lib/systemd/user/systemd-tmpfiles-setup.service'.
&>/dev/null
from commands in systemd's RPM scriptlets; it would be great if some of these could be dropped altogether, as they make debugging issues like this very painfulAh, I should add that in my case this is with systemd-255.5-1.fc41.x86_64
that the netinst ISO pulls in, but the OpenQA results seem to be with the build from this update.
The dbus-broker service is indeed disabled in the final system installed by the netinst ISO:
# systemctl status dbus-broker.service --no-pager
○ dbus-broker.service - D-Bus System Message Bus
Loaded: loaded (/usr/lib/systemd/system/dbus-broker.service; disabled; preset: enabled)
Drop-In: /usr/lib/systemd/system/service.d
└─10-timeout-abort.conf
Active: inactive (dead)
TriggeredBy: × dbus.socket
Docs: man:dbus-broker-launch(1)
Even though presets say otherwise.
So, the netinst issue is caused (at least in my case) by missing /etc/systemd/system/dbus.service
symlink:
# journalctl -o short-monotonic -b --no-pager
...
[ 6.004806] fedora systemd[1]: Reached target timers.target - Timer Units.
[ 6.005493] fedora systemd[1]: Listening on avahi-daemon.socket - Avahi mDNS/DNS-SD Stack Activation Socket.
[ 6.006534] fedora systemd[1]: dbus.socket: Socket service dbus.service not loaded, refusing.
[ 6.006581] fedora systemd[1]: Failed to listen on dbus.socket - D-Bus System Message Bus Socket.
[ 6.006622] fedora systemd[1]: Dependency failed for avahi-daemon.service - Avahi mDNS/DNS-SD Stack.
[ 6.006646] fedora systemd[1]: avahi-daemon.service: Job avahi-daemon.service/start failed with result 'dependency'.
[ 6.006666] fedora systemd[1]: Dependency failed for firewalld.service - firewalld - dynamic firewall daemon.
[ 6.006687] fedora systemd[1]: firewalld.service: Job firewalld.service/start failed with result 'dependency'.
[ 6.006712] fedora systemd[1]: Dependency failed for systemd-homed.service - Home Area Manager.
[ 6.006739] fedora systemd[1]: systemd-homed.service: Job systemd-homed.service/start failed with result 'dependency'.
...
# ls -l /etc/systemd/system
total 28
drwxr-xr-x. 1 root root 34 Apr 28 11:40 bluetooth.target.wants
lrwxrwxrwx. 1 root root 41 Apr 28 11:40 dbus-org.bluez.service -> /usr/lib/systemd/system/bluetooth.service
lrwxrwxrwx. 1 root root 41 Apr 28 11:40 dbus-org.fedoraproject.FirewallD1.service -> /usr/lib/systemd/system/firewalld.service
lrwxrwxrwx. 1 root root 44 Apr 28 11:40 dbus-org.freedesktop.Avahi.service -> /usr/lib/systemd/system/avahi-daemon.service
lrwxrwxrwx. 1 root root 45 Apr 28 11:40 dbus-org.freedesktop.home1.service -> /usr/lib/systemd/system/systemd-homed.service
lrwxrwxrwx. 1 root root 57 Apr 28 11:40 dbus-org.freedesktop.nm-dispatcher.service -> /usr/lib/systemd/system/NetworkManager-dispatcher.service
lrwxrwxrwx. 1 root root 44 Apr 28 11:40 dbus-org.freedesktop.oom1.service -> /usr/lib/systemd/system/systemd-oomd.service
lrwxrwxrwx. 1 root root 41 Apr 28 11:42 default.target -> /usr/lib/systemd/system/multi-user.target
drwxr-xr-x. 1 root root 30 Apr 28 11:40 graphical.target.wants
drwxr-xr-x. 1 root root 426 Apr 28 11:40 multi-user.target.wants
drwxr-xr-x. 1 root root 68 Apr 28 11:40 network-online.target.wants
drwxr-xr-x. 1 root root 136 Apr 28 11:40 sockets.target.wants
drwxr-xr-x. 1 root root 108 Apr 28 11:40 sysinit.target.wants
drwxr-xr-x. 1 root root 60 Apr 28 11:40 systemd-homed.service.wants
drwxr-xr-x. 1 root root 110 Apr 28 11:40 timers.target.wants
# systemctl status dbus
Unit dbus.service could not be found.
[root@fedora anaconda]# systemctl status dbus.socket
× dbus.socket - D-Bus System Message Bus Socket
Loaded: loaded (/usr/lib/systemd/system/dbus.socket; disabled; preset: ena>
Active: failed (Result: start-limit-hit)
Triggers: ● dbus.service
Listen: /run/dbus/system_bus_socket (Stream)
Apr 28 11:44:13 fedora systemd[1]: Failed to listen on dbus.socket - D-Bus Syst>
Apr 28 11:44:13 fedora systemd[1]: dbus.socket: Socket service dbus.service not>
Apr 28 11:44:13 fedora systemd[1]: Failed to listen on dbus.socket - D-Bus Syst>
Apr 28 11:44:13 fedora systemd[1]: dbus.socket: Socket service dbus.service not>
Apr 28 11:44:13 fedora systemd[1]: Failed to listen on dbus.socket - D-Bus Syst>
Apr 28 11:44:13 fedora systemd[1]: dbus.socket: Socket service dbus.service not>
Apr 28 11:44:13 fedora systemd[1]: Failed to listen on dbus.socket - D-Bus Syst>
Apr 28 11:44:13 fedora systemd[1]: dbus.socket: Start request repeated too quic>
Apr 28 11:44:13 fedora systemd[1]: dbus.socket: Failed with result 'start-limit>
Apr 28 11:44:13 fedora systemd[1]: Failed to listen on dbus.socket - D-Bus Syst>
But if I call systemctl enable dbus-broker.service
manually, the symlink is created, strange.
Yeah, I was convinced that dracut 060 already has that patch, but turns out that was merged after 060 was released.
However, systemd is broken in this update as well, since currently PID 1 is not reexec'ed when upgrading from v255 to v256-rc1, see https://github.com/systemd/systemd/issues/32508#issuecomment-2079991745.
As mentioned in https://bodhi.fedoraproject.org/updates/FEDORA-2024-28601a6bb0#comment-3391708 it would be great to have this in Rawhide ASAP, since currently anything that depends on bpftool (or any other stuff from the former kernel-tools meta package) now FTBFS.
Our systemd/SELinux Vagrant job is functional once again, so adding my +1, thanks!
Seems to work as expected, but I came across the same issue @nucleo already mentioned - https://dl.fedoraproject.org/ now returns:
But it worked fine with
firefox-130.0-3.fc39.x86_64
.