Comments

19 Comments
karma

Seems to work as expected, but I came across the same issue @nucleo already mentioned - https://dl.fedoraproject.org/ now returns:


Secure Connection Failed

An error occurred during a connection to dl.fedoraproject.org. Peer’s Certificate has been revoked.

Error code: SEC_ERROR_REVOKED_CERTIFICATE

But it worked fine with firefox-130.0-3.fc39.x86_64.

Test Case firefox addons
Test Case firefox browse
Test Case firefox media

When I try to upgrade to this build in my test Rawhide VMs, it manages to bring down the user dbus instance during the upgrade:

# rpm -q systemd
systemd-255.5-1.fc41.x86_64
# journalctl --cursor-file=/tmp/cursor -n0 -q
# dnf upgrade systemd
Updating and loading repositories:
 Beaker Client - Fedorarawhide                                                                                                                            100% |  25.1 KiB/s |   4.6 KiB |  00m00s
 Copr repo for qa-tools owned by lpol                                                                                                                     100% |   1.5 MiB/s |  68.9 KiB |  00m00s
 Beaker harness                                                                                                                                           100% | 200.8 KiB/s |   9.0 KiB |  00m00s
 Fedora rawhide openh264 (From Cisco) - x86_64                                                                                                            100% |   2.9 KiB/s |   4.8 KiB |  00m02s
 Fedora - Rawhide - Developmental packages for the next Fedora release                                                                                    100% |   8.7 MiB/s |  20.6 MiB |  00m02s
Repositories loaded.
Package                                                           Arch           Version                                                            Repository                                Size
Upgrading:                                                                                                                                                                                        
 systemd                                                          x86_64         256~rc1^20240509git1781de1-2.fc41                                  rawhide                               15.7 MiB
  replacing systemd                                               x86_64         255.5-1.fc41                                                       updates                               14.6 MiB
 systemd-libs                                                     x86_64         256~rc1^20240509git1781de1-2.fc41                                  rawhide                                2.0 MiB
  replacing systemd-libs                                          x86_64         255.5-1.fc41                                                       updates                                1.9 MiB
 systemd-networkd                                                 x86_64         256~rc1^20240509git1781de1-2.fc41                                  rawhide                                2.1 MiB
  replacing systemd-networkd                                      x86_64         255.5-1.fc41                                                       updates                                2.0 MiB
 systemd-pam                                                      x86_64         256~rc1^20240509git1781de1-2.fc41                                  rawhide                                1.1 MiB
  replacing systemd-pam                                           x86_64         255.5-1.fc41                                                       updates                                1.0 MiB
 systemd-resolved                                                 x86_64         256~rc1^20240509git1781de1-2.fc41                                  rawhide                              667.1 KiB
  replacing systemd-resolved                                      x86_64         255.5-1.fc41                                                       updates                              638.4 KiB
 systemd-udev                                                     x86_64         256~rc1^20240509git1781de1-2.fc41                                  rawhide                               11.7 MiB
   replacing systemd-udev                                         x86_64         255.5-1.fc41                                                       updates                               11.5 MiB

Transaction Summary:
 Upgrading:         6 packages
 Replacing:         6 packages
...
# journalctl -o short-monotonic --no-hostname --cursor-file=/tmp/cursor
...

[65859.779939] systemd[1]: Reexecuting requested from client PID 2348 ('systemctl') (unit session-4.scope)...
[65859.783141] systemd[1]: Reexecuting.
[65859.844531] audit: BPF prog-id=29 op=UNLOAD
[65859.875868] systemd[1]: systemd 256~rc1^20240509git1781de1-2.fc41 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPT>
[65859.875932] systemd[1]: Detected virtualization kvm.
[65859.875945] systemd[1]: Detected architecture x86-64.
...
[65860.682906] systemd[1973]: Created slice session.slice - User Core Session Slice.
[65860.685684] systemd[2417]: /usr/lib/systemd/systemd-executor (deleted): error while loading shared libraries: libsystemd-core-255.5-1.fc41.so: cannot open shared object file: No such file or directory
[65860.689210] systemd[1973]: Starting dbus-broker.service - D-Bus User Message Bus...
[65860.689476] systemd[1973]: dbus-broker.service: Main process exited, code=exited, status=127/n/a
[65860.689670] systemd[1973]: dbus-broker.service: Failed with result 'exit-code'.
[65860.689907] systemd[1973]: Failed to start dbus-broker.service - D-Bus User Message Bus.
[65860.691601] systemd[2418]: /usr/lib/systemd/systemd-executor (deleted): error while loading shared libraries: libsystemd-core-255.5-1.fc41.so: cannot open shared object file: No such file or directory
[65860.691889] systemd[1973]: Starting dbus-broker.service - D-Bus User Message Bus...
[65860.692133] systemd[1973]: dbus-broker.service: Main process exited, code=exited, status=127/n/a
[65860.692245] systemd[1973]: dbus-broker.service: Failed with result 'exit-code'.
[65860.692546] systemd[1973]: Failed to start dbus-broker.service - D-Bus User Message Bus.
[65860.693735] systemd[2419]: /usr/lib/systemd/systemd-executor (deleted): error while loading shared libraries: libsystemd-core-255.5-1.fc41.so: cannot open shared object file: No such file or directory
[65860.693956] systemd[1973]: Starting dbus-broker.service - D-Bus User Message Bus...
[65860.694072] systemd[1973]: dbus-broker.service: Main process exited, code=exited, status=127/n/a
[65860.694156] systemd[1973]: dbus-broker.service: Failed with result 'exit-code'.
[65860.694394] systemd[1973]: Failed to start dbus-broker.service - D-Bus User Message Bus.
[65860.695430] systemd[2420]: /usr/lib/systemd/systemd-executor (deleted): error while loading shared libraries: libsystemd-core-255.5-1.fc41.so: cannot open shared object file: No such file or directory
[65860.695636] systemd[1973]: Starting dbus-broker.service - D-Bus User Message Bus...
[65860.695697] systemd[1973]: dbus-broker.service: Main process exited, code=exited, status=127/n/a
[65860.695793] systemd[1973]: dbus-broker.service: Failed with result 'exit-code'.
[65860.696054] systemd[1973]: Failed to start dbus-broker.service - D-Bus User Message Bus.
[65860.697080] systemd[2421]: /usr/lib/systemd/systemd-executor (deleted): error while loading shared libraries: libsystemd-core-255.5-1.fc41.so: cannot open shared object file: No such file or directory
[65860.696963] systemd[1973]: Starting dbus-broker.service - D-Bus User Message Bus...
[65860.697263] systemd[1973]: dbus-broker.service: Main process exited, code=exited, status=127/n/a
[65860.697344] systemd[1973]: dbus-broker.service: Failed with result 'exit-code'.
[65860.697507] systemd[1973]: Failed to start dbus-broker.service - D-Bus User Message Bus.
[65860.697701] systemd[1973]: dbus-broker.service: Start request repeated too quickly.
[65860.697720] systemd[1973]: dbus-broker.service: Failed with result 'exit-code'.
[65860.697751] systemd[1973]: Failed to start dbus-broker.service - D-Bus User Message Bus.
[65860.697810] systemd[1973]: dbus.socket: Failed with result 'service-start-limit-hit'.
# systemctl --user status dbus-broker
Warning: The unit file, source configuration file or drop-ins of dbus-broker.service changed on disk. Run 'systemctl --user daemon-reload' to reload units.
× dbus-broker.service - D-Bus User Message Bus
     Loaded: loaded (/usr/lib/systemd/user/dbus-broker.service; enabled; preset: enabled)
    Drop-In: /usr/lib/systemd/user/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: exit-code) since Wed 2024-05-15 05:33:25 EDT; 2min 13s ago
 Invocation: 7a968b77b67b4520aa847c56d90c2de9
TriggeredBy: × dbus.socket
       Docs: man:dbus-broker-launch(1)
    Process: 2421 ExecStart=/usr/bin/dbus-broker-launch --scope user (code=exited, status=127)
   Main PID: 2421 (code=exited, status=127)
        CPU: 0

May 15 05:33:25 xxx systemd[1973]: Starting dbus-broker.service - D-Bus User Message Bus...
May 15 05:33:25 xxx systemd[1973]: dbus-broker.service: Main process exited, code=exited, status=127/n/a
May 15 05:33:25 xxx systemd[1973]: dbus-broker.service: Failed with result 'exit-code'.
May 15 05:33:25 xxx systemd[1973]: Failed to start dbus-broker.service - D-Bus User Message Bus.
May 15 05:33:25 xxx systemd[1973]: dbus-broker.service: Start request repeated too quickly.
May 15 05:33:25 xxx systemd[1973]: dbus-broker.service: Failed with result 'exit-code'.
May 15 05:33:25 xxx systemd[1973]: Failed to start dbus-broker.service - D-Bus User Message Bus.

I think bind mounting the whole host's /run into the chroot is wrong exactly for this reason, but I'm pretty sure there are reasons behind it (which I'm not familiar with), so I can't make any statements about it.

The quickest "fix" could be temporarily building systemd without the new ssh-generator stuff (-Dsshconfdir=no and probably removing the systemd-ssh-generator after build, since it seems to be always built, ugh), and then figuring out if it would be faster to deal with the issue on anaconda's side or our side (the preset thingy; this would be great to have in general anyway). IIRC Zbyszek should be back tomorrow, so I'll leave the decision to him.

I was wading through anaconda logs and came across this:

DEBUG:blivet:                     DirectoryDevice._set_format: /run ; type: bind ; current: None ;
DEBUG:blivet:updated /run size to 960 B (960 B)

Which, if it indeed means what I think it does, would explain where this severed link comes from. Meaning - if /run from the host is bind-mounted into the new chroot, the error is quite easy to reproduce, because the symlink generated by systemd-ssh-generator (/run/systemd/generator/sshd-vsock@.service) becomes severed in the chroot, if sshd is not part of the initial transaction:

# rpm -q systemd
systemd-256~rc1-99.fc41.x86_64
## Current selinux-policy prevents systemd-ssh-generator from working properly, hence the setenforce & daemon-reexec
# setenforce 0
# systemctl daemon-reexec
# ls -l /run/systemd/generator/sshd-vsock@.service
lrwxrwxrwx. 1 root root 37 Apr 28 16:00 /run/systemd/generator/sshd-vsock@.service -> /usr/lib/systemd/system/sshd@.service
# mkdir -p minimal/{proc,run,sys}
# mount -t proc proc minimal/proc
# mount -t sysfs sysfs minimal/sys
# mount --bind /run minimal/run/
# dnf --installroot $PWD/minimal --nogpgcheck --releasever rawhide -y install @core |& tee log.txt
...
Creating group 'systemd-journal' with GID 190.
Creating group 'systemd-network' with GID 192.
Creating user 'systemd-network' (systemd Network Management) with UID 192 and GID 192.
Creating group 'systemd-oom' with GID 999.
Creating user 'systemd-oom' (systemd Userspace OOM Killer) with UID 999 and GID 999.
Creating group 'systemd-resolve' with GID 193.
Creating user 'systemd-resolve' (systemd Resolver) with UID 193 and GID 193.
-rw-r--r--. 1 root root  547 Apr 16 00:00 dbus-broker.service
lrwxrwxrwx. 1 root root   25 Apr 28 00:00 dbus-org.freedesktop.hostname1.service -> systemd-hostnamed.service
lrwxrwxrwx. 1 root root   23 Apr 28 00:00 dbus-org.freedesktop.locale1.service -> systemd-localed.service
lrwxrwxrwx. 1 root root   22 Apr 28 00:00 dbus-org.freedesktop.login1.service -> systemd-logind.service
lrwxrwxrwx. 1 root root   25 Apr 28 00:00 dbus-org.freedesktop.timedate1.service -> systemd-timedated.service
-rw-r--r--. 1 root root  137 Jan 24 00:00 dbus.socket
drwxr-xr-x. 1 root root    0 Apr 28 00:00 dbus.target.wants
total 0
-- systemctl preset-all BEGIN --
Preset files say disable session-1.scope.
Preset files say disable -.mount.
Preset files say disable boot.mount.
Preset files say disable home.mount.
Preset files say disable var.mount.
Preset files say disable boot-efi.mount.
Cannot find unit sshd@.service.
Failed to preset unit sshd-vsock@.service: Link has been severed
-- systemctl preset-all END--
-- systemctl --global preset-all BEGIN --
Created symlink '/etc/systemd/user/sockets.target.wants/dbus.socket' → '/usr/lib/systemd/user/dbus.socket'.
Created symlink '/etc/systemd/user/dbus.service' → '/usr/lib/systemd/user/dbus-broker.service'.
Created symlink '/etc/systemd/user/timers.target.wants/grub-boot-success.timer' → '/usr/lib/systemd/user/grub-boot-success.timer'.
Created symlink '/etc/systemd/user/timers.target.wants/systemd-tmpfiles-clean.timer' → '/usr/lib/systemd/user/systemd-tmpfiles-clean.timer'.
Created symlink '/etc/systemd/user/basic.target.wants/systemd-tmpfiles-setup.service' → '/usr/lib/systemd/user/systemd-tmpfiles-setup.service'.
-- systemctl --global preset-all END--
...
# stat minimal/etc/systemd/system/dbus.service
stat: cannot statx 'minimal/etc/systemd/system/dbus.service': No such file or directory

I'm not really familiar with how anaconda does the actual installation, so please take this with a grain of salt, but it would explain what's going wrong in the netinstall ISO.

The sshd-vsock@.service unit was introduced in systemd v256 (https://github.com/systemd/systemd/commit/631cdd59e11b5a3147aa2cf120e9c4a24566cd01).

I rebuilt the RPM from this update with some more logging* and the systemctl preset-all command, that should create the dbus.service symlink (among others), seem to choke on the systemd-vsock@.service unit:

INFO:anaconda.modules.payloads.payload.dnf.transaction_progress:Configuring (running scriptlet for): systemd-256~rc1-99.fc41.x86_64 1714302944 ca3c023ab30f90a3da9987d3b60e9cbb16b91b65fa6bdea00173c18332a2d626
INFO:dnf.rpm:Creating group 'input' with GID 104.
Creating group 'kvm' with GID 36.
Creating group 'render' with GID 105.
Creating group 'sgx' with GID 106.
Creating group 'systemd-journal' with GID 190.
Creating group 'systemd-network' with GID 192.
Creating user 'systemd-network' (systemd Network Management) with UID 192 and GID 192.
Creating group 'systemd-oom' with GID 999.
Creating user 'systemd-oom' (systemd Userspace OOM Killer) with UID 999 and GID 999.
Creating group 'systemd-resolve' with GID 193.
Creating user 'systemd-resolve' (systemd Resolver) with UID 193 and GID 193.
lrwxrwxrwx. 1 root root   25 Apr 28 00:00 dbus-org.freedesktop.hostname1.service -> systemd-hostnamed.service
lrwxrwxrwx. 1 root root   23 Apr 28 00:00 dbus-org.freedesktop.locale1.service -> systemd-localed.service
lrwxrwxrwx. 1 root root   22 Apr 28 00:00 dbus-org.freedesktop.login1.service -> systemd-logind.service
lrwxrwxrwx. 1 root root   25 Apr 28 00:00 dbus-org.freedesktop.timedate1.service -> systemd-timedated.service
Failed to preset unit sshd-vsock@.service: Link has been severed
Created symlink '/etc/systemd/user/sockets.target.wants/dbus.socket' → '/usr/lib/systemd/user/dbus.socket'.
Created symlink '/etc/systemd/user/dbus.service' → '/usr/lib/systemd/user/dbus-broker.service'.
Created symlink '/etc/systemd/user/timers.target.wants/grub-boot-success.timer' → '/usr/lib/systemd/user/grub-boot-success.timer'.
Created symlink '/etc/systemd/user/timers.target.wants/systemd-tmpfiles-clean.timer' → '/usr/lib/systemd/user/systemd-tmpfiles-clean.timer'.
Created symlink '/etc/systemd/user/basic.target.wants/systemd-tmpfiles-setup.service' → '/usr/lib/systemd/user/systemd-tmpfiles-setup.service'.
  • I just dropped the various &>/dev/null from commands in systemd's RPM scriptlets; it would be great if some of these could be dropped altogether, as they make debugging issues like this very painful

Ah, I should add that in my case this is with systemd-255.5-1.fc41.x86_64 that the netinst ISO pulls in, but the OpenQA results seem to be with the build from this update.

The dbus-broker service is indeed disabled in the final system installed by the netinst ISO:

# systemctl status dbus-broker.service --no-pager
○ dbus-broker.service - D-Bus System Message Bus
     Loaded: loaded (/usr/lib/systemd/system/dbus-broker.service; disabled; preset: enabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: inactive (dead)
TriggeredBy: × dbus.socket
       Docs: man:dbus-broker-launch(1)

Even though presets say otherwise.

So, the netinst issue is caused (at least in my case) by missing /etc/systemd/system/dbus.service symlink:

# journalctl -o short-monotonic -b --no-pager
...
[    6.004806] fedora systemd[1]: Reached target timers.target - Timer Units.
[    6.005493] fedora systemd[1]: Listening on avahi-daemon.socket - Avahi mDNS/DNS-SD Stack Activation Socket.
[    6.006534] fedora systemd[1]: dbus.socket: Socket service dbus.service not loaded, refusing.
[    6.006581] fedora systemd[1]: Failed to listen on dbus.socket - D-Bus System Message Bus Socket.
[    6.006622] fedora systemd[1]: Dependency failed for avahi-daemon.service - Avahi mDNS/DNS-SD Stack.
[    6.006646] fedora systemd[1]: avahi-daemon.service: Job avahi-daemon.service/start failed with result 'dependency'.
[    6.006666] fedora systemd[1]: Dependency failed for firewalld.service - firewalld - dynamic firewall daemon.
[    6.006687] fedora systemd[1]: firewalld.service: Job firewalld.service/start failed with result 'dependency'.
[    6.006712] fedora systemd[1]: Dependency failed for systemd-homed.service - Home Area Manager.
[    6.006739] fedora systemd[1]: systemd-homed.service: Job systemd-homed.service/start failed with result 'dependency'.
...

# ls -l /etc/systemd/system
total 28
drwxr-xr-x. 1 root root  34 Apr 28 11:40 bluetooth.target.wants
lrwxrwxrwx. 1 root root  41 Apr 28 11:40 dbus-org.bluez.service -> /usr/lib/systemd/system/bluetooth.service
lrwxrwxrwx. 1 root root  41 Apr 28 11:40 dbus-org.fedoraproject.FirewallD1.service -> /usr/lib/systemd/system/firewalld.service
lrwxrwxrwx. 1 root root  44 Apr 28 11:40 dbus-org.freedesktop.Avahi.service -> /usr/lib/systemd/system/avahi-daemon.service
lrwxrwxrwx. 1 root root  45 Apr 28 11:40 dbus-org.freedesktop.home1.service -> /usr/lib/systemd/system/systemd-homed.service
lrwxrwxrwx. 1 root root  57 Apr 28 11:40 dbus-org.freedesktop.nm-dispatcher.service -> /usr/lib/systemd/system/NetworkManager-dispatcher.service
lrwxrwxrwx. 1 root root  44 Apr 28 11:40 dbus-org.freedesktop.oom1.service -> /usr/lib/systemd/system/systemd-oomd.service
lrwxrwxrwx. 1 root root  41 Apr 28 11:42 default.target -> /usr/lib/systemd/system/multi-user.target
drwxr-xr-x. 1 root root  30 Apr 28 11:40 graphical.target.wants
drwxr-xr-x. 1 root root 426 Apr 28 11:40 multi-user.target.wants
drwxr-xr-x. 1 root root  68 Apr 28 11:40 network-online.target.wants
drwxr-xr-x. 1 root root 136 Apr 28 11:40 sockets.target.wants
drwxr-xr-x. 1 root root 108 Apr 28 11:40 sysinit.target.wants
drwxr-xr-x. 1 root root  60 Apr 28 11:40 systemd-homed.service.wants
drwxr-xr-x. 1 root root 110 Apr 28 11:40 timers.target.wants

# systemctl status dbus
Unit dbus.service could not be found.
[root@fedora anaconda]# systemctl status dbus.socket
× dbus.socket - D-Bus System Message Bus Socket
     Loaded: loaded (/usr/lib/systemd/system/dbus.socket; disabled; preset: ena>
     Active: failed (Result: start-limit-hit)
   Triggers: ● dbus.service
     Listen: /run/dbus/system_bus_socket (Stream)

Apr 28 11:44:13 fedora systemd[1]: Failed to listen on dbus.socket - D-Bus Syst>
Apr 28 11:44:13 fedora systemd[1]: dbus.socket: Socket service dbus.service not>
Apr 28 11:44:13 fedora systemd[1]: Failed to listen on dbus.socket - D-Bus Syst>
Apr 28 11:44:13 fedora systemd[1]: dbus.socket: Socket service dbus.service not>
Apr 28 11:44:13 fedora systemd[1]: Failed to listen on dbus.socket - D-Bus Syst>
Apr 28 11:44:13 fedora systemd[1]: dbus.socket: Socket service dbus.service not>
Apr 28 11:44:13 fedora systemd[1]: Failed to listen on dbus.socket - D-Bus Syst>
Apr 28 11:44:13 fedora systemd[1]: dbus.socket: Start request repeated too quic>
Apr 28 11:44:13 fedora systemd[1]: dbus.socket: Failed with result 'start-limit>
Apr 28 11:44:13 fedora systemd[1]: Failed to listen on dbus.socket - D-Bus Syst>

But if I call systemctl enable dbus-broker.service manually, the symlink is created, strange.

Yeah, I was convinced that dracut 060 already has that patch, but turns out that was merged after 060 was released.

However, systemd is broken in this update as well, since currently PID 1 is not reexec'ed when upgrading from v255 to v256-rc1, see https://github.com/systemd/systemd/issues/32508#issuecomment-2079991745.

As mentioned in https://bodhi.fedoraproject.org/updates/FEDORA-2024-28601a6bb0#comment-3391708 it would be great to have this in Rawhide ASAP, since currently anything that depends on bpftool (or any other stuff from the former kernel-tools meta package) now FTBFS.

karma

Our systemd/SELinux Vagrant job is functional once again, so adding my +1, thanks!

BZ#2060045 Issues with networking in the latest Rawhide Vagrant (libvirt) boxes