We're not seeing test results and the gating status isn't changing because greenwave is giving us a 502 response with the error "Unexpected error while fetching remote policies". I actually thought I'd made Bodhi display these errors :( Will have to see why that's not working, and why we're getting the error in the first place.

@kevin says he caused this: "I mistakenly made that build from a src.rpm... (which you can accidentally do if you are a koji admin)"

so he'll fix that bit, and I'll see if it makes sense to have greenwave be a bit more robust or at least communicate the error through to Bodhi so we don't have to go schlep through greenwave logs...

https://github.com/release-engineering/greenwave/issues/218 is the greenwave issue here. Kevin has fixed the build and edited the update, so we're now seeing the test results and the gating status has updated. though, er, it probably shouldn't be 'passed'...hmm. I think it got sent to 'passed' immediately due to the previous test passes from when the tests ran days ago, and hasn't yet flipped to failed because tests are failing now, probably because they've been restarted...

Oh, the tests are now failing for the same reason Bodhi is complaining - some of the builds in the update are now old and should be rebuilt again, I guess.

libvirt and qemu have newer builds already in f40 tag I am removing these builds from the update.

requested fresh builds of libvirt and qemu in the side tag

I'll do it.

qemu build is failing on something related to xen...

FAILED: libqemu-aarch64-softmmu.fa.p/hw_arm_xen_arm.c.o 
gcc -m64 -mcx16 -Ilibqemu-aarch64-softmmu.fa.p -I. -I.. -Itarget/arm -I../target/arm -Iqapi -Itrace -Iui -Iui/shader -I/usr/include/pixman-1 -I/usr/include/capstone -I/usr/include/spice-server -I/usr/include/spice-1 -I/usr/include/nss3 -I/usr/include/nspr4 -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -I/usr/include/cacard -I/usr/include/sysprof-6 -I/usr/include/PCSC -fdiagnostics-color=auto -Wall -Winvalid-pch -std=gnu11 -O2 -g -fstack-protector-strong -Wundef -Wwrite-strings -Wmissing-prototypes -Wstrict-prototypes -Wredundant-decls -Wold-style-declaration -Wold-style-definition -Wtype-limits -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wempty-body -Wnested-externs -Wendif-labels -Wexpansion-to-defined -Wimplicit-fallthrough=2 -Wmissing-format-attribute -Wno-missing-include-dirs -Wno-shift-negative-value -Wno-psabi -Wshadow=local -isystem /builddir/build/BUILD/qemu-8.2.0-rc2/linux-headers -isystem linux-headers -iquote . -iquote /builddir/build/BUILD/qemu-8.2.0-rc2 -iquote /builddir/build/BUILD/qemu-8.2.0-rc2/include -iquote /builddir/build/BUILD/qemu-8.2.0-rc2/host/include/x86_64 -iquote /builddir/build/BUILD/qemu-8.2.0-rc2/host/include/generic -iquote /builddir/build/BUILD/qemu-8.2.0-rc2/tcg/i386 -Wno-unused-function -pthread -DSTAP_SDT_V2 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -fno-strict-aliasing -fno-common -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Werror=implicit-function-declaration -Werror=implicit-int -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -O2 -fexceptions -g -grecord-gcc-switches -Wall -Wno-complain-wrong-lang -Werror=format-security -Werror=implicit-function-declaration -Werror=implicit-int -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fPIE -isystem../linux-headers -isystemlinux-headers -DNEED_CPU_H '-DCONFIG_TARGET="aarch64-softmmu-config-target.h"' '-DCONFIG_DEVICES="aarch64-softmmu-config-devices.h"' -MD -MQ libqemu-aarch64-softmmu.fa.p/hw_arm_xen_arm.c.o -MF libqemu-aarch64-softmmu.fa.p/hw_arm_xen_arm.c.o.d -o libqemu-aarch64-softmmu.fa.p/hw_arm_xen_arm.c.o -c ../hw/arm/xen_arm.c
../hw/arm/xen_arm.c: In function ‘xen_create_virtio_mmio_devices’:
../hw/arm/xen_arm.c:74:5: error: ‘GUEST_VIRTIO_MMIO_SPI_LAST’ undeclared (first use in this function)
   74 |    (GUEST_VIRTIO_MMIO_SPI_LAST - GUEST_VIRTIO_MMIO_SPI_FIRST)
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~
../hw/arm/xen_arm.c:87:21: note: in expansion of macro ‘NR_VIRTIO_MMIO_DEVICES’
   87 |     for (i = 0; i < NR_VIRTIO_MMIO_DEVICES; i++) {
      |                     ^~~~~~~~~~~~~~~~~~~~~~
../hw/arm/xen_arm.c:74:5: note: each undeclared identifier is reported only once for each function it appears in
   74 |    (GUEST_VIRTIO_MMIO_SPI_LAST - GUEST_VIRTIO_MMIO_SPI_FIRST)
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~
../hw/arm/xen_arm.c:87:21: note: in expansion of macro ‘NR_VIRTIO_MMIO_DEVICES’
   87 |     for (i = 0; i < NR_VIRTIO_MMIO_DEVICES; i++) {
      |                     ^~~~~~~~~~~~~~~~~~~~~~
../hw/arm/xen_arm.c:74:34: error: ‘GUEST_VIRTIO_MMIO_SPI_FIRST’ undeclared (first use in this function)
   74 |    (GUEST_VIRTIO_MMIO_SPI_LAST - GUEST_VIRTIO_MMIO_SPI_FIRST)
      |                                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~
../hw/arm/xen_arm.c:87:21: note: in expansion of macro ‘NR_VIRTIO_MMIO_DEVICES’
   87 |     for (i = 0; i < NR_VIRTIO_MMIO_DEVICES; i++) {
      |                     ^~~~~~~~~~~~~~~~~~~~~~
../hw/arm/xen_arm.c:88:23: error: ‘GUEST_VIRTIO_MMIO_BASE’ undeclared (first use in this function)
   88 |         hwaddr base = GUEST_VIRTIO_MMIO_BASE + i * VIRTIO_MMIO_DEV_SIZE;
      |                       ^~~~~~~~~~~~~~~~~~~~~~

Yes, we know. I think there is an easy fix but I haven't heard back from the qemu team yet.

those things it's trying to use are defined (only) in include/hw/xen/interface/arch-arm.h, which is copied from Xen itself. that is (I guess) included when building on ARM, but not when building on x86. Trying to always include it in xen_arm.c (whatever platform we're building on) just causes failures because now lots of stuff has competing definitions. Not sure how to proceed there (aside from reverting the qemu 8.2.0 commit that caused the problem...)

One thing I don't understand - how did @crobinso 's original 8.2.0rc2 build work? I'm rather confused about that.

The 8.2.0.rc2 qemu builds works for xen 4.17 but not for xen 4.18 because there is a xen version condition in include/hw/xen/xen_native.h which is in my opinion broken.