This is a weird error, mostly because the IPA version is less than our minimum. I think we need to wait for FreeIPA to update and then we can retest.

It's not really a 'weird error', you are doing a bad thing: creating an update that is incompatible with the distribution as it stands. If this PKI needs a newer FreeIPA than F28 currently contains, then either that FreeIPA should also be in this update, or a FreeIPA update should be created first, pushed stable, and then PKI can be updated. If the two newer versions are interdependent (neither works with the old version of the other), they must be updated together in a single update.

BTW, some more info on precisely how the test failed: in this update, the pki-server package had a Conflicts: freeipa-server < 4.7.1 line added. However, none of the other pki-core subpackages had this done, and there are no dependencies between the various subpackages of pki-core which force them to always be of equal versions. So when the test asked DNF to install the group @freeipa-server, what DNF wound up doing to satisfy the PKI deps was installing version 10.6.8-1.fc28 of pki-base, pki-base-java, python3-pki, pki-symkey and pki-tools, but version 10.6.6-1.fc28 (the old version from the stable repos) of pki-server. There's nothing that makes this 'invalid' so far as dependencies are concerned; it's a correct solution to the request. Obviously, things don't work well with old FreeIPA, old pki-server, but new everything-else-pki. :)

Sure, but I think calling this a "weird error" is acceptable as you just pointed out: we're doing a bad thing, and the error we see isn't that we're doing a bad thing, its that something else, later, way down the line, is broken.

In OpenQA, after the bodhi update installation stage, I'd expect to verify at least that the packages in this bodhi update were installed at the versions in the update. e.g., rpm -qa shows them as being installed at the versions here.

Put another way, if I take $CURRENT_VERSION of $PKG and package $CURRENT_VERSION+1 as $CURRENT_VERSION with a new line Conflicts: $something_else < k in the spec file, and $something_else >= k hasn't been released and so we get $CURRENT_VERSION installed in Bodhi (instead of $CURRENT_VERSION+1), I'd expect Bodhi to fail with a package version error. I would not expect it to pass (assuming that $PKG @ $CURRENT_VERSION passes Bodhi previously and will pass again).

(In the last paragraph, s/Bodhi/OpenQA/g, sorry).

Sure, but I think calling this a "weird error" is acceptable as you just pointed out: we're doing a bad thing, and the error we see isn't that we're doing a bad thing, its that something else, later, way down the line, is broken.

Are you sure that you are not doing a bad thing? OpenQA do the same as normal user would do. Just call dnf update and the result is combination of packages which cause failures to install freeIPA.

And it is obviously caused by this bodhi update because it pass without updates-testing. Sure; error reporting could be better. But assumption about $CURRENT_VERSION+1 is difficult. You can obsolete some packages; replace ... So it is would be a challenge to guess which set of packages should be installed.

I would appreciate if you could either un-push this update from updates-testing or add freeipa-4.7.2 to this update. I am not sure whether you have permissions for that.

I've added updated pki-core and freeipa packages together on this Bodhi update so the conflict issues should now be resolved.

It's arguable both ways. As @lslebodn points out, openQA's basic goal is to test how things work as a user would encounter them. He's correct that, had this update been pushed stable, a user would've seen exactly what openQA did: they'd run an update, it would apparently work just fine, then suddenly FreeIPA would break.

However, we can improve things so that openQA provides more information to detect this kind of problem, certainly. The reason it doesn't at present is simply that I hadn't considered that this scenario might be possible; dependency issues in updates usually tend to manifest as errors in package install or update, not this kind of 'the operation succeeds but doesn't install what we expect it to' case. This is the first case like this I've come across in an openQA test.

FreeIPA 4.7 PR-CI is passing with pki-core-10.6.8-3, https://github.com/freeipa/freeipa/pull/2646. PR-CI uses F29 while Travis CI tests on F28. I was also able to install an FreeIPA cluster with two servers on Fedora 28 successfully.