Created attachment 1423275 [details]
tarball containing whole of /var/log after test fails

URL of an image to test with, for convenience: https://kojipkgs.fedoraproject.org/compose/branched/Fedora-28-20180417.n.0/compose/AtomicWorkstation/x86_64/iso/Fedora-AtomicWorkstation-ostree-x86_64-28-20180417.n.0.iso

note openQA tests in a qemu VM with 'std' graphics (same as 'vga', I think). Results may differ on bare metal or with a different virtual video device, I guess.

<mclasen> adamw: did mesa-libEGL go missing again ?
<mclasen> there was some glitch where soft deps were interpreted differently in tree compose, or something
<mclasen> https://pagure.io/fedora-comps/pull-request/250
<mclasen> maybe that libsolv change that colin was theorizing about has hit f28 ?
<adamw> who knows!
<adamw> we can apply that comps change to f28 easily enough, though, if that should fix it...

It does look like mesa-libEGL went missing:

$ rpm-ostree db diff 5344360af3807647a728366dc59ef23aa5100b4b83013316e0f1c9c9d0031103 d3461fff6aa51d6167ce57919368688cc2512f080048a3e0784e4161a208c701
ostree diff commit old: 5344360af3807647a728366dc59ef23aa5100b4b83013316e0f1c9c9d0031103
ostree diff commit new: d3461fff6aa51d6167ce57919368688cc2512f080048a3e0784e4161a208c701
...
Removed:
  authconfig-7.0.1-5.fc28.x86_64
  gnome-themes-standard-3.27.90-1.fc28.x86_64
  mesa-libEGL-18.0.0-4.fc28.x86_64
  mesa-libGL-18.0.0-4.fc28.x86_64

Should mutter depend on mesa-libEGL and mesa-libGL?

ajax just did a libglvnd build that adds back the mesa-libGL and mesa-libEGL hard requires.

libglvnd-1.0.1-0.5.20180327git5baa1e5.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-d48d3c17a2

I just opened two PRs (one for rawhide and one for f28) to sync the package list for FAW with comps. Note the rawhide one adds in mesa-libEGL, but the f28 one doesn't. So the comps differ slightly for rawhide vs f28, which might be why we are seeing an issue?

https://pagure.io/workstation-ostree-config/pull-request/83

https://pagure.io/workstation-ostree-config/pull-request/84

Dusty: that's exactly what the discussion in comment #3 was about, I think. Note that PR mclasen linked to changed only f29, not f28.

(In reply to Adam Williamson from comment #9)
> Dusty: that's exactly what the discussion in comment #3 was about, I think.
> Note that PR mclasen linked to changed only f29, not f28.

ok. if someone does decide to create that PR for f28 let me know and I can update comps again.

So, I managed to join the dots on exactly *why* the change from Requires to Recommends caused mesa-libEGL to go missing from the ostree, I think. Basically, because lorax doesn't pull Recommends into install trees, and the FAW ostree is built using the Workstation install tree for its base repo. Full details here: https://bugzilla.redhat.com/show_bug.cgi?id=1569242

So we kinda have three proposed fixes for this now: the libglvnd update, the clutter update in https://bugzilla.redhat.com/show_bug.cgi?id=1568881 , and the idea of making F28's comps specify mesa-libEGL explicitly like Rawhide's.

We probably don't really need all three of those. What do we think is the most actually-correct combination of the three to do? And should we synchronize whatever we decide on with Rawhide?

libglvnd-1.0.1-0.5.20180327git5baa1e5.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-d48d3c17a2

(In reply to Adam Williamson from comment #12)
> So we kinda have three proposed fixes for this now: the libglvnd update, the
> clutter update in https://bugzilla.redhat.com/show_bug.cgi?id=1568881 , and
> the idea of making F28's comps specify mesa-libEGL explicitly like Rawhide's.
> 
> We probably don't really need all three of those. What do we think is the
> most actually-correct combination of the three to do? And should we
> synchronize whatever we decide on with Rawhide?

I'd prefer going down the hard dep route for F28 GA. We did this for F27 and it's just reverting to a known good state. If we do the comps fix, adding mesa-libEGL to Workstation comps, it probably fixes Workstation well enough, but who knows if it fixes all other spins and distro upgrades. I'd sleep better if we have a hard dep for F28 :)

For rawhide and short term, I'd say let's keep the exact same thing as we do for F28, whichever it ends up to be -- if we go for hard dep, then revert the comps change that already went in.

For longer term for rawhide, I'd like to go back to libglvnd having Recommends on mesa-libEGL and figure out the install tree issues so that recommended deps are pulled in. I don't think this is something we should play with during F28 freeze though.

There are still three permutations of "the hard dep route", though: change libglvnd, change clutter, or change both. Which of those do you prefer?

I think both are needed. libglvnd update pulls in mesa-libGL, clutter update pulls in mesa-dri-drivers.

OK, in that case, I'm +1 FE for both this and 1568881 . Thanks.

+1 FE

+1 FE

+1 FE

That's +4, setting Accepted.

+1 FE

libglvnd-1.0.1-0.5.20180327git5baa1e5.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.

On https://bugzilla.redhat.com/show_bug.cgi?id=1568881#c23 , Petr Pisar wrote:

"After adding mesa-libGL dependency to libglvnd, gtk clients running against Xvfb crash. Probably because of bug #1568644."

As this bug was actually the bug we treated as being 'for libglvnd', let's move that discussion over here. I'm not yet sure how big of a problem this is for F28 release, trying to look into it ATM.

If I'm understanding ajax correctly, he thinks #1568644 is only really a problem if mesa-dri-drivers isn't installed. AFAICT it is installed in all the scenarios that are a problem for F28 release: it's in the installer, it's in desktop live images and so on. (In comps, it's in base-x, and all the desktop environment groups include base-x). I tested a VNC install and it worked OK. So unless we're missing something, we don't think there's a release-critical issue there. The issue with buildroots is a real one, but doesn't need to be a release blocker or FE.