Created attachment 1197959 [details]
system journal after hang happened

Created attachment 1197960 [details]
systemctl status --all during the hang

Created attachment 1197961 [details]
rpm -qa output

Created attachment 1197962 [details]
pstack on gsd process 1

Created attachment 1197963 [details]
pstack on gsd process 2

How long did you wait to log in to your session, are you sure it doesn't just take very long?

I've recently seen an issue in g-s-d which causes login to hang for a very long time (like 30 seconds) until login finally succeeds. You might want to try the workaround: Downgrade cups-libs to 2.1.4 and try to login again.

Source and possible duplicate of this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1366775

(In reply to Christian Stadelmann from comment #6)
> How long did you wait to log in to your session, are you sure it doesn't
> just take very long?

I waited for about 4 minutes.

> I've recently seen an issue in g-s-d which causes login to hang for a very
> long time (like 30 seconds) until login finally succeeds. You might want to
> try the workaround: Downgrade cups-libs to 2.1.4 and try to login again.

I masked cups.service and it didn't help. Seems to be a separate issue (unless masking is not supposed to avoid it).

(In reply to Kamil Páral from comment #7)
> (In reply to Christian Stadelmann from comment #6)
> > I've recently seen an issue in g-s-d which causes login to hang for a very
> > long time (like 30 seconds) until login finally succeeds. You might want to
> > try the workaround: Downgrade cups-libs to 2.1.4 and try to login again.
> 
> I masked cups.service and it didn't help. Seems to be a separate issue
> (unless masking is not supposed to avoid it).

Masking the cups.service is not supposed to avoid the problem about which Christian is talking. Downgrading CUPS or explicit start of cups.service fixes the problem.

After 2 hours of bisecting, I found out the culprit is gstreamer1-vaapi, not gnome components. This transaction breaks the desktop for me:

Command Line   : distro-sync gstreamer1-vaapi
Transaction performed with:
    Installed     dnf-1.1.10-1.fc25.noarch        @@commandline
    Installed     rpm-4.13.0-0.rc1.46.fc25.x86_64 @@commandline
Packages Altered:
    Upgraded gstreamer1-vaapi-1.9.1-1.fc25.x86_64 @fedora
    Upgrade                   1.9.2-1.fc25.x86_64 @updates-testing


My hardware is:
00:02.0 VGA compatible controller [0300]: Intel Corporation HD Graphics 5500 [8086:1616] (rev 09)

In case this is somehow related to vdpau (I think there's some relation between vaapi and vdpau), I have these packages installed:
libva-vdpau-driver-0.7.4-14.fc24.x86_64
libvdpau-1.1.1-3.fc24.x86_64
mesa-vdpau-drivers-12.0.1-2.fc25.x86_64
vdpauinfo-1.0-5.fc24.x86_64

Afaik libva (=VAAPI) uses vdpau on Nvidia or some AMD GPU hardware if libva-vdpau-driver is installed. In case you have an intel hardware, the libva-vdpau-driver shouldn't be used at all.
I've had some issues with libva-intel-driver so I uninstalled it too (on F24).

See also: https://bugzilla.redhat.com/show_bug.cgi?id=1123536

Can we have any info about why gst is even run on the start of a wayland session ?

Can you reproduce with libva-vdpau-driver removed ?

(In reply to Nicolas Chauvet (kwizart) from comment #11)
> Can you reproduce with libva-vdpau-driver removed ?

I tried removing libva-vdpau-driver, and then also installing libva-intel-driver. None of that helped, same issue.

If I remove gstreamer1-vaapi and keep libva* packages installed, there's no problem during login.

Can you try to load a video with another player (such as vlc) using vaapi.
Then with gstreamer ?

Which component tries to load a video "at" login ?

(In reply to Nicolas Chauvet (kwizart) from comment #14)
> Can you try to load a video with another player (such as vlc) using vaapi.
> Then with gstreamer ?

When using totem with gstreamer1-vaapi, I see only a black screen. Totem prints this:

$ totem sintel_trailer-480p.mp4 
libva info: VA-API version 0.39.2
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib64/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_39
libva info: va_openDriver() returns 0
libva info: VA-API version 0.39.2
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib64/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_39
libva info: va_openDriver() returns 0
libva info: VA-API version 0.39.2
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib64/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_39
libva info: va_openDriver() returns 0
... <unrelated output>


Vlc plays the video fine whichever output type I choose, but I suspect it does not use hardware acceleration in any of those cases. The output is always the same:

$ vlc -v sintel_trailer-480p.mp4 
VLC media player 3.0.0-git Vetinari (revision 2.2.0-git-8681-g749293f)
[000055a6a2aab1c8] core libvlc: Running vlc with the default interface. Use 'cvlc' to use vlc without interface.
[00007f6558c01728] mp4 stream warning: unknown box type ccpy (incompletely loaded)
[00007f6558c01728] mp4 stream warning: unknown box type desc (incompletely loaded)
[00007f6558c018f8] mp4 demux warning: elst box found
[00007f6558c018f8] mp4 demux warning: STTS table of 1 entries
[00007f6558c018f8] mp4 demux warning: CTTS table of 1059 entries
[00007f6558c018f8] mp4 demux warning: STTS table of 1 entries
[00007f6558cc15f8] avcodec decoder warning: thread type 1: disabling hardware acceleration
[00007f6558d7ae88] faad decoder warning: decoded zero sample
Failed to open VDPAU backend libvdpau_nvidia.so: cannot open shared object file: No such file or directory
[00007f6558cc15f8] avcodec decoder warning: plane 0 not aligned: disabling direct rendering
[000055a6a2b91c58] pulse audio output warning: starting late (-12197 us)

(In reply to Kamil Páral from comment #7)
> (In reply to Christian Stadelmann from comment #6)
> > How long did you wait to log in to your session, are you sure it doesn't
> > just take very long?
> 
> I waited for about 4 minutes.

On an SSD or a rotating disk? I'm asking because of the access times. I am running into this freeze every time I log in for the first time after boot and on a SSD it takes me 5…30 seconds.

> > I've recently seen an issue in g-s-d which causes login to hang for a very
> > long time (like 30 seconds) until login finally succeeds. You might want to
> > try the workaround: Downgrade cups-libs to 2.1.4 and try to login again.
> 
> I masked cups.service and it didn't help. Seems to be a separate issue
> (unless masking is not supposed to avoid it).

Ok, the cups issue is completely unrelated, see the bug report there.

Yes, this is about gstreamer1-vaapi, not cups.

I don't know if it's the same or not, but I've got the same effect right after upgrading from F24 to F25 Beta, but (I think) it was caused by GNOME Shell because after I logged into GNOME on Xorg, SELinux troubleshooter gave me a notification (I ignored it at first, but I couldn't log in to the wayland session without fixing it).

I reported the issue in Bug #1385090. and I fixed the issue with commands:
# ausearch -c 'gnome-shell' --raw | audit2allow -M my-gnomeshell
# semodule -X 300 -i my-gnomeshell.pp

https://bodhi.fedoraproject.org/updates/FEDORA-2016-a3eee5e16a seems to fix this. No login issues and totem now correctly plays videos.

I see this again with gstreamer1-vaapi-1.10.1-1.fc25 from https://bodhi.fedoraproject.org/updates/FEDORA-2016-344df1623c . This is in journal related to the frozen login:

Dec 05 10:14:12 dryad gst-plugin-scan[2074]: Couldn't g_module_open libpython. Reason: /usr/lib64/libpython3.5m.so: cannot open shared object file: No such file or directory
...
Dec 05 10:15:42 dryad gnome-session[1893]: gnome-session-binary[1893]: WARNING: Application 'gnome-settings-daemon.desktop' failed to register before timeout
Dec 05 10:15:42 dryad gnome-session-binary[1893]: WARNING: Application 'gnome-settings-daemon.desktop' failed to register before timeout
Dec 05 10:15:42 dryad gnome-session-binary[1893]: Unrecoverable failure in required component gnome-settings-daemon.desktop

I asked for gnome-settings-daemon to be more resilient to gstreamer1-vaapi issues here:
https://bugzilla.gnome.org/show_bug.cgi?id=775619

Which component use system-python-libs (libpython3.5m.so) in gstreamer ? (not gstreamer1-vaapi at least).
I don't find it installed by default on my current system (f24)

(In reply to Kamil Páral from comment #20)
> I see this again with gstreamer1-vaapi-1.10.1-1.fc25 from
> https://bodhi.fedoraproject.org/updates/FEDORA-2016-344df1623c . This is in
> journal related to the frozen login:
> 
> Dec 05 10:14:12 dryad gst-plugin-scan[2074]: Couldn't g_module_open
> libpython. Reason: /usr/lib64/libpython3.5m.so: cannot open shared object
> file: No such file or directory
> ...
> Dec 05 10:15:42 dryad gnome-session[1893]: gnome-session-binary[1893]:
> WARNING: Application 'gnome-settings-daemon.desktop' failed to register
> before timeout
> Dec 05 10:15:42 dryad gnome-session-binary[1893]: WARNING: Application
> 'gnome-settings-daemon.desktop' failed to register before timeout
> Dec 05 10:15:42 dryad gnome-session-binary[1893]: Unrecoverable failure in
> required component gnome-settings-daemon.desktop

Can you supply the snipped bit? There's a library in python3-gstreamer1 that would cause the error you're seeing, but it should fail fast if it can't find the binaries it's expecting.

Also, please retry without gstreamer1-vaapi installed, to confirm that it's gstreamer1-vaapi and not python3-gstreamer1 causing the bug. If the presence or absence of the bug is directly tied to gstreamer1-vaapi being installed, then we at least know it's not the message you're seeing that causes the freeze.

So, after further debugging I found out this problem is related to EasyScreenCast extension [1]. If I disable it, the login works fine - probably because nothing touches vaapi during login. But if EasyScreenCast is enabled, the login is stuck. Either downgrading gstreamer1-vaapi to 1.10.0-1.fc25, or removing it completely solves the problem.

[1] https://extensions.gnome.org/extension/690/easyscreencast/

Created attachment 1228071 [details]
journal during frozen login

Created attachment 1228072 [details]
rpm -qa output

This is getting ridiculous, but after experimenting a bit I can't login at all even with gstreamer1-vaapi-1.10.0-1.fc25, with EasyScreenCast enabled. I either need to disable the extension, use X11 or uninstall gstreamer1-vaapi. I think this got triggered by logging in once using X11, and since that time Wayland login stopped working even for gstreamer1-vaapi 1.10.0. It's a mess, and I'm getting a suspicion this could have the same root cause as bug 1394755. However, I experimented with ~/.cache/gstreamer-1.0/registry.x86_64.bin and still could not get it working.

After reading https://bugzilla.gnome.org/show_bug.cgi?id=776041#c1 I believe this is essentially the same problem as bug 1394755. The gstreamer cache registry gets updated due to the a gstreamer plugin updated (in this case vaapi) and causes a deadlock in gnome-shell. Therefore probably not related to vaapi plugin at all. Closing as a duplicate.

*** This bug has been marked as a duplicate of bug 1394755 ***

Confirmed with Rui that this is indeed a duplicate. During login, gst-plugin-scanner is running (waiting) with this stack trace:

Thread 1 (Thread 0x7f09333b8700 (LWP 3608)):
#0  0x00007f0931e31ff0 in __poll_nocancel () at /lib64/libc.so.6
#1  0x00007f092fabb7c9 in wl_display_poll () at /lib64/libwayland-client.so.0
#2  0x00007f092fabcdac in wl_display_dispatch_queue () at /lib64/libwayland-client.so.0
#3  0x00007f092fabd00b in wl_display_roundtrip_queue () at /lib64/libwayland-client.so.0
#4  0x00007f093137fcb2 in gst_vaapi_display_wayland_setup () at /usr/lib64/gstreamer-1.0/libgstvaapi.so
#5  0x00007f09313459d9 in gst_vaapi_display_new () at /usr/lib64/gstreamer-1.0/libgstvaapi.so
#6  0x00007f093132129a in gst_vaapi_create_test_display () at /usr/lib64/gstreamer-1.0/libgstvaapi.so
#7  0x00007f093131bd17 in plugin_init () at /usr/lib64/gstreamer-1.0/libgstvaapi.so
#8  0x00007f0932f0ec67 in gst_plugin_register_func () at /lib64/libgstreamer-1.0.so.0
#9  0x00007f0932f10bb6 in _priv_gst_plugin_load_file_for_registry () at /lib64/libgstreamer-1.0.so.0
#10 0x00007f0932f135d3 in exchange_packets () at /lib64/libgstreamer-1.0.so.0
#11 0x00007f0932f145e8 in _gst_plugin_loader_client_run () at /lib64/libgstreamer-1.0.so.0
#12 0x000055742258d914 in main ()
Detaching from program: /usr/libexec/gstreamer-1.0/gst-plugin-scanner, process 3608