Kevin, Paul,

Paul provided me a machine so I'd be able to give it a try.
In order to bisect what may be causing the issue (and unblock you guys), I'd like to confirm that the hanging process is the sssd that's installed in the mounted fs, not in the system one, right?

Considering that's the case, what's the easiest way to change the sssd version installed in the mounted fs?

Yeah, it's the one in the image that appliance-creator is making. 

It makes a file, loop mounts it and installs into it. At the end it should umount it and finish up, but this bug is causing that loop mount to be busy and so it cannot umount the loop and it fails. 

It should use your enabled repos on the machine. So perhaps exclude sssd from the normal repos and add a local repo and put sssd versions in there?

koji calls it like so: 

'/usr/bin/appliance-creator', '-c', '/chroot_tmpdir/koji-image-f29-build-27652379.ks', '-d', '-v', '--logfile', '/chroot_tmpdir/appliance.log', '--cache', '/chroot_tmpdir/koji-appliance', '-o', 'app-output', '--format', 'raw', '--name', 'Fedora-Minimal-armhfp-Rawhide-20180616.n.0', '--version', 'Rawhide', '--release', '20180616.n.0'

You can find a ks at: https://kojipkgs.fedoraproject.org//work/tasks/2379/27652379/koji-image-f29-build-27652379.ks

Thanks much for looking into this! Let me know if I can assist any further...

Kevin,

Paul has shared that. Retreading it I realized I can simply tweak the .ks file to use my own version of sssd (hopefully). I will take a look at that.

Thanks to both of you!

Hi,

I think 

@@ -909,6 +859,9 @@ done
 %attr(750,root,root) %dir %{_var}/log/%{name}
 %attr(700,root,root) %dir %{_sysconfdir}/sssd
 %attr(711,root,root) %dir %{_sysconfdir}/sssd/conf.d
+%if (0%{?use_openssl} == 1)
+%attr(711,sssd,sssd) %dir %{_sysconfdir}/sssd/pki
+%endif
 %ghost %attr(0600,root,root) %config(noreplace) %{_sysconfdir}/sssd/sssd.conf
 %dir %{_sysconfdir}/logrotate.d
 %config(noreplace) %{_sysconfdir}/logrotate.d/sssd

is causing the issue.

Fabiano, can you change the owner of the pki directory to root so that it reads:

%if (0%{?use_openssl} == 1)
%attr(711,root,root) %dir %{_sysconfdir}/sssd/pki
%endif

(same permissions and owner as eg. ../sssd/conf.d)


While installing the package the user 'sssd' has to be looked up to set the permission. But since the appliance-creator is running changed root in the image directory at this point and the sssd user does not exist in /etc/passwd of the image glibc's nss code has to check all configured nss modules (sss, files and systemd here). It looks like sss and file where already load while appliance-creator was not running in the changed root environment to do other lookup. But systemd is loaded for the first time and since appliance-creator is currently in the change root environment libnss_system.so is loaded from the change root image and not from the main system and stays open because the nss modules are not unloaded only when the process exits.

I found https://pagure.io/appliance-tools/c/398360b2b5e86072cee058e0e3d2eb9a74eb158e?branch=master which I guess was added to fix a similar issue.

To avoid this issue in future libnss_system.so.2 can be loaded in the same way at startup or the dlopen can be replaced with a lookup for an non-existing user like e.g.:

+def do_unknown_user_hack():
+    import pwd as forgettable
+    try:
+        forgettable.getpwnam('fwefwkejkgre')
+    except:
+        pass
+    del forgettable
+    return
+
 if __name__ == "__main__":
-    hack = do_nss_sss_hack()
+    do_unknown_user_hack()
     sys.exit(main())


On the other hand you might not want to change anything and consider the failure as a test for unknown users in the installed packages.

HTH

bye,
Sumit

P.S. I think I somehow broke the test system you provided because I've see 'umount: /var/tmp/imgcreate-ZwmDIR/install_root/sys: target is busy' in my last tests as well.

Awesome detective work. Thanks for digging into this.

Kevin,

I've pushed the fix suggested by Sumit. I've tested it in the machine provided by Paul and it works like a charm.

The build is on its way and from Tomorrow (or later Today) you should be unblocked.

I'll keep this bug opened till I hear a positive feedback from you. Okay?

Sure. :) If we get a rawhide compose tomorrow I think we can close it as fixed.

This looks fine now. Thanks for all your work fixing it!