Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1756143

Summary: after running 'mock', /proc/self/cgroup contains name=systemd. intentional?
Product: [Fedora] Fedora Reporter: Cole Robinson <crobinso>
Component: systemdAssignee: systemd-maint
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 31CC: ego.cordatus, lnykryn, msekleta, praiskup, ssahani, s, systemd-maint, vascom2, yaneti, zbyszek
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: systemd-243-3.gitef67743.fc31 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-10-26 17:24:42 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Cole Robinson 2019-09-26 20:45:45 UTC
I'm not sure if this is a bug or intentional behavior, so I'm looking for some advice.

$ uname -a
Linux worklaptop 5.3.0-1.fc31.x86_64 #1 SMP Mon Sep 16 12:34:42 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
$ rpm -q systemd
systemd-243-2.gitfab6f01.fc31.x86_64
$ mount | grep cgroup
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,seclabel,nsdelegate)
$ cat /proc/self/cgroup
0::/user.slice/user-1000.slice/...

After I run a mock build (via fedpkg mockbuild), the last bit changes
$ cat /proc/self/cgroup
1:name=systemd:/
0::/user.slice/user-1000.slice/...

This broke libvirt VM startup: https://bugzilla.redhat.com/show_bug.cgi?id=1751120

podman container startup using crun also seems broken:

$ podman run -it alpine sh
Error: creating cgroup directory '/sys/fs/cgroup/name=systemd/user.slice/user-1000.slice/user/user.slice/libpod-69952a2d396138de9ffa0d1a5e096ae2d6382c6e17c93a0baf53b762a8f83ff9.scope': Permission denied: OCI runtime error

So is this a systemd bug, or do both those apps need to adjust their expectations? Also is their some way to 'undo' this situation without rebooting the host, as a temporary workaround?

Comment 1 Zbigniew Jędrzejewski-Szmek 2019-09-27 13:03:48 UTC
mock doesn't matter here, plain systemd-nspawn invocation is enough to cause this
(with the right image).

systemd-nspawn has code to detect if the image it is about to start has an init that
has support for the unified cgroup hierarchy. Specifically, it checks if systemd>230
is installed. If it detects no support, if falls back to mounting legacy hierarchy.
This is a bit ugly, but in the beginning almost nothing had support for the unified
hierarchy, and we didn't want to break spawning of existing images. Maybe this should
be revisited, but I don't think there's any easy solution if we want to keep compatibility
with older images.

With recent buildroot dependency cleanups, systemd may no longer be installed by mock,
so suddenly systemd-nspawn detects no support and falls back to legacy in the booted
image. Once that happens, I don't think it is possible to undo without rebooting.

It is possible to override the detection by setting UNIFIED_CGROUP_HIERARCHY=yes.
You can use that as a temporary work-around.

That said, mock actually calls systemd-nspawn with --as-pid2/-a. In this mode, nspawn
"knows" that a compatible init will be running (because it is it), so we should
skip the detection based on the image. I'll prep a patch.

Comment 2 Fedora Update System 2019-10-10 20:46:35 UTC
FEDORA-2019-5cb48e5111 has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2019-5cb48e5111

Comment 3 Fedora Update System 2019-10-11 16:53:38 UTC
systemd-243-3.gitef67743.fc31 has been pushed to the Fedora 31 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-5cb48e5111

Comment 4 Artem 2019-10-14 07:07:23 UTC
RHBZ#1756143 not fixed. I did ~40 builds and now it hangs again.

Comment 5 Zbigniew Jędrzejewski-Szmek 2019-10-14 07:40:19 UTC
@Artem:
This bug is about a very simple thing: whether 'systemd-nspawn --as-pid2' mounts
v1 or v2 hierarchy in the container. It is fully deterministic, without any race conditions,
so if you're seeing hangs when doing a large number of builds, then it's most likely
some different thing.

Comment 6 Artem 2019-10-14 07:43:54 UTC
Oh, sorry. Maybe we should reopen #1754807 then?

https://bugzilla.redhat.com/show_bug.cgi?id=1754807

@Vascom wrote that it still happens as well.

Comment 7 Vasiliy Glazov 2019-10-14 07:46:02 UTC
#1754807 still persist.

Comment 8 Fedora Update System 2019-10-26 17:24:42 UTC
systemd-243-3.gitef67743.fc31 has been pushed to the Fedora 31 stable repository. If problems still persist, please make note of it in this bug report.