Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1708249 - Updating rawhide container fails due to filesystem update problem
Summary: Updating rawhide container fails due to filesystem update problem
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: buildah
Version: 29
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Giuseppe Scrivano
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-05-09 13:05 UTC by Alex Scheel
Modified: 2020-08-07 06:35 UTC (History)
13 users (show)

Fixed In Version: buildah-1.8.2-1.gite23314b.fc30 buildah-1.8.2-1.gite23314b.fc29
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-05-16 17:47:35 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
Full output of dnf update on a Fedora rawhide container (75.93 KB, text/plain)
2019-05-09 13:09 UTC, Alex Scheel
no flags Details

Description Alex Scheel 2019-05-09 13:05:39 UTC
Description of problem:

I've pulled the rawhide image and tried doing a dnf --update:


FROM fedora:rawhide

# Update the container
RUN true \
        && dnf update -y --refresh \
        && true

CMD true \
        && bash -i \
        && true


When executing this under buildah as a regular user, I get the following:



$ buildah bud -f bugreports/fedora_filesystem --tag fedora_filesystem:rawhide .
STEP 1: FROM fedora:rawhide
STEP 2: RUN true         && dnf update -y --refresh         && true
Fedora - Modular Rawhide - Developmental packages for the next Fedora release                          147 kB/s | 198 kB     00:01    
Fedora - Rawhide - Developmental packages for the next Fedora release                                  4.0 MB/s |  56 MB     00:13    
Dependencies resolved.
=======================================================================================================================================
 Package                                   Architecture         Version                                    Repository             Size
=======================================================================================================================================
Upgrading:

<snip>

 filesystem                                x86_64               3.11-1.fc31                                rawhide               1.1 M

<snip>

  Upgrading        : filesystem-3.11-1.fc31.x86_64                                                                              59/144 
Error unpacking rpm package filesystem-3.11-1.fc31.x86_64
error: unpacking of archive failed on file /proc: cpio: chown

  Installing       : xkeyboard-config-2.24-5.fc30.noarch                                                                        60/144 
error: filesystem-3.11-1.fc31.x86_64: install failed

<snip>


Failed:
  filesystem-3.11-1.fc31.x86_64                                                                                                        

Error: Transaction failed



Version-Release number of selected component (if applicable):

filesystem-3.11-1.fc31.x86_64


How reproducible:

Very, see above. 


Steps to Reproduce:
1. Write above Dockerfile to disk.
2. Run buildah bud -f ThisBZDockerFile --tag fedora_filesystem:rawhide .
3. Notice that filesystem fails to update.

Actual results:

filesystem fails to update because of an error unpacking file

Expected results:

Fedora rawhide should be able to update.

Additional info:

See attached error.log for a complete text. This is using stock Rawhide

Comment 1 Alex Scheel 2019-05-09 13:09:40 UTC
Created attachment 1566144 [details]
Full output of dnf update on a Fedora rawhide container

Comment 2 Lukas Slebodnik 2019-05-09 18:39:11 UTC
I do not think bug is in the filesystem package.
Build of image passed for me without any problem.

What is version of fedora on host? 

Could you provide output of  following commands from your machine ?

sh# ls -ld /sys/
sh# rpm -V filesystem

Comment 3 Alex Scheel 2019-05-09 18:55:06 UTC
Sure, happy to:



-- Host system #1 -- 

$ lsb_release -a
LSB Version:	:core-4.1-amd64:core-4.1-noarch
Distributor ID:	Fedora
Description:	Fedora release 30 (Thirty)
Release:	30
Codename:	Thirty

$ ls -ld /sys/
dr-xr-xr-x. 13 root root 0 Apr 30 18:26 /sys/

$ rpm -V filesystem
$ rpm -qa | grep ^filesystem
filesystem-3.10-1.fc30.x86_64

$ sudo dnf update --refresh -y
Fedora Modular 30 - x86_64                               2.6 kB/s |  15 kB     00:05    
Fedora Modular 30 - x86_64 - Updates                      31 kB/s |  15 kB     00:00    
Fedora 30 - x86_64 - Updates                              67 kB/s |  17 kB     00:00    
Fedora 30 - x86_64                                        52 kB/s |  15 kB     00:00    
google-chrome                                            7.9 kB/s | 1.3 kB     00:00    
RCM Tools for Fedora 30 (RPMs)                           3.6 kB/s | 3.8 kB     00:01    
RPM Fusion for Fedora 30 - Free - Updates                5.9 kB/s | 3.6 kB     00:00    
RPM Fusion for Fedora 30 - Free                           13 kB/s | 3.2 kB     00:00    
RPM Fusion for Fedora 30 - Nonfree - Updates             4.4 kB/s | 3.7 kB     00:00    
RPM Fusion for Fedora 30 - Nonfree                       5.3 kB/s | 3.2 kB     00:00   
Dependencies resolved.
Nothing to do.
Complete!

$ buildah images
docker.io/library/fedora                                 rawhide              b994d52a3692         Apr 24, 2019 17:21     330 MB

-- Host system #2 --

$ lsb_release -a
LSB Version:	:core-4.1-amd64:core-4.1-noarch
Distributor ID:	Fedora
Description:	Fedora release 29 (Twenty Nine)
Release:	29
Codename:	TwentyNine

$ ls -ld /sys/
dr-xr-xr-x. 13 root root 0 Jan 29 12:00 /sys/

$ rpm -V filesystem
$ rpm -qa | grep ^filesystem
filesystem-3.9-2.fc29.x86_64

$ sudo dnf update --refresh -y
Copr repo for master owned by @pki                        11 kB/s | 3.5 kB     00:00    
ccutil for Fedora 29                                      32 kB/s | 2.9 kB     00:00    
Fedora Modular 29 - x86_64                                49 kB/s |  18 kB     00:00    
Fedora Modular 29 - x86_64 - Updates                      98 kB/s |  15 kB     00:00    
Fedora 29 - x86_64 - Updates                              29 kB/s |  16 kB     00:00    
Fedora 29 - x86_64                                        51 kB/s |  18 kB     00:00    
RCM Tools for Fedora 29 (RPMs)                            42 kB/s | 3.8 kB     00:00    
RPM Fusion for Fedora 29 - Free - Updates                 18 kB/s | 3.6 kB     00:00    
RPM Fusion for Fedora 29 - Free                          4.4 kB/s | 3.2 kB     00:00    
RPM Fusion for Fedora 29 - Nonfree - Updates             8.6 kB/s | 3.7 kB     00:00    
RPM Fusion for Fedora 29 - Nonfree                        29 kB/s | 3.2 kB     00:00    
Dependencies resolved.
Nothing to do.
Complete!

$ buildah images
docker.io/library/fedora                                 rawhide              eff97b05aaa6         Mar 11, 2019 20:20     311 MB

Comment 4 Lukas Slebodnik 2019-05-09 19:35:00 UTC
The issue is in buildah when running in unprivileged mode

Owner and group for /proc is nobody inside container for unprivileged user

I prepared simpler and deterministic reproducer

[test@hp-dl180g6-01 ~]$ cat Dockerfile 
FROM docker.io/fedora@sha256:ac62b7859a2e8f1339c6ec4bc4bbafcd352ffc385d632c9819e6f8942ff91416

# Update the container
RUN true \
        && capsh --print \
        && ls -ld /proc/ \
        && dnf update -y filesystem
[test@hp-dl180g6-01 ~]$ ls -ld /proc/
dr-xr-xr-x. 216 root root 0 May  8 18:07 /proc/
[test@hp-dl180g6-01 ~]$ buildah bud --tag fedora_filesystem:rawhide .
STEP 1: FROM docker.io/fedora@sha256:ac62b7859a2e8f1339c6ec4bc4bbafcd352ffc385d632c9819e6f8942ff91416
STEP 2: RUN true         && capsh --print         && ls -ld /proc/         && dnf update -y filesystem
Current: = cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap+eip
Bounding set =cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap
Ambient set =cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap
Securebits: 00/0x0/1'b0
 secure-noroot: no (unlocked)
 secure-no-suid-fixup: no (unlocked)
 secure-keep-caps: no (unlocked)
 secure-no-ambient-raise: no (unlocked)
uid=0(root)
gid=0(root)
groups=
dr-xr-xr-x. 222 nobody nobody 0 May  9 19:32 /proc/
Fedora - Modular Rawhide - Developmental packages for the next Fedora rel  42 kB/s | 198 kB     00:04    
Fedora - Rawhide - Developmental packages for the next Fedora release     1.1 MB/s |  56 MB     00:50    
Dependencies resolved.
==========================================================================================================
 Package                   Architecture          Version                     Repository              Size
==========================================================================================================
Upgrading:
 filesystem                x86_64                3.11-1.fc31                 rawhide                1.1 M

Transaction Summary
==========================================================================================================
Upgrade  1 Package

Total download size: 1.1 M
Downloading Packages:
filesystem-3.11-1.fc31.x86_64.rpm                                         378 kB/s | 1.1 MB     00:02    
----------------------------------------------------------------------------------------------------------
Total                                                                     356 kB/s | 1.1 MB     00:03     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Running scriptlet: filesystem-3.11-1.fc31.x86_64                                                    1/1 
  Preparing        :                                                                                  1/1 
  Upgrading        : filesystem-3.11-1.fc31.x86_64                                                    1/2 
Error unpacking rpm package filesystem-3.11-1.fc31.x86_64
error: unpacking of archive failed on file /proc: cpio: chown

  Cleanup          : filesystem-3.10-1.fc30.x86_64                                                    2/2 
error: filesystem-3.11-1.fc31.x86_64: install failed

  Running scriptlet: filesystem-3.10-1.fc30.x86_64                                                    2/2 
  Verifying        : filesystem-3.11-1.fc31.x86_64                                                    1/2 
  Verifying        : filesystem-3.10-1.fc30.x86_64                                                    2/2 

Failed:
  filesystem-3.11-1.fc31.x86_64                                                                           

Error: Transaction failed
error building at step {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin DISTTAG=f31container FGC=f31 FBR=f31] Command:run Args:[true         && capsh --print         && ls -ld /proc/         && dnf update -y filesystem] Flags:[] Attrs:map[] Message:RUN true         && capsh --print         && ls -ld /proc/         && dnf update -y filesystem Original:RUN true         && capsh --print         && ls -ld /proc/         && dnf update -y filesystem}: error while running runtime: exit status 1
ERRO[0106] exit status 1

Comment 5 Lukas Slebodnik 2019-05-09 19:37:36 UTC
Workaround is to run "buildah bud" as a root
or to use newer image for rawhide (registry.fedoraproject.org/fedora:rawhide)
The filesystem package is up to date there

Comment 6 Lukas Slebodnik 2019-05-09 19:46:50 UTC
I would guess it is caused by fuse-overlayfs vs overlay for /

Comment 7 Alex Scheel 2019-05-09 19:57:22 UTC
I can confirm that running buildah as root is a workaround for this problem from my end. Thanks!

Comment 8 Fedora Update System 2019-05-10 20:19:45 UTC
buildah-1.8.2-1.gite23314b.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-204dcd7630

Comment 9 Fedora Update System 2019-05-10 20:19:53 UTC
buildah-1.8.2-1.gite23314b.fc30 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-e4d7ff8447

Comment 10 Daniel Walsh 2019-05-10 21:26:59 UTC
Giuseppe, do you think this is an error in fuse-overlay?

Comment 11 Giuseppe Scrivano 2019-05-10 21:58:58 UTC
this is more of a problem in the rpm package that is trying to chown /proc, which is not handled by fuse-overlayfs.  That is not permitted for an unprivileged user.

Comment 12 Fedora Update System 2019-05-11 02:11:08 UTC
buildah-1.8.2-1.gite23314b.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-e4d7ff8447

Comment 13 Fedora Update System 2019-05-11 04:24:13 UTC
buildah-1.8.2-1.gite23314b.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-204dcd7630

Comment 14 Lukas Slebodnik 2019-05-11 17:37:29 UTC
(In reply to Giuseppe Scrivano from comment #11)
> this is more of a problem in the rpm package that is trying to chown /proc,
> which is not handled by fuse-overlayfs.  That is not permitted for an
> unprivileged user.

It works well when running as root so it cannot be problem in rpm.
The problem is that UID/GID is not expected in unprivileged mode
and moreover it si not allowed to fix that in case of upgrade.

Comment 15 Daniel Walsh 2019-05-12 09:53:42 UTC
Lukas what UID/GID is it using?  Is it a huge UID?

Comment 16 Lukas Slebodnik 2019-05-15 11:31:16 UTC
(In reply to Daniel Walsh from comment #15)
> Lukas what UID/GID is it using?  Is it a huge UID?

You can see it in the comment#4.
It's nobody user which is usually high

And here is output from following Dockerfile

FROM docker.io/fedora@sha256:ac62b7859a2e8f1339c6ec4bc4bbafcd352ffc385d632c9819e6f8942ff91416
# Update the container
RUN true \
        && capsh --print \
        && ls -lnd /* \
        && rpm -V filesystem \
        && dnf update -y filesystem



sh$ buildah bud --tag fedora_filesystem:rawhide .
STEP 1: FROM docker.io/fedora@sha256:ac62b7859a2e8f1339c6ec4bc4bbafcd352ffc385d632c9819e6f8942ff91416
STEP 2: RUN true         && capsh --print         && ls -lnd /*         && rpm -V filesystem         && dnf update -y filesystem
Current: = cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap+eip
Bounding set =cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap
Ambient set =cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap
Securebits: 00/0x0/1'b0
 secure-noroot: no (unlocked)
 secure-no-suid-fixup: no (unlocked)
 secure-keep-caps: no (unlocked)
 secure-no-ambient-raise: no (unlocked)
uid=0(root)
gid=0(root)
groups=
lrwxrwxrwx.   1     0     0    7 Feb 11 13:47 /bin -> usr/bin
dr-xr-xr-x.   2     0     0    6 Feb 11 13:47 /boot
drwxr-xr-x.   5     0     0  360 May 15 11:29 /dev
drwxr-xr-x.   3     0     0 4096 Apr 22 11:39 /etc
drwxr-xr-x.   2     0     0    6 Feb 11 13:47 /home
lrwxrwxrwx.   1     0     0    7 Feb 11 13:47 /lib -> usr/lib
lrwxrwxrwx.   1     0     0    9 Feb 11 13:47 /lib64 -> usr/lib64
drwx------.   2     0     0    6 Apr 22 11:38 /lost+found
drwxr-xr-x.   2     0     0    6 Feb 11 13:47 /media
drwxr-xr-x.   2     0     0    6 Feb 11 13:47 /mnt
drwxr-xr-x.   2     0     0    6 Feb 11 13:47 /opt
dr-xr-xr-x. 171 65534 65534    0 May 15 11:29 /proc
dr-xr-x---.   2     0     0  162 Apr 22 11:39 /root
drwxr-xr-x.   2     0     0  174 Apr 22 11:39 /run
lrwxrwxrwx.   1     0     0    8 Feb 11 13:47 /sbin -> usr/sbin
drwxr-xr-x.   2     0     0    6 Feb 11 13:47 /srv
dr-xr-xr-x.  13 65534 65534    0 May 15 01:33 /sys
drwxrwxrwt.   2     0     0   32 Apr 22 11:39 /tmp
drwxr-xr-x.   7     0     0  144 Apr 22 11:39 /usr
drwxr-xr-x.   3     0     0  235 Apr 22 11:39 /var
.M.......    /
.....UG..    /proc
.....UG..    /sys
error building at step {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin DISTTAG=f31container FGC=f31 FBR=f31] Command:run Args:[true         && capsh --print         && ls -lnd /*         && rpm -V filesystem         && dnf update -y filesystem] Flags:[] Attrs:map[] Message:RUN true         && capsh --print         && ls -lnd /*         && rpm -V filesystem         && dnf update -y filesystem Original:RUN true         && capsh --print         && ls -lnd /*         && rpm -V filesystem         && dnf update -y filesystem}: error while running runtime: exit status 1
ERRO[0000] exit status 1

So it is not just about /proc but also for /sys

Comment 17 Daniel Walsh 2019-05-15 19:41:33 UTC
I take it that this works if you run buildah bud as root but not in rootless mode.

Comment 18 Daniel Walsh 2019-05-15 19:46:52 UTC
What is attempting to install the filesystem package?  In the default rawhide container, there is no file system package, and it looks like something is attempting to pull it in?

Comment 19 Alex Scheel 2019-05-15 21:53:51 UTC
Dan,


I'm not sure I follow:

$ podman run -ti registry.fedoraproject.org/fedora:rawhide /bin/bash -c 'rpm -qa | grep filesystem'
Trying to pull registry.fedoraproject.org/fedora:rawhide...Getting image source signatures
Copying blob 05af05297975 done
Copying config fc02a8623a done
Writing manifest to image destination
Storing signatures
libreport-filesystem-2.10.0-3.fc31.noarch
filesystem-3.11-1.fc31.x86_64

$ podman run -ti registry.hub.docker.com/library/fedora:rawhide /bin/bash -c 'rpm -qa | grep filesystem'
Trying to pull registry.hub.docker.com/library/fedora:rawhide...Getting image source signatures
Copying blob 5036ff40a678 skipped: already exists
Copying config b994d52a36 done
Writing manifest to image destination
Storing signatures
libreport-filesystem-2.10.0-1.fc30.noarch
filesystem-3.10-1.fc30.x86_64


This isn't "attempting to install" -- this is a core package and I doubt Fedora could function without it. 

Description  : The filesystem package is one of the basic packages that is installed
             : on a Linux system. Filesystem contains the basic directory layout
             : for a Linux operating system, including the correct permissions for
             : the directories.

This includes a bunch of default filesystem directories such as everything in / (/root, /etc, ...)

(now, skipping the update for this package is probably fine, but I'm more annoyed with it breaking dnf update as a result) 




And yes:
$ podman run -ti registry.hub.docker.com/library/fedora:rawhide dnf update --refresh -y
^ fails when run as a regular user

But:
$ sudo podman run -ti registry.hub.docker.com/library/fedora:rawhide dnf update --refresh -y
^ works when run as root via sudo.

And:
$ sudo docker run -ti registry.hub.docker.com/library/fedora:rawhide dnf update --refresh -y
^ works as well.

I'm too lazy to set up docker as not-root, I'll leave that as an exercise for the reader.




The lazy solution would be to poke $SOMEBODY to update the Dockerhub image, which I'm perfectly fine with. Or have podman/buildah detect that you're trying to load an unqualified "fedora:<version>" and pull that from Fedora's registry instead (which'd require more discussion probably).

Comment 20 Giuseppe Scrivano 2019-05-16 06:26:19 UTC
/proc is owned by root in the initial namespace, so a chown on /proc in a rootless container will never succeed when it is mounted.

You can workaround it by mounting something else on /proc (hopefully yum/rpm won't need it), but the correct fix would be to not attempt to chown it.  Can cpio chown errors be ignored?

Comment 21 Lukas Slebodnik 2019-05-16 07:33:19 UTC
(In reply to Giuseppe Scrivano from comment #20)
> /proc is owned by root in the initial namespace, so a chown on /proc in a
> rootless container will never succeed when it is mounted.
> 
> You can workaround it by mounting something else on /proc (hopefully yum/rpm
> won't need it), but the correct fix would be to not attempt to chown it. 
> Can cpio chown errors be ignored?

You cannot ignore chown errors in rpm. You want to fix owner/permissions in chase of spec file change. The problem is that /proc and /sys use nobody user and not root.
But that probably cannot be changed for rootless mode.

Comment 22 Daniel Walsh 2019-05-16 14:24:30 UTC
Well the /proc and /sys are actually owned by real root, but in side of the user namespace, real root is not mapped, so files owned by UID=0 are shown as NOBODY.

The rawhide image comes without a filesytem package inside of it, so these tools are attempting to install it.

sudo podman run fedora:rawhide rpm -q filesystem
[sudo] password for dwalsh: 
filesystem-3.10-1.fc30.x86_64

The problem is the rawhide fedora container image is screwed up right now.

sudo podman run fedora:rawhide rpm -qf /etc/fedora-release /proc
fedora-release-common-31-0.5.noarch
filesystem-3.10-1.fc30.x86_64

The release is 31, but the filesystem package is 30, which is triggering the filesystem package to require an update.

Probably most releases this package never gets updated, so no one ever sees this issue.

Once fedora:rawhide actually has packages built for rawhide, the issue should go away.

Comment 23 Daniel Walsh 2019-05-16 14:26:56 UTC
Who is in charge of creating the base images?  How do I reassign this to tell Fedora to fix the base image.

Comment 24 Lukas Slebodnik 2019-05-16 15:17:15 UTC
(In reply to Daniel Walsh from comment #23)
> Who is in charge of creating the base images?  How do I reassign this to
> tell Fedora to fix the base image.

That is just workaround and the same situation will happen next time in case of new rawhide koji build for filesystem package.

Comment 25 Lukas Slebodnik 2019-05-16 15:19:19 UTC
Clement,
could you update rawhide image in docker hub ?

Comment 26 Lukas Slebodnik 2019-05-16 15:22:28 UTC
(In reply to Daniel Walsh from comment #22)
> Well the /proc and /sys are actually owned by real root, but in side of the
> user namespace, real root is not mapped, so files owned by UID=0 are shown
> as NOBODY.
> 
> The rawhide image comes without a filesytem package inside of it, so these
> tools are attempting to install it.
> 

The rawhide image comes *WITH* the filesystem package.

sh# docker run -ti --rm docker.io/fedora:rawhide rpm -q filesystem
filesystem-3.10-1.fc30.x86_64

And the filesystem pacakge is pulled in as a dependency of bash/gawk

sh# docker run -ti --rm docker.io/fedora:rawhide rpm -e filesystem
error: Failed dependencies:
        filesystem >= 3 is needed by (installed) bash-5.0.2-1.fc30.x86_64
        filesystem >= 3 is needed by (installed) gawk-4.2.1-6.fc31.x86_64

Comment 27 Daniel Walsh 2019-05-16 16:48:53 UTC
Perhaps a workaround, but since this can not be fixed, and is fairly unusual, it is the best we can do.

If you need to update a package that attempts to modify files owned by root within a usernamespaced container
it will fail.  Luckily the only exposed files by default are /proc and /sys.

Comment 28 Daniel Walsh 2019-05-16 16:50:30 UTC
Filesystem package within the rawhide container image should be from f31 not f30.  All packages in that image should be checked to make sure they are update.

Comment 29 Alex Scheel 2019-05-16 17:40:02 UTC
This is just a normal rawhide process.

Until the F31 mass rebuild (2019-07-24 according to the schedule [0]), only things which have been explicitly updated (such as filesystem) will have fc31. Everything else which is the same since F30 branched (and which haven't been updated in rawhide) will have the fc30 tag.



The bug is due to the mapping of root->NOBODY. Shouldn't root be mapped to "root" inside the container, with changes not being propagated to outside the container? I'm sure you'll tell me this isn't possible due to using bind mounting or something, but that satisfies the principle of least surprise: I have root==uid=0,gid=0 inside the container, so I should be able to chown darn near everything inside the container.


At the very least, even if chown doesn't work, the owner of /proc and /sys *inside* the container should appear as being root. But I don't know what's required to get there or what else that'd break.


My 2c.


[0]: https://fedoraproject.org/wiki/Releases/31/Schedule

Comment 30 Daniel Walsh 2019-05-16 17:46:36 UTC
Alex, I understand, I have worked on Fedora since the fedora core 1.
But the filesystem package is available now, and has not been built in the rawhide image yet.

I am not arguing that this package should be allowed to be updated and is working correctly.

I am explaining that running container runtime tools like podman and buildah as non root 
will not work in all circumstances.  We have a long list of them.  

The one we are hitting here is a root file inside of a user namespace without UID=0 being mounted
inside of the container.

When you run rootless containers usually your UID is root inside of the container or the first UID listed
in /etc/subuid.  But UID=0 on the host is not mapped into the container, and even it it was you would not 
be allowed to modify the root owned files like the /proc and /sys directories.

User Namespace says that any UID on the file systme that is not mapped into th user namespace is treated as the 
NOBODY user and is only accessible via WORLD permissions.

Comment 31 Daniel Walsh 2019-05-16 17:47:35 UTC
Bottom line we can not fix this.
If you need to install/update the filesystem package you have to run podman/buildah as root.

Comment 32 Fedora Update System 2019-05-17 01:04:27 UTC
buildah-1.8.2-1.gite23314b.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.

Comment 33 Fedora Update System 2019-05-19 10:27:09 UTC
buildah-1.8.2-1.gite23314b.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.

Comment 34 Petr Pisar 2020-08-06 15:32:18 UTC
*** Bug 1548403 has been marked as a duplicate of this bug. ***

Comment 35 Pavel Raiskup 2020-08-06 15:52:02 UTC
(In reply to Daniel Walsh from comment #31)
> Bottom line we can not fix this.
> If you need to install/update the filesystem package you have to run
> podman/buildah as root.

Is this still valid statement?  If not, do we have more info?

Comment 36 Daniel Walsh 2020-08-06 21:23:05 UTC
Pavel it is always going to be a valid statement.  A non root process running as your user is not going to be able to chown a file system /proc and /sys owned by root.

There is always going to be some things we can not do in rootless mode, since we are under the constraints of not being root.

here is a whole page of things we can not do in rootless mode.

https://github.com/containers/podman/blob/master/rootless.md

Now maybe in the future the kernel will support a rootless user mounting the /proc and /sys file system in their user namespace, and this will be owned by root of the user namespace, but I would not bet on this happening soon.

Comment 37 Daniel Walsh 2020-08-06 21:24:41 UTC
At least the initial problem has been fixed.

$ podman run fedora:rawhide rpm -q filesystem
filesystem-3.14-2.fc32.x86_64

Comment 38 Pavel Raiskup 2020-08-07 06:35:24 UTC
Thanks for the explanation!  I was just confused by the "CANTFIX => ERRATA"
status change.  Mostly triggered by buildah bodhi update.

> $ podman run fedora:rawhide rpm -q filesystem
> filesystem-3.14-2.fc32.x86_64

Yeah, I think the current devel discussion is happening because that one is
already obsolete, there should be fc33 package in Rawhide.


Note You need to log in before you can comment on or make changes to this bug.