1887445 – dnf fails to run due to python time error in a 32 bit container

Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1887445 - dnf fails to run due to python time error in a 32 bit container

Summary: dnf fails to run due to python time error in a 32 bit container

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	None
Product:	Fedora
Classification:	Fedora
Component:	buildah
Sub Component:
Version:	33
Hardware:	armv7hl
OS:	Linux
Priority:	unspecified
Severity:	urgent
Target Milestone:	---
Assignee:	Lokesh Mandvekar
QA Contact:	Fedora Extras Quality Assurance
Docs Contact:
URL:
Whiteboard:
Duplicates (1):	1888681 (view as bug list)
Depends On:
Blocks:	ARMTracker
TreeView+	depends on / blocked

Reported:	2020-10-12 13:49 UTC by Dennis Gilmore
Modified:	2021-06-11 15:49 UTC (History)
CC List:	26 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2021-06-11 15:49:05 UTC
Type:	Bug
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Description Dennis Gilmore 2020-10-12 13:49:25 UTC

Description of problem:

trying to build a container on 32 bit arm dnf fails to run to to a fatal python error

[dennis@korolev coursera-dl]$ buildah bud Dockerfile.f33 tag coursera-dl
STEP 1: FROM fedora:33
STEP 2: LABEL maintainer "https://github.com/coursera-dl/"
STEP 3: RUN dnf -y update &&     dnf install gcc gcc-c++ openssl-devel python3-devel &&     dnf clean all
Fatal Python error: init_interp_main: can't initialize time
Python runtime state: core initialized
PermissionError: [Errno 1] Operation not permitted

Current thread 0xb6ff0010 (most recent call first):
<no Python frame>
error building at STEP "RUN dnf -y update &&     dnf -y install gcc gcc-c++ openssl-devel python3-devel &&     dnf clean all": error while running runtime: exit status 1
ERRO exit status 1

Version-Release number of selected component (if applicable):
STEP 3: RUN rpm -q python3 dnf 
python3-3.9.0-1.fc33.armv7hl
dnf-4.2.23-2.fc33.noarch

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Victor Stinner 2020-10-12 14:10:20 UTC

> Fatal Python error: init_interp_main: can't initialize time
> Python runtime state: core initialized
> PermissionError: [Errno 1] Operation not permitted

This error comes from the private Python function _PyTime_Init(). This function checks that 3 clocks are working:

* system clock: clock_gettime(CLOCK_REALTIME) and clock_getres(CLOCK_REALTIME)
* monotonic clock: clock_gettime(CLOCK_MONOTONIC) and clock_getres(clk_id)
* performance counter clock: same than the system clock

A clock_gettime() or clock_getres() function call failed with EPERM (errno 1).

IMO buildah blocked the access to one of these syscalls. I suggest you to investigate how buildah builds a sandbox to build the package, and allow these syscalls there.

Comment 2 Victor Stinner 2020-10-12 14:49:38 UTC

I reassign the issue to the buildah component. Python just wants to read two clocks (CLOCK_REALTIME, CLOCK_MONOTONIC) and their resolution. I don't see why such safe option would be denied by a sandbox by default. I guess that clock_getres() was just forgotten in the default sandbox policy.

Comment 3 Miro Hrončok 2020-10-15 17:10:53 UTC

*** Bug 1888681 has been marked as a duplicate of this bug. ***

Comment 4 Daniel Walsh 2020-10-20 13:43:49 UTC

Could you try this as root?  To see if this is User Namespace causing it.

If it works as root, we know we have a user namespace issue, if not, could you try in permissive mode?

If that works, then SELinux.

After that we get to Capabilities and seccomp.

BTW make sure you are fully up2date on containers-common package, which has seccomp and default capabilities fixes.

Comment 5 Dennis Gilmore 2020-10-20 14:37:17 UTC

[root@korolev coursera-dl]# buildah bud  --tag courseradl/courseradl --build-arg VERSION=0.11.5 Dockerfile.f33
STEP 1: FROM fedora:33
Getting image source signatures
Copying blob 6a60f7c2ff89 done  
Copying config 43578cc1e0 done  
Writing manifest to image destination
Storing signatures
STEP 2: LABEL maintainer "https://github.com/coursera-dl/"
STEP 3: RUN dnf -y update &&     dnf install gcc gcc-c++ openssl-devel python3-devel &&     dnf clean all
Fatal Python error: init_interp_main: can't initialize time
Python runtime state: core initialized
PermissionError: [Errno 1] Operation not permitted

Current thread 0xb6fa0010 (most recent call first):
<no Python frame>
error building at STEP "RUN dnf -y update &&     dnf install gcc gcc-c++ openssl-devel python3-devel &&     dnf clean all": error while running runtime: exit status 1
[root@korolev coursera-dl]# setenforce 0
[root@korolev coursera-dl]# buildah bud  --tag courseradl/courseradl --build-arg VERSION=0.11.5 Dockerfile.f33
STEP 1: FROM fedora:33
STEP 2: LABEL maintainer "https://github.com/coursera-dl/"
STEP 3: RUN dnf -y update &&     dnf install gcc gcc-c++ openssl-devel python3-devel &&     dnf clean all
Fatal Python error: init_interp_main: can't initialize time
Python runtime state: core initialized
PermissionError: [Errno 1] Operation not permitted

Current thread 0xb6f7d010 (most recent call first):
<no Python frame>
error building at STEP "RUN dnf -y update &&     dnf install gcc gcc-c++ openssl-devel python3-devel &&     dnf clean all": error while running runtime: exit status 1

Comment 6 Dennis Gilmore 2020-10-20 14:39:54 UTC

containers-common-1.2.0-3.fc33.armv7hl

Comment 7 Daniel Walsh 2020-10-20 15:34:25 UTC

Could you see if this is a seccomp issue.


buildah bud --security-opt seccomp=unconfined /tmp/

You should also look in /var/log/audit/audit.log for SECCOMP failures

ausearch -m seccomp

Comment 8 Dennis Gilmore 2020-10-20 16:43:00 UTC

[root@korolev coursera-dl]# buildah bud --tag coursera-dl --build-arg VERSION=0.11.5 --security-opt seccomp=unconfined Dockerfile.f33
STEP 1: FROM fedora:33
STEP 2: LABEL maintainer "https://github.com/coursera-dl/"
STEP 3: RUN dnf -y update &&     dnf install gcc gcc-c++ openssl-devel python3-devel &&     dnf clean all
Fedora 33 openh264 (From Cisco) - armhfp                                                                                                                                                                      3.1 kB/s | 2.6 kB     00:00    
Fedora Modular 33 - armhfp                                                                                                                                                                                    2.8 MB/s | 3.2 MB     00:01    
Fedora Modular 33 - armhfp - Updates                                                                                                                                                                          633  B/s | 257  B     00:00    
Fedora 33 - armhfp - Updates                                                                                                                                                                                  379  B/s | 257  B     00:00


lots of instances in /var/log/audit/audit.log like the following:
type=SECCOMP msg=audit(1602768936.153:3304): auid=217600001 uid=217600001 gid=217600001 ses=6 subj=system_u:system_r:container_t:s0:c38,c641 pid=11414 comm="sh" exe="/usr/bin/bash" sig=0 arch=40000028 syscall=403 compat=0 ip=0xb6e175bc code=0x50000AUID="dennis" UID="dennis" GID="dennis" ARCH=armeb SYSCALL=unknown-syscall(403)
type=SECCOMP msg=audit(1602768936.156:3305): auid=217600001 uid=217600001 gid=217600001 ses=6 subj=system_u:system_r:container_t:s0:c38,c641 pid=11414 comm="sh" exe="/usr/bin/bash" sig=0 arch=40000028 syscall=403 compat=0 ip=0xb6e175bc code=0x50000AUID="dennis" UID="dennis" GID="dennis" ARCH=armeb SYSCALL=unknown-syscall(403)
type=SECCOMP msg=audit(1602768936.208:3306): auid=217600001 uid=217600001 gid=217600001 ses=6 subj=system_u:system_r:container_t:s0:c38,c641 pid=11420 comm="dnf" exe="/usr/bin/python3.9" sig=0 arch=40000028 syscall=403 compat=0 ip=0xb6bb15bc code=0x50000AUID="dennis" UID="dennis" GID="dennis" ARCH=armeb SYSCALL=unknown-syscall(403)
type=SECCOMP msg=audit(1603204489.929:5145): auid=0 uid=0 gid=0 ses=7 subj=system_u:system_r:container_t:s0:c187,c958 pid=1391 comm="sh" exe="/usr/bin/bash" sig=0 arch=40000028 syscall=403 compat=0 ip=0xb6e3e5bc code=0x50000AUID="root" UID="root" GID="root" ARCH=armeb SYSCALL=unknown-syscall(403)
type=SECCOMP msg=audit(1603204489.948:5146): auid=0 uid=0 gid=0 ses=7 subj=system_u:system_r:container_t:s0:c187,c958 pid=1394 comm="dnf" exe="/usr/bin/python3.9" sig=0 arch=40000028 syscall=403 compat=0 ip=0xb6bf45bc code=0x50000AUID="root" UID="root" GID="root" ARCH=armeb SYSCALL=unknown-syscall(403)

Comment 9 Daniel Walsh 2020-10-21 20:07:43 UTC

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=951012
talks about a similar issue.

=======================================================================
I looked into this, this is is new time64 syscall, I'll be going ahead
and whitelist all new time64 syscalls in 1.9.10.

403: clock_gettime64
404: clock_settime64
405: clock_adjtime64
406: clock_getres_time64
407: clock_nanosleep_time64
408: timer_gettime64
409: timer_settime64
410: timerfd_gettime64
411: timerfd_settime64
412: utimensat_time64
413: pselect6_time64
414: ppoll_time64

=========================================================================

Comment 10 Daniel Walsh 2020-10-21 20:29:19 UTC

https://github.com/containers/common/pull/331

Comment 11 Paul Whalen 2020-11-04 20:59:36 UTC

This looks like it should be fixed in containers-common-1.2.0-6.fc33.armv7hl, but still hitting the error

...
Python runtime state: core initialized
PermissionError: [Errno 1] Operation not permitted

Current thread 0xb6f93010 (most recent call first):
<no Python frame>
Error: error building at STEP "RUN /usr/bin/dnf install -y httpd": error while running runtime: exit status 1


[root@wand ~]# rpm -q containers-common
containers-common-1.2.0-6.fc33.armv7hl

----
time->Wed Nov  4 15:53:13 2020
type=SECCOMP msg=audit(1604523193.829:547): auid=0 uid=0 gid=0 ses=1 subj=system_u:system_r:container_t:s0:c617,c656 pid=1527 comm="sh" exe="/usr/bin/bash" sig=0 arch=40000028 syscall=403 compat=0 ip=0xb6dce5bc code=0x50000
----
time->Wed Nov  4 15:53:13 2020
type=SECCOMP msg=audit(1604523193.835:548): auid=0 uid=0 gid=0 ses=1 subj=system_u:system_r:container_t:s0:c617,c656 pid=1527 comm="sh" exe="/usr/bin/bash" sig=0 arch=40000028 syscall=403 compat=0 ip=0xb6dce5bc code=0x50000
----
time->Wed Nov  4 15:53:13 2020
type=SECCOMP msg=audit(1604523193.837:549): auid=0 uid=0 gid=0 ses=1 subj=system_u:system_r:container_t:s0:c617,c656 pid=1527 comm="sh" exe="/usr/bin/bash" sig=0 arch=40000028 syscall=403 compat=0 ip=0xb6dce5bc code=0x50000
----
time->Wed Nov  4 15:53:13 2020
type=SECCOMP msg=audit(1604523193.868:550): auid=0 uid=0 gid=0 ses=1 subj=system_u:system_r:container_t:s0:c617,c656 pid=1527 comm="dnf" exe="/usr/bin/python3.9" sig=0 arch=40000028 syscall=403 compat=0 ip=0xb6bec5bc code=0x50000

Comment 12 Daniel Walsh 2020-11-05 19:58:32 UTC

Do you see clock_gettime64 defined in /usr/share/containers/seccomp.json?  Do you have an /etc/containers/seccomp.json?

Comment 13 Miguel Angel Ajo 2020-11-06 20:58:37 UTC

I can reproduce this also with docker on an ARMv7

docker container run -ti --rm fedora:33 python3
Fatal Python error: init_interp_main: can't initialize time
Python runtime state: core initialized
PermissionError: [Errno 1] Operation not permitted

Current thread 0xb6f11010 (most recent call first):
<no Python frame>

Comment 14 Daniel Walsh 2020-11-07 11:45:47 UTC

Miquel are you seeing this issue with Podman?  If yes could you verify if the field is in the seccomp.json file?
https://bugzilla.redhat.com/show_bug.cgi?id=1887445#c12

Comment 15 Miguel Angel Ajo 2020-11-11 17:43:20 UTC

root@rpi4:~#  docker run --rm -ti fedora:33 python3
Fatal Python error: init_interp_main: can't initialize time
Python runtime state: core initialized
PermissionError: [Errno 1] Operation not permitted

Current thread 0xb6f69010 (most recent call first):
<no Python frame>


root@rpi4:~#  podman run --rm -ti fedora:33 python3
Fatal Python error: init_interp_main: can't initialize time
Python runtime state: core initialized
PermissionError: [Errno 1] Operation not permitted

Current thread 0xb6f52010 (most recent call first):
<no Python frame>


Daniel, I don't have clock_gettime64 in /usr/share/containers/seccomp.json



But shouldn't the fedora image work also in the default docker environment? :?

Comment 16 Miguel Angel Ajo 2020-11-11 17:45:08 UTC

After adding clock_gettime64 in /usr/share/containers/seccomp.json, it's still the same.


root@rpi4:~# grep gettime /usr/share/containers/seccomp.json
				"clock_gettime",
                                "clock_gettime64",
				"gettimeofday",
				"timer_gettime",
				"timerfd_gettime",


root@rpi4:~#  podman run --rm -ti fedora:33 python3
Fatal Python error: init_interp_main: can't initialize time
Python runtime state: core initialized
PermissionError: [Errno 1] Operation not permitted

Current thread 0xb6f44010 (most recent call first):
<no Python frame>

Comment 17 Daniel Walsh 2020-11-11 22:33:06 UTC

Any messages in /var/log/audit/audit.log?

Comment 18 Daniel Walsh 2020-11-11 22:35:02 UTC

Are you running this on a 32 bit platform?

Want to make sure

 podman run --rm -ti --security-opt seccomp=unconfined fedora:33 python3

Works.

Comment 19 Huang, Chih-Wei 2020-11-13 02:33:00 UTC

I also have this issue. I'm using a Rockchip rk3288 board (32-bit arm system) and running Debian 9 with docker 19.03.13.

/# docker run --rm -ti fedora:33 python3
Fatal Python error: init_interp_main: can't initialize time
Python runtime state: core initialized
PermissionError: [Errno 1] Operation not permitted

Current thread 0xb6f65010 (most recent call first):
<no Python frame>

Exactly the same error as #15.

Comment 20 Huang, Chih-Wei 2020-11-13 02:50:31 UTC

OK. I found the answer:
https://askubuntu.com/questions/1263284/apt-update-throws-signature-error-in-ubuntu-20-04-container-on-arm

It's an issue in libseccomp2 of Debian 9. Update it as the above link suggested:

/# wget http://ftp.tw.debian.org/debian/pool/main/libs/libseccomp/libseccomp2_2.5.0-3_armhf.deb
/# dpkg -i libseccomp2_2.5.0-3_armhf.deb

Then python3 in fedora docker works fine:

/# docker run --rm -ti fedora python3                                                                                               
Python 3.9.0 (default, Oct  6 2020, 00:00:00) 
[GCC 10.2.1 20200826 (Red Hat 10.2.1-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 

And I can run 'dnf update' in fedora docker now.

Comment 21 Dennis Gilmore 2020-11-17 16:50:23 UTC

It seems to work here

[root@korolev ~]# yum install https://kojipkgs.fedoraproject.org//packages/skopeo/1.2.0/6.fc33/armv7hl/containers-common-1.2.0-6.fc33.armv7hl.rpm https://kojipkgs.fedoraproject.org//packages/skopeo/1.2.0/6.fc33/armv7hl/skopeo-1.2.0-6.fc33.armv7hl.rpm
Last metadata expiration check: 2:24:10 ago on Tue 17 Nov 2020 02:23:04 PM UTC.
containers-common-1.2.0-6.fc33.armv7hl.rpm                                                                                                                                                                  304 kB/s |  52 kB     00:00    
skopeo-1.2.0-6.fc33.armv7hl.rpm                                                                                                                                                                             9.3 MB/s | 5.4 MB     00:00    
Dependencies resolved.
============================================================================================================================================================================================================================================
 Package                                                      Architecture                                       Version                                                      Repository                                               Size
============================================================================================================================================================================================================================================
Installing:
 skopeo                                                       armv7hl                                            1:1.2.0-6.fc33                                               @commandline                                            5.4 M
Upgrading:
 containers-common                                            armv7hl                                            1:1.2.0-6.fc33                                               @commandline                                             52 k

Transaction Summary
============================================================================================================================================================================================================================================
Install  1 Package
Upgrade  1 Package

Total size: 5.4 M
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                    1/1 
  Upgrading        : containers-common-1:1.2.0-6.fc33.armv7hl                                                                                                                                                                           1/3 
  Installing       : skopeo-1:1.2.0-6.fc33.armv7hl                                                                                                                                                                                      2/3 
  Cleanup          : containers-common-1:1.2.0-3.fc33.armv7hl                                                                                                                                                                           3/3 
  Running scriptlet: containers-common-1:1.2.0-3.fc33.armv7hl                                                                                                                                                                           3/3 
  Verifying        : skopeo-1:1.2.0-6.fc33.armv7hl                                                                                                                                                                                      1/3 
  Verifying        : containers-common-1:1.2.0-6.fc33.armv7hl                                                                                                                                                                           2/3 
  Verifying        : containers-common-1:1.2.0-3.fc33.armv7hl                                                                                                                                                                           3/3 

Upgraded:
  containers-common-1:1.2.0-6.fc33.armv7hl                                                                                                                                                                                                  

Installed:
  skopeo-1:1.2.0-6.fc33.armv7hl                                                                                                                                                                                                             

Complete!
[root@korolev ~]# podman run --rm -it fedora:33 /usr/bin/bash
[root@fa3fa2fa5c02 /]# dnf update
Fedora 33 openh264 (From Cisco) - armhfp                                                                                                                                                                    4.2 kB/s | 2.6 kB     00:00    
Fedora Modular 33 - armhfp                                                                                                                                                                                  2.2 MB/s | 3.2 MB     00:01    
Fedora Modular 33 - armhfp - Updates                                                                                                                                                                        1.6 MB/s | 2.3 MB     00:01    
Fedora 33 - armhfp - Updates                                                                                                                                                                                5.4 MB/s |  14 MB     00:02

Comment 22 Dennis Gilmore 2020-11-17 16:57:30 UTC

grep clock_gettime64  /usr/share/containers/seccomp.json /etc/containers/seccomp.json
/usr/share/containers/seccomp.json:				"clock_gettime64",
grep: /etc/containers/seccomp.json: No such file or directory

Comment 23 Dennis Gilmore 2020-11-17 22:18:12 UTC

it seems that buildah is working fine, however, there is still some time issues in podman. following the test case https://fedoraproject.org/wiki/QA:Testcase_Podman#Create_a_container_from_Dockerfile I have a failure to install the packages 

[dennis@korolev test]$ podman build -t fedora-httpd $(pwd)
STEP 1: FROM registry.fedoraproject.org/fedora:latest
Getting image source signatures
Copying blob f978f3f3fcf3 [--------------------------------------] 0.0b / 0.0b
Copying config e5e1502f65 done  
Writing manifest to image destination
Storing signatures
STEP 2: RUN /usr/bin/dnf install -y httpd
Fatal Python error: init_interp_main: can't initialize time
Python runtime state: core initialized
PermissionError: [Errno 1] Operation not permitted

Current thread 0xb6fb7010 (most recent call first):
<no Python frame>
Error: error building at STEP "RUN /usr/bin/dnf install -y httpd": error while running runtime: exit status 1

however, running buildah works

[dennis@korolev test]$ buildah bud -t fedora-httpd
STEP 1: FROM registry.fedoraproject.org/fedora:latest
STEP 2: RUN /usr/bin/dnf install -y httpd
Fedora 33 openh264 (From Cisco) - armhfp                                                                                                                                                                    3.2 kB/s | 2.6 kB     00:00    
Fedora Modular 33 - armhfp                                                                                                                                                                                  2.3 MB/s | 3.2 MB     00:01    
Fedora Modular 33 - armhfp - Updates                                                                                                                                                                        782 kB/s | 2.3 MB     00:02    
Fedora 33 - armhfp - Updates                                                                                                                                                                                1.2 MB/s |  14 MB     00:12
<snip>

containers-common-1.2.0-6.fc33.armv7hl
podman-2.1.1-12.fc33.armv7hl
buildah-1.16.2-1.fc33.armv7hl

Comment 24 Dennis Gilmore 2020-11-17 22:20:45 UTC

----
time->Tue Nov 17 19:27:10 2020
type=SECCOMP msg=audit(1605641230.869:2318): auid=217600001 uid=0 gid=0 ses=7 subj=system_u:system_r:container_t:s0:c242,c634 pid=9618 comm="sh" exe="/usr/bin/bash" sig=0 arch=40000028 syscall=403 compat=0 ip=0xb6e005bc code=0x50000
----
time->Tue Nov 17 19:27:10 2020
type=SECCOMP msg=audit(1605641230.882:2319): auid=217600001 uid=0 gid=0 ses=7 subj=system_u:system_r:container_t:s0:c242,c634 pid=9618 comm="sh" exe="/usr/bin/bash" sig=0 arch=40000028 syscall=403 compat=0 ip=0xb6e005bc code=0x50000
----
time->Tue Nov 17 19:27:10 2020
type=SECCOMP msg=audit(1605641230.884:2320): auid=217600001 uid=0 gid=0 ses=7 subj=system_u:system_r:container_t:s0:c242,c634 pid=9618 comm="sh" exe="/usr/bin/bash" sig=0 arch=40000028 syscall=403 compat=0 ip=0xb6e005bc code=0x50000
----
time->Tue Nov 17 19:27:11 2020
type=SECCOMP msg=audit(1605641231.152:2321): auid=217600001 uid=0 gid=0 ses=7 subj=system_u:system_r:container_t:s0:c242,c634 pid=9618 comm="yum" exe="/usr/bin/python3.9" sig=0 arch=40000028 syscall=403 compat=0 ip=0xb6b895bc code=0x50000

Comment 25 Daniel Walsh 2020-11-18 11:32:36 UTC

I believe this is clock_gettime64.

Comment 26 Daniel Walsh 2021-01-28 14:47:28 UTC

Dennis, do you see clock_gettim64 within /usr/share/containers/seccomp.json?

Comment 27 Daniel Walsh 2021-06-11 15:49:05 UTC

FIxed in current release.

Note You need to log in before you can comment on or make changes to this bug.

amurdaca
cstratak
cwhuang
debarshir
dmalcolm
dwalsh
gscrivan
jnovy
lbalhar
lsm5
majopela
m.cyprian
mhroncok
mike
nalin
pbrobinson
pviktori
pwhalen
rh.container.bot
rkuska
santiago
shcherbina.iryna
slavek.kabrda
thrnciar
tomspur
torsava