Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1878924 - mock fails in _copy_config with FileNotFoundError for /etc/resolv.conf in some circumstances
Summary: mock fails in _copy_config with FileNotFoundError for /etc/resolv.conf in som...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: mock
Version: 33
Hardware: All
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Pavel Raiskup
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: openqa
Depends On:
Blocks: F34BetaBlocker 1834846
TreeView+ depends on / blocked
 
Reported: 2020-09-14 22:41 UTC by Adam Williamson
Modified: 2020-10-01 01:41 UTC (History)
11 users (show)

Fixed In Version: mock-2.6-1.fc32 mock-2.6-1.el7 mock-2.6-1.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-09-19 22:39:10 UTC
Type: Bug
Embargoed:

Attachments (Terms of Use)

Description Adam Williamson 2020-09-14 22:41:30 UTC 
There's an openQA test run on updates which does the following:

* Set up a side repo with the update in it
* Update the SUT from that repo, reboot
* Create mock config which also uses that repo, and a live kickstart which also uses that repo
* Install some packages into the mock chroot and then copy the kickstart into it (not using --old-chroot or --enable-networking)
* Try to create a live image in the mock chroot, this time using --old-chroot and --enable-networking, using that mock config and live kickstart

this process is designed to replicate as close as possible what Koji does when creating official live images.

When run on https://bodhi.fedoraproject.org/updates/FEDORA-2020-8c3c5881b2 - systemd-246.4-2.fc33 - that test fails at the point where we try to run the livemedia-creator command in the mock chroot, with this error:

...
  File "/usr/lib/python3.9/site-packages/mockbuild/buildroot.py", line 286, in _copy_config
    shutil.copy2(orig_conf_file, etcdir)
  File "/usr/lib64/python3.9/shutil.py", line 432, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib64/python3.9/shutil.py", line 261, in copyfile
    with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/mock/fedora-33-x86_64/root/etc/resolv.conf'

The same test run on earlier and later updates does not fail, suggesting it is this update specifically that causes the problem. The change in the update is this:

https://src.fedoraproject.org/rpms/systemd/c/b98737cec484f1c8dcb839e97192809b99f77c89?branch=f33

AFAIK, that's making systemd %post try harder (on initial install, not on update) to create /etc/resolv.conf as a symlink to /run/systemd/resolve/stub-resolv.conf . Which I guess may not exist in a mock environment, which may be the issue here.

Here's the failure:
https://openqa.fedoraproject.org/tests/664882
Comment 1 Adam Williamson 2020-09-14 23:38:08 UTC 
This also appears to have broken today's Rawhide compose:
https://pagure.io/releng/failed-composes/issue/1761
Comment 2 Pavel Raiskup 2020-09-15 06:07:11 UTC 
Thank you for the report, Adam.

What SUT stands for?

> https://src.fedoraproject.org/rpms/systemd/c/b98737cec484f1c8dcb839e97192809b99f77c89?branch=f33

Great analysis, that indeed triggered the problems.  This all is caused by (to me) unexpected
shutil.copy*() behavior:

    $ mkdir /tmp/a
    $ ln -s non-existing-file /tmp/a/resolv.conf
    $ mkdir /tmp/b
    $ echo content > /tmp/b/resolv.conf
    $ ipyton3
    In [1]: import shutil

    In [2]: shutil.copy2("/tmp/b/resolv.conf", "/tmp/a")
    Out[2]: '/tmp/a/resolv.conf'

    In [3]:
    Do you really want to exit ([y]/n)? y

    $ cat /tmp/a/non-existing-file
    content

IOW, shutil.copy() follows symlinks in dst=, and creates the non-existing file.  Seems
like we have to unlink the destination file first.
Comment 3 Zbigniew Jędrzejewski-Szmek 2020-09-15 06:23:14 UTC 
Heh, shutil.copy2() even has a follow_symlinks= argument, but it doesn't work!
With follow_symlinks=False, the symlink in /tmp/a is still followed.

_copy_config() already removes the destination path, but it uses os.path.exists()
which returns false for broken symlinks. Maybe use os.access() instead?
Comment 4 Pavel Raiskup 2020-09-15 06:24:40 UTC 
There's e.g. https://bugs.python.org/issue34374 - and per the report,
/bin/cp indeed behaves the same, except for the "dangling" symlink situation:

  $ cp b/resolv.conf a/resolv.conf 
  cp: not writing through dangling symlink 'a/resolv.conf'

I reported that.
Comment 5 Pavel Raiskup 2020-09-15 06:30:11 UTC 
> _copy_config() already removes the destination path, but it uses os.path.exists()
> which returns false for broken symlinks. Maybe use os.access() instead?

That sounds racy (as well as the current behavior).  Is it any better to just
try to remove the destination file, and ignore FileNotFoundError?
Comment 6 Pavel Raiskup 2020-09-15 06:50:00 UTC 
https://github.com/rpm-software-management/mock/pull/631
Comment 7 Adam Williamson 2020-09-15 15:22:31 UTC 
So, I guess we need a new mock build for Rawhide at least. If we are still planning to have the change in F33, we need a new mock build for F33 as well and it should be added to the systemd update, I think.
Comment 8 Fedora Update System 2020-09-15 17:06:55 UTC 
FEDORA-2020-f7ce2b3a71 has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2020-f7ce2b3a71
Comment 9 Fedora Update System 2020-09-15 17:08:20 UTC 
FEDORA-2020-8b8b3ade7b has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2020-8b8b3ade7b
Comment 10 Fedora Update System 2020-09-16 05:21:36 UTC 
FEDORA-EPEL-2020-07a27d3e56 has been submitted as an update to Fedora EPEL 8. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-07a27d3e56
Comment 11 Fedora Update System 2020-09-16 05:21:37 UTC 
FEDORA-EPEL-2020-07a27d3e56 has been submitted as an update to Fedora EPEL 8. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-07a27d3e56
Comment 12 Fedora Update System 2020-09-16 05:22:45 UTC 
FEDORA-EPEL-2020-0996fb7a3c has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-0996fb7a3c
Comment 13 Fedora Update System 2020-09-16 14:30:10 UTC 
FEDORA-2020-f7ce2b3a71 has been pushed to the Fedora 32 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-f7ce2b3a71`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-f7ce2b3a71

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
Comment 14 Fedora Update System 2020-09-16 14:40:01 UTC 
FEDORA-EPEL-2020-0996fb7a3c has been pushed to the Fedora EPEL 7 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-0996fb7a3c

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
Comment 15 Fedora Update System 2020-09-16 14:58:24 UTC 
FEDORA-EPEL-2020-07a27d3e56 has been pushed to the Fedora EPEL 8 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-07a27d3e56

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
Comment 16 Fedora Update System 2020-09-16 15:36:02 UTC 
FEDORA-2020-ef6074cb37 has been pushed to the Fedora 31 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-ef6074cb37`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-ef6074cb37

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
Comment 17 Fedora Update System 2020-09-16 18:55:53 UTC 
FEDORA-2020-8b8b3ade7b has been pushed to the Fedora 33 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-8b8b3ade7b`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-8b8b3ade7b

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
Comment 18 Fedora Update System 2020-09-19 22:39:10 UTC 
FEDORA-2020-f7ce2b3a71 has been pushed to the Fedora 32 stable repository.
If problem still persists, please make note of it in this bug report.
Comment 19 Fedora Update System 2020-09-23 18:10:28 UTC 
FEDORA-2020-ef6074cb37 has been pushed to the Fedora 31 stable repository.
If problem still persists, please make note of it in this bug report.
Comment 20 Fedora Update System 2020-09-25 16:43:56 UTC 
FEDORA-2020-8b8b3ade7b has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.
Comment 21 Fedora Update System 2020-10-01 00:30:52 UTC 
FEDORA-EPEL-2020-0996fb7a3c has been pushed to the Fedora EPEL 7 stable repository.
If problem still persists, please make note of it in this bug report.
Comment 22 Fedora Update System 2020-10-01 01:41:51 UTC 
FEDORA-EPEL-2020-07a27d3e56 has been pushed to the Fedora EPEL 8 stable repository.
If problem still persists, please make note of it in this bug report.
Note You need to log in before you can comment on or make changes to this bug.