Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1331787 - race condition on builder caused by createrepo_c?
Summary: race condition on builder caused by createrepo_c?
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Internal Copr
Classification: Internal
Component: backend
Version: unspecified
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Copr Team
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-04-29 13:42 UTC by Pavel Raiskup
Modified: 2020-12-14 10:44 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-12-11 11:52:29 UTC
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1355720 0 unspecified CLOSED crash: double-free with --ignore-lock 2022-05-16 11:32:56 UTC

Internal Links: 1355720

Description Pavel Raiskup 2016-04-29 13:42:37 UTC 
This issue seems to be caused by concurrent run of createrepo_c with build.
I still need to have a look at it.

...
DEBUG util.py:563:  child environment: None
DEBUG util.py:495:  Executing command: ['/bin/mount', '-n', '-t', 'tmpfs', 'tmpfs', '/var/lib/mock/fedora-23-x86_64-mockbuilder-5704/root/dev/shm'] with env {'HOSTNAME': 'mock', 'HOME': '/builddir', 'TERM': 'vt100', 'SHELL': '/bin/sh', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'LANG': 'en_US.UTF-8'} and shell False
DEBUG util.py:542:  Child return code was: 0
DEBUG util.py:563:  child environment: None
DEBUG util.py:495:  Executing command: ['/bin/mount', '-n', '-t', 'devpts', '-o', 'gid=5,mode=0620,ptmxmode=0666,newinstance', 'devpts', '/var/lib/mock/fedora-23-x86_64-mockbuilder-5704/root/dev/pts'] with env {'HOSTNAME': 'mock', 'HOME': '/builddir', 'TERM': 'vt100', 'SHELL': '/bin/sh', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'LANG': 'en_US.UTF-8'} and shell False
DEBUG util.py:542:  Child return code was: 0
DEBUG util.py:563:  child environment: None
DEBUG util.py:495:  Executing command: ['/bin/mount', '-n', '--bind', '/var/cache/mock/fedora-23-x86_64/yum_cache/', '/var/lib/mock/fedora-23-x86_64-mockbuilder-5704/root/var/cache/yum/'] with env {'HOSTNAME': 'mock', 'HOME': '/builddir', 'TERM': 'vt100', 'SHELL': '/bin/sh', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'LANG': 'en_US.UTF-8'} and shell False
DEBUG util.py:542:  Child return code was: 0
DEBUG util.py:563:  child environment: None
DEBUG util.py:495:  Executing command: ['/bin/mount', '-n', '--bind', '/tmp/mock-selinux-plugin.gvbfzpxj', '/var/lib/mock/fedora-23-x86_64-mockbuilder-5704/root/proc/filesystems'] with env {'HOSTNAME': 'mock', 'HOME': '/builddir', 'TERM': 'vt100', 'SHELL': '/bin/sh', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'LANG': 'en_US.UTF-8'} and shell False
DEBUG util.py:542:  Child return code was: 0
DEBUG buildroot.py:138:  rootdir = /var/lib/mock/fedora-23-x86_64-mockbuilder-5704/root
DEBUG buildroot.py:139:  resultdir = /var/tmp/mockremote-xcsUn/build/results/fedora-23-x86_64/cov-im-utils-1.0.2.20160429.144027.gef57d34-1.fc23
DEBUG util.py:97:  ensuring that dir exists: /var/lib/mock/fedora-23-x86_64-mockbuilder-5704/root/etc/pki/mock
DEBUG util.py:100:  creating dir: /var/lib/mock/fedora-23-x86_64-mockbuilder-5704/root/etc/pki/mock
DEBUG util.py:97:  ensuring that dir exists: /var/lib/mock/fedora-23-x86_64-mockbuilder-5704/root/etc/yum/pluginconf.d
DEBUG util.py:100:  creating dir: /var/lib/mock/fedora-23-x86_64-mockbuilder-5704/root/etc/yum/pluginconf.d
DEBUG package_manager.py:199:  configure yum
DEBUG package_manager.py:205:  configure yum priorities
DEBUG package_manager.py:207:  configure yum rhnplugin
DEBUG util.py:97:  ensuring that dir exists: /var/lib/mock/fedora-23-x86_64-mockbuilder-5704/root/etc
DEBUG package_manager.py:96:  ['/usr/bin/yum-deprecated', '--installroot', '/var/lib/mock/fedora-23-x86_64-mockbuilder-5704/root/', '--releasever', '23', 'install', '@buildsys-build']
DEBUG util.py:563:  child environment: None
DEBUG util.py:495:  Executing command: ['/usr/bin/yum-deprecated', '--installroot', '/var/lib/mock/fedora-23-x86_64-mockbuilder-5704/root/', '--releasever', '23', 'install', '@buildsys-build', '--setopt=tsflags=nocontexts'] with env {'PROMPT_COMMAND': 'printf "\x1b]0;<mock-chroot>\x07<mock-chroot>"', 'HOSTNAME': 'mock', 'LANG': 'en_US.UTF-8', 'LC_MESSAGES': 'C', 'TERM': 'vt100', 'HOME': '/builddir', 'SHELL': '/bin/bash', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin'} and shell False
DEBUG util.py:417:  Yum command has been deprecated, use dnf instead.
DEBUG util.py:417:  See 'man dnf' and 'man yum2dnf' for more information.
DEBUG util.py:417:  http://coprbe.devel.redhat.com/repos/kdudka/covscan-testing/fedora-23-x86_64/devel/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found
DEBUG util.py:417:  Trying other mirror.
DEBUG util.py:417:   One of the configured repositories failed (updates),
DEBUG util.py:417:   and yum doesn't have enough cached data to continue. At this point the only
DEBUG util.py:417:   safe thing yum can do is fail. There are a few ways to work "fix" this:
DEBUG util.py:417:       1. Contact the upstream for the repository and get them to fix the problem.
DEBUG util.py:417:       2. Reconfigure the baseurl/etc. for the repository, to point to a working
DEBUG util.py:417:          upstream. This is most often useful if you are using a newer
DEBUG util.py:417:          distribution release than is supported by the repository (and the
DEBUG util.py:417:          packages for the previous distribution release still work).
DEBUG util.py:417:       3. Disable the repository, so yum won't use it by default. Yum will then
DEBUG util.py:417:          just ignore the repository until you permanently enable it again or use
DEBUG util.py:417:          --enablerepo for temporary usage:
DEBUG util.py:417:              yum-config-manager --disable updates
DEBUG util.py:417:       4. Configure the failing repository to be skipped, if it is unavailable.
DEBUG util.py:417:          Note that yum will try to contact the repo. when it runs most commands,
DEBUG util.py:417:          so will have to try and fail each time (and thus. yum will be be much
DEBUG util.py:417:          slower). If it is a very temporary problem though, this is often a nice
DEBUG util.py:417:          compromise:
DEBUG util.py:417:              yum-config-manager --save --setopt=updates.skip_if_unavailable=true
DEBUG util.py:417:  Cannot retrieve metalink for repository: updates/23/x86_64. Please verify its path and try again
DEBUG util.py:542:  Child return code was: 1
DEBUG util.py:176:  kill orphans
Comment 1 Pavel Raiskup 2016-07-12 11:13:27 UTC 
Maybe related bug report 1355720; again reported by Kamil, thanks.
Comment 2 Miroslav Suchý 2016-07-12 13:57:22 UTC 
The real cause was:

DEBUG util.py:417:  Cannot retrieve metalink for repository: updates/23/x86_64. Please verify its path and try again

So probably bad fedora metalink state.
Comment 3 Kamil Dudka 2016-07-12 14:18:12 UTC 
(In reply to Miroslav Suchý from comment #2)
> The real cause was:
>
> DEBUG util.py:417:  Cannot retrieve metalink for repository:
> updates/23/x86_64. Please verify its path and try again
> 
> So probably bad fedora metalink state.

Cloud be ... but how does it explain the following part of the output?

DEBUG util.py:417:  http://coprbe.devel.redhat.com/repos/kdudka/covscan-testing/fedora-23-x86_64/devel/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found

I was just building updates for an existing copr repository (yes, in parallel).  The repodata file was not found.  It should have existed all the time and only be atomically replaced with its new version.
Comment 4 Miroslav Suchý 2016-07-13 08:06:13 UTC 
See
https://fedorahosted.org/copr/wiki/UserDocs#Iseesomestrangeerroraboutdevelrepodatainlogs.
Comment 5 Kamil Dudka 2016-07-13 08:22:25 UTC 
I did not realize there was "devel/" inserted in the URL.  Anyway, why is copr passing a URL to mock despite it knows there is no repository at that URL?
Comment 6 Miroslav Suchý 2016-07-13 08:32:48 UTC 
(In reply to Kamil Dudka from comment #5)
> Anyway, why is
> copr passing a URL to mock despite it knows there is no repository at that
> URL?

No particular reason. Just because no one implemented the if-condition, which would query the current state and pass the /devel/ only if manual repo creation is enabled.
Comment 7 Pavel Raiskup 2018-09-18 09:02:23 UTC 
FTR: I still need to document the 'metalink' issue somewhere.
Comment 8 Pavel Raiskup 2020-12-11 11:52:29 UTC 
This was about two issues:
1. The 'metalink' problem got resolved by mock configuration on Copr builders, we now
   re-try several times before we give up (so the metalink usually get's fixed in
   before subsequent attempts).

2. The devel repo download failure is non-blocking, but there are several things
   that need to be fixed about our createrepo processes, this is tracked basically
   in https://pagure.io/copr/copr/issue/1148
Comment 9 Pavel Raiskup 2020-12-14 10:44:01 UTC 
Ah, this is the correct issue for the devel repo: https://pagure.io/copr/copr/issue/1145
Note You need to log in before you can comment on or make changes to this bug.