Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1580435
Summary: | rubygem-mongo: "Inappropriate ioctl for device" for only mock new chroot | ||||||||
---|---|---|---|---|---|---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Jun Aruga <jaruga> | ||||||
Component: | mock | Assignee: | Miroslav Suchý <msuchy> | ||||||
Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||||
Severity: | unspecified | Docs Contact: | |||||||
Priority: | unspecified | ||||||||
Version: | 29 | CC: | jdisnard, jkeating, mebrown, msekleta, msimacek, mskalick, msuchy, omajid, praiskup, rjanekov, tdawson, vondruch, williams | ||||||
Target Milestone: | --- | ||||||||
Target Release: | --- | ||||||||
Hardware: | Unspecified | ||||||||
OS: | Unspecified | ||||||||
Whiteboard: | |||||||||
Fixed In Version: | mock-1.4.13-1.fc28 mock-1.4.13-1.fc27 mock-1.4.13-1.el7 | Doc Type: | If docs needed, set a value | ||||||
Doc Text: | Story Points: | --- | |||||||
Clone Of: | Environment: | ||||||||
Last Closed: | 2018-08-16 08:06:47 UTC | Type: | Bug | ||||||
Regression: | --- | Mount Type: | --- | ||||||
Documentation: | --- | CRM: | |||||||
Verified Versions: | Category: | --- | |||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||
Embargoed: | |||||||||
Attachments: |
|
Description
Jun Aruga
2018-05-21 12:59:38 UTC
Sorry but I can not reproduce the issue on Fedora 27 using new (systemd-nspawn based chroot, with -r fedora-rawhide-x86_64). Build (%check phase) took a considerable amount of time but the process completed successfully. I will retry on Fedora 28 and report back. I can see the problem on F28. rspec is running in %check but it looks like a lot of test cases are timing out and eventually entire test suite fails (exact reason seems to be different than in your case). However, the root cause doesn't seem to be mock or systemd-nspawn. In my case mongod that is spawned before running rspec crashed. Unfortunately, the test suite doesn't terminate right away. Instead, it continues for some time and fails with a rather cryptic error message. I captured the coredump of mongod but it has almost 300MB so I think that is too big to be attached as bugzilla attachment. One last note, if the root cause is really crash of mongod during %check (in both cases) then the system on the builder shouldn't really matter. However, I didn't observe the issue on F27. Created attachment 1440951 [details]
GDB backtrace of Mongo
Testing this on my Rawhide, the following test case reliably shots down MongoDB when using --new-chroot while it is passing just fine with --old-chroot:
~~~
CI=1 EXTERNAL_DISABLED=1 rspec spec/mongo/address/unix_spec.rb:37
~~~
See the attached log for backtrace.
~~~
$ rpm -qf /usr/bin/systemd-nspawn
systemd-container-238-7.fc29.1.x86_64
~~~
Marek, any idea what is going on? Hi, when I've tried to build rubygem-mongo, mongod reported fail because of 'mlock' failure - https://github.com/mongodb/mongo/blob/v3.6/src/mongo/base/secure_allocator.cpp#L243 Is it possible that systemd-nspawn prohibits this somehow? Whom to ask? This piece of mongodb code is almost the same in F27 mongodb version and F28. So I guess that systemd-nspawn changed in F28. (In reply to Marek Skalický from comment #7) > Is it possible that systemd-nspawn prohibits this somehow? Whom to ask? > This piece of mongodb code is almost the same in F27 mongodb version and > F28. So I guess that systemd-nspawn changed in F28. @Michal: any hint please? Thank you for the investigating, everyone. As Michal said, this issue happens on only f28, when running mock (--new-root) on f28. I have never experienced it when running mock on f27. When I tested with below dummy spec file, same error was happened. ``` $ cat spec/foo_spec.rb require 'spec_helper' describe 'foo' do it 'bar' do expect(true).to be_truthy end end ``` ``` CI=1 EXTERNAL_DISABLED=1 rspec spec/foo.spec ``` The problem is with default seccomp filter that is used by systemd-nspawn in F28. mlock() (and other memory locking related) syscalls are by default disabled. In order to enable them, you have to either use --system-call-filter=@memlock or --capability=cap_ipc_lock on nspawn command line. https://github.com/systemd/systemd/blob/master/src/nspawn/nspawn-seccomp.c#L56 I think that disallowing module loading and preventing containers from mocking with system time is sensible. Not quite sure about mlock() and friends. @Michal, thx for investigation. So lets see what mock maitainers thinks about it. *** Bug 1594397 has been marked as a duplicate of this bug. *** --system-call-filter is not recognized on RHEL7. --capability=cap_ipc_lock should be OK. Fixed in commit: * f2282d6 (HEAD -> devel, origin/devel) enable cap_ipc_lock in nspawn container [RHBZ#1580435] For everyone hitting this issue, you can try: https://copr.fedorainfracloud.org/coprs/g/mock/mock/ which has this fix. mock-core-configs-29.1-1.fc28 distribution-gpg-keys-1.22-1.fc28 mock-1.4.13-1.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-a6fe7bfae0 distribution-gpg-keys-1.22-1.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-adfd38b8b9 mock-core-configs-29.1-1.fc27 distribution-gpg-keys-1.22-1.fc27 mock-1.4.13-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-4aeec4e04e mock-core-configs-29.1-1.el7 distribution-gpg-keys-1.22-1.el7 mock-1.4.13-1.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-68b5e45991 This bug appears to have been reported against 'rawhide' during the Fedora 29 development cycle. Changing version to '29'. distribution-gpg-keys-1.22-1.fc27, mock-1.4.13-1.fc27, mock-core-configs-29.1-1.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-4aeec4e04e distribution-gpg-keys-1.22-1.el7, mock-1.4.13-1.el7, mock-core-configs-29.1-1.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-68b5e45991 distribution-gpg-keys-1.22-1.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-adfd38b8b9 distribution-gpg-keys-1.22-1.fc28, mock-1.4.13-1.fc28, mock-core-configs-29.1-1.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-a6fe7bfae0 distribution-gpg-keys-1.22-1.fc28, mock-1.4.13-1.fc28, mock-core-configs-29.1-1.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report. Thank you for fixing this issue. I appreciate it. It's very helpful. I tried to run a scratch build for the rubygem-mongo on a rawhide package. The running mock version is 1.3.4. It is older version. I wish the mock version becomes the latest version. https://koji.fedoraproject.org/koji/taskinfo?taskID=29110478 https://kojipkgs.fedoraproject.org//work/tasks/478/29110478/build.log > Mock Version: 1.3.4 distribution-gpg-keys-1.22-1.el7 mock-1.4.13-1.el7 mock-core-configs-29.2-1.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-68b5e45991 distribution-gpg-keys-1.22-1.fc27 mock-1.4.13-1.fc27 mock-core-configs-29.2-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-4aeec4e04e distribution-gpg-keys-1.22-1.fc27, mock-1.4.13-1.fc27, mock-core-configs-29.2-1.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-4aeec4e04e As I got an error when doing local mock build on rawhide with the latest version: mock 4.13-1, I checked it by strace, adding below debug code. Then I still found the "Inappropriate ioctl for device" message in the log file. This error happens both new chroot and old chroot (--old-chroot option) cases. But this error does not happen on f28. Both local mock build and scratch build are success on f28. https://src.fedoraproject.org/fork/jaruga/rpms/rubygem-mongo/c/1b5816649ee143d786de6c7b1acbadfbefda5430?branch=hotfix%2Finappropriate-ictl rubygem-mongo.spec ``` $ CI=1 EXTERNAL_DISABLED=1 strace -o strace.log rspec spec --fail-fast=1 ``` ``` $ rpm -q mock mock-1.4.13-1.fc28.noarch $ rpm -q systemd systemd-238-8.git0e0aa59.fc28.x86_64 $ mock -r fedora-rawhide-x86_64 *.rpm $ view /path/to/strace.log ... ioctl(1, TCGETS, 0x7ffce9d27e90) = -1 ENOTTY (Inappropriate ioctl for device) ... ``` Created attachment 1476632 [details]
strace_part.log
I would attach the strace.log's only first 500 lines, as the actual log is big size, more than 40000 lines.
distribution-gpg-keys-1.22-1.el7, mock-1.4.13-1.el7, mock-core-configs-29.2-1.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-68b5e45991 distribution-gpg-keys-1.22-1.fc27, mock-1.4.13-1.fc27, mock-core-configs-29.2-1.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report. distribution-gpg-keys-1.23-1.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-07adf03b88 distribution-gpg-keys-1.23-1.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-07adf03b88 distribution-gpg-keys-1.22-1.el7, mock-1.4.13-1.el7, mock-core-configs-29.2-1.el7 has been pushed to the Fedora EPEL 7 stable repository. If problems still persist, please make note of it in this bug report. distribution-gpg-keys-1.23-1.el6 has been pushed to the Fedora EPEL 6 stable repository. If problems still persist, please make note of it in this bug report. |