Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1580435 - rubygem-mongo: "Inappropriate ioctl for device" for only mock new chroot
Summary: rubygem-mongo: "Inappropriate ioctl for device" for only mock new chroot
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: mock
Version: 29
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Miroslav Suchý
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1594397 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-05-21 12:59 UTC by Jun Aruga
Modified: 2018-10-10 20:14 UTC (History)
13 users (show)

Fixed In Version: mock-1.4.13-1.fc28 mock-1.4.13-1.fc27 mock-1.4.13-1.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-08-16 08:06:47 UTC
Type: Bug
Embargoed:

Attachments (Terms of Use)
GDB backtrace of Mongo (13.46 KB, text/plain)
2018-05-24 06:19 UTC, Vít Ondruch 		no flags Details
strace_part.log (39.23 KB, text/plain)
2018-08-17 14:05 UTC, Jun Aruga 		no flags Details
View All

Description Jun Aruga 2018-05-21 12:59:38 UTC 
Description of problem:

Below issue happens for only mock new chroot environment (systemd-nspawn).
It does not happen for old chroot environment (--old-chroot) and koji environment.
I reported the mock upstream project to ask.
But I would like to file the issue here too.
Because I guess that depends on systemd version. It is meaningful to file here.

```
$ mock -r fedora-rawhide-x86_64 *.src.rpm
...
BUILDSTDERR: ioctl(1, TCGETS, 0x7ffe94a72140)        = -1 ENOTTY (Inappropriate ioctl for device)
An error occurred in a `before(:suite)` hook. 
Failure/Error: raise Error::SocketError, e.message
Mongo::Error::SocketError:
  end of file reached
# ./lib/mongo/socket.rb:227:in `rescue in handle_errors'
...
# EOFError:
#   end of file reached
#   /usr/share/ruby/socket.rb:452:in `__read_nonblock'
```

Version-Release number of selected component (if applicable):
rubygem-mongo-2.5.1-1.fc29.src.rpm

$ rpm -q mock
mock-1.4.10-1.fc28.noarch

$ rpm -qf /usr/bin/systemd-nspawn
systemd-container-238-8.git0e0aa59.fc28.x86_64


How reproducible:

See https://github.com/rpm-software-management/mock/issues/186


Actual results:

```
$ mock -r fedora-rawhide-x86_64 *.src.rpm

# EOFError:
#   end of file reached
#   /usr/share/ruby/socket.rb:452:in `__read_nonblock'
```

Expected results:

$ mock -r fedora-rawhide-x86_64 *.src.rpm
finish successfully.

Additional info:
Comment 1 Michal Sekletar 2018-05-23 14:30:31 UTC 
Sorry but I can not reproduce the issue on Fedora 27 using new (systemd-nspawn based chroot, with -r fedora-rawhide-x86_64). Build (%check phase) took a considerable amount of time but the process completed successfully.

I will retry on Fedora 28 and report back.
Comment 2 Michal Sekletar 2018-05-23 15:10:06 UTC 
I can see the problem on F28. rspec is running in %check but it looks like a lot of test cases are timing out and eventually entire test suite fails (exact reason seems to be different than in your case).

However, the root cause doesn't seem to be mock or systemd-nspawn. In my case mongod that is spawned before running rspec crashed. Unfortunately, the test suite doesn't terminate right away. Instead, it continues for some time and fails with a rather cryptic error message.
Comment 3 Michal Sekletar 2018-05-23 15:18:39 UTC 
I captured the coredump of mongod but it has almost 300MB so I think that is too big to be attached as bugzilla attachment.
Comment 4 Michal Sekletar 2018-05-23 15:20:48 UTC 
One last note, if the root cause is really crash of mongod during %check (in both cases) then the system on the builder shouldn't really matter. However, I didn't observe the issue on F27.
Comment 5 Vít Ondruch 2018-05-24 06:19:59 UTC 
Created attachment 1440951 [details]
GDB backtrace of Mongo

Testing this on my Rawhide, the following test case reliably shots down MongoDB when using --new-chroot while it is passing just fine with --old-chroot:

~~~
CI=1 EXTERNAL_DISABLED=1 rspec spec/mongo/address/unix_spec.rb:37
~~~

See the attached log for backtrace.

~~~
$ rpm -qf /usr/bin/systemd-nspawn 
systemd-container-238-7.fc29.1.x86_64
~~~
Comment 6 Vít Ondruch 2018-05-24 06:20:42 UTC 
Marek, any idea what is going on?
Comment 7 Marek Skalický 2018-05-24 14:52:33 UTC 
Hi,
when I've tried to build rubygem-mongo, mongod reported fail because of 'mlock' failure - https://github.com/mongodb/mongo/blob/v3.6/src/mongo/base/secure_allocator.cpp#L243

Is it possible that systemd-nspawn prohibits this somehow? Whom to ask?
This piece of mongodb code is almost the same in F27 mongodb version and F28. So I guess that systemd-nspawn changed in F28.
Comment 8 Vít Ondruch 2018-05-25 08:43:15 UTC 
(In reply to Marek Skalický from comment #7)
> Is it possible that systemd-nspawn prohibits this somehow? Whom to ask?
> This piece of mongodb code is almost the same in F27 mongodb version and
> F28. So I guess that systemd-nspawn changed in F28.

@Michal: any hint please?
Comment 9 Jun Aruga 2018-05-28 13:19:51 UTC 
Thank you for the investigating, everyone.

As Michal said, this issue happens on only f28, when running mock (--new-root) on f28. I have never experienced it when running mock on f27.

When I tested with below dummy spec file, same error was happened.

```
$ cat spec/foo_spec.rb 
require 'spec_helper'

describe 'foo' do
  it 'bar' do
    expect(true).to be_truthy
  end
end
```

```
CI=1 EXTERNAL_DISABLED=1 rspec spec/foo.spec
```
Comment 10 Michal Sekletar 2018-05-29 15:55:18 UTC 
The problem is with default seccomp filter that is used by systemd-nspawn in F28. mlock() (and other memory locking related) syscalls are by default disabled. In order to enable them, you have to either use --system-call-filter=@memlock or --capability=cap_ipc_lock on nspawn command line.

https://github.com/systemd/systemd/blob/master/src/nspawn/nspawn-seccomp.c#L56

I think that disallowing module loading and preventing containers from mocking with system time is sensible. Not quite sure about mlock() and friends.
Comment 11 Vít Ondruch 2018-05-30 08:39:59 UTC 
@Michal, thx for investigation. So lets see what mock maitainers thinks about it.
Comment 12 Miroslav Suchý 2018-06-25 07:21:44 UTC 
*** Bug 1594397 has been marked as a duplicate of this bug. ***
Comment 13 Miroslav Suchý 2018-06-25 17:10:16 UTC 
--system-call-filter is not recognized on RHEL7. --capability=cap_ipc_lock should be OK.

Fixed in commit:
* f2282d6 (HEAD -> devel, origin/devel) enable cap_ipc_lock in nspawn container [RHBZ#1580435]

For everyone hitting this issue, you can try:
https://copr.fedorainfracloud.org/coprs/g/mock/mock/
which has this fix.
Comment 14 Fedora Update System 2018-08-13 19:52:30 UTC 
mock-core-configs-29.1-1.fc28 distribution-gpg-keys-1.22-1.fc28 mock-1.4.13-1.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-a6fe7bfae0
Comment 15 Fedora Update System 2018-08-13 19:52:45 UTC 
distribution-gpg-keys-1.22-1.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-adfd38b8b9
Comment 16 Fedora Update System 2018-08-13 19:52:59 UTC 
mock-core-configs-29.1-1.fc27 distribution-gpg-keys-1.22-1.fc27 mock-1.4.13-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-4aeec4e04e
Comment 17 Fedora Update System 2018-08-13 19:53:14 UTC 
mock-core-configs-29.1-1.el7 distribution-gpg-keys-1.22-1.el7 mock-1.4.13-1.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-68b5e45991
Comment 18 Jan Kurik 2018-08-14 11:19:08 UTC 
This bug appears to have been reported against 'rawhide' during the Fedora 29 development cycle.
Changing version to '29'.
Comment 19 Fedora Update System 2018-08-14 20:36:32 UTC 
distribution-gpg-keys-1.22-1.fc27, mock-1.4.13-1.fc27, mock-core-configs-29.1-1.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-4aeec4e04e
Comment 20 Fedora Update System 2018-08-14 21:19:45 UTC 
distribution-gpg-keys-1.22-1.el7, mock-1.4.13-1.el7, mock-core-configs-29.1-1.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-68b5e45991
Comment 21 Fedora Update System 2018-08-14 21:43:04 UTC 
distribution-gpg-keys-1.22-1.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-adfd38b8b9
Comment 22 Fedora Update System 2018-08-14 22:42:05 UTC 
distribution-gpg-keys-1.22-1.fc28, mock-1.4.13-1.fc28, mock-core-configs-29.1-1.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-a6fe7bfae0
Comment 23 Fedora Update System 2018-08-16 08:06:47 UTC 
distribution-gpg-keys-1.22-1.fc28, mock-1.4.13-1.fc28, mock-core-configs-29.1-1.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.
Comment 24 Jun Aruga 2018-08-16 13:10:36 UTC 
Thank you for fixing this issue. I appreciate it.
It's very helpful.

I tried to run a scratch build for the rubygem-mongo on a rawhide package.
The running mock version is 1.3.4. It is older version.
I wish the mock version becomes the latest version.

https://koji.fedoraproject.org/koji/taskinfo?taskID=29110478
https://kojipkgs.fedoraproject.org//work/tasks/478/29110478/build.log
> Mock Version: 1.3.4
Comment 25 Fedora Update System 2018-08-16 15:28:02 UTC 
distribution-gpg-keys-1.22-1.el7 mock-1.4.13-1.el7 mock-core-configs-29.2-1.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-68b5e45991
Comment 26 Fedora Update System 2018-08-16 15:28:25 UTC 
distribution-gpg-keys-1.22-1.fc27 mock-1.4.13-1.fc27 mock-core-configs-29.2-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-4aeec4e04e
Comment 27 Fedora Update System 2018-08-17 13:35:31 UTC 
distribution-gpg-keys-1.22-1.fc27, mock-1.4.13-1.fc27, mock-core-configs-29.2-1.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-4aeec4e04e
Comment 28 Jun Aruga 2018-08-17 13:50:22 UTC 
As I got an error when doing local mock build on rawhide with the latest version: mock 4.13-1, I checked it by strace, adding below debug code.
Then I still found the "Inappropriate ioctl for device" message in the log file.

This error happens both new chroot and old chroot (--old-chroot option) cases.
But this error does not happen on f28. Both local mock build and scratch build are success on f28.

https://src.fedoraproject.org/fork/jaruga/rpms/rubygem-mongo/c/1b5816649ee143d786de6c7b1acbadfbefda5430?branch=hotfix%2Finappropriate-ictl

rubygem-mongo.spec

```
$ CI=1 EXTERNAL_DISABLED=1 strace -o strace.log rspec spec --fail-fast=1
```

```
$ rpm -q mock
mock-1.4.13-1.fc28.noarch

$ rpm -q systemd
systemd-238-8.git0e0aa59.fc28.x86_64

$ mock -r fedora-rawhide-x86_64 *.rpm

$ view /path/to/strace.log
...
ioctl(1, TCGETS, 0x7ffce9d27e90)        = -1 ENOTTY (Inappropriate ioctl for device)
...
```
Comment 29 Jun Aruga 2018-08-17 14:05:05 UTC 
Created attachment 1476632 [details]
strace_part.log

I would attach the strace.log's only first 500 lines, as the actual log is big size, more than 40000 lines.
Comment 30 Fedora Update System 2018-08-17 15:55:40 UTC 
distribution-gpg-keys-1.22-1.el7, mock-1.4.13-1.el7, mock-core-configs-29.2-1.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-68b5e45991
Comment 31 Fedora Update System 2018-08-30 03:51:53 UTC 
distribution-gpg-keys-1.22-1.fc27, mock-1.4.13-1.fc27, mock-core-configs-29.2-1.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.
Comment 32 Fedora Update System 2018-09-14 12:00:09 UTC 
distribution-gpg-keys-1.23-1.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-07adf03b88
Comment 33 Fedora Update System 2018-09-14 23:15:37 UTC 
distribution-gpg-keys-1.23-1.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-07adf03b88
Comment 34 Fedora Update System 2018-09-22 22:56:38 UTC 
distribution-gpg-keys-1.22-1.el7, mock-1.4.13-1.el7, mock-core-configs-29.2-1.el7 has been pushed to the Fedora EPEL 7 stable repository. If problems still persist, please make note of it in this bug report.
Comment 35 Fedora Update System 2018-10-10 20:14:48 UTC 
distribution-gpg-keys-1.23-1.el6 has been pushed to the Fedora EPEL 6 stable repository. If problems still persist, please make note of it in this bug report.
Note You need to log in before you can comment on or make changes to this bug.