Bug 1558249

Created attachment 1410162 [details]
File: backtrace

Created attachment 1410163 [details]
File: cgroup

Created attachment 1410164 [details]
File: core_backtrace

Created attachment 1410165 [details]
File: cpuinfo

Created attachment 1410166 [details]
File: dso_list

Created attachment 1410167 [details]
File: environ

Created attachment 1410168 [details]
File: limits

Created attachment 1410169 [details]
File: maps

Created attachment 1410170 [details]
File: mountinfo

Created attachment 1410171 [details]
File: open_fds

Created attachment 1410172 [details]
File: proc_pid_status

This is the same backtrace as in bug #1544429, which was confirmed to be fixed in findutils-4.6.0-16.fc27.  This could be caused by some inconsistency in the file system data as you suggest.

According to attachment #1410170 [details] /home/scott.a.clark/Documents is a cifs mount.  Could it be the cause of the crash?

From the backtrace it is not clear which path exactly was being processed while the assertion failure was triggered.  Could you please capture strace's output if the crash reoccurs?

I unmounted the CIFS and still got the issue. strace shows that the error is in the home partition. I am attaching the strace output here.

Created attachment 1410476 [details]
strace on findutils

Do I understand it correctly that the crash is still reproducible?  Would you be able to run a modified build of findutils and attach its debugging output?

Yes, still reproducible. This is a workstation laptop, so I can definitely install a debugging build of findutils. I'm beginning to think, though, that I may have filesystem problems on this installation. Maybe a debug would bear that out?

That is difficult to tell at this point.  You said that you ran fsck.  Did it find any errors?  Was it able to fix them?

Please try findutils-4.6.0-16.1.fc27 from the following copr:

https://copr.fedorainfracloud.org/coprs/kdudka/findutils-debug

The following commands will do it for you:

# dnf copr enable kdudka/findutils-debug
# dnf update findutils

Then please re-run the 'find' command with -D fts:

$ find -D fts ...

OK, made some progress. I do believe the CIFS mount has something to do with this. I got a core dump when it was mounted, but unmounting it I got no errors, even scanning the entire filesystem. Excerpt:

On CIFS:
=== pre-pop ========== 7
2: 5
  2-leaving: /home/scott.a.clark/Public
find: ===== check ===== cwd: 6
fts_read: p=/home/scott.a.clark/Public
  entering: /home/scott.a.clark/Documents
fts_read: p=/home/scott.a.clark/Documents
  4-leaving: /home/scott.a.clark/Documents
find: ===== check ===== cwd: 7
=== post-push ========== 5
2: 5
fts_read: p=/home/scott.a.clark/Documents/UnifiedGlobalServerNaming.xls
fts_read: p=/home/scott.a.clark/Documents/Pictures
  4-leaving: /home/scott.a.clark/Documents/Pictures
Aborted

CIFS Unmounted:
=== post-push ========== 5
3: 5
=== pre-pop ========== 7
3: 5
  2-leaving: /var/yp
find: ===== check ===== cwd: 6
fts_read: p=/var/yp
fts_read: p=/var/.updated
  3-leaving: /var
find: ===== check ===== cwd: 5
fts_read: p=/var
=== post-push ========== 6
3: 6
  3-leaving: /
find: ===== check ===== cwd: -1
fts_read: p=/

This looks suspicious.  The debugging output should contain the line:

    entering: /home/scott.a.clark/Documents/Pictures

... before the line:

    4-leaving: /home/scott.a.clark/Documents/Pictures

Could you please retry it with 'find -noleaf ...' ?

According to attachment #1410170 [details] /home/scott.a.clark/Documents is a cifs mount.  However, in attachment #1410476 [details] fstatfs() returns f_type=XFS_SB_MAGIC for that directory.  Is the file system auto-mounted?

Yes, the CIFS is auto-mounted via fstab. I can consistently reproduce core dump when using find on the Documents mount; if I unmount, it works.

[root@kelesis ~]# find / -type f -name mime
find: ‘/run/user/1000/doc’: Permission denied
find: ‘/run/user/1000/gvfs’: Permission denied
/var/lib/docker/overlay2/66c58816cdaee70ce6f6adea90d092116b3febf2a184c47cb906dc9d2e5354da/diff/usr/share/terminfo/m/mime
/var/lib/docker/overlay2/c9dfb741a113d5fbe51aab168691e30059bac9d015c5d33ae9ba66b667c7e67e/diff/usr/share/terminfo/m/mime
[root@kelesis ~]# mount /home/scott.a.clark/Documents/
[root@kelesis ~]# find / -type f -name mime
Aborted (core dumped)

(In reply to Scott Clark from comment #20)
> Yes, the CIFS is auto-mounted via fstab.

By auto-mounted, I meant "lazily" mounted on access by the automount service.

Could you please retry it with 'find -noleaf ...' ?

No, it's not using the automount service since it requires either being on my corporate network or VPN. It's just an fstab mount that either mounts at boot or I mount manually after connecting.

Here's the noleaf run:

[root@kelesis ~]# find / -noleaf -type f -name mime
find: ‘/run/user/1000/doc’: Permission denied
find: ‘/run/user/1000/gvfs’: Permission denied
/var/lib/docker/overlay2/66c58816cdaee70ce6f6adea90d092116b3febf2a184c47cb906dc9d2e5354da/diff/usr/share/terminfo/m/mime
/var/lib/docker/overlay2/c9dfb741a113d5fbe51aab168691e30059bac9d015c5d33ae9ba66b667c7e67e/diff/usr/share/terminfo/m/mime

This succeeded with the CIFS mount active.

Could you please try findutils-4.6.0-18.1.fc27 from my findutils-debug copr?

# dnf copr enable kdudka/findutils-debug
# dnf update findutils

Does it work now without the -noleaf option, too?

Seeing this same bug, although bizarrely, it seems like it's happening where a CIFS mount /used/ to be, not where it is now?

(In reply to Geoffrey Wiseman from comment #24)
> Seeing this same bug, although bizarrely, it seems like it's happening where
> a CIFS mount /used/ to be, not where it is now?

Geoffrey, which version of findutils are you referring to?  The testing build mentioned in comment #23 or a stable release?

Fedora 27 before and after dnf upgrade, but not including the testing build. Would you like me to try the testing build as well?

(In reply to Geoffrey Wiseman from comment #26)
> Would you like me to try the testing build as well?

That would be much appreciated!

Kamil, I'm still getting a core dump when CIFS is mounted and -noleaf is not specified.

[root@kelesis ~]# find / -type f -name mime
Aborted (core dumped)
[root@kelesis ~]# find / -noleaf -type f -name mime
find: ‘/run/user/1000/gvfs’: Permission denied
/var/lib/docker/overlay2/66c58816cdaee70ce6f6adea90d092116b3febf2a184c47cb906dc9d2e5354da/diff/usr/share/terminfo/m/mime
/var/lib/docker/overlay2/c9dfb741a113d5fbe51aab168691e30059bac9d015c5d33ae9ba66b667c7e67e/diff/usr/share/terminfo/m/mime
/var/lib/docker/overlay2/8f945510efb85334fcf5d3a6c1274ed1b5f6d6805237476ca4431ae06e418e48/diff/usr/share/terminfo/m/mime
/var/lib/docker/overlay2/bcd450a5c349b6e51cd3c07af97337135376425850fb29f361b7b7452b4748aa/diff/usr/share/terminfo/m/mime
[root@kelesis ~]# dnf list findutils
Last metadata expiration check: 2:43:50 ago on Mon 09 Apr 2018 09:42:15 AM CDT.
Installed Packages
findutils.x86_64                                                                             1:4.6.0-18.1.fc27                                                                              @kdudka-findutils-debug
Available Packages
findutils.src                                                                                1:4.6.0-18.1.fc27                                                                              kdudka-findutils-debug

Thanks for reply!  Could you please attach a pair of strace outputs with/without the -noleaf option enabled?

Created attachment 1419687 [details]
straces for find with and without noleaf in a .tgz

Same results as Scott -- with `-noleaf` I don't get a core dump without it, I do. And, yes, looks like it's in the folder with the CIFS mount that I get the core dump -- not the folder where it used to be. I couldn't tell from the output, but I narrowed down the folder structure used in the tests until it broke and it's definitely in `/mnt`.

Created attachment 1419866 [details]
sclark stacktrace 4-10

Thank you for providing the strace outputs!  I will forward it upstream and, in case they do not see an obvious bug, I will try to create a local reproducer and debug it myself.

Paul Eggert from upstream has identified some bugs in the fts gnulib module and created a patch to fix them.  Could you pleas retry it with findutils-4.6.0-18.2 from my copr:

# dnf copr enable kdudka/findutils-debug
# dnf update findutils

Does it work now without the -noleaf option, too?

Yes, this seems to make the core dump vanish for me at least.

(In reply to Geoffrey Wiseman from comment #35)
> Yes, this seems to make the core dump vanish for me at least.

Thanks for confirmation!

upstream commits:
http://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=commitdiff;h=2e53df54
http://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=commitdiff;h=81b8c0d3

Sorry for the delay, I can also confirm that the latest update works without the -noleaf option.

Thanks for the confirmation!  Official updates coming soon...

findutils-4.6.0-19.fc27 coreutils-8.27-21.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-72c7737ac6

coreutils-8.29-6.fc28 findutils-4.6.0-19.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-8998b98f9c

coreutils-8.27-21.fc27, findutils-4.6.0-19.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-72c7737ac6

coreutils-8.29-6.fc28, findutils-4.6.0-19.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-8998b98f9c

Maybe not available yet?

```
$ sudo dnf install coreutils findutils --enablerepo=updates-testing
Last metadata expiration check: 0:01:30 ago on Sat 21 Apr 2018 04:16:46 PM EDT.
Package coreutils-8.27-20.fc27.x86_64 is already installed, skipping.
Package findutils-1:4.6.0-18.2.fc27.x86_64 is already installed, skipping.
Dependencies resolved.
Nothing to do.
Complete!

```

(In reply to Geoffrey Wiseman from comment #43)
> Maybe not available yet?

It takes some time till the mirrors are synchronized.  You can always install the packages directly from Koji:

$ sudo dnf install https://kojipkgs.fedoraproject.org//packages/coreutils/8.27/21.fc27/x86_64/coreutils-8.27-21.fc27.x86_64.rpm https://kojipkgs.fedoraproject.org//packages/coreutils/8.27/21.fc27/x86_64/coreutils-common-8.27-21.fc27.x86_64.rpm https://kojipkgs.fedoraproject.org//packages/findutils/4.6.0/19.fc27/x86_64/findutils-4.6.0-19.fc27.x86_64.rpm

How long is 'some time'? :) 

Still seems not to be available for me. It's not super-important, even the original core dump wasn't causing that much problem, and the test build has fixed that, so I'm not blocked by this, I'm just curious how long it takes, 'cause it seems to be significantly longer than I expected.

Maybe a stupid question but did you try 'dnf update' instead of 'dnf install'?

coreutils-8.27-21.fc27, findutils-4.6.0-19.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.

coreutils-8.29-6.fc28, findutils-4.6.0-19.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.