Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 2125355 - Calling `fixfiles -M relabel` twice in quick succession can delete your entire root filesystem
Summary: Calling `fixfiles -M relabel` twice in quick succession can delete your entir...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: policycoreutils
Version: 36
Hardware: All
OS: All
unspecified
urgent
Target Milestone: ---
Assignee: Petr Lautrbach
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-09-08 17:37 UTC by Anthony Barone
Modified: 2023-03-25 02:01 UTC (History)
11 users (show)

Fixed In Version: policycoreutils-3.5-0.rc2.1.fc38 policycoreutils-3.5-1.fc37
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-01-23 08:04:02 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Anthony Barone 2022-09-08 17:37:55 UTC
Description of problem:

Running `fixfiles -M relabel` will relabel your entire filesystem, and will bind-mount any mounts with files selected for relabeling into `/tmp`. 

If you start this operation and then realize you wanted to change something (say, you wanted to select 'y' when asked to clear out the `/tmp` directory, you might stop it by pressing `<Crtl> + c`. This gives you back the terminal, but importantly doesnt actually stop the relabeling process (note that there is no indication given that fixfiles is still running - you would have to check a system monitoring tool like `top` to discover that it hadn't stopped)

You then call `fixfiles -M relabel` a 2nd time and select 'y' when asked to clear out the `/tmp` directory.

You likely wont realize it for a few minutes, but you just unknowingly ran the equivalent of `sudo \rm -rf /`.

Because the initial `fixfiles -M relabel` call is still running the filesystem mounts that you are relabeling are still bind-mounted to `/tmp/<...>` and *cannot* be unmounted, since they are busy. When the 2nd `fixfiles -M relabel` call then clears out the `/tmp` directory is apparently does so without performing any checks about, say, whether or not your entire filesystem is bind-mounted somewhere on `/tmp`. As such, it proceeds to delete your entire filesystem as it is cleaning out `/tmp`.


Version-Release number of selected component (if applicable): Current fixfiles version for Fedora 36. Id give you the exact version, but I'm still trying to put my system back together...


How reproducible: Very. 

Note: It should go without saying, but be aware that the system you reproduce this on wont be usable (or, well, exist at all, if left to finish uninterrupted) afterwards.

Steps to Reproduce:
1. run `sudo fixfiles -M relabel`. It shouldn't matter if you select `y` or `n` to clear out `/tmp` here.
2. stop it by pressing `<ctrl> + c`
3. run `sudo fixfiles [-M] relabel` a second time. Select `y` when asked to clear out `/tmp`. Using the `-M` flag shouldnt matter on the 2nd fixfiles call.
4. Watch your entire system delete itself as you are powerless to stop it, except for cutting off power to the machine / doing a hard reboot

Expected results: A relabeled filesystem

Actual results: Your filesystem no longer needs relabeling. Since it no longer exists.

Additional info: perhaps it was an unwise choice to have a program that both:

1. creates bind mounts in /tmp that cannot be unmounted for a long period of time (minutes to hours - until fixfiles finishes relabeling everything under that mount), and 

2. gives users a very easy (single character response) way to (as root) delete everything (including stuff under bind mounts) on `/tmp`

It seems like this could be avoided by, say, making a directory and then mounting a tmpfs at `/tmp-relabel` and then bind-mounting filesystems there. It could also be avoided by adding a check for bind-mounted filesystems under `/tmp` and excluding them from the `\rm -rf /tmp` call that `fixfiles [-M] relabel` seems to make.

If nothing else at least give people a warning if they unknowingly do what is described above...

Comment 1 Petr Lautrbach 2022-09-15 09:15:09 UTC
This https://github.com/bachradsusi/SELinuxProject-selinux/commit/a4cb900cfb65fa0ed7b3c9928a7a4634be4a70ff could help.

I'm going to test it a bit and propose it upstream.

Comment 2 Peder Stray 2022-09-15 15:44:29 UTC
Wouldn't this happen if anything else tries to clean up in /tmp too?  like tmpcleaning cron-jobs?  Mounting anything of value under /tmp seems like a really bad idea.

Comment 3 Anthony Barone 2022-09-18 08:07:16 UTC
(In reply to Peder Stray from comment #2)
> Wouldn't this happen if anything else tries to clean up in /tmp too?  like tmpcleaning cron-jobs?  Mounting anything of value under /tmp seems like a really bad idea.


For what it's worth, I would tend to agree with this. I've always gone by the rule of thumb that you shouldn't put anything in /tmp that you would be sad to not have after a reboot. Granted bind mounting is less risky and just rebooting won't cause bind mounted data to be lost, but since stuff in /tmp is intended to be transient and disposable it's not all that uncommon for programs (including fixfiles) to treat it as such and delete it all.


(In reply to Petr Lautrbach from comment #1)
> This https://github.com/bachradsusi/SELinuxProject-selinux/commit/a4cb900cfb65fa0ed7b3c9928a7a4634be4a70ff could help.
> 
> I'm going to test it a bit and propose it upstream.


Id be curious to hear the results of your tests, but I can envision a few potential problems with this:

1. I dont think that the running `setfiles` command will allow you to just umount the bind mounted filesystem, since it is still running the restore operation on it. I suspect you might be able to solve this by killing the setfiles process first or by using `umount -l`, though these both sound like the type of thing that might open the possibility of filesystem corruption. I dont think repeatedly re-calling the umount command or waiting for it would work either, since that would end up waiting for the relabel command to finish, and chances are if the user was going to re-run the fixfiles command they already would have by then.

2. Even if the above point is actually not a problem, this approach would, I believe, only guard against pressing <ctrl>+<c>. This is probably the most common way that the fixfiles command might get interrupted, but is by no means the only way.


---------------------------------------------------------------------------------------------------------------------------

If I might suggest an alternate (untested) solution - the same loop you modify in your linked commit, change to this instead:

```
TMP_MOUNT_BASEDIR="$(mktemp -p /mnt -d)"     # doesnt have to be at /mnt, but shouldnt be anywhere under /tmp
mount -t tmpfs tmpfs "${TMP_MOUNT_BASEDIR}"

for m in `echo $FILESYSTEMSRW`; do
	TMP_MOUNT="$(mktemp -p "${TMP_MOUNT_BASEDIR}" -d)"
	test -z ${TMP_MOUNT+x} && echo "Unable to find temporary directory!" && exit 1

	mkdir -p "${TMP_MOUNT}${m}" || exit 1
	mount --bind "${m}" "${TMP_MOUNT}${m}" || exit 1
	${SETFILES} ${VERBOSE} ${EXCLUDEDIRS} ${FORCEFLAG} ${THREADS} $* -q ${FC} -r "${TMP_MOUNT}" "${TMP_MOUNT}${m}"
	umount "${TMP_MOUNT}${m}" || exit 1
	rm -rf "${TMP_MOUNT}" || echo "Error cleaning up."
done;

umount "${TMP_MOUNT_BASEDIR}"
rm -rf "${TMP_MOUNT_BASEDIR}"
```

Basically, use mktemp to get a unique directory somewhere that isnt under /tmp, then create the directory and mount a tmpfs to it, then run the setfiles loop (but have mktemp create temp dirs under your newly mounted tmpfs instead of under /tmp), then when setfiles is done umount the tmpfs and remove the directory.

I think this would work, so long as $TMP_MOUNT_BASEDIR is chosen somewhere that fixfiles has permissions to create a directory and mount a tmpfs to it (but that isnt under /tmp).

Comment 4 Petr Lautrbach 2022-09-18 16:53:37 UTC
There's a thread on mailing upstream list starting with https://lore.kernel.org/selinux/CAJ2a_DeBWkHziE4+DsRqqLULtGkdX68c8jdU3Hxs++84NoPpsQ@mail.gmail.com/T/#m264867de1eb0fe872a1191d74a9b6e3166493d5e with v3 version of the patch which uses '/run' instead of '/tmp' - https://lore.kernel.org/selinux/CAJ2a_DeBWkHziE4+DsRqqLULtGkdX68c8jdU3Hxs++84NoPpsQ@mail.gmail.com/T/#mf5e06bf175ee04b4b8a4c3984be87afdabd45e06

It would be great if the discussion continue only there in order not to duplicate messages and suggestions.

> 
> (In reply to Petr Lautrbach from comment #1)
> > This https://github.com/bachradsusi/SELinuxProject-selinux/commit/a4cb900cfb65fa0ed7b3c9928a7a4634be4a70ff could help.
> > 
> > I'm going to test it a bit and propose it upstream.
> 
> 
> Id be curious to hear the results of your tests, but I can envision a few
> potential problems with this:
> 
> 1. I dont think that the running `setfiles` command will allow you to just
> umount the bind mounted filesystem, since it is still running the restore
> operation on it. I suspect you might be able to solve this by killing the
> setfiles process first or by using `umount -l`, though these both sound like
> the type of thing that might open the possibility of filesystem corruption.
> I dont think repeatedly re-calling the umount command or waiting for it
> would work either, since that would end up waiting for the relabel command
> to finish, and chances are if the user was going to re-run the fixfiles
> command they already would have by then.


When I tried to reproduce this problem and hit Ctr-C there was no setfiles process left, only the mountpoint and therefore umount simply worked.

What shell or kind of terminal do you use?

 
> 2. Even if the above point is actually not a problem, this approach would, I
> believe, only guard against pressing <ctrl>+<c>. This is probably the most
> common way that the fixfiles command might get interrupted, but is by no
> means the only way.
> 

The v3 version mentioned above already trap's on all EXIT signals.

Comment 5 Petr Lautrbach 2022-09-19 09:25:48 UTC
> When I tried to reproduce this problem and hit Ctr-C there was no setfiles
> process left, only the mountpoint and therefore umount simply worked.
> 


[root@P1 ~]# fixfiles -M relabel

    Files in the /tmp directory may be labeled incorrectly, this command
    can remove all files in /tmp.  If you choose to remove files from /tmp,
    a reboot will be required after completion.

    Do you wish to clean out the /tmp directory [N]? 
Relabeling / /boot /dev /dev/hugepages /dev/mqueue /dev/pts /dev/shm /home /run /run/user/1000 /sys /sys/fs/cgroup /sys/fs/pstore /sys/kernel/debug /sys/kernel/tracing /tmp /var
6k^C

[root@P1 ~]# ps ax | grep setfiles
  99107 pts/2    S+     0:00 grep --color=auto setfiles

[root@P1 ~]# mount | grep /tmp/tmp
/dev/mapper/luks-123e-32c7-4d1f-323b-23c7440558c on /tmp/tmp.0sXcAjjdKd type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)

Comment 6 Anthony Barone 2022-10-25 13:07:19 UTC
(In reply to Petr Lautrbach from comment #4)
 
> When I tried to reproduce this problem and hit Ctr-C there was no setfiles
> process left, only the mountpoint and therefore umount simply worked.
> 
> What shell or kind of terminal do you use?


Sorry for the delay in replying....life has been extra hectic lately.

I use the Fedora KDE spin and this was run on "konsole" - the standard KDE terminal.

Unfortunately, I can't just go back and check logs (being they got deleted and all), but trying to remember back I *might* have run fixfiles using the parallel relabeling flag (`-T <N>`) on the first run. Perhaps the reason that `<ctrl> + <c>` didn't stop the first fixfiles run was because the -T flag made fixfiles fork off the fixfiles processes that actually touched the filesystem, and these were not stopped when the main fixfiles process was interrupted?

Comment 7 Fedora Update System 2023-01-16 16:34:42 UTC
FEDORA-2023-398d7812ee has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-398d7812ee

Comment 8 Fedora Update System 2023-01-23 08:04:02 UTC
FEDORA-2023-398d7812ee has been pushed to the Fedora 38 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 9 Fedora Update System 2023-02-27 18:52:05 UTC
FEDORA-2023-568a3be5b8 has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2023-568a3be5b8

Comment 10 Fedora Update System 2023-02-28 02:54:10 UTC
FEDORA-2023-568a3be5b8 has been pushed to the Fedora 37 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-568a3be5b8`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-568a3be5b8

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 11 Fedora Update System 2023-03-25 02:01:32 UTC
FEDORA-2023-568a3be5b8 has been pushed to the Fedora 37 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.