Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1190933 - kernel 3.18: read-only root due to journal_async_commit in /etc/fstab
Summary: kernel 3.18: read-only root due to journal_async_commit in /etc/fstab
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: 21
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Eric Sandeen
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: 1186363
TreeView+ depends on / blocked
 
Reported: 2015-02-10 02:54 UTC by Conrad S
Modified: 2015-03-14 09:15 UTC (History)
7 users (show)

Fixed In Version: kernel-3.18.9-100.fc20
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-03-09 08:16:44 UTC
Type: Bug


Attachments (Terms of Use)
dmesg (58.30 KB, text/plain)
2015-02-10 03:01 UTC, Conrad S
no flags Details

Description Conrad S 2015-02-10 02:54:01 UTC
Description of problem:

Booting kernel 3.18 with journal_async_commit for / in /etc/fstab causes the root file system to be read-only.

Version-Release number of selected component (if applicable):
kernel-3.18.5-201.fc21.x86_64
3.18.3 also has this problem.

How reproducible:
Always while using kernel 3.18.  No problems with kernel 3.17.

Steps to Reproduce:
1. Change the entry for / in /etc/fstab to:
UUID=...  / ext4    defaults,nodiratime,noatime,journal_async_commit 1 1
2. Upgrade from kernel 3.17 to 3.18 
3. Reboot
4. Try to write to /

Actual results:
/ is mounted read-only at boot, causing all sorts of problems (eg. wifi no longer connects)

Expected results:
/ is mounted as read-write

Additional info:
kernel-3.17.7-300.fc21.x86_64 works without problems: / is mounted as read-write at boot with journal_async_commit

Comment 1 Conrad S 2015-02-10 03:01:24 UTC
Created attachment 989921 [details]
dmesg

dmesg from kernel 3.18.5

Comment 2 Conrad S 2015-02-10 03:05:25 UTC
A symptom of the problem (extracted from dmesg)
[    4.129492] EXT4-fs (sdb1): changing journal_checksum during remount not supported

Changing the /etc/fstab entry to also contain "journal_checksum" seems not to help.

Comment 3 Josh Boyer 2015-02-10 13:40:14 UTC
That is upstream commit

commit 6b992ff25658367089db4a82666e232b65d55eae
Author: Darrick J. Wong <darrick.wong>
Date:   Thu Oct 30 10:53:16 2014 -0400

    ext4: disallow changing journal_csum option during remount
    
    ext4 does not permit changing the metadata or journal checksum feature
    flag while mounted.  Until we decide to support that, don't allow a
    remount to change the journal_csum flag (right now we silently fail to
    change anything).
    
    Signed-off-by: Darrick J. Wong <darrick.wong>
    Signed-off-by: Theodore Ts'o <tytso>


which indeed landed in 3.18.  Note that "works without problems" on 3.17 in this case means it didn't do anything at all.

I'm not sure this is a bug, but I'll let the fs people weigh in.

Comment 4 Conrad S 2015-02-10 14:46:12 UTC
Technically this is not supposed to be a remount, but the "first" mount at boot time.

Obviously there is the cyclic dependency of requiring /etc to be accessible in order to read /etc/fstab, with /etc usually being part of the / file system. So / needs to be mounted in order to read /etc/fstab.

However, if we can't enable journal_async_commit at boot time, then what's the point of putting options into /etc/fstab ?  If the journal_csum flag can't be enabled at boot or after mounting, then there is no way of enabling journal_async_commit.

The manual for mount says:
journal_async_commit - Commit block can be written to disk without waiting for descriptor blocks. If enabled, older kernels cannot mount the device. This will enable 'journal_checksum' internally.

So how to enable journal_async_commit at boot time?

Comment 5 Eric Sandeen 2015-02-10 14:50:46 UTC
I'll chime in more a bit later, but the rootflags=journal_async_commit parameter on the kernel commandline should do the trick.  This is in general the way to specify non-remountable options for the root fs.

-Eric

Comment 6 Conrad S 2015-02-10 15:05:31 UTC
If the journal_async_commit and journal_csum options are problematic for /, having a big fat warning in ext4 documentation (and the man page for mount) would be very useful -- it'll save a lot of headaches and head scratching.

I originally thought that kernel 3.18 had problems in its wifi drivers (as wifi no longer connected), only to discover by accident that this was really a file system problem.

Note that specifying fs options as kernel parameters does defeat the purpose of /etc/fstab. Is there a better solution to this?  (For example, permanently setting journal_async_commit at the file system level?)

Comment 7 Eric Sandeen 2015-02-10 16:02:38 UTC
They are not problematic for root, exactly.  They are problematic when specified during remounting.  Which is what happens for root. ;)

It's just that the root fs gets remounted with the options in fstab, and so those non-remountable options cannot be specified directly via fstab.

> Note that specifying fs options as kernel parameters does defeat the purpose of /etc/fstab.

well, no, not really.  Actually, it may be that whatever sets up /boot/init* environment should pick up the options out of fstab - I'm not sure.  If it doesn't, that might make for a nice RFE.

And yes, you can probably also set options into the filesystem with tune2fs; there is tune2fs -o mount-option as well as -E mount_opts="string"

Anyway, it would have been better if these options just got ignored with a warning on a remount, rather than failing the mount.  I'll propose that upstream.

Comment 8 Conrad S 2015-02-10 17:04:25 UTC
$ sudo tune2fs -o journal_async_commit /dev/sdb1
tune2fs 1.42.11 (09-Jul-2014)
Invalid mount option set: journal_async_commit

I'm not sure whether the -E option would work under 3.18. From manual page for tune2fs states:
-E extended-options
 ...
mount_opts=mount_option_string
Set  a  set  of default mount options which will be used when the file system is mounted.  Unlike the bitmask-based default mount options which can be specified with the -o option, mount_option_string is an arbitrary string with a maximum length of 63 bytes, which is stored in the superblock.

The ext4 file system driver will *** first apply the bitmask-based default options ***, and then parse the mount_option_string, before parsing the mount options passed from the mount(8) program.

I've added *** for emphasis.  If the journal_csum flag is not set in the default options, wouldn't 3.18 then refuse to apply journal_async_commit ?

Comment 9 Eric Sandeen 2015-02-10 17:44:10 UTC
If there are no bitmask-based defaults, then nothing is done in your *** step ***, and then the mount_option_string will be applied.

It's not a remount, it's just multiple steps in the original option parsing.

Sigh, this is made further confusing by the fact that the option is not shown in /proc/mounts due to lack of handling in ext4_show_options ...  o_O let me send a patch for that, too....

Anyway: AFAICT, there is not really a bug here; behavior was intentionally changed.  I'll argue for changing it back to be more forgiving, and fix up the other cruft that has shaken out while looking at it.

Comment 10 Eric Sandeen 2015-02-10 18:00:16 UTC
Oh, no - actually - show_options *does not show* options if they are set by default.

So if you set journal_async_commit with "-E mount_opts=" then it *won't* show up in /proc/mounts.  Of course ... o_O

but also! of course! it _does_ show up in 

# cat /proc/fs/ext4/loop0/options 
rw
delalloc
journal_checksum
journal_async_commit

...

Anyway, setting the default does work.  dmesg and the proc file shows that it mounts the way you want.

Comment 11 Conrad S 2015-02-11 06:02:59 UTC
Thanks for looking into this.


(In reply to Eric Sandeen from comment #9)
> Sigh, this is made further confusing by the fact that the option is not
> shown in /proc/mounts due to lack of handling in ext4_show_options ...  o_O
> let me send a patch for that, too....
> 
> Anyway: AFAICT, there is not really a bug here; behavior was intentionally
> changed.  I'll argue for changing it back to be more forgiving, and fix up
> the other cruft that has shaken out while looking at it.

Comment 12 Conrad S 2015-02-24 05:18:12 UTC
Is the fix going to be backported to kernel 3.18 and/or 3.19 ?

Comment 13 Josh Boyer 2015-02-24 13:55:10 UTC
I've not seen a fix yet.  If there is one, we can look at backporting it.

Comment 14 Eric Sandeen 2015-02-24 16:07:59 UTC
Sorry.  This should fix it:

http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/fs/ext4?id=2d5b86e048780c5efa7f7d9708815555919e7b05

I didn't cc: stable :(

Comment 15 Josh Boyer 2015-02-24 16:10:09 UTC
OK, thanks Eric.  I'll get this into f20/f21 today.

Comment 16 Fedora Update System 2015-03-02 12:34:25 UTC
kernel-3.18.8-201.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/kernel-3.18.8-201.fc21

Comment 17 Fedora Update System 2015-03-02 12:36:13 UTC
kernel-3.18.8-100.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/kernel-3.18.8-100.fc20

Comment 18 Fedora Update System 2015-03-04 10:24:41 UTC
Package kernel-3.18.8-100.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing kernel-3.18.8-100.fc20'
as soon as you are able to, then reboot.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-2990/kernel-3.18.8-100.fc20
then log in and leave karma (feedback).

Comment 19 Fedora Update System 2015-03-09 08:16:44 UTC
kernel-3.18.8-201.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 20 Fedora Update System 2015-03-09 21:13:53 UTC
kernel-3.18.9-100.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/kernel-3.18.9-100.fc20

Comment 21 Fedora Update System 2015-03-14 09:15:07 UTC
kernel-3.18.9-100.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.