Created attachment 989921 [details]
dmesg

dmesg from kernel 3.18.5

A symptom of the problem (extracted from dmesg)
[    4.129492] EXT4-fs (sdb1): changing journal_checksum during remount not supported

Changing the /etc/fstab entry to also contain "journal_checksum" seems not to help.

That is upstream commit

commit 6b992ff25658367089db4a82666e232b65d55eae
Author: Darrick J. Wong <darrick.wong>
Date:   Thu Oct 30 10:53:16 2014 -0400

    ext4: disallow changing journal_csum option during remount
    
    ext4 does not permit changing the metadata or journal checksum feature
    flag while mounted.  Until we decide to support that, don't allow a
    remount to change the journal_csum flag (right now we silently fail to
    change anything).
    
    Signed-off-by: Darrick J. Wong <darrick.wong>
    Signed-off-by: Theodore Ts'o <tytso>


which indeed landed in 3.18.  Note that "works without problems" on 3.17 in this case means it didn't do anything at all.

I'm not sure this is a bug, but I'll let the fs people weigh in.

Technically this is not supposed to be a remount, but the "first" mount at boot time.

Obviously there is the cyclic dependency of requiring /etc to be accessible in order to read /etc/fstab, with /etc usually being part of the / file system. So / needs to be mounted in order to read /etc/fstab.

However, if we can't enable journal_async_commit at boot time, then what's the point of putting options into /etc/fstab ?  If the journal_csum flag can't be enabled at boot or after mounting, then there is no way of enabling journal_async_commit.

The manual for mount says:
journal_async_commit - Commit block can be written to disk without waiting for descriptor blocks. If enabled, older kernels cannot mount the device. This will enable 'journal_checksum' internally.

So how to enable journal_async_commit at boot time?

I'll chime in more a bit later, but the rootflags=journal_async_commit parameter on the kernel commandline should do the trick.  This is in general the way to specify non-remountable options for the root fs.

-Eric

If the journal_async_commit and journal_csum options are problematic for /, having a big fat warning in ext4 documentation (and the man page for mount) would be very useful -- it'll save a lot of headaches and head scratching.

I originally thought that kernel 3.18 had problems in its wifi drivers (as wifi no longer connected), only to discover by accident that this was really a file system problem.

Note that specifying fs options as kernel parameters does defeat the purpose of /etc/fstab. Is there a better solution to this?  (For example, permanently setting journal_async_commit at the file system level?)

They are not problematic for root, exactly.  They are problematic when specified during remounting.  Which is what happens for root. ;)

It's just that the root fs gets remounted with the options in fstab, and so those non-remountable options cannot be specified directly via fstab.

> Note that specifying fs options as kernel parameters does defeat the purpose of /etc/fstab.

well, no, not really.  Actually, it may be that whatever sets up /boot/init* environment should pick up the options out of fstab - I'm not sure.  If it doesn't, that might make for a nice RFE.

And yes, you can probably also set options into the filesystem with tune2fs; there is tune2fs -o mount-option as well as -E mount_opts="string"

Anyway, it would have been better if these options just got ignored with a warning on a remount, rather than failing the mount.  I'll propose that upstream.

$ sudo tune2fs -o journal_async_commit /dev/sdb1
tune2fs 1.42.11 (09-Jul-2014)
Invalid mount option set: journal_async_commit

I'm not sure whether the -E option would work under 3.18. From manual page for tune2fs states:
-E extended-options
 ...
mount_opts=mount_option_string
Set  a  set  of default mount options which will be used when the file system is mounted.  Unlike the bitmask-based default mount options which can be specified with the -o option, mount_option_string is an arbitrary string with a maximum length of 63 bytes, which is stored in the superblock.

The ext4 file system driver will *** first apply the bitmask-based default options ***, and then parse the mount_option_string, before parsing the mount options passed from the mount(8) program.

I've added *** for emphasis.  If the journal_csum flag is not set in the default options, wouldn't 3.18 then refuse to apply journal_async_commit ?

If there are no bitmask-based defaults, then nothing is done in your *** step ***, and then the mount_option_string will be applied.

It's not a remount, it's just multiple steps in the original option parsing.

Sigh, this is made further confusing by the fact that the option is not shown in /proc/mounts due to lack of handling in ext4_show_options ...  o_O let me send a patch for that, too....

Anyway: AFAICT, there is not really a bug here; behavior was intentionally changed.  I'll argue for changing it back to be more forgiving, and fix up the other cruft that has shaken out while looking at it.

Oh, no - actually - show_options *does not show* options if they are set by default.

So if you set journal_async_commit with "-E mount_opts=" then it *won't* show up in /proc/mounts.  Of course ... o_O

but also! of course! it _does_ show up in 

# cat /proc/fs/ext4/loop0/options 
rw
delalloc
journal_checksum
journal_async_commit

...

Anyway, setting the default does work.  dmesg and the proc file shows that it mounts the way you want.

Thanks for looking into this.


(In reply to Eric Sandeen from comment #9)
> Sigh, this is made further confusing by the fact that the option is not
> shown in /proc/mounts due to lack of handling in ext4_show_options ...  o_O
> let me send a patch for that, too....
> 
> Anyway: AFAICT, there is not really a bug here; behavior was intentionally
> changed.  I'll argue for changing it back to be more forgiving, and fix up
> the other cruft that has shaken out while looking at it.

Is the fix going to be backported to kernel 3.18 and/or 3.19 ?

I've not seen a fix yet.  If there is one, we can look at backporting it.

Sorry.  This should fix it:

http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/fs/ext4?id=2d5b86e048780c5efa7f7d9708815555919e7b05

I didn't cc: stable :(

OK, thanks Eric.  I'll get this into f20/f21 today.

kernel-3.18.8-201.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/kernel-3.18.8-201.fc21

kernel-3.18.8-100.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/kernel-3.18.8-100.fc20

Package kernel-3.18.8-100.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing kernel-3.18.8-100.fc20'
as soon as you are able to, then reboot.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-2990/kernel-3.18.8-100.fc20
then log in and leave karma (feedback).

kernel-3.18.8-201.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.

kernel-3.18.9-100.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/kernel-3.18.9-100.fc20

kernel-3.18.9-100.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.