Bug 2238982

Created attachment 1988841 [details]
screencast

Created attachment 1988842 [details]
journal

Propose as a Final FE as it acts like a disaster,and may cause data loss.Besides,it kind of violates :https://fedoraproject.org/wiki/Fedora_39_Final_Release_Criteria#Default_application_functionality

Can you test if this was the same in F38? I wonder if it's to do with the compression feature...

Can I get a btrfs filesystem usage /home?

> Can you test if this was the same in F38? I wonder if it's to do with the compression feature...

Yes,I also saw this problem in F38

> Can I get a btrfs filesystem usage /home?

Sure,here it is:

lnie@localhost-live:~$ sudo btrfs filesystem usage /home
Overall:
    Device size:   8.00GiB
    Device allocated:   8.00GiB
    Device unallocated:   1.00MiB
    Device missing:     0.00B
    Device slack:     0.00B
    Used:   6.00GiB
    Free (estimated):   1.99GiB (min: 1.99GiB)
    Free (statfs, df):   1.99GiB
    Data ratio:      1.00
    Metadata ratio:      1.00
    Global reserve:   5.98MiB (used: 0.00B)
    Multiple profiles:        no

Data,single: Size:7.99GiB, Used:6.00GiB (75.09%)
   /dev/vda1   7.99GiB

Metadata,single: Size:8.00MiB, Used:6.58MiB (82.23%)
   /dev/vda1   8.00MiB

System,single: Size:4.00MiB, Used:16.00KiB (0.39%)
   /dev/vda1   4.00MiB

Unallocated:
   /dev/vda1   1.00MiB

Created attachment 1989292 [details]
screencast from /home <10G system

Created attachment 1989293 [details]
screencast from /home > 10G system

As you can see from the attached screencasts,on home<10G system,when you try to remove/create some file, the horrible "Read-only system" message is printed,though  you will find the file you were trying to remove is removed after a reboot. It's pretty confusing in  a pretty bad way.I'm thinking maybe we should also consider this as a blocker:(
For developer,I saw "writing error in swap file" is printed when I try to save a file on home>10G system,while "can't open file for writing" on home<10G system

Huh interesting, we're not allocating enough metadata.  We have regression tests for this, let me see if I can reproduce it locally.

Can I see the output from the following commands

mount
swapon
btrfs inspect-internal dump-super <device that /home lives on>

and then for the case that the file system flips read only can I get the dmesg output as well?

I'm seeing a few issues here, and I've nailed down a few of them, and some of them I'm just guessing at because I don't have all the information.

1) There's a reporting problem.  In the case where it shows you have 2gib free, you have no metadata space, and that's why you're getting an -ENOSPC.  'df' is not showing the right amount because you filled up metadata space so much that it messed up our internal "does this fs have enough metadata to do anything?" logic and we're showing you the full data you have free, but should be showing 0 because there's no metadata.  This is why we recommend using "btrfs filesystem usage", as it shows the whole answer.  However this is a bug, and I've sent the patch upstream, the patch is here https://lore.kernel.org/linux-btrfs/a9e6b02e9eb7a0532c401a898661b0511c31d0e8.1695047676.git.josef@toxicpanda.com/

2) We really shouldn't let you get that close to full on metadata, we're failing to anticipate the data usage.  In your case you're using a 512mb chunk size, and this results in a lot of checksums.  But it also creates this racy environment where we think we have plenty of slack space, until suddenly we don't.  For data we assume reservation size == actual use (which it is in this case, but isn't in compression), so we're allowing metadata to overcommit in a situation where we could lose that slack space and then we're in trouble.  I was able to reproduce this particular behavior and fixed the underlying issue here https://lore.kernel.org/linux-btrfs/b97e47ce0ce1d41d221878de7d6090b90aa7a597.1695065233.git.josef@toxicpanda.com/

3) Flipping read only.  This is concerning and I don't have the dmesg to validate my theory, but I assume it is a result of the situation that is caused by problem #2.  In my testing I got myself down to <1mib of free metadata, and if you hit the timing just right we could still think we have room for the overcommit and then fail to make a metadata allocation and then flip read only because of that.  The fix is to fix the underlying cause, which is what #2 fixes.  However that's just my best guess, I'd need to see the dmesg to validate we're flipping readonly in the allocation path.

Created attachment 1989483 [details]
mount swapon btrfs inspect-internal dump-super and dmesg

I have a maybe silly question,as you can see on a /home< 10G btrfs system, metadata-size/data-size is  ~ 0.00097,while on a /home < 10G btrfs system that number is 0.0178, ten times? isn't that a problem?

lnie@fedora:~$  sudo btrfs filesystem usage /home
[sudo] password for lnie:
Overall:
    Device size:  15.00GiB
    Device allocated:  15.00GiB
    Device unallocated:   1.00MiB
    Device missing:     0.00B
    Device slack:     0.00B
    Used:  14.74GiB
    Free (estimated):   8.01MiB (min: 8.01MiB)
    Free (statfs, df):   8.01MiB
    Data ratio:      1.00
    Metadata ratio:      1.00
    Global reserve:  15.23MiB (used: 0.00B)
    Multiple profiles:        no

Data,single: Size:14.74GiB, Used:14.73GiB (99.95%)
   /dev/vda1  14.74GiB

Metadata,single: Size:264.00MiB, Used:15.69MiB (5.94%)
   /dev/vda1 264.00MiB

System,single: Size:4.00MiB, Used:16.00KiB (0.39%)
   /dev/vda1   4.00MiB

Unallocated:
   /dev/vda1   1.00MiB

Ok perfect dmesg shows that you ran out of metadata space, so my second fix will resolve that.

The >10gib fs is doing what I want, allocating another metadata block group.

Btrfs divides the disk up into chunks (called block groups) and dedicates these chunks to either Data, Metadata, or System chunks.  For smaller file systems like yours we try to make the metadata chunks smaller, 256mib at a time.  As you've noticed this means that you're losing about 1% of your disk to metadata.  In the case where you're just blowing large data chunks onto your device this doesn't make much sense, unfortunately we don't know that's what's going to happen so we have to make it work for the general case.  If you decided to take a bunch of snapshots you'd use more metadata, or if you had lots of small files you'd also use a lot more metadata.  We avoid allocating new chunks except when we absolutely need to in order to reduce the amount of metadata overhead there is.  In your case we actually were too conservative and ended up causing other problems.  It's a tricky balancing act with different tradeoffs on either side.

The two fixes Josef mentioned are queued up for 6.5.5: https://gitlab.com/cki-project/kernel-ark/-/commits/fedora-6.5?ref_type=heads

FEDORA-2023-0defc5c6ec has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-0defc5c6ec

FEDORA-2023-4c8291ba6a has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-4c8291ba6a

FEDORA-2023-3100e4d61c has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2023-3100e4d61c

FEDORA-2023-3100e4d61c has been pushed to the Fedora 37 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-3100e4d61c`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-3100e4d61c

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2023-0defc5c6ec has been pushed to the Fedora 39 stable repository.
If problem still persists, please make note of it in this bug report.

FEDORA-2023-4c8291ba6a has been pushed to the Fedora 38 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-4c8291ba6a`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-4c8291ba6a

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2023-3100e4d61c has been pushed to the Fedora 37 stable repository.
If problem still persists, please make note of it in this bug report.

FEDORA-2023-4c8291ba6a has been pushed to the Fedora 38 stable repository.
If problem still persists, please make note of it in this bug report.