Bug 2121106

Created attachment 1907386 [details]
anaconda.log

Created attachment 1907387 [details]
dbus.log

Created attachment 1907388 [details]
journal.log

Created attachment 1907389 [details]
ks-script-srl91e7l.log

Created attachment 1907390 [details]
program.log

Created attachment 1907391 [details]
storage.log

Created attachment 1907392 [details]
anaconda-ks.cfg

Created attachment 1907393 [details]
journal (from installed system)

Created attachment 1907394 [details]
rpm -qa (from installed system)

Proposing as a Final blocker:
"If a particular keyboard layout has been configured for the system, that keyboard layout must be used:
    When unlocking encrypted storage volumes during boot (but see footnotes)
    When logging in at a console"
https://fedoraproject.org/wiki/Fedora_37_Final_Release_Criteria#Keyboard_layout_configuration

Discussed during the 2022-08-29 blocker review meeting: [0]

The decision to classify this bug as both an "AcceptedBlocker (Final)" and an "AcceptedFreezeException (Final)" was made as it is a clear violation of the Final criteria, and since it affects the install process so can't be fixed with aupdate and would be annoying for Czech users, we also grant it Beta FE status.

[0] https://meetbot.fedoraproject.org/fedora-blocker-review/2022-08-29/f37-blocker-review.2022-08-29-16.01.txt

Edit to comment 11: "AcceptedFreezeException (Beta)"

From journal.log:

Aug 24 08:45:11 localhost-live org.fedoraproject.Anaconda.Modules.Localization[2468]: DEBUG:anaconda.modules.localization.localization:X Layouts are set to ['cz (qwerty)', 'us'].
...
Aug 24 08:45:32 localhost-live org.fedoraproject.Anaconda.Modules.Localization[2468]: INFO:anaconda.threading:Running Thread: AnaTaskThread-GetMissingKeyboardConfigurationTask-1 (140290654168768)
Aug 24 08:45:32 localhost-live org.fedoraproject.Anaconda.Modules.Localization[2468]: INFO:anaconda.modules.common.task.task:Get missing keyboard settings.
...
Aug 24 08:45:33 localhost-live systemd-localed[3288]: Changed X11 keyboard layout to 'cz,us' model '' variant 'qwerty,' options ''
Aug 24 08:45:33 localhost-live systemd-localed[3288]: Changing virtual console keymap to 'cz-us-qwertz' toggle ''
...
Aug 24 08:45:33 localhost-live org.fedoraproject.Anaconda.Modules.Localization[2468]: DEBUG:anaconda.modules.localization.localed:Missing virtual console keymap value cz-us-qwertz converted from ['cz (qwerty)', 'us'] X layouts
Aug 24 08:45:33 localhost-live org.fedoraproject.Anaconda.Modules.Localization[2468]: INFO:anaconda.threading:Thread Done: AnaTaskThread-GetMissingKeyboardConfigurationTask-1 (140290654168768)
Aug 24 08:45:33 localhost-live org.fedoraproject.Anaconda.Modules.Localization[2468]: DEBUG:anaconda.modules.localization.localization:Virtual console keymap is set to cz-us-qwertz.
Aug 24 08:45:33 localhost-live org.fedoraproject.Anaconda.Modules.Localization[2468]: DEBUG:anaconda.modules.localization.localization:X Layouts are set to ['cz (qwerty)', 'us'].

We call SetX11Keyboard of the org.freedesktop.locale1 DBus service [0] to convert the X11 keyboard layout to a virtual console keymap. The service chooses cz-us-qwertz, so I am not sure how to fix this on our side. Reassigning to systemd for further investigation.

[0] https://www.freedesktop.org/wiki/Software/systemd/localed/

Kamil, do you know when this last worked as expected?

OK, so, my guess is "probably never", because this ultimately comes around to the good old janky handwritten list of "matching" console and X layouts that used to be part of system-config-keyboard and which we're *still* lugging around as part of systemd:

https://github.com/systemd/systemd/blob/main/src/locale/kbd-model-map

as you can see, that has the console layout 'cz-us-qwertz' marked as a match for the X11 layout 'cz,us', with no 'variant' included.

I don't read Czech, so I don't know for sure, but I *think* the console map 'cz-qwerty' is the one you want, right? That should be 'switched' cz/us with QWERTY layout? If so, we should probably add a line to kbd-model-map:

cz-qwerty		cz,us	pc105		qwerty		terminate:ctrl_alt_bksp,grp:shifts_toggle,grp_led:scroll

and that should make it work as expected. I think.

(In reply to Adam Williamson from comment #15)
> OK, so, my guess is "probably never"

I tested Fedora 30 and it's broken as well, so "probably never" might be a decent estimate.

I'll look at the low-level details shortly.

Yeah, I forgot to elaborate on that a bit. So basically systemd will first look for one of the auto-converted xkb maps that is an exact match for the config. For the config you asked for, it will look for a map called "cz,us-qwerty.map", and not surprisingly, it won't find one. So after that it'll look for a legacy layout, and it does that by looking at that kbd-model-map file. It has a heuristic for parsing that file. First off, it'll give an entry that matches the 'layout' value - in our case, "cz,us" - 10 points, so right off the bat, the one existing line that matches that:

# consolelayout		xlayout	xmodel		xvariant	xoptions
cz-us-qwertz		cz,us	pc105		-		terminate:ctrl_alt_bksp,grp:shifts_toggle,grp_led:scroll

gets ten points. Any line that matches just the first component, split by commas, gets five points - so this line gets five points:

cz-lat2			cz	pc105		qwerty		terminate:ctrl_alt_bksp

Also if there was a line with the xlayout value as e.g. "cz,fr" it'd get one point, but that's not the case here. After that, the line gets one extra point if the model matches or the requested model is empty; two extra points if the model check passes and the variant matches; or three extra points if the model check passes and the variant and options match.

I *think* in this case, the cz-us-qwertz line gets 11 points and the cz-lat2 line gets 6. Looking at your logs, the variant is shown as "qwerty," (with a trailing comma). The variant in the cz-lat2 line is "qwerty". So that doesn't match, and there's no fuzzing for commas in the systemd matching code. We'll have to look into that in more detail. If those did match, the cz-lat2 line would get 7 points, so it still would not overtake the cz-us-qwertz line.

systemd then takes the line with the highest score. If two lines have the same score, the first encountered will win. If the score is 10 or higher, it goes ahead and uses that line's specified console layout (cz-us-qwertz, in this case). If the score is less than 10, it will go back and check auto-converted layouts again, this time looking for one that just matches the first part of the layout specifier, and if it finds one, it'll prefer that. In our case, it'll go with cz-us-qwertz .

Now if we added a line like this:

cz-qwerty		cz,us	pc105		qwerty,		terminate:ctrl_alt_bksp,grp:shifts_toggle,grp_led:scroll

(note I included the trailing comma this time, but we really need to figure out what's going on with that) - that line should score 12 points, because now the variant matches the query. So that line would out-score the cz-us-qwertz line, and systemd would pick this new line, and go with the `cz-qwerty` console layout.

All this stuff is in src/locale/localed-util.c and src/locale/localed.c in systemd source.

The *purpose* of all this logic is so we can actually find and prefer the "legacy" switched console layouts, when appropriate. For cases like Russian it's absolutely necessary, because the "native" layout can't type ASCII characters. So we *really need* a way to say "hmm, if the requested X config was ru,us , that means we need a switched Russian/US console layout". Otherwise we'd just wind up giving people the xkb-converted ru.map.gz layout, which can't type ASCII characters.

By selecting both Czech and US layouts, you're effectively triggering that mechanism, because now the X layout specifier is "cz,us". I'm assuming that what you'd logically want at the console in this case is a CZ/US switched qwerty console layout, and AFAICS, the legacy cz-qwerty.map.gz fits that bill.

In typing this up, I notice another complication here: there are two console cz-qwerty layouts. The xkb-converted /usr/lib/kbd/keymaps/xkb/cz-qwerty.map.gz and the "legacy" switched /usr/lib/kbd/keymaps/legacy/i386/qwerty/cz-qwerty.map.gz . I don't know what happens in this case if you just request "cz-qwerty" - I don't know which one will get loaded. I also don't know if we have a way to express a preference for the legacy one. :| We might have to do something like dropping the xkb-converted one in this case...

(My ignorance here, but why would you want to have both cz,us rather than just cz?)

(In reply to Adam Williamson from comment #17)
> Yeah, I forgot to elaborate on that a bit. So basically ...

Thanks a lot for very detailed description. This is so... bananas. We should all go grow vegetables and wait for the industry to start from scratch.


> Now if we added a line like this:
> 
> cz-qwerty		cz,us	pc105		qwerty,	
> terminate:ctrl_alt_bksp,grp:shifts_toggle,grp_led:scroll

I edited /usr/share/systemd/kbd-model-map on Workstation Live and added this line directly below the "cz-us-qwertz" line. Then installed the system. It worked - after a reboot, I have "cz-qwerty" keymap active both during disk unlock and in the booted system.

$ localectl 
   System Locale: LANG=en_US.UTF-8
       VC Keymap: cz-qwerty
      X11 Layout: cz,us
     X11 Variant: qwerty,

$ cat /etc/vconsole.conf 
KEYMAP="cz-qwerty"
FONT="eurlatgr"


> (note I included the trailing comma this time, but we really need to figure
> out what's going on with that)

If I didn't include the comma, systemd selected "cz-us-qwertz".

> By selecting both Czech and US layouts, you're effectively triggering that
> mechanism, because now the X layout specifier is "cz,us". I'm assuming that
> what you'd logically want at the console in this case is a CZ/US switched
> qwerty console layout, and AFAICS, the legacy cz-qwerty.map.gz fits that
> bill.

When it comes to keyboard layouts, I have expectations of anyone's grandma. I.e. it should just work (tm). Pressing the keys should make the right letters appear. I don't even know how to switch keymaps in a VT, and people who know it can probably configure it correctly themselves. So the important part is, I believe, is that the primary X11 keymap should exactly match the chosen vconsole keymap (without any switching). Not sure if that's currently possible.

> In typing this up, I notice another complication here: there are two console
> cz-qwerty layouts. The xkb-converted
> /usr/lib/kbd/keymaps/xkb/cz-qwerty.map.gz and the "legacy" switched
> /usr/lib/kbd/keymaps/legacy/i386/qwerty/cz-qwerty.map.gz . I don't know what
> happens in this case if you just request "cz-qwerty" - I don't know which
> one will get loaded.

I installed the system with just "Czech (qwerty)" keymap, and it correctly selected cz-qwerty vconsole map. But I don't know how to distinguish which one. Either way, things seem to work as expected in this case.

$ cat /etc/vconsole.conf 
KEYMAP="cz-qwerty"
FONT="eurlatgr"

$ localectl 
   System Locale: LANG=en_US.UTF-8
       VC Keymap: cz-qwerty
      X11 Layout: cz
     X11 Variant: qwerty


(In reply to Jens Petersen from comment #18)
> (My ignorance here, but why would you want to have both cz,us rather than
> just cz?)

I believe it's quite common here. If I want to type lots of numbers or do coding (i.e. use a lots of different brackets, dollars and similar symbols), it's much easier to do with a US keyboard. Also many people are used to the US keyboard from the past when the internationalization of keyboards and operating systems wasn't that good or was outright missing. So it's common to have both CZ and US keyboards installed and switch depending on user's preference or current activity.

> Thanks a lot for very detailed description. This is so... bananas. We should all go grow vegetables and wait for the industry to start from scratch.

Yeah, the fundamental problem is this mismatch between how console layouts work and how xkb works. The current console keyboard stuff doesn't really have a concept of quickly toggling between different layouts, except by literally running `loadkeys whatever` (assuming you can type that on the current layout). The way "switched" console layouts really work is through the "levels" concept...like, when you hold down shift or hit caps lock, the layout is on a different "level", meaning the keys produce different characters. Switched console layouts have four (usually) levels - layout 1 regular, layout 1 shifted, layout 2 regular, layout 2 shifted - and the 'toggle' key combo just kinda "locks" the layout to the layout 2 levels, in the same sort of way that capslock "locks" to the shifted level. So we have this fundamental problem that if you want a "switched" layout, for kbd that *has* to be one single kbd layout that defines both; for xkb it *has* to be done by configuring two layouts and a way to quickly switch between them. We can never make the two match.

The best "fix" for this whole mess would be for someone to redo the kernel VT stuff so it uses xkb layouts and supports xkb-style switching. Then we could just have one set of layouts for everything. Unfortunately, projects along these lines have shown up but never seem to get finished or merged. kmscon got abandoned in 2014, systemd-consoled in 2015. https://github.com/systemd/systemd/pull/747 seems to be the informal discussion point for "wouldn't it be nice if we could finally do that" conversations.

I also dug up this old blog post of mine that more or less explains how things work currently: https://www.happyassassin.net/posts/2013/11/23/keyboard-layouts-in-fedora-20-and-previously/

> When it comes to keyboard layouts, I have expectations of anyone's grandma. I.e. it should just work (tm). Pressing the keys should make the right letters appear. I don't even know how to switch keymaps in a VT, and people who know it can probably configure it correctly themselves. So the important part is, I believe, is that the primary X11 keymap should exactly match the chosen vconsole keymap (without any switching). Not sure if that's currently possible.

For kbd layouts, the switch combo is defined *in the layout itself* and varies from layout to layout. I think knowing how to switch is kinda folk knowledge for Linux users in each country, more or less. The details of how the layout works are usually documented at the top of the file. For the legacy cz-qwerty layout, it has this in Czech:

# POZOR: Tato klavesova mapa obsahuje ve skutecnosti 2 (dve) klavesnice
#        Primarni je CESKA
#        Sekundarni je US
#        Prepinani se provadi pomoci klavesy "Pause"
#                  ktera funguje jako "ShiftR_Lock"
#        CESKA: Control-Klavesa, Alt-Klavesa, Alt-Shift-Klavesa
#               => funguje stejne jako v US klavesnici
#        US:    AltGr-Klavesa, AltGr-Shift-Klavesa
#               => funguje stejne jako v CESKE klavesnici
#           (i dead klavesy na AltGr-2 az AltGr-9, AltGr-0, AltGr--, AltGr-=)
#               Navic klavesa "PrintScreen" funguje jako carka a hacek

which I guess means Czech is the primary layout, US is the switched layout, and at least according to Google, says you press Pause to switch.

> I installed the system with just "Czech (qwerty)" keymap, and it correctly selected cz-qwerty vconsole map. But I don't know how to distinguish which one. Either way, things seem to work as expected in this case.

Per my blog post above, I would guess you probably got the xkb-converted layout, unfortunately. Which would mean it's not switchable. I guess you could tell by pressing the Pause key and seeing if anything changes. To test using the legacy layout instead, I think you can pass `loadkeys` an exact filename: `loadkeys /usr/lib/kbd/keymaps/legacy/i386/qwerty/cz-qwerty.map.gz`. Then see how that one works, and if switching works.

If the 'legacy' cz-qwerty works as well as the xkb-converted one and also allows switching, we could potentially delete or rename the xkb-converted one in the kbd spec so the legacy one would be preferred. Although there's the problem that some people might not install the 'legacy' package since it's, well, legacy. (And I just noticed recently that Silverblue doesn't include it, which is kind of a bug I should get fixed). Sigh, I hate all this stuff.

> press Pause to switch

Those funny keys, who still has them? (Turns out Fn+P sends Pause on Thinkpads...)

> Per my blog post above, I would guess you probably got the xkb-converted layout, unfortunately. Which would mean it's not switchable.

I can confirm this, I get the non-switchable one.


So if I understand this properly, we need to:
1) add cz-qwerty into /usr/share/systemd/kbd-model-map (and optionally figure out why the extra comma is needed)
2) figure out if the xkb-converted cz-qwerty keymap should be deleted, the legacy one preferred or what

Ad 2), please note that I can write all English characters using the xkb-converted keymap, when pressing Shift, AltGr or Shift+AltGr and the right key. So perhaps 2) is really not a problem?

Resetting the blocker discussion:
https://pagure.io/fedora-qa/blocker-review/issue/867#comment-816832

Adam or Kamil, can you submit a pull request upstream with the additional line?
I'll take care of propagating the patch back into Fedora.

I'm still kinda thinking about exactly what to do, honestly. This is a pretty fuzzy case. It's not as 'simple' as e.g. Russian, where we pretty much know everyone will want the legacy switched layout. In this case the switching is more of a convenience than a necessity.

It's also more complicated because this isn't one of our "pre-canned" scenarios, where we're dealing with the layout config you get just by picking a language in the anaconda list (we can be pretty opinionated about what happens then); this is a "roll your own" scenario, where Kamil picked a custom preferred X layout config and we're essentially trying to guess what console layout he'd like to match that.

My kinda immediate choice would be to wipe the xkb-converted 'cz-qwerty' and use the "legacy" one instead. If it does everything the converted one does *plus* let you switch to a US layout for convenience, it's better, right? Except, a couple of things...

1. What if someone not expecting switching hits Pause by accident and gets very confused?
2. We split off kbd-legacy. I really don't like that, I think it was a mistake. But we did it. That means if we do this, somebody could wind up with 'cz-qwerty' in vconsole.conf but no 'cz-qwerty' layout actually installed. Though really, that applies to *any* case where we use kbd-model-map to configure a layout that's in -legacy, which is why I think this was a bad idea. As a side note on this, I really need to check what happens if you install Silverblue in Russian; I'm guessing it's bad.
3. Doing this would have another 'interesting' consequence: somebody who configured *only* the Czech qwerty xkb layout (without also a US xkb layout) would *also* get the switched console layout. On that path, systemd's "find an xkb-converted layout" code looks for a layout called "cz-qwerty". And of course, if we wipe the xkb-converted 'cz-qwerty' (which would be the "right thing" to find, on that path), loadkeys will load the legacy one, with switching. So we have the potential for both points 1 and 2 to happen *without the user even having requested a switched layout at all*, in that case.

So, uh, I'm really not sure. I'm probably going to file a bug asking that we revert the -legacy split, honestly. The more I think about it the more I think that was a terrible idea. In some ways I like the idea of renaming the 'legacy' layout to cz-qwerty-switched and having kbd-model-map select that layout in this case, but the problem with that is we'd have to get it renamed or at least included under both names in upstream kbd, or else we'd have to carry the kbd-model-map line as a downstream patch as it wouldn't be safe for any other distro that didn't do the rename. Ugh.

https://bugzilla.redhat.com/show_bug.cgi?id=2127513 - bug to drop the -legacy subpackage split for kbd.

Discussed during the 2022-09-19 blocker review meeting: [0]

The decision to classify this bug as a "RejectedBlocker (Final)" and an "AcceptedFreezeException (Final)" was made as the new details about how specific this bug is (you have to manually choose this exact combination of X layouts to trigger it), and the difficulty of fixing it fully, we agreed this is too conditional of a violation to count as a blocker. However, we grant it a freeze exception so we have time to land at least the moderate improvement that should be possible.

[0] https://meetbot.fedoraproject.org/fedora-blocker-review/2022-09-19/f37-blocker-review.2022-09-19-16.00.txt

So, https://bodhi.fedoraproject.org/updates/FEDORA-2022-ac8991ae06 makes kbd hard-require kbd-legacy again, which should avoid concerns about the layout from -legacy disappearing.

So, at this point, *technically* the 'best' option would I think be to do something like this in kbd-model-map:

cz-qwerty-switched	cz,us	pc105		qwerty,		terminate:ctrl_alt_bksp,grp:shifts_toggle,grp_led:scroll

and rename the legacy layout to 'cz-qwerty-switched'. That would get people exactly what they asked for.

The problem with this is, we would either have to convince kbd upstream to rename the layout (which seems highly unlikely) or carry the systemd change downstream forever. There's an obvious alternative:

cz-qwerty		cz,us	pc105		qwerty,		terminate:ctrl_alt_bksp,grp:shifts_toggle,grp_led:scroll
cz-qwerty-xkb		cz	pc105		qwerty,		terminate:ctrl_alt_bksp,grp:shifts_toggle,grp_led:scroll

and rename the xkb-converted layout to 'cz-qwerty-xkb'. But there are problems with that too.

I think in the end it might be the best idea after all to just do:

cz-qwerty		cz,us	pc105		qwerty,		terminate:ctrl_alt_bksp,grp:shifts_toggle,grp_led:scroll

and wipe the xkb-converted layout in the kbd spec file, so anyone who picks cz with qwerty should end up with the switched legacy layout, and just live with the potential of someone who doesn't expect switching hitting the Pause key. This, after all, would've been a possibility before we started doing the xkb-converted layouts in the first place...

I also still need to figure out where the damn comma is coming from...

OK, I think I see where the comma comes from, it's a bug in anaconda. PR coming.

https://github.com/rhinstaller/anaconda/pull/4355 should fix the comma problem.

and https://github.com/systemd/systemd/pull/24795 adds the line to kbd-model-map.

Thanks, Adam. I've played with cz-qwerty again, xkb and legacy one, and I think we might actually want to leave the preference as it is (xkb gets chosen). While it's true that the legacy one is switchable, which adds certain convenience for people who know how to perform the switch, it is much worse when it comes to typing US non-letter characters when the CZ layout is active. You see, this is what a typical physical Czech keyboard looks like:
https://i.imgur.com/YxCgLTn.jpg

On non-letter keys, you can see two columns, the left one being a typical US layout and the right one being a CZ layout. You can type most of the US characters from the CZ layout by pressing AltGr(+Shift) + the right key from the US layout. So AltGr+4 is $, AltGr+7 is &, etc. You don't need to remember anything, you just look at the physical keys. This works the same on both Linux and Windows. In VT, this also works with the xkb-converted keymap, because it's, well, converted from xkb. But with the legacy keymap, the approach to write these characters (without switching to the full US keymap through Pause) is completely different. $ is AltGr+; , & is AltGr+C , [] are on AltGr+FG and {} on AltGr+BN instead of where you can physically see them on the picture (right of the letter P).

So, for a modern day Linux user, who is used to the xkb keymap from a graphical session and ends up on a VT by accident or very rarely, I believe the xkb-converted keymap is much better, because the layout matches what he/she is used to.

So if we leave xkb+legacy as it is right now (xkb preferred), I think all we need to do is to make the two PRs above accepted, and we should be done? Thanks a lot for investigating all this.

"So if we leave xkb+legacy as it is right now (xkb preferred), I think all we need to do is to make the two PRs above accepted, and we should be done? Thanks a lot for investigating all this."

Yep, that's correct. I was going to send a PR proposing that we delete the xkb-converted layout, but based on your explanation, I agree that seems like a bad idea and we should just keep and prefer the xkb-converted one.

FEDORA-2022-a3bf337a61 has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2022-a3bf337a61

FEDORA-2022-a3bf337a61 has been pushed to the Fedora 38 stable repository.
If problem still persists, please make note of it in this bug report.

Re-opening as this is filed against 37. Zbigniew, can we please get this backported for F37?

FEDORA-2022-7cf1869073 has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2022-7cf1869073

The same comment as in #2129343: Bodhi closes bugs automatically when the update for F38 is created.
I wish there was a way to instruct bodhi not to do that in some cases. If somebody knows how to this
or could make it happen, that'd make it much nicer to submit updates that fix bugs that apply to
multiple releases.

It's an option in Bodhi, but I don't know if there's a way to set it for auto-created Rawhide updates :/

FEDORA-2022-7cf1869073 has been pushed to the Fedora 37 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2022-7cf1869073`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-7cf1869073

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2022-7cf1869073 has been pushed to the Fedora 37 stable repository.
If problem still persists, please make note of it in this bug report.

I tested the latest Live image and I can confirm this is fixed now.