(In reply to Adam Williamson from comment #0)
> When upgrading a system with glibc-all-langpacks installed from Fedora 29 or
> Fedora 30 to Fedora Rawhide, the file /usr/lib/locale/locale-archive somehow
> disappears entirely.
> 
> To reproduce, simply install Fedora 29 or Fedora 30, ensure
> glibc-all-langpacks is installed, and upgrade to Fedora Rawhide (using dnf
> system-upgrade). Then run 'rpm -V glibc-all-langpacks' and 'ls -l
> /usr/lib/locale/locale-archive.tmpl'. These show that the file is missing.
> 
> This seems to be the cause of
> https://bugzilla.redhat.com/show_bug.cgi?id=1716691 .

Thanks for filing this Adam.

As I noted here:
https://bugzilla.redhat.com/show_bug.cgi?id=1716691#c22

This is probably a transitional issue.

I think we're going to have to find a way to transition between the old and new package.

Perhaps by releasing changes in the current active Fedora branches that allow the upgrade to happen smoothly.

Oh. Hey. I bet I know what this is.

Before https://src.fedoraproject.org/rpms/glibc/c/50bcae98df7d383733d4d6d896b9e1e829914eb0?branch=master , this was glibc-all-langpacks %postun:

%postun all-langpacks -p <lua>
-- In the postun we always remove the locale cache.
-- We are being uninstalled and if this is an upgrade
-- then the new packages template will be used to
-- recreate a new copy of the cache.
os.remove("%{_prefix}/lib/locale/locale-archive")

now let's remind ourselves of RPM scriptlet ordering:

https://docs.fedoraproject.org/en-US/packaging-guidelines/Scriptlets/

so when we update from an old-style glibc-all-langpacks (e.g. the ones in F29 or F30) to the new-style one in Rawhide, we do %pretrans, the new package %pre, then we install the new package (with its /usr/lib/locale/locale-archive file), then we do the new package's %post, then the old package's %preun, then we remove the old package...then we do the old package's %postun, which deletes /usr/lib/locale/locale-archive .

Whoops.

Off the top of my head I dunno how we should fix this, but that's what I reckon the problem is.

Proposing as a Beta blocker, as I'd be pretty surprised if all locales suddenly disappearing during an upgrade didn't cause lots of problems...

I've also see a variant of this which results in no new languages being brought in.

 Upgrading        : glibc-all-langpacks-2.29.9000-23.fc31.x86_64                               1/38 
warning: /usr/lib/locale/locale-archive saved as /usr/lib/locale/locale-archive.rpmsave

(In reply to Adam Williamson from comment #4)
> Proposing as a Beta blocker, as I'd be pretty surprised if all locales
> suddenly disappearing during an upgrade didn't cause lots of problems...

Agreed, we'll have this sorted before then.

I saw that line in the logs for my case also (and that .rpmsave file really is created and is there on the system when the upgrade completes). I think it's basically rpm getting confused by you changing the file from %ghost %config to...you know...just a regular file, thinking it's going away, and 'saving' it. You normally get that message when you have a package with a %config file, and you upgrade to a newer version of the package with that file removed - rpm saves the file just in case you might've needed anything from it.

I don't think *that* particular mechanism kicking in would cause any problems (besides this unnecessary .rpmsave file wasting a couple hundred megs of disk space), but hey, IMBW.

Actually, thinking it through a bit more, it might be entirely intentional/correct that this mechanism kicks in when you change a file from %config to a regular file. After all, that means the file will now get overwritten; if it was a config file, it's reasonable to think it may have some local changes that the user might still need, and so it should not be thrown away entirely but the rpmsave backup created.

I think we should just patch glibc to use a different path for the locale archive.  It's not a public interface because the format is undocumented and internal to glibc.

Carlos, what do you think?

(In reply to Florian Weimer from comment #9)
> I think we should just patch glibc to use a different path for the locale
> archive.  It's not a public interface because the format is undocumented and
> internal to glibc.
> 
> Carlos, what do you think?

This is one of the simplest options.

No user should be depending on the location of the locale archive.

I suggest we keep it in the same path but rename the name of the file e.g. locale-archive.2, something obviously the same as the locale-archive but marked as being "the second" and the tooling will use this new named file.

Then configure the glibc build to use the new alternative name (we just need to plumb that through).

I've looked at altering the existing installs to honour this and we introduce more complexity than a simple file rename.

For example, who deletes the old locale-archive?

When you remove glibc-all-langpacks the %ghost'd locale-archive is removed vi os.remove in %post. If we don't do this then the file stays around and that's wrong. So we we would need a marker to indicate "new style" locale-archive and avoid the removal in that case, this marker might as well be the name of the file. This is already much more complicated than changing the name to locale_archive.

I support a name change and I propose "locale-archive.2" and we can bump to .3 or .4 in the future if we need to change format or semantics.

(In reply to Carlos O'Donell from comment #10)
> I support a name change and I propose "locale-archive.2" and we can bump to
> .3 or .4 in the future if we need to change format or semantics.

I'm testing a build which just uses locale-archive.%{version}-%{release}, so that we don't have to maintain that number.  This will also help with format changes during upgrades.

It's not nice for statically linked, localized binaries, but we hopefully do not have many of those.  They would also have been affected by the lack of format compatibility for the locale data anyway.

(In reply to Florian Weimer from comment #11)
> (In reply to Carlos O'Donell from comment #10)
> > I support a name change and I propose "locale-archive.2" and we can bump to
> > .3 or .4 in the future if we need to change format or semantics.
> 
> I'm testing a build which just uses locale-archive.%{version}-%{release}, so
> that we don't have to maintain that number.  This will also help with format
> changes during upgrades.

Looks good to me.

> It's not nice for statically linked, localized binaries, but we hopefully do
> not have many of those.  They would also have been affected by the lack of
> format compatibility for the locale data anyway.

Correct. We have always said that static binaries may not be able to load locales
because of format changes. We may need to do a better job versioning that data
to avoid problems.

Patch posted upstream: https://sourceware.org/ml/libc-alpha/2019-06/msg00038.html

Adam, would you please test this again if you can?  The build has just finished.  Thanks.

It'll get tested automatically with the next Rawhide compose, if I get around to doing it manually before that I'll let you know.

Upstream pointed out that the patch reduces compatibility of statically linked binaries.  We will need a different solution for this.

*** Bug 1718785 has been marked as a duplicate of this bug. ***

glibc-2.29.9000-26.fc31 reverts the original patch and now simply restores the deleted file on %posttrans, by shipping the contents as a hard-linked file.

The actual bug fix is in the old glibc-all-langpacks, see bug 1717367.  We will keep the workaround now in rawhide for a couple of Fedora releases.

I tried -26 release. Locales seems to be back but I still issues

sh$ rpm -qa glibc \*langpack\*
glibc-all-langpacks-2.29.9000-26.fc31.x86_64
glibc-2.29.9000-26.fc31.x86_64
libreoffice-langpack-en-6.2.3.2-5.fc31.x86_64

sh$  locale -a | wc -l
865

sh-5.0$ locale
LANG=C.utf8
LC_CTYPE="C.utf8"
LC_NUMERIC="C.utf8"
LC_TIME="C.utf8"
LC_COLLATE="C.utf8"
LC_MONETARY="C.utf8"
LC_MESSAGES="C.utf8"
LC_PAPER="C.utf8"
LC_NAME="C.utf8"
LC_ADDRESS="C.utf8"
LC_TELEPHONE="C.utf8"
LC_MEASUREMENT="C.utf8"
LC_IDENTIFICATION="C.utf8"
LC_ALL=

sh-5.0$ rpm -q --changelog openssl | head
* Mon Jun 03 2019 Tomáš Mráz <tmraz> 1.1.1c-3
- add upstream patch to defer sending KeyUpdate after
  pending writes are complete

* Thu May 30 2019 Tomáš Mráz <tmraz> 1.1.1c-2
- fix use of uninitialized memory

* Wed May 29 2019 Tomáš Mráz <tmraz> 1.1.1c-1
- update to the 1.1.1c release

(In reply to Lukas Slebodnik from comment #19)
> I tried -26 release. Locales seems to be back but I still issues

Did you reboot or otherwise restart your terminal?

(In reply to Florian Weimer from comment #20)
> (In reply to Lukas Slebodnik from comment #19)
> > I tried -26 release. Locales seems to be back but I still issues
> 
> Did you reboot or otherwise restart your terminal?

I rebooted machine to be sure.

But it seems to be some problem with xterm because xfce4-terminal works well.

(In reply to Lukas Slebodnik from comment #21)
> (In reply to Florian Weimer from comment #20)
> > (In reply to Lukas Slebodnik from comment #19)
> > > I tried -26 release. Locales seems to be back but I still issues
> > 
> > Did you reboot or otherwise restart your terminal?
> 
> I rebooted machine to be sure.
> 
> But it seems to be some problem with xterm because xfce4-terminal works well.

Right, it's probably an unrelated issue.

(In reply to Florian Weimer from comment #22)
> (In reply to Lukas Slebodnik from comment #21)
> > (In reply to Florian Weimer from comment #20)
> > > (In reply to Lukas Slebodnik from comment #19)
> > > > I tried -26 release. Locales seems to be back but I still issues
> > > 
> > > Did you reboot or otherwise restart your terminal?
> > 
> > I rebooted machine to be sure.
> > 
> > But it seems to be some problem with xterm because xfce4-terminal works well.
> 
> Right, it's probably an unrelated issue.

And it might be partially related also to this BZ and not just issue with xterm
(https://bugzilla.redhat.com/show_bug.cgi?id=1718934 in case you are interested)

I've just realized that I did not configure LANG=C.utf8.
And my desktop manager properly set LANG=en_US.UTF-8

OR is there issue with en_US.UTF-8 vs en_US.utf8 ?

sh$ locale -a | grep en_US
en_US
en_US.iso88591
en_US.iso885915
en_US.utf8

sh$ pgrep lightdm | while read pid; do echo ---$pid---; cat /proc/$pid/environ | tr '\0' '\n' | sort; done
---1389---
INVOCATION_ID=5bf5bfc9dc124bb0ace55a3a7372e7f0
JOURNAL_STREAM=9:32378
LANG=en_US.UTF-8
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
---3510---
INVOCATION_ID=5bf5bfc9dc124bb0ace55a3a7372e7f0
JOURNAL_STREAM=9:32378
LANG=en_US.UTF-8
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin

(In reply to Lukas Slebodnik from comment #23)
> OR is there issue with en_US.UTF-8 vs en_US.utf8 ?

They are the same thing.  It's en_US.utf8 on disk, but en_US.UTF-8 is a valid alias.

OK,

I still try to debug what has change a and why my xsession use C.utf8

The locale C.utf8 is present on my system
sh$ rpm -qf /usr/lib/locale/C.utf8/*
glibc-common-2.29.9000-26.fc31.x86_64
glibc-common-2.29.9000-26.fc31.x86_64
glibc-common-2.29.9000-26.fc31.x86_64
glibc-common-2.29.9000-26.fc31.x86_64
glibc-common-2.29.9000-26.fc31.x86_64
glibc-common-2.29.9000-26.fc31.x86_64
glibc-common-2.29.9000-26.fc31.x86_64
glibc-common-2.29.9000-26.fc31.x86_64
glibc-common-2.29.9000-26.fc31.x86_64
glibc-common-2.29.9000-26.fc31.x86_64
glibc-common-2.29.9000-26.fc31.x86_64
glibc-common-2.29.9000-26.fc31.x86_64

sh$ ls -l /usr/lib/locale/C.utf8/
total 1852
-rw-r--r--. 1 root root     131 Jun  5 12:27 LC_ADDRESS
-rw-r--r--. 1 root root 1519058 Jun  5 12:27 LC_COLLATE
-rw-r--r--. 1 root root  339912 Jun  5 12:27 LC_CTYPE
-rw-r--r--. 1 root root     252 Jun  5 12:27 LC_IDENTIFICATION
-rw-r--r--. 1 root root      23 Jun  5 12:27 LC_MEASUREMENT
drwxr-xr-x. 1 root root      30 Jun 10 14:11 LC_MESSAGES
-rw-r--r--. 1 root root     282 Jun  5 12:27 LC_MONETARY
-rw-r--r--. 1 root root      62 Jun  5 12:27 LC_NAME
-rw-r--r--. 1 root root      50 Jun  5 12:27 LC_NUMERIC
-rw-r--r--. 1 root root      34 Jun  5 12:27 LC_PAPER
-rw-r--r--. 1 root root      47 Jun  5 12:27 LC_TELEPHONE
-rw-r--r--. 1 root root    3360 Jun  5 12:27 LC_TIME


But I could see in ~/.xsession-errors that xrdb does not know it
sh-5.0$ locale
LANG=C.utf8
LC_CTYPE="C.utf8"
LC_NUMERIC="C.utf8"
LC_TIME="C.utf8"
LC_COLLATE="C.utf8"
LC_MONETARY="C.utf8"
LC_MESSAGES="C.utf8"
LC_PAPER="C.utf8"
LC_NAME="C.utf8"
LC_ADDRESS="C.utf8"
LC_TELEPHONE="C.utf8"
LC_MEASUREMENT="C.utf8"
LC_IDENTIFICATION="C.utf8"
LC_ALL=
sh-5.0$ xrdb -merge /home/alcik/.Xresources
/home/alcik/.Xresources:0: warning: Unknown encoding: C.utf8
    sh-5.0$ 
sh-5.0$ LC_ALL=C.utf8 xrdb -merge /home/alcik/.Xresources
/home/alcik/.Xresources:0: warning: Unknown encoding: C.utf8
    sh-5.0$ 
sh-5.0$ LC_ALL=en_US.utf8 xrdb -merge /home/alcik/.Xresources
sh-5.0$ echo $?
0

sh-5.0$ rpm -V glibc-common
sh-5.0$ echo $?
0

(In reply to Lukas Slebodnik from comment #25)
> sh-5.0$ xrdb -merge /home/alcik/.Xresources
> /home/alcik/.Xresources:0: warning: Unknown encoding: C.utf8

This is a bug in mcpp. Please file it there.

383     strcpy( norm, name);
384     if (norm[ 5] == '.')
385         memmove( norm, norm + 5, strlen( norm + 5) + 1);
386         /* Remove initial 'xxxxx.' as 'ja_JP.', 'en_US.' or any other   */
387     conv_case( norm, norm + strlen( norm), LOWER);
388     strip_bar( norm);

It assumes a 5 char locale, and C or eo (Esperanto without country) don't conform to that.

Then it tries to lookup 'cutf8' and fails.

A quick hack would be to add 'cutf8' as an alias for the UTF8 line of encoding_name.