(In reply to Adam Williamson from comment #0)
> initdb: error: invalid locale settings; check LANG and LC_* environment
> variables

Looking at the source code, this error indicates that setlocale(LC_something, "")
failed (returned NULL).  The hint in the error message seems dead on given that.

> this is working fine in Fedora 33 at present, so it seems like it's likely a
> difference between 12.3 (in F33) and 12.4 (in Rawhide). That, or something
> else changed in Rawhide but not in Branched which affects postgresql's
> behaviour.

I don't see any changes in that part of the PG code between 12.3 and 12.4.

hmm, yeah, it may not be postgresql itself as 12.4 has now been submitted as an update for F32 (but not F33? It should be sent out for F33 too...) and passed the same test there:

https://openqa.fedoraproject.org/tests/647377

so, this may be caused by something else that changed between Fedora-Rawhide-20200817.n.0 and Fedora-Rawhide-20200823.n.2 (that's the delta we have here between last success and first failure - we didn't get a complete compose between those two).

So, yeah, I have a new suspect, this change to glibc:

https://src.fedoraproject.org/rpms/glibc/c/b1869f947af08ef9c8fba2a70c4a71315e838ca0?branch=master

I'm guessing glibc-minimal-langpack might be too minimal for postgresql to work with. Will investigate a bit further.

Yup, that's the issue. If I make the test do 'dnf -y install glibc-all-langpacks' before postgresql-setup, it works.

Re-assigning to glibc for now, but this is kind of a complicated thing. Will talk to sgallagh a bit about it.

This could be a bug in the test script, namely that it's not being careful that LANG/LC_ALL are set to something that glibc will accept even with the minimal langpack.  The other interpretation seems to be that glibc-minimal-langpack is too minimal for practical use.

A third possibility is that an incorrect LC_*/LANG environment variable leaks into the invocation, e.g. because it was propagated over SSH, or the default environment was not set correctly.

Installing glibc-all-langpacks papers over such issues.

(I do not think this particular initdb check will fail with the C locale is used.)

couple more test notes:

1. Just having glibc-minimal-langpack and doing `LC_ALL=C.UTF-8 /usr/bin/postgresql-setup --initdb` fails
2. Installing glibc-langpack-en before postgresql-setup works
3. Installing glibc-langpack-fr before postgresql-setup fails

So postgresql-setup either needs the en langpack installed to work, or needs the langpack that matches the current LANG installed to work, I guess. The first test was one sgallagh suggested, thinking it might make postgresql-setup work, but it didn't.

Tom: Florian: the test here simply does a default Fedora Server install and then tries to deploy postgresql in a typical way. It does not specifically try and set any locale related variables because there's no reason we should expect a sysadmin to do that in order to simply deploy a postgresql server on a default Fedora Server install.

The test is not running over ssh. It's literally entering keypresses (via VNC) into a console on a default Fedora Server install running in a VM.

BTW, I checked the actual state of the locale-related environment variables right before the test runs postgresql-setup, if we don't install any additional langpacks, with:

printenv | grep LC
printenv | grep LANG

the first shows nothing (i.e. no variables with LC in the name are set at all). The second shows LANG=en_US.UTF-8 .

The LANG setting is the problem, that's not correct for glibc-minimal-langpack. You need at least glibc-langpack-en for that.

Yeah, I guessed that might be it - I'm running a test right now to see if `LANG=C.UTF-8 /usr/bin/postgresql-setup --initdb` works. But if that's the problem, it's still a bug, because that's the setting we're getting OOTB on a default install. I'll see if I can find out where it comes from.

Hmm, so...

`LANG=C.UTF-8 /usr/bin/postgresql-setup --initdb` doesn't work. Nor does `export LANG=C.UTF-8; /usr/bin/postgresql-setup --initdb`. Not sure why not.

Digging into "why do we have LANG=en_US.UTF-8 but no langpack"...the answer is basically: anaconda. anaconda has code to write a /etc/locale.conf with LANG=(the locale chosen for the install) , and that's getting done here. anaconda *also* has some code which I think is meant to ensure relevant langpacks get installed, but I think it's broken/outdated:

    def langpacks(self):
        # get all available languages in repos
        available_langpacks = self._base.sack.query().available() \
            .filter(name__glob="langpacks-*")
        alangs = [p.name.split('-', 1)[1] for p in available_langpacks]

        langpacks = []
        # add base langpacks into transaction
        localization_proxy = LOCALIZATION.get_proxy()
        for lang in [localization_proxy.Language] + localization_proxy.LanguageSupport:
            loc = pyanaconda.localization.find_best_locale_match(lang, alangs)
            if not loc:
                log.warning("Selected lang %s does not match "
                            "any available langpack", lang)
                continue
            langpacks.append("langpacks-" + loc)
        return langpacks

note it's looking for "langpacks-*", which doesn't look like the right query now (if it ever was). And indeed when I check the install logs, we're hitting the warning:

var/log/anaconda/journal.log:Aug 31 11:44:47 fedora anaconda[1655]: packaging: Selected lang en_US.UTF-8 does not match any available langpack
var/log/anaconda/journal.log:Aug 31 11:44:47 fedora anaconda[1655]: anaconda: payload.requirement: no package requirement added for langpacks

hmm, okay, so those are different langpacks, this thing:

https://koji.fedoraproject.org/koji/packageinfo?packageID=21858

I think there may still be a bug there as I'd expect we should be finding a match, but those 'langpacks' don't pull in glibc's langpacks AFAICS. So I don't think anaconda has any code that's expected to pull in the glibc langpack matching the LANG setting it writes to /etc/locale.conf , unless I'm missing something.

FWIW I think anaconda doesn't find any langpacks because almost none of the langpacks packages are in the Server DVD repo - only langpacks-core-font-en is. Not sure if we'd consider that a bug.

OOOH. It *is* the langpacks thing, it was just a bit tricky to find. Check this:

[adamw@adam dnf (master %)]$ rpm -q --supplements glibc-langpack-en
(glibc and (langpacks-core-en or langpacks-core-en_AG or langpacks-core-en_AU or langpacks-core-en_BW or langpacks-core-en_CA or langpacks-core-en_DK or langpacks-core-en_GB or langpacks-core-en_HK or langpacks-core-en_IE or langpacks-core-en_IL or langpacks-core-en_IN or langpacks-core-en_NG or langpacks-core-en_NZ or langpacks-core-en_PH or langpacks-core-en_SC or langpacks-core-en_SG or langpacks-core-en_US or langpacks-core-en_ZA or langpacks-core-en_ZM or langpacks-core-en_ZW))

i.e., glibc-langpack-en "supplements" glibc together with any langpacks-core-en* package. That means if glibc *and* a langpacks-core-en package are being pulled into a transaction, dnf will also want to pull in glibc-langpack-en.

I did another test that confirms this: I did a default install from the Server *netinst*. The difference between that and a default install from the Server DVD is that a default install from the DVD has only the packages on the DVD itself available for install; a default install from the Server netinst has all the packages from the whole Everything repo available for install (we *used* to have variant netinsts use the repo from the appropriate variant installroot rather than Everything, but we stopped doing that a while back). Along with several other differences - the Server netinst default install is 50 packages bigger than the Server DVD default install! - the Server netinst install gets glibc-langpack-en instead of glibc-langpack-minimal . And I'm sure this is the reason why, because the packaging logs from that install *also* show that the langpack detection code worked - it found and selected langpacks-en (and hence its dependencies, langpacks-core-en and langpacks-core-font-en) for install. So we get glibc and langpacks-core-en in the transaction, which triggers the "Supplements" for glibc-langpack-en and pulls it in too.

So, here's a recap of the key points so far:

1. `postgresql-setup --initdb` doesn't work if LANG is set to 'en_US.UTF-8' but glibc-langpacks-en is not installed (or, I guess, analogous situation with any other language)

2. anaconda tries to pick a locale based on the language chosen for installation, and if this succeeds in the installer environment, writes a /etc/locale.conf with "LANG=(that locale)". It also makes an effort to install a corresponding langpacks package for the locale it selects, but if that fails, it *still* writes the locale.conf

3. The repo on the Server DVD is missing most of the binary packages from the langpacks RPM, meaning when installing from that repo, anaconda fails to install a langpacks package corresponding to the locale used for the install and written to locale.conf

4. glibc used to recommend glibc-all-langpacks, but as of 2.32.9000-3 only recommends glibc-langpack-minimal. Workstation and KDE comps groups explicitly include glibc-all-langpacks (also the anaconda-tools group does, which pulls it into other live images), but Server group doesn't

Oh, I confirmed 1. by having the test wipe `/etc/locale.conf` then log out and back in. After doing that, it works. So it seems just changing the $LANG definition at the time of running the command isn't enough to fix it, you need a new session.

The most obvious fix for this I can think of, off the top of my head, is: add the langpacks* packages to the Server DVD repo. However I'm not sure what's the best way to achieve that without impacting anything else. There's 'additional_packages' in pungi-fedora fedora.conf that might do the trick. I don't think we can necessarily add it to anaconda-tools in comps, because that I think would make live images include glibc-all-langpacks *and* then all the separate langpacks and glibc-langpacks packages; right now lives have glibc-all-langpacks and langpacks-en and its dependencies.

So I'm still not 100%, but from looking through the pungi code, I think either editing additional_packages in pungi-fedora fedora.conf or the Server variant definition in pungi-fedora variants-fedora.xml would be the right way to do this. It should pull them into the Server installroot, and hence the DVD, and nothing else. I *think*.

I don't think sgallagh's suggestion to edit fedora-disk-base.ks in fedora-kickstarts is right, I don't think that or any kickstart is used in determining the contents of the install root and hence the DVD repo.

But I could of course be wrong!

Is there anything here that we should be doing on the glibc side? This bug is still assigned to glibc, after all.

No, I think Adam is correct. I'll take ownership of this.

I have a proposed fix under consideration:

https://pagure.io/pungi-fedora/pull-request/878

The fix here is merged, we can mark this ON_QA. Rawhide composes are failing, but they *are* getting far enough to produce a Server DVD, so we should be able to test this. Here's the latest:
https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20200909.n.0/compose/Server/x86_64/iso/Fedora-Server-dvd-x86_64-Rawhide-20200909.n.0.iso
I just downloaded it to test something else, so I'll try and check this too.

Unfortunately that doesn't boot, so can't test. Will try again with the next one.

Good news: we got a successful Rawhide compose and the test passed: https://openqa.fedoraproject.org/tests/662159

So, closing. I do think postgresql could potentially make this work - I did notice from the test videos that DNF actually works out it's in this situation, and copes with it - but that would be a separate and lower-priority bug.