Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at
Bug 1872511 - postgresql-setup --initdb fails with "error: invalid locale settings" if only glibc-minimal-langpack is installed
Summary: postgresql-setup --initdb fails with "error: invalid locale settings" if only...
Alias: None
Product: Fedora
Classification: Fedora
Component: distribution
Version: rawhide
Hardware: All
OS: Linux
Target Milestone: ---
Assignee: Stephen Gallagher
QA Contact: Fedora Extras Quality Assurance
Whiteboard: openqa
Depends On:
Blocks: F34BetaBlocker
TreeView+ depends on / blocked
Reported: 2020-08-25 23:31 UTC by Adam Williamson
Modified: 2020-09-11 20:56 UTC (History)
23 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2020-09-11 20:56:48 UTC
Type: Bug

Attachments (Terms of Use)

Description Adam Williamson 2020-08-25 23:31:48 UTC
On the most recent Fedora Rawhide compose - Fedora-Rawhide-20200823.n.2 - the openQA test which tests postgresql server deployment failed:

the test fails when running `/usr/bin/postgresql-setup --initdb` after installing the server package and configuring the firewall. initdb_postgresql.log shows this:


The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

initdb: error: invalid locale settings; check LANG and LC_* environment variables


this is working fine in Fedora 33 at present, so it seems like it's likely a difference between 12.3 (in F33) and 12.4 (in Rawhide). That, or something else changed in Rawhide but not in Branched which affects postgresql's behaviour.

I don't see any related SELinux denials in the logs.

Proposing as an F34 Beta blocker as a violation of "It must be possible to configure a Fedora Server system installed according to the above criteria as a postgresql database server, using the official deployment tools provided in the distribution postgresql packages" - .

Comment 1 Tom Lane 2020-08-25 23:47:52 UTC
(In reply to Adam Williamson from comment #0)
> initdb: error: invalid locale settings; check LANG and LC_* environment
> variables

Looking at the source code, this error indicates that setlocale(LC_something, "")
failed (returned NULL).  The hint in the error message seems dead on given that.

> this is working fine in Fedora 33 at present, so it seems like it's likely a
> difference between 12.3 (in F33) and 12.4 (in Rawhide). That, or something
> else changed in Rawhide but not in Branched which affects postgresql's
> behaviour.

I don't see any changes in that part of the PG code between 12.3 and 12.4.

Comment 2 Adam Williamson 2020-09-01 18:07:05 UTC
hmm, yeah, it may not be postgresql itself as 12.4 has now been submitted as an update for F32 (but not F33? It should be sent out for F33 too...) and passed the same test there:

so, this may be caused by something else that changed between Fedora-Rawhide-20200817.n.0 and Fedora-Rawhide-20200823.n.2 (that's the delta we have here between last success and first failure - we didn't get a complete compose between those two).

Comment 3 Adam Williamson 2020-09-01 18:21:42 UTC
So, yeah, I have a new suspect, this change to glibc:

I'm guessing glibc-minimal-langpack might be too minimal for postgresql to work with. Will investigate a bit further.

Comment 4 Adam Williamson 2020-09-01 18:36:30 UTC
Yup, that's the issue. If I make the test do 'dnf -y install glibc-all-langpacks' before postgresql-setup, it works.

Re-assigning to glibc for now, but this is kind of a complicated thing. Will talk to sgallagh a bit about it.

Comment 5 Tom Lane 2020-09-01 18:44:51 UTC
This could be a bug in the test script, namely that it's not being careful that LANG/LC_ALL are set to something that glibc will accept even with the minimal langpack.  The other interpretation seems to be that glibc-minimal-langpack is too minimal for practical use.

Comment 6 Florian Weimer 2020-09-01 19:03:30 UTC
A third possibility is that an incorrect LC_*/LANG environment variable leaks into the invocation, e.g. because it was propagated over SSH, or the default environment was not set correctly.

Installing glibc-all-langpacks papers over such issues.

(I do not think this particular initdb check will fail with the C locale is used.)

Comment 7 Adam Williamson 2020-09-01 19:03:48 UTC
couple more test notes:

1. Just having glibc-minimal-langpack and doing `LC_ALL=C.UTF-8 /usr/bin/postgresql-setup --initdb` fails
2. Installing glibc-langpack-en before postgresql-setup works
3. Installing glibc-langpack-fr before postgresql-setup fails

So postgresql-setup either needs the en langpack installed to work, or needs the langpack that matches the current LANG installed to work, I guess. The first test was one sgallagh suggested, thinking it might make postgresql-setup work, but it didn't.

Comment 8 Adam Williamson 2020-09-01 19:05:23 UTC
Tom: Florian: the test here simply does a default Fedora Server install and then tries to deploy postgresql in a typical way. It does not specifically try and set any locale related variables because there's no reason we should expect a sysadmin to do that in order to simply deploy a postgresql server on a default Fedora Server install.

The test is not running over ssh. It's literally entering keypresses (via VNC) into a console on a default Fedora Server install running in a VM.

Comment 9 Adam Williamson 2020-09-01 19:10:37 UTC
BTW, I checked the actual state of the locale-related environment variables right before the test runs postgresql-setup, if we don't install any additional langpacks, with:

printenv | grep LC
printenv | grep LANG

the first shows nothing (i.e. no variables with LC in the name are set at all). The second shows LANG=en_US.UTF-8 .

Comment 10 Florian Weimer 2020-09-01 19:12:15 UTC
The LANG setting is the problem, that's not correct for glibc-minimal-langpack. You need at least glibc-langpack-en for that.

Comment 11 Adam Williamson 2020-09-01 19:14:06 UTC
Yeah, I guessed that might be it - I'm running a test right now to see if `LANG=C.UTF-8 /usr/bin/postgresql-setup --initdb` works. But if that's the problem, it's still a bug, because that's the setting we're getting OOTB on a default install. I'll see if I can find out where it comes from.

Comment 12 Adam Williamson 2020-09-01 19:27:26 UTC
Hmm, so...

`LANG=C.UTF-8 /usr/bin/postgresql-setup --initdb` doesn't work. Nor does `export LANG=C.UTF-8; /usr/bin/postgresql-setup --initdb`. Not sure why not.

Digging into "why do we have LANG=en_US.UTF-8 but no langpack"...the answer is basically: anaconda. anaconda has code to write a /etc/locale.conf with LANG=(the locale chosen for the install) , and that's getting done here. anaconda *also* has some code which I think is meant to ensure relevant langpacks get installed, but I think it's broken/outdated:

    def langpacks(self):
        # get all available languages in repos
        available_langpacks = self._base.sack.query().available() \
        alangs = ['-', 1)[1] for p in available_langpacks]

        langpacks = []
        # add base langpacks into transaction
        localization_proxy = LOCALIZATION.get_proxy()
        for lang in [localization_proxy.Language] + localization_proxy.LanguageSupport:
            loc = pyanaconda.localization.find_best_locale_match(lang, alangs)
            if not loc:
                log.warning("Selected lang %s does not match "
                            "any available langpack", lang)
            langpacks.append("langpacks-" + loc)
        return langpacks

note it's looking for "langpacks-*", which doesn't look like the right query now (if it ever was). And indeed when I check the install logs, we're hitting the warning:

var/log/anaconda/journal.log:Aug 31 11:44:47 fedora anaconda[1655]: packaging: Selected lang en_US.UTF-8 does not match any available langpack
var/log/anaconda/journal.log:Aug 31 11:44:47 fedora anaconda[1655]: anaconda: payload.requirement: no package requirement added for langpacks

Comment 13 Adam Williamson 2020-09-01 19:31:21 UTC
hmm, okay, so those are different langpacks, this thing:

I think there may still be a bug there as I'd expect we should be finding a match, but those 'langpacks' don't pull in glibc's langpacks AFAICS. So I don't think anaconda has any code that's expected to pull in the glibc langpack matching the LANG setting it writes to /etc/locale.conf , unless I'm missing something.

Comment 14 Adam Williamson 2020-09-01 19:48:14 UTC
FWIW I think anaconda doesn't find any langpacks because almost none of the langpacks packages are in the Server DVD repo - only langpacks-core-font-en is. Not sure if we'd consider that a bug.

Comment 15 Adam Williamson 2020-09-01 21:15:20 UTC
OOOH. It *is* the langpacks thing, it was just a bit tricky to find. Check this:

[adamw@adam dnf (master %)]$ rpm -q --supplements glibc-langpack-en
(glibc and (langpacks-core-en or langpacks-core-en_AG or langpacks-core-en_AU or langpacks-core-en_BW or langpacks-core-en_CA or langpacks-core-en_DK or langpacks-core-en_GB or langpacks-core-en_HK or langpacks-core-en_IE or langpacks-core-en_IL or langpacks-core-en_IN or langpacks-core-en_NG or langpacks-core-en_NZ or langpacks-core-en_PH or langpacks-core-en_SC or langpacks-core-en_SG or langpacks-core-en_US or langpacks-core-en_ZA or langpacks-core-en_ZM or langpacks-core-en_ZW))

i.e., glibc-langpack-en "supplements" glibc together with any langpacks-core-en* package. That means if glibc *and* a langpacks-core-en package are being pulled into a transaction, dnf will also want to pull in glibc-langpack-en.

I did another test that confirms this: I did a default install from the Server *netinst*. The difference between that and a default install from the Server DVD is that a default install from the DVD has only the packages on the DVD itself available for install; a default install from the Server netinst has all the packages from the whole Everything repo available for install (we *used* to have variant netinsts use the repo from the appropriate variant installroot rather than Everything, but we stopped doing that a while back). Along with several other differences - the Server netinst default install is 50 packages bigger than the Server DVD default install! - the Server netinst install gets glibc-langpack-en instead of glibc-langpack-minimal . And I'm sure this is the reason why, because the packaging logs from that install *also* show that the langpack detection code worked - it found and selected langpacks-en (and hence its dependencies, langpacks-core-en and langpacks-core-font-en) for install. So we get glibc and langpacks-core-en in the transaction, which triggers the "Supplements" for glibc-langpack-en and pulls it in too.

Comment 16 Adam Williamson 2020-09-01 21:25:59 UTC
So, here's a recap of the key points so far:

1. `postgresql-setup --initdb` doesn't work if LANG is set to 'en_US.UTF-8' but glibc-langpacks-en is not installed (or, I guess, analogous situation with any other language)

2. anaconda tries to pick a locale based on the language chosen for installation, and if this succeeds in the installer environment, writes a /etc/locale.conf with "LANG=(that locale)". It also makes an effort to install a corresponding langpacks package for the locale it selects, but if that fails, it *still* writes the locale.conf

3. The repo on the Server DVD is missing most of the binary packages from the langpacks RPM, meaning when installing from that repo, anaconda fails to install a langpacks package corresponding to the locale used for the install and written to locale.conf

4. glibc used to recommend glibc-all-langpacks, but as of 2.32.9000-3 only recommends glibc-langpack-minimal. Workstation and KDE comps groups explicitly include glibc-all-langpacks (also the anaconda-tools group does, which pulls it into other live images), but Server group doesn't

Comment 17 Adam Williamson 2020-09-01 21:38:12 UTC
Oh, I confirmed 1. by having the test wipe `/etc/locale.conf` then log out and back in. After doing that, it works. So it seems just changing the $LANG definition at the time of running the command isn't enough to fix it, you need a new session.

Comment 18 Adam Williamson 2020-09-02 23:51:43 UTC
The most obvious fix for this I can think of, off the top of my head, is: add the langpacks* packages to the Server DVD repo. However I'm not sure what's the best way to achieve that without impacting anything else. There's 'additional_packages' in pungi-fedora fedora.conf that might do the trick. I don't think we can necessarily add it to anaconda-tools in comps, because that I think would make live images include glibc-all-langpacks *and* then all the separate langpacks and glibc-langpacks packages; right now lives have glibc-all-langpacks and langpacks-en and its dependencies.

Comment 19 Adam Williamson 2020-09-04 06:35:39 UTC
So I'm still not 100%, but from looking through the pungi code, I think either editing additional_packages in pungi-fedora fedora.conf or the Server variant definition in pungi-fedora variants-fedora.xml would be the right way to do this. It should pull them into the Server installroot, and hence the DVD, and nothing else. I *think*.

I don't think sgallagh's suggestion to edit fedora-disk-base.ks in fedora-kickstarts is right, I don't think that or any kickstart is used in determining the contents of the install root and hence the DVD repo.

But I could of course be wrong!

Comment 20 Florian Weimer 2020-09-04 06:54:07 UTC
Is there anything here that we should be doing on the glibc side? This bug is still assigned to glibc, after all.

Comment 21 Stephen Gallagher 2020-09-04 12:56:10 UTC
No, I think Adam is correct. I'll take ownership of this.

Comment 22 Stephen Gallagher 2020-09-04 14:24:59 UTC
I have a proposed fix under consideration:

Comment 23 Adam Williamson 2020-09-09 19:49:40 UTC
The fix here is merged, we can mark this ON_QA. Rawhide composes are failing, but they *are* getting far enough to produce a Server DVD, so we should be able to test this. Here's the latest:
I just downloaded it to test something else, so I'll try and check this too.

Comment 24 Adam Williamson 2020-09-09 20:48:08 UTC
Unfortunately that doesn't boot, so can't test. Will try again with the next one.

Comment 25 Adam Williamson 2020-09-11 20:56:48 UTC
Good news: we got a successful Rawhide compose and the test passed:

So, closing. I do think postgresql could potentially make this work - I did notice from the test videos that DNF actually works out it's in this situation, and copes with it - but that would be a separate and lower-priority bug.

Note You need to log in before you can comment on or make changes to this bug.