1933863 – Broken hostname setting via reverse lookup with NetworkManager + systemd-resolved

Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1933863 - Broken hostname setting via reverse lookup with NetworkManager + systemd-resolved

Summary: Broken hostname setting via reverse lookup with NetworkManager + systemd-reso...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Fedora
Classification:	Fedora
Component:	NetworkManager
Sub Component:
Version:	34
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	---
Assignee:	Lubomir Rintel
QA Contact:	Fedora Extras Quality Assurance
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2021-03-01 21:24 UTC by Jonathan Lebon
Modified:	2021-03-19 20:14 UTC (History)
CC List:	11 users (show)
Fixed In Version:	NetworkManager-1.30.2-1.fc34
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2021-03-19 20:14:03 UTC
Type:	Bug
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
freedesktop.org Gitlab	NetworkManager/NetworkManager - merge_requests 770	0	None	None	None	2021-03-04 15:25:47 UTC

Description Jonathan Lebon 2021-03-01 21:24:06 UTC

Description of problem:

In Fedora CoreOS 34, we've noticed that hostname setting via reverse DNS doesn't work because of what seems to be a conceptual broken dependency between the lookup and updating DNS.

In my test, I'm using static IP to force NM not to get the hostname from DHCP and to fallback to reverse DNS (because AFAICT the NM in f34 doesn't yet have `hostname.from-dhcp` from https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/669). Here's the connection file I'm using in a libvirt VM:

[ipv4]
address1=192.168.122.171/24,192.168.122.1
dns=192.168.122.1;
dns-search=
method=manual

I have dnsmasq set up on my host to return `foobar` when querying the IP 192.168.122.171 (by adding the relevant line to `/etc/hosts`).

Here's the relevant logs from NetworkManager (with trace logging):

```
Mar 01 21:03:18 localhost NetworkManager[674]: <info>  [1614632598.9301] manager: NetworkManager state is now CONNECTED_SITE
Mar 01 21:03:18 localhost NetworkManager[674]: <info>  [1614632598.9308] policy: set 'Wired connection 1' (ens3) as default for IPv4 routing and DNS
Mar 01 21:03:18 localhost NetworkManager[674]: <debug> [1614632598.9310] manager: PrimaryConnection now Wired connection 1
Mar 01 21:03:18 localhost NetworkManager[674]: <trace> [1614632598.9320] policy: set-hostname: updating hostname (routing and dns)
Mar 01 21:03:18 localhost NetworkManager[674]: <trace> [1614632598.9321] policy: get-hostname: "localhost" (from dbus)
...
Mar 01 21:03:18 localhost NetworkManager[674]: <trace> [1614632598.9328] device[3c55bb4a7b7a290e] (ens3): hostname-from-dns: ipv4 resolver state wait-address, old address (null), new address 192.168.122.171
Mar 01 21:03:18 localhost NetworkManager[674]: <trace> [1614632598.9329] device[3c55bb4a7b7a290e] (ens3): hostname-from-dns: starting lookup for address 192.168.122.171
Mar 01 21:03:18 localhost NetworkManager[674]: <debug> [1614632598.9358] dns-mgr: (update_routing_and_dns): DNS configuration changed
Mar 01 21:03:18 localhost NetworkManager[674]: <debug> [1614632598.9361] dns-mgr: (update_routing_and_dns): no DNS changes to commit (1)
Mar 01 21:03:18 localhost NetworkManager[674]: <debug> [1614632598.9362] dns-mgr: (device_state_changed): DNS configuration changed
Mar 01 21:03:18 localhost NetworkManager[674]: <debug> [1614632598.9363] dns-mgr: (device_state_changed): committing DNS changes (0)
Mar 01 21:03:18 localhost NetworkManager[674]: <debug> [1614632598.9364] dns-mgr: update-dns: not updating resolv.conf
Mar 01 21:03:18 localhost NetworkManager[674]: <trace> [1614632598.9365] dns-mgr: config:      100 best    v4 2     : 192.168.122.1
Mar 01 21:03:18 localhost NetworkManager[674]: <trace> [1614632598.9367] dns-mgr: plugin: add domain <auto-default> (i=2, p=100)
Mar 01 21:03:18 localhost NetworkManager[674]: <trace> [1614632598.9368] dns-mgr: plugin: settings: ifindex=2, priority=100, default-route=1, search=, reverse=122.168.192.in-addr.arpa,171.122.168.192.in-addr.arpa
Mar 01 21:03:18 localhost NetworkManager[674]: <debug> [1614632598.9370] dns-mgr: update-dns: updating plugin systemd-resolved
Mar 01 21:03:18 localhost NetworkManager[674]: <trace> [1614632598.9371] dns-sd-resolved[d26e64a84b337b7a]: send-updates: start 5 requests
Mar 01 21:03:18 localhost systemd-resolved[638]: ens3: Bus client set default route setting: yes
Mar 01 21:03:18 localhost systemd-resolved[638]: ens3: Bus client set DNS server list to: 192.168.122.1
Mar 01 21:03:18 localhost NetworkManager[674]: <trace> [1614632598.9434] dns-mgr: update-resolv-no-stub: '/run/NetworkManager/no-stub-resolv.conf' successfully written
Mar 01 21:03:18 localhost NetworkManager[674]: <trace> [1614632598.9438] dns-mgr: update-resolv-conf: write internal file /run/NetworkManager/resolv.conf succeeded
Mar 01 21:03:18 localhost NetworkManager[674]: <trace> [1614632598.9439] dns-mgr: current configuration: [{'nameservers': <['192.168.122.1']>, 'interface': <'ens3'>, 'priority': <100>, 'vpn': <false>}]
...
Mar 01 21:03:18 localhost NetworkManager[674]: <debug> [1614632598.9867] device[3c55bb4a7b7a290e] (ens3): hostname-from-dns: lookup done for 192.168.122.171, result "localhost"
...
Mar 01 21:03:18 localhost NetworkManager[674]: <trace> [1614632598.9887] policy: set-hostname: hostname already set to 'localhost' (from address lookup)
```

So what I'm seeing here is that NM is starting the reverse DNS lookup *before* it actually tells systemd-resolved what the new DNS server is, hence causing address lookup to fail (and stay on localhost).

If I do `systemctl restart NetworkManager`, NM correctly finds the `foobar` hostname from reverse DNS because DNS has already been set up:

```
Mar 01 21:03:34 localhost NetworkManager[796]: <trace> [1614632614.7156] device[4fe1b4205a9fa185] (ens3): hostname-from-dns: ipv4 resolver state wait-address, old address (null), new address 192.168.122.171
Mar 01 21:03:34 localhost NetworkManager[796]: <trace> [1614632614.7157] device[4fe1b4205a9fa185] (ens3): hostname-from-dns: starting lookup for address 192.168.122.171
...
Mar 01 21:03:34 localhost NetworkManager[796]: <debug> [1614632614.7367] device[4fe1b4205a9fa185] (ens3): hostname-from-dns: lookup done for 192.168.122.171, result "foobar"
Mar 01 21:03:34 localhost NetworkManager[796]: <trace> [1614632614.7369] policy: set-hostname: updating hostname (lookup finished)
Mar 01 21:03:34 localhost NetworkManager[796]: <trace> [1614632614.7370] policy: get-hostname: "localhost" (from dbus)
Mar 01 21:03:34 localhost NetworkManager[796]: <trace> [1614632614.7372] policy: device hostname info:
Mar 01 21:03:34 localhost NetworkManager[796]: <trace> [1614632614.7376] policy:   - prio:  100 ipv4 (def) dhcp  dns  dev:ens3
Mar 01 21:03:34 localhost NetworkManager[796]: <trace> [1614632614.7377] policy:   - prio:  100 ipv6       dhcp  dns  dev:ens3
Mar 01 21:03:34 localhost NetworkManager[796]: <trace> [1614632614.7378] device[4fe1b4205a9fa185] (ens3): hostname-from-dns: ipv4 resolver state done, old address 192.168.122.171, new address 192.168.122.171
Mar 01 21:03:34 localhost NetworkManager[796]: <trace> [1614632614.7380] policy: get-hostname: "localhost" (from dbus)
Mar 01 21:03:34 localhost NetworkManager[796]: <info>  [1614632614.7381] policy: set-hostname: set hostname to 'foobar' (from address lookup)
...
Mar 01 21:03:34 foobar systemd-hostnamed[685]: Hostname set to <foobar> (transient)
Mar 01 21:03:34 foobar systemd-resolved[638]: System hostname changed to 'foobar'.
```

So it seems like NM needs to be tweaked to delay hostname-from-dns until after DNS has been updated?

Version-Release number of selected component (if applicable):

# rpm -q NetworkManager systemd
NetworkManager-1.30.0-1.fc34.x86_64
systemd-248~rc2-1.fc34.x86_64

How reproducible:

100%

Steps to Reproduce:
1. See above

Actual results:

Hostname setting via reverse DNS doesn't work until NetworkManager.service is restarted.

Expected results:

Hostname setting via reverse DNS works.

Additional info:

Fedora CoreOS currently neuters systemd-resolved so that it just directly points to the underlying DNS server. We also set the fallback hostname to `localhost` and the `nsswitch.conf` config to match f32 (where `dns` is ordered higher than `myhostname` and `resolve` isn't in the list)  See all that here (with links to background):

https://github.com/coreos/fedora-coreos-config/blob/72a2e7e5b89746b7ea320ee9da057bd8ccba52e0/manifests/fedora-coreos-base.yaml#L82-L110

Comment 1 Jonathan Lebon 2021-03-02 17:34:42 UTC

OK right, I think what's going on here is that in the systemd-resolved case, NM considers /etc/resolv.conf unmanaged. This means that when the dns-manager updates systemd-resolved with the new DNS config, it doesn't emit a CONFIG_CHANGED signal. And so the hostname lookup isn't restarted.

So... should that code be relaxed a bit so that CONFIG_CHANGED is emitted even if we're not managing /etc/resolv.conf but we updated a caching plugin? Something like:

diff --git a/src/core/dns/nm-dns-manager.c b/src/core/dns/nm-dns-manager.c
index d30c3fa549..5e03b102b7 100644
--- a/src/core/dns/nm-dns-manager.c
+++ b/src/core/dns/nm-dns-manager.c
@@ -1815,8 +1815,8 @@ plugin_skip:;
                            NM_DNS_MANAGER_RESOLV_CONF_MAN_UNMANAGED);
     }

-    /* signal that resolv.conf was changed */
-    if (do_update && result == SR_SUCCESS)
+    /* signal that DNS resolution configs were changed */
+    if ((do_update || caching) && result == SR_SUCCESS)
         g_signal_emit(self, signals[CONFIG_CHANGED], 0);

     nm_clear_pointer(&priv->config_variant, g_variant_unref);

Comment 2 Beniamino Galvani 2021-03-04 10:08:29 UTC

Hi Jonathan, thanks for looking into this issue.

The patch looks correct to me. Would you like to open a merge request at [1] for it or do you prefer if I do it?

[1] https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests

Comment 3 Jonathan Lebon 2021-03-04 15:25:48 UTC

Thanks for confirming Beniamino! I've tested the patch and confirmed it fixes the issue. Posted an MR.

Comment 4 Jonathan Lebon 2021-03-05 15:18:07 UTC

Thanks for merging!

Are you planning to rebase NM in f34 before GA? If not, would it be possible to backport this patch to f34? I can draft up the dist-git PR for that if you'd like.

Comment 5 Beniamino Galvani 2021-03-08 09:20:26 UTC

We will do a 1.30.2 minor release this week and we will rebase the F34 package to this new version (which includes your commit).

Comment 6 Fedora Update System 2021-03-12 17:07:01 UTC

FEDORA-2021-fb173bc82f has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2021-fb173bc82f

Comment 7 Fedora Update System 2021-03-12 18:57:22 UTC

FEDORA-2021-fb173bc82f has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-fb173bc82f`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-fb173bc82f

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 8 Fedora Update System 2021-03-19 20:14:03 UTC

FEDORA-2021-fb173bc82f has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.