Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1940715 - resolved (with caching enabled) returns only AAAA answer to an A query
Summary: resolved (with caching enabled) returns only AAAA answer to an A query
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: systemd
Version: 34
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: systemd-maint
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1940732 1940993 (view as bug list)
Depends On:
Blocks: F34FinalBlocker
TreeView+ depends on / blocked
 
Reported: 2021-03-19 00:36 UTC by Adam Williamson
Modified: 2021-03-25 11:43 UTC (History)
26 users (show)

Fixed In Version: systemd-248~rc4-3.fc34
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-03-25 00:18:50 UTC
Type: Bug
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github systemd systemd issues 19049 0 None closed resolved (with caching enabled) returns IPv6 IP for e.g. retrace.fedoraproject.org despite lack of global IPv6 connectiv... 2021-03-23 13:05:05 UTC

Description Adam Williamson 2021-03-19 00:36:29 UTC 
With systemd-248~rc2-8.fc34.x86_64, I'm getting only IPv6 IPs for some DNS lookups despite not having any global IPv6 connectivity. retrace.fedoraproject.org is one name that causes issues. Here's the results of several commands suggested by mcatanzaro:

[adamw@xps13k ~]$ ping -v retrace.fedoraproject.org
ping: connect: Network is unreachable
[adamw@xps13k ~]$ resolvectl query retrace.fedoraproject.org
retrace.fedoraproject.org: 2620:52:3:1:dead:beef:cafe:c005 -- link: wlp58s0
                           (retrace03.rdu-cc.fedoraproject.org)

-- Information acquired via protocol DNS in 2.3ms.
-- Data is authenticated: no; Data was acquired via local or encrypted transport: no
-- Data from: cache
[adamw@xps13k ~]$ dig @127.0.0.53 retrace.fedoraproject.org

; <<>> DiG 9.16.11-RedHat-9.16.11-5.fc34 <<>> @127.0.0.53 retrace.fedoraproject.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 47990
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;retrace.fedoraproject.org.	IN	A

;; ANSWER SECTION:
retrace.fedoraproject.org. 53	IN	CNAME	retrace03.rdu-cc.fedoraproject.org.
retrace03.rdu-cc.fedoraproject.org. 6953 IN A	8.43.85.61

;; AUTHORITY SECTION:
retrace03.rdu-cc.fedoraproject.org. 53 IN AAAA	2620:52:3:1:dead:beef:cafe:c005

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Thu Mar 18 13:32:35 PDT 2021
;; MSG SIZE  rcvd: 129

[adamw@xps13k ~]$ dig @1.1.1.1 retrace.fedoraproject.org

; <<>> DiG 9.16.11-RedHat-9.16.11-5.fc34 <<>> @1.1.1.1 retrace.fedoraproject.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5324
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;retrace.fedoraproject.org.	IN	A

;; ANSWER SECTION:
retrace.fedoraproject.org. 300	IN	CNAME	retrace03.rdu-cc.fedoraproject.org.
retrace03.rdu-cc.fedoraproject.org. 86400 IN A	8.43.85.61

;; Query time: 715 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Thu Mar 18 13:32:43 PDT 2021
;; MSG SIZE  rcvd: 101

With -7, things work OK. -7 had the same backport as -8, but still (by mistake) had the config snippet to disable caching; the only difference between -7 and -8 is that -7 has caching disabled, -8 has it enabled. So this definitely breaks when caching is enabled.

https://bugzilla.redhat.com/show_bug.cgi?id=1938460 actually looks like a very similar report, but it's for Fedora 33, so I'm not sure if it's actually the same bug. Filing separately for now.
Comment 1 Adam Williamson 2021-03-19 00:37:00 UTC 
This could cause almost anything network-related not to work, so it seems like a clear Final blocker under various criteria.
Comment 2 Chris Murphy 2021-03-19 06:58:42 UTC 
*** Bug 1940732 has been marked as a duplicate of this bug. ***
Comment 3 Adam Williamson 2021-03-20 02:49:56 UTC 
I've edited https://bodhi.fedoraproject.org/updates/FEDORA-2021-f26baf322b to include a build with caching disabled via config snippet, again, to work around this for now.
Comment 4 Fedora Update System 2021-03-23 07:03:32 UTC 
FEDORA-2021-ea92e5703f has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2021-ea92e5703f
Comment 5 Petr Menšík 2021-03-23 09:11:04 UTC 
Wait, wasn't one of primary reasons to enable systemd-resolved ability for caching? With caching disabled, it would allow just dumb split-DNS configuration, but nothing more?

Do we block DNSSEC in default installation just for possibility of split-DNS, which not everyone might need?

It seems systemd-resolved has chosen wrong section to add also AAAA record. It might use ADDITIONAL, but it uses AUTHORITY. That is supposed to be used to be used only for SOA record.
Comment 6 Petr Menšík 2021-03-23 11:12:15 UTC 
Note: returning IPv6 address in DNS, even when IPv6 default route does not exist, is correct. When AAAA record is requested and exists, it must be returned to that query.
This bug is not really about returning IPV6 address also, but omitting IPv4 response on A query, where only IPv4 or no answer should be returned. Both address families is returned by getaddrinfo() even without systemd-resolved active and it should work the same way.

systemd-resolved might be free to adapt resolvectl query <name>, where address family is not specified. But it must not change address family queries depending on IPv6 route presence, when specific address family was requested. Especially not DNS requests received over stub resolver, where no active RFC specifies a way to query multiple families in single query. It is up to stub resolver to choose what queries to make, DNS cache must just answer them appropriately.

I think bug title should be modified to reflect the real issue solved.
Comment 7 Michael Catanzaro 2021-03-23 13:08:39 UTC 
(In reply to Petr Menšík from comment #5)
> Wait, wasn't one of primary reasons to enable systemd-resolved ability for
> caching? With caching disabled, it would allow just dumb split-DNS
> configuration, but nothing more?

Yes. Disabling the cache is an emergency workaround only. It will be reenabled soon, I hope. Unfortunately systemd-resolved 248 release candidates have all been very unstable, and this is now the third emergency bug in a row. I thought the cache should be good to reenable with this update, but it seems Zbigniew wants further testing, which is pretty understandable.

> Do we block DNSSEC in default installation just for possibility of
> split-DNS, which not everyone might need?

Does the DNSSEC support depend on the DNS cache?

Split DNS was the primary requirement for me, since it affects correctness of results. Without the DNS cache, my DNS is a little slower. Without split DNS, I cannot access any redhat.com internal resources, since requests for internal domains would be sent to my privacy VPN provider instead. I'm not willing to turn off my privacy VPN due to my ISP's unsatisfactory privacy policy.

> It seems systemd-resolved has chosen wrong section to add also AAAA record.
> It might use ADDITIONAL, but it uses AUTHORITY. That is supposed to be used
> to be used only for SOA record.

Thanks for noting this. Can you report it upstream please?

(In reply to Petr Menšík from comment #6)
> Note: returning IPv6 address in DNS, even when IPv6 default route does not
> exist, is correct. When AAAA record is requested and exists, it must be
> returned to that query.
> This bug is not really about returning IPV6 address also, but omitting IPv4
> response on A query, where only IPv4 or no answer should be returned. Both
> address families is returned by getaddrinfo() even without systemd-resolved
> active and it should work the same way.
> 
> systemd-resolved might be free to adapt resolvectl query <name>, where
> address family is not specified. But it must not change address family
> queries depending on IPv6 route presence, when specific address family was
> requested. Especially not DNS requests received over stub resolver, where no
> active RFC specifies a way to query multiple families in single query. It is
> up to stub resolver to choose what queries to make, DNS cache must just
> answer them appropriately.
> 
> I think bug title should be modified to reflect the real issue solved.

I was wondering about that. Again, it would be helpful to report further problems upstream to try to get this as good as possible.
Comment 8 Zbigniew Jędrzejewski-Szmek 2021-03-23 17:00:37 UTC 
This seems to be fixed with the latest set of patches. If there are no reports to the
contrary, I think we can reenable the cache in the next update.

I only see a minor issue: for AAAA?, we return the A address in authority section, and to an
A?, we don't return AAAA at all. I think that the section shouldn't be authority but additional,
and also the asymmetry in handling of A and AAAA is suspicious.

https://tools.ietf.org/html/rfc1034#section-3.7 doesn't really answer if this is wrong or not.
If someone has a better pointer, please share.
Comment 9 Michael Catanzaro 2021-03-23 17:05:36 UTC 
(In reply to Zbigniew Jędrzejewski-Szmek from comment #8)
> I only see a minor issue: for AAAA?, we return the A address in authority
> section, and to an
> A?, we don't return AAAA at all. I think that the section shouldn't be
> authority but additional,
> and also the asymmetry in handling of A and AAAA is suspicious.

Petr recommended using the ADDITIONAL section in comment #5.
Comment 10 Zbigniew Jędrzejewski-Szmek 2021-03-23 17:19:38 UTC 
*** Bug 1940993 has been marked as a duplicate of this bug. ***
Comment 11 Fedora Update System 2021-03-24 02:44:18 UTC 
FEDORA-2021-ea92e5703f has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-ea92e5703f`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-ea92e5703f

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
Comment 12 Fedora Update System 2021-03-25 00:18:50 UTC 
FEDORA-2021-ea92e5703f has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.
Comment 13 Zbigniew Jędrzejewski-Szmek 2021-03-25 11:43:38 UTC 
(In reply to Michael Catanzaro from comment #9)
> (In reply to Zbigniew Jędrzejewski-Szmek from comment #8)
> > I only see a minor issue: for AAAA?, we return the A address in authority
> > section, and to an
> > A?, we don't return AAAA at all. I think that the section shouldn't be
> > authority but additional,
> > and also the asymmetry in handling of A and AAAA is suspicious.
> 
> Petr recommended using the ADDITIONAL section in comment #5.

This part is fixed by https://github.com/systemd/systemd/pull/19112.
Note You need to log in before you can comment on or make changes to this bug.