Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1725437 - dirmngr endlessly receives service unavailable 503 error
Summary: dirmngr endlessly receives service unavailable 503 error
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: gnupg2
Version: 29
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Red Hat Crypto Team
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-06-30 15:56 UTC by Stan King
Modified: 2020-11-05 10:34 UTC (History)
4 users (show)

Fixed In Version: gnupg2-2.2.17-1.fc30 gnupg2-2.2.17-1.fc29
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-07-18 17:56:11 UTC
Type: Bug
Embargoed:

Attachments (Terms of Use)
First 500 lines of dirmngr.log, debug level set to advanced. (58.18 KB, text/plain)
2019-06-30 15:56 UTC, Stan King 		no flags Details
View All

Description Stan King 2019-06-30 15:56:35 UTC 
Created attachment 1586026 [details]
First 500 lines of dirmngr.log, debug level set to advanced.

Description of problem:

I observed an unusual amount of network traffic, and traced it to a dirmngr process.  It's receiving a 503 service unavailable error, and endlessly retries the operation, multiple times per second.

I couldn't find fixes related to this in the release notes at https://gnupg.org/download/release_notes.html, nor did I find any similar bug reports on the Redhat Bugzilla.

Version-Release number of selected component (if applicable):

gnupg2-2.2.13-1.fc29.x86_64

How reproducible:

I'm not sure what triggers the execution of dirmngr.  The system was unattended when it happened this morning.  However, when it's running, it's generally affected by this.


Steps to Reproduce:
1. wait until dirmngr starts running
2.
3.

Actual results:
Network traffic of about 10KB/second down, 5KB/second up, attempting to connect to https://keys.fedoraproject.org:443/ but receiving error 503 service unavailable each time.

Expected results:
I'm unfamiliar with the function of dirmngr, but presumably this will keep it from doing its regular job.


Additional info:

I'm attaching the first 500 lines from dirmngr.log, with debug level set to advanced.
Comment 1 Tomas Mraz 2019-07-01 08:00:04 UTC 
Do you have the keys.fedoraproject.org specified in .gnupg/dirmngr.conf?

In general I would recommend filling a bug in the upstream issue tracker at https://dev.gnupg.org/ as that would speed up the investigation and fix. Unless I am able to reproduce the problem locally I would be just a messenger.
Comment 2 Stan King 2019-07-01 11:18:21 UTC 
Tomas,

I do not specify keys in .gnupg/dirmngr.conf.  It's using the default value.

keys.fedoraproject.org is resolving to CNAME keys01.fedoraproject.org., A 140.211.169.207, which is the host returning error 503.  The whois command reports this IP range belongs to Network for Education and Research in Oregon (NERO), Eugene, OR.  Contact information is provided there also.

Aside from that element of the fedoraproject.org infrastructure providing an error code, the second problem is that dirmngr immediately retrying the operation that's doomed to fail, rather than delaying or trying another keys host.  The CNAME of keys01.fedoraproject.org makes me think there was an intention to also have keys02, keys03, etc. and rotate among them as is done with pool.ntp.org, but that was not done.  keys02.fedoraproject.org resolves to another IP within the same range as keys01.  There is no keys03.fedoraproject.org.  It seems strange that there's a dependency on a single IP range within the state of Oregon, given the international user base of Fedora.

My logfile indicates that it's receiving commands from an instance of /usr/bin/gpg2 that seems to have been started by thunderbird, in the middle of the night.

It is possible to run it manually from the command line.  Just type "dirmngr", and after producing some output ending in "OK Dirmngr 2.2.13 at your service", it will await commands.  The two commands you need to reproduce the behavior are

KEYSERVER --clear hkps://keys.fedoraproject.org:443

and

KS_GET -- 0x8657980D9AB51E50

after which you should see the effects of its retry loop.
Comment 3 Tomas Mraz 2019-07-01 16:20:47 UTC 
OK, I've created the upstream issue:

https://dev.gnupg.org/T4600
Comment 4 Stan King 2019-07-02 00:14:50 UTC 
Thank you, Tomas.  I'm sure those who use dirmngr will appreciate it.

Will you let the server continue to generate 503 until the software is fixed?  Otherwise, how will we know if the problem is repaired?  Do you have a test server than can provide error 503 on demand?

Again, thank you for reporting this problem upstream.  I'm just looking forward to verifying it when it comes downstream.
Comment 5 Tomas Mraz 2019-07-02 07:28:10 UTC 
Unfortunately I do not manage the broken keyserver.

I suppose the gnupg upstream will test the fix appropriately.
Comment 6 Todd Zullinger 2019-07-13 16:36:37 UTC 
It looks like the dirmngr issue was fixed in the recently released gnupg-2.2.17.

As far as keys.fedoraproject.org, it seems likely that the service will be discontinued, so anyone who has configured it in either gpg.conf or dirmngr.conf will need to adjust their configurations.

References:
https://pagure.io/fedora-infrastructure/issue/7988 (keys.fp.o is down?)
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/thread/G4WTK5J6VU6HVOSCNSVKL54GYU7IKZNE/ (Retire keys.fedoraproject.org?)
Comment 7 Fedora Update System 2019-07-15 10:19:56 UTC 
FEDORA-2019-28a3675529 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-28a3675529
Comment 8 Fedora Update System 2019-07-15 10:20:00 UTC 
FEDORA-2019-2f259a6c0a has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-2f259a6c0a
Comment 9 Fedora Update System 2019-07-16 00:54:12 UTC 
gnupg2-2.2.17-1.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-2f259a6c0a
Comment 10 Fedora Update System 2019-07-16 02:35:37 UTC 
gnupg2-2.2.17-1.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-28a3675529
Comment 11 Stan King 2019-07-16 02:37:22 UTC 
Unfortunately, I don't have an arrangement that can generate a 503 error to test this properly.
Comment 12 Fedora Update System 2019-07-18 17:56:11 UTC 
gnupg2-2.2.17-1.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.
Comment 13 Fedora Update System 2019-08-06 01:55:09 UTC 
gnupg2-2.2.17-1.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.
Note You need to log in before you can comment on or make changes to this bug.