Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1935084 - [abrt] systemd: greedy_realloc(): systemd-resolved killed by SIGABRT
Summary: [abrt] systemd: greedy_realloc(): systemd-resolved killed by SIGABRT
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: systemd
Version: 33
Hardware: x86_64
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: systemd-maint
QA Contact: Fedora Extras Quality Assurance
URL: https://retrace.fedoraproject.org/faf...
Whiteboard: abrt_hash:1a045c90f9752cf34a934316127...
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-03-04 11:03 UTC by Colin.Simpson
Modified: 2021-04-01 15:20 UTC (History)
11 users (show)

Fixed In Version: systemd-248~rc3-1.fc35 systemd-248~rc4-3.fc34 systemd-246.13-1.fc33
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-03-25 00:18:47 UTC
Type: ---
Embargoed:

Attachments (Terms of Use)
File: backtrace (27.46 KB, text/plain)
2021-03-04 11:03 UTC, Colin.Simpson 		no flags Details
File: core_backtrace (6.83 KB, text/plain)
2021-03-04 11:03 UTC, Colin.Simpson 		no flags Details
File: cpuinfo (2.52 KB, text/plain)
2021-03-04 11:03 UTC, Colin.Simpson 		no flags Details
File: dso_list (4.67 KB, text/plain)
2021-03-04 11:03 UTC, Colin.Simpson 		no flags Details
File: environ (359 bytes, text/plain)
2021-03-04 11:03 UTC, Colin.Simpson 		no flags Details
File: limits (1.29 KB, text/plain)
2021-03-04 11:03 UTC, Colin.Simpson 		no flags Details
File: maps (30.33 KB, text/plain)
2021-03-04 11:03 UTC, Colin.Simpson 		no flags Details
File: mountinfo (7.74 KB, text/plain)
2021-03-04 11:03 UTC, Colin.Simpson 		no flags Details
File: open_fds (1.74 KB, text/plain)
2021-03-04 11:03 UTC, Colin.Simpson 		no flags Details
File: proc_pid_status (1.33 KB, text/plain)
2021-03-04 11:03 UTC, Colin.Simpson 		no flags Details
Valgrind output (7.62 KB, text/plain)
2021-03-08 10:46 UTC, Colin.Simpson 		no flags Details
Valgrind with debuginfo (8.50 KB, text/plain)
2021-03-08 12:10 UTC, Colin.Simpson 		no flags Details
View All

Description Colin.Simpson 2021-03-04 11:03:22 UTC 
Description of problem:
Using systemd-resolved in the mode where /etc/resolv.conf is a symlink to stub-resolv.conf.

The only thing maybe slightly unusual about this is this is a corporate environment where there are DNS resolves to 
find AD DC's etc so more (and slightly unusual) TXT records than a home user say.

Version-Release number of selected component:
systemd-246.10-1.fc33

Additional info:
reporter:       libreport-2.14.0
backtrace_rating: 4
cgroup:         0::/system.slice/systemd-resolved.service
cmdline:        /usr/lib/systemd/systemd-resolved
crash_function: greedy_realloc
executable:     /usr/lib/systemd/systemd-resolved
journald_cursor: s=fdd891952425460bbca7beed6c664b5f;i=1cec55;b=3378b7098c7842098a06f8019f356a34;m=13d56f1;t=5bc3ab2ec26f0;x=887c91819e0b76e6
kernel:         5.10.17-200.fc33.x86_64
rootdir:        /
runlevel:       unknown
type:           CCpp
uid:            193

Truncated backtrace:
Thread no. 1 (10 frames)
 #6 greedy_realloc at ../src/basic/alloc-util.c:63
 #7 bus_match_parse at ../src/libsystemd/sd-bus/bus-match.c:791
 #8 bus_add_match_full at ../src/libsystemd/sd-bus/sd-bus.c:3319
 #9 sd_bus_add_match_async at ../src/libsystemd/sd-bus/sd-bus.c:3405
 #10 sd_bus_track_add_name at ../src/libsystemd/sd-bus/bus-track.c:223
 #11 dns_query_bus_track at ../src/resolve/resolved-dns-query.c:1019
 #12 bus_method_resolve_address at ../src/resolve/resolved-bus.c:499
 #13 method_callbacks_run at ../src/libsystemd/sd-bus/bus-objects.c:415
 #14 object_find_and_run at ../src/libsystemd/sd-bus/bus-objects.c:1325
 #15 bus_process_object at ../src/libsystemd/sd-bus/bus-objects.c:1445
Comment 1 Colin.Simpson 2021-03-04 11:03:28 UTC 
Created attachment 1760648 [details]
File: backtrace
Comment 2 Colin.Simpson 2021-03-04 11:03:29 UTC 
Created attachment 1760649 [details]
File: core_backtrace
Comment 3 Colin.Simpson 2021-03-04 11:03:31 UTC 
Created attachment 1760650 [details]
File: cpuinfo
Comment 4 Colin.Simpson 2021-03-04 11:03:32 UTC 
Created attachment 1760651 [details]
File: dso_list
Comment 5 Colin.Simpson 2021-03-04 11:03:34 UTC 
Created attachment 1760652 [details]
File: environ
Comment 6 Colin.Simpson 2021-03-04 11:03:35 UTC 
Created attachment 1760653 [details]
File: limits
Comment 7 Colin.Simpson 2021-03-04 11:03:37 UTC 
Created attachment 1760654 [details]
File: maps
Comment 8 Colin.Simpson 2021-03-04 11:03:39 UTC 
Created attachment 1760655 [details]
File: mountinfo
Comment 9 Colin.Simpson 2021-03-04 11:03:40 UTC 
Created attachment 1760656 [details]
File: open_fds
Comment 10 Colin.Simpson 2021-03-04 11:03:42 UTC 
Created attachment 1760657 [details]
File: proc_pid_status
Comment 11 Colin.Simpson 2021-03-04 11:07:06 UTC 
This is happening maybe 100 times a day!
Comment 12 Zbigniew Jędrzejewski-Szmek 2021-03-05 16:24:45 UTC 
The crash is in greedy_realloc(), but it is the first allocation of that pointer, not a reallocation.
It seems we are calling malloc() and glibc detects a previous corruption. Unfortunately this means
that the backtrace is not terribly useful: it tells us where the error was detected, but not where
it happened.

I don't see anything wrong in the code. I also wrote a fuzzer for that part of the code, but so
far it hasn't found anything useful (https://github.com/systemd/systemd/pull/18890).

Since this is repeatable for you, maybe you could run the code under valgrind? That'd be very
helpful. In a terminal, do:

  sudo systemctl stop systemd-resolved && sudo valgrind /usr/lib/systemd/systemd-resolved

If it print outs any errors, please attach them here. You can kill it with ^C at any time.
Do 'sudo systemctl start systemd-resolved' to restart the service afterwards.
Comment 13 Colin.Simpson 2021-03-08 10:45:51 UTC 
Thanks for following up.

This bombed pretty quick for me:

[sae]csimpson: systemctl stop systemd-resolved && sudo valgrind --log-file=/root/resolved-valgrind.txt /usr/lib/systemd/systemd-resolved 
Positive Trust Anchors:
. IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Using system hostname 'sae'.
Assertion 'p->n_ref > 0' failed at src/resolve/resolved-dns-query.c:69, function dns_query_candidate_unref(). Aborting.
Aborted


I'll attach the Valgrind output.
Comment 14 Colin.Simpson 2021-03-08 10:46:51 UTC 
Created attachment 1761551 [details]
Valgrind output
Comment 15 Zbigniew Jędrzejewski-Szmek 2021-03-08 11:17:23 UTC 
Thanks for the report. Unfortunately it is not useful without debuginfo. Please add them:

  sudo dnf debuginfo-install systemd

(The version of systemd.rpm and systemd-debuginfo.rpm should match.)
Comment 16 Colin.Simpson 2021-03-08 12:09:56 UTC 

systemctl stop systemd-resolved && sudo valgrind --log-file=/root/resolved-valgrind.txt /usr/lib/systemd/systemd-resolved 
Positive Trust Anchors:
. IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Using system hostname 'sae'.
Assertion 'p->n_ref > 0' failed at src/resolve/resolved-dns-query.c:69, function dns_query_candidate_unref(). Aborting.
Aborted
Comment 17 Colin.Simpson 2021-03-08 12:10:58 UTC 
Created attachment 1761589 [details]
Valgrind with debuginfo
Comment 18 Zbigniew Jędrzejewski-Szmek 2021-03-08 13:46:59 UTC 
This should hopefully be fixed by https://github.com/systemd/systemd/pull/18832. I'll need to backport that
patch to systemd-246 though, so it'll be a few days. I'll ask you test it then.
Comment 19 Colin.Simpson 2021-03-08 14:54:27 UTC 
Great, I look forward to testing.
Comment 20 Zbigniew Jędrzejewski-Szmek 2021-03-09 13:39:49 UTC 
*** Bug 1936559 has been marked as a duplicate of this bug. ***
Comment 21 Fedora Update System 2021-03-23 11:35:31 UTC 
FEDORA-2021-1c1a870ceb has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2021-1c1a870ceb
Comment 22 Fedora Update System 2021-03-23 14:07:01 UTC 
FEDORA-2021-ea92e5703f has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2021-ea92e5703f
Comment 23 Fedora Update System 2021-03-24 02:44:16 UTC 
FEDORA-2021-ea92e5703f has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-ea92e5703f`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-ea92e5703f

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
Comment 24 Fedora Update System 2021-03-24 11:57:49 UTC 
FEDORA-2021-1c1a870ceb has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2021-1c1a870ceb
Comment 25 Fedora Update System 2021-03-25 00:18:47 UTC 
FEDORA-2021-ea92e5703f has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.
Comment 26 Fedora Update System 2021-03-25 01:31:43 UTC 
FEDORA-2021-1c1a870ceb has been pushed to the Fedora 33 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-1c1a870ceb`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-1c1a870ceb

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
Comment 27 Fedora Update System 2021-03-27 01:11:04 UTC 
FEDORA-2021-1c1a870ceb has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.
Comment 28 Colin.Simpson 2021-04-01 15:20:15 UTC 
With 
systemd-246.13-1.fc33.x86_64

I'm still seeing this crashing
traps: systemd-resolve[212290] general protection fault ip:7f4fd55982d7 sp:7ffc171684b0 error:0 in libsystemd-shared-246.so[7f4fd54c4000+1a8000]

Do you need me to re-run valgrind against this?
Note You need to log in before you can comment on or make changes to this bug.