without having backtrace I doubt it's possible to do anything about this. Most probably this bug has been fixed in DNF 2.x (or related components)...

Please, provide backtrace. (probably via ABRT).

I managed to re-produce it with DNF 2.2.0

$ sudo dnf update
OS                                                                                                                                                       83 MB/s |  59 MB     00:00    
Openh264                                                                                                                                                 23 kB/s | 3.1 kB     00:00    
Segmentation fault                                                        0% [                                                                        ] ---  B/s |   0  B     --:-- ETA


dmesg -T | grep segfault
[Wed Apr  5 11:19:26 2017] dnf[25601]: segfault at 300010080 ip 0000000300010080 sp 00007ffe1945fb98 error 14 in system-python[55bc6444d000+1000]


$ dnf --version
2.2.0
  Installed: dnf-0:2.3.0-0.16g2620114.fc25.noarch at 2017-04-05 10:21
  Built    :  at 2017-04-04 01:50

  Installed: rpm-0:4.13.0.1-1.fc25.x86_64 at 2017-03-27 08:53
  Built    : Fedora Project at 2017-02-24 12:48

Because it doesn't happen every time, it's hard to provide a backtrace. I will try to produce it next time I run dnf command.

Please can you provide a core dump? Or investigate core dump by your self? Here are some hints:

# to change core bump size
ulimit -c unlimited

# Change core dump file location
cat /proc/sys/kernel/core_pattern
echo "core.%e.%p" > /proc/sys/kernel/core_pattern

Then run app that fails

sudo gdb /usr/bin/python3.5 core.dnf.9299
then: "bt"

Thank you for this info. I will get it sorted.

Yesterday I've updated one of my systems to Fedora 26 beta and I had the same problem. Interestingly when I run it via gdb:
gdb -ex r --args /usr/libexec/system-python /usr/bin/dnf system-upgrade download --refresh --releasever=26 --allowerasing -y

it didn't crash. The args line:
/usr/libexec/system-python /usr/bin/dnf system-upgrade download --refresh --releasever=26 --allowerasing -y

was extracted via ps ax when running dnf directly.

Today I was upgrading one of the F25 machines via dnf and again it was segfaulting.

At least there is an easy way to re-produce it on multiple systems.

Created attachment 1288389 [details]
debug_trace

s dnf reinstall `rpm -qa` -y crashed it pretty quick. Log attached.

Please can you install missing debuginfo and then provide output again, because the most important line has ??:

Missing separate debuginfos, use: dnf debuginfo-install gpgme-1.8.0-12.fc26.x86_64 libassuan-2.4.3-2.fc26.x86_64 libattr-2.4.47-18.fc26.x86_64 libblkid-2.29.1-2.fc26.x86_64 libcap-2.25-5.fc26.x86_64 libcom_err-1.43.4-2.fc26.x86_64 libcomps-0.1.8-3g01a4759.fc26.x86_64 libffi-3.1-10.fc26.x86_64 libgpg-error-1.25-2.fc26.x86_64 libmount-2.29.1-2.fc26.x86_64 libpsl-0.17.0-2.fc26.x86_64 librepo-1.7.20-12g2865c01.fc26.x86_64 libsolv-0.6.27-23g3538163.fc26.x86_64 libssh2-1.8.0-2.fc26.x86_64 libunistring-0.9.7-1.fc26.x86_64 libuuid-2.29.1-2.fc26.x86_64 libxml2-2.9.4-2.fc26.x86_64 lz4-libs-1.7.5-3.fc26.x86_64 ncurses-libs-6.0-8.20170212.fc26.x86_64 popt-1.16-8.fc26.x86_64 python3-gpg-1.8.0-12.fc26.x86_64 python3-libcomps-0.1.8-3g01a4759.fc26.x86_64 python3-librepo-1.7.20-12g2865c01.fc26.x86_64

Also any additional information about setting of in .repo files or Katello server could be helpful.

Thanks a lot.

this bug is either in libcurl or in nss... definitely not in DNF

Unfortunately I've got them all installed:

dnf debuginfo-install gpgme-1.8.0-12.fc26.x86_64 libassuan-2.4.3-2.fc26.x86_64 libattr-2.4.47-18.fc26.x86_64 libblkid-2.29.1-2.fc26.x86_64 libcap-2.25-5.fc26.x86_64 libcom_err-1.43.4-2.fc26.x86_64 libcomps-0.1.8-3g01a4759.fc26.x86_64 libffi-3.1-10.fc26.x86_64 libgpg-error-1.25-2.fc26.x86_64 libmount-2.29.1-2.fc26.x86_64 libpsl-0.17.0-2.fc26.x86_64 librepo-1.7.20-12g2865c01.fc26.x86_64 libsolv-0.6.27-23g3538163.fc26.x86_64 libssh2-1.8.0-2.fc26.x86_64 libunistring-0.9.7-1.fc26.x86_64 libuuid-2.29.1-2.fc26.x86_64 libxml2-2.9.4-2.fc26.x86_64 lz4-libs-1.7.5-3.fc26.x86_64 ncurses-libs-6.0-8.20170212.fc26.x86_64 popt-1.16-8.fc26.x86_64 python3-gpg-1.8.0-12.fc26.x86_64 python3-libcomps-0.1.8-3g01a4759.fc26.x86_64 python3-librepo-1.7.20-12g2865c01.fc26.x86_64
enabling updates-testing-debuginfo repository
enabling rpmfusion-free-updates-testing-debuginfo repository
enabling rpmfusion-free-debuginfo repository
Last metadata expiration check: 0:00:00 ago on Mon 19 Jun 2017 10:52:54 BST.
Package glibc-debuginfo-2.25-5.fc26.x86_64 is already installed, skipping.
Package libselinux-debuginfo-2.6-6.fc26.x86_64 is already installed, skipping.
Package pcre-debuginfo-8.40-7.fc26.x86_64 is already installed, skipping.
Package libidn2-debuginfo-2.0.2-1.fc26.x86_64 is already installed, skipping.
Package openssl-debuginfo-1:1.1.0f-3.fc26.x86_64 is already installed, skipping.
Package curl-debuginfo-7.53.1-7.fc26.x86_64 is already installed, skipping.
Package krb5-debuginfo-1.15.1-8.fc26.x86_64 is already installed, skipping.
Package keyutils-debuginfo-1.5.10-1.fc26.x86_64 is already installed, skipping.
Package openldap-debuginfo-2.4.44-10.fc26.x86_64 is already installed, skipping.
Package nspr-debuginfo-4.14.0-2.fc26.x86_64 is already installed, skipping.
Package nss-debuginfo-3.30.2-1.1.fc26.x86_64 is already installed, skipping.
Package nss-util-debuginfo-3.30.2-1.0.fc26.x86_64 is already installed, skipping.
Package cyrus-sasl-debuginfo-2.1.26-32.fc26.x86_64 is already installed, skipping.
Package nss-softokn-debuginfo-3.30.2-1.0.fc26.x86_64 is already installed, skipping.
Package libdb-debuginfo-5.3.28-21.fc26.x86_64 is already installed, skipping.
Package nghttp2-debuginfo-1.21.1-1.fc26.x86_64 is already installed, skipping.
Package gcc-debuginfo-7.1.1-2.fc26.x86_64 is already installed, skipping.
Package glib2-debuginfo-2.52.2-2.fc26.x86_64 is already installed, skipping.
Package libsolv-debuginfo-0.6.27-2.fc26.x86_64 is already installed, skipping.
Package rpm-debuginfo-4.13.0.1-4.fc26.x86_64 is already installed, skipping.
Package acl-debuginfo-2.2.52-14.fc26.x86_64 is already installed, skipping.
Package elfutils-debuginfo-0.169-1.fc26.x86_64 is already installed, skipping.
Package lua-debuginfo-5.3.4-3.fc26.x86_64 is already installed, skipping.
Package python3-debuginfo-3.6.1-6.fc26.x86_64 is already installed, skipping.
Package gdbm-debuginfo-1.13-1.fc26.x86_64 is already installed, skipping.
Package sqlite-debuginfo-3.19.1-1.fc26.x86_64 is already installed, skipping.
Dependencies resolved.
Nothing to do.
Complete!

My repos config:

/etc/yum.repos.d/redhat.repo 
#
# Certificate-Based Repositories
# Managed by (rhsm) subscription-manager
#
# *** This file is auto-generated.  Changes made here will be over-written. ***
# *** Use "subscription-manager repo-override --help" if you wish to make changes. ***
#
# If this file is empty and this system is subscribed consider 
# a "yum repolist" to refresh available repos
#

[World_Programming_Ltd_Fedora_26_Updates]
metadata_expire = 1
sslclientcert = /etc/pki/entitlement/1571189452804602295.pem
baseurl = https://puppet.teamwpc.local/pulp/repos/World_Programming_Ltd/DEV/OS_Updates_Fedora_26/custom/Fedora_26/Updates
sslverify = 1
name = Updates
sslclientkey = /etc/pki/entitlement/1571189452804602295-key.pem
enabled = 1
sslcacert = /etc/rhsm/ca/katello-server-ca.pem
gpgcheck = 0

[World_Programming_Ltd_Fedora_26_Openh264]
metadata_expire = 1
sslclientcert = /etc/pki/entitlement/1571189452804602295.pem
baseurl = https://puppet.teamwpc.local/pulp/repos/World_Programming_Ltd/DEV/OS_Updates_Fedora_26/custom/Fedora_26/Openh264
sslverify = 1
name = Openh264
sslclientkey = /etc/pki/entitlement/1571189452804602295-key.pem
enabled = 1
sslcacert = /etc/rhsm/ca/katello-server-ca.pem
gpgcheck = 0

[World_Programming_Ltd_Fedora_26_OS]
metadata_expire = 1
sslclientcert = /etc/pki/entitlement/1571189452804602295.pem
baseurl = https://puppet.teamwpc.local/pulp/repos/World_Programming_Ltd/DEV/OS_Updates_Fedora_26/custom/Fedora_26/OS
sslverify = 1
name = OS
sslclientkey = /etc/pki/entitlement/1571189452804602295-key.pem
enabled = 1
sslcacert = /etc/rhsm/ca/katello-server-ca.pem
gpgcheck = 0


My CA is a 4096 bit sha512WithRSAEncryption certificate.

We think the bug is in curl. Can you (gyus from curl) confirm this? Thanks.

(In reply to Martin Hatina from comment #10)
> We think the bug is in curl. Can you (gyus from curl) confirm this?

Nope.  I cannot see any obvious bug of curl here.  Is the core file available?

Could you please check the value of data->set.fdebug inside Curl_debug()?

Is the value equal to the address that the process attempted to jump to?

Does librepo use the CURLOPT_DEBUGFUNCTION option of libcurl?

Hi,

Fedora 26 is now out and we'll upgrading a few machines. Please let me know if there's any progress or workaround for this issue.

Thanks,
Rob

Do not expect any progress from me unless the questions from comment #11 are answered.  Switching to librepo for further analysis...

Not sure if that's related or a completely different issue, but after the update to Fedora 26 dnf just hangs now. If I remove katello subscription and let it work against default fedora repos, it's all fine.

Strace:

stat("/etc/rhsm/ca/katello-server-ca.pem", {st_mode=S_IFREG|0644, st_size=2553, ...}) = 0
open("/etc/rhsm/ca/katello-server-ca.pem", O_RDONLY) = 21
fstat(21, {st_mode=S_IFREG|0644, st_size=2553, ...}) = 0
read(21, "-----BEGIN CERTIFICATE-----\nMIIH"..., 2553) = 2553
close(21)                               = 0
stat("/etc/pki/entitlement/3267844884140372615.pem", {st_mode=S_IFREG|0644, st_size=2790, ...}) = 0
stat("/etc/pki/entitlement/3267844884140372615.pem", {st_mode=S_IFREG|0644, st_size=2790, ...}) = 0
open("/etc/pki/entitlement/3267844884140372615.pem", O_RDONLY) = 21
fstat(21, {st_mode=S_IFREG|0644, st_size=2790, ...}) = 0
read(21, "-----BEGIN CERTIFICATE-----\nMIIE"..., 2790) = 2790
close(21)                               = 0
fcntl(18, F_SETLK, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=1073741824, l_len=1}) = 0
fcntl(18, F_SETLK, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=1073741826, l_len=510}) = 0
fcntl(18, F_SETLK, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=1073741824, l_len=1}) = 0
stat("/etc/pki/nssdb/cert9.db-journal", 0x7fff76a0e790) = -1 ENOENT (No such file or directory)
fstat(18, {st_mode=S_IFREG|0644, st_size=9216, ...}) = 0
pread64(18, "\0\0\0\2\0\0\0\t\0\0\0\0\0\0\0\0", 16, 24) = 16
fstat(18, {st_mode=S_IFREG|0644, st_size=9216, ...}) = 0
stat("/etc/pki/nssdb/cert9.db-wal", 0x7fff76a0e790) = -1 ENOENT (No such file or directory)
fstat(18, {st_mode=S_IFREG|0644, st_size=9216, ...}) = 0
pread64(18, "\n\0\0\0\0\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024
fcntl(18, F_SETLK, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0
futex(0x55b573563610, FUTEX_WAIT_PRIVATE, 2, NULL

(In reply to Rob Sanders from comment #14)
> Not sure if that's related or a completely different issue, but after the
> update to Fedora 26 dnf just hangs now.

I do not think it is related, sounds like bug #1470352 to me...

Created attachment 1300924 [details]
dnf ccpp

Hi Kamil,

Yes, you're right it was related to the other ticket. Unfortunately upgrading nss didn't fix this issue.

# rpm -i nss-3.31.0-1.1.fc26.x86_64.rpm 
# dnf update
Updates                                                                                                                                                  28 MB/s | 8.1 MB     00:00    
Openh264                                                                                                                                                 14 kB/s | 3.1 kB     00:00    
Segmentation fault (core dumped)                                          0% [                                                                        ] ---  B/s |   0  B     --:-- ETA


I would appreciate any help in getting to the bottom of this ticket. Please let me know what do you require and I'll do my best to provide you with all the information.

I've attacked ccpp dump from this crash.

Created attachment 1300925 [details]
backtrace from the last crash

bt full from attached coredump.

Comment on attachment 1300924 [details]
dnf ccpp

Thank you for sharing the core dump.  According to dso_list, librepo.so.0 is loaded from librepo-1.7.20-12g2865c01.fc26, which I am not able to find in Koji.  Could you please retest with librepo-1.7.20-3.fc26?

It must have been a version from dnf 2.x repo. After downgrading librepo to 1.7.20-3, I can still reproduce the bug.

(In reply to Rob Sanders from comment #19)
> It must have been a version from dnf 2.x repo. After downgrading librepo to
> 1.7.20-3, I can still reproduce the bug.

Then please upload the coredump from the crash with librepo-1.7.20-3.fc26.  I am not able to fully map attachment #1300924 [details] because of the library version mismatch.  Neither I know which dnf 2.x repo you refer to.  I am just a libcurl developer trying to help (although I am almost sure that libcurl is not the cause of this bug).

Created attachment 1301128 [details]
dnf ccpp

It looks like the compressed log file was 24MB this time so I couldn't add it here. I've uploaded it to a first free online uploader services google found. It's available here:
https://ufile.io/snu6x

Also it turns out this bug also affects packagekit - same function.

Thanks!

The data of the easy handle looks like garbage.  So I suspect that in some case curl_easy_cleanup() is called without calling curl_multi_remove_handle() first.  Unfortunately, I do not know librepo internals enough to confirm the hypothesis.  Could you please try to rebuild librepo with the following patch applied?

diff --git a/librepo/fastestmirror.c b/librepo/fastestmirror.c
index 42dee1f..a7395f6 100644
--- a/librepo/fastestmirror.c
+++ b/librepo/fastestmirror.c
@@ -63,8 +63,10 @@ lr_lrfastestmirror_free(LrFastestMirror *mirror)
 {
     if (!mirror)
         return;
+#if 0
     if (mirror->curl)
         curl_easy_cleanup(mirror->curl);
+#endif
     g_free(mirror);
 }

diff --git a/librepo/handle.c b/librepo/handle.c
index ccea79b..941eb06 100644
--- a/librepo/handle.c
+++ b/librepo/handle.c
@@ -118,8 +118,10 @@ lr_handle_free(LrHandle *handle)
 {
     if (!handle)
         return;
+#if 0
     if (handle->curl_handle)
         curl_easy_cleanup(handle->curl_handle);
+#endif
     if (handle->mirrorlist_fd != -1)
         close(handle->mirrorlist_fd);
     if (handle->metalink_fd != -1)

Unfortunately the same problem after the patch.

I have identified a possible bug in libcurl source code.  Could you please try libcurl from the following scratch build?

https://koji.fedoraproject.org/koji/taskinfo?taskID=20614836

Looks like you've nailed it!

I can no longer re-produce this issue after upgrading curl/libcurl and trust me, I've tried. Previously I could reproduce it every few seconds:

[Wed Jul 19 17:06:16 2017] yum[26175]: segfault at a1 ip 00000000000000a1 sp 00007ffd5a113e18 error 14 in system-python[55e3ef392000+2000]
[Wed Jul 19 17:06:28 2017] traps: yum[26320] general protection ip:7ff60b330aee sp:7ffe5c0a8040 error:0 in libc-2.25.so[7ff60b2be000+1c7000]
[Wed Jul 19 17:06:35 2017] traps: yum[26354] general protection ip:7fe03c3a1063 sp:7ffcf206fa28 error:0 in libcurl.so.4.4.0[7fe03c389000+7c000]
[Wed Jul 19 17:06:41 2017] traps: yum[26413] general protection ip:7fa9483df063 sp:7ffc4bbe5a08 error:0 in libcurl.so.4.4.0[7fa9483c7000+7c000]



Is it possible to push this change to F25? This will save me - and others - lots of manual patching before bigger F26 rollout.

Perfect.  Thanks for confirmation!  I will prepare builds for f25/f26 tomorrow.  It is interesting that you discovered the bug now because it was introduced in 2009 (exactly 8 years ago) by this commit:

https://github.com/curl/curl/commit/curl-7_19_5-204-g5f0cae803

Fix pushed upstream:

https://github.com/curl/curl/commit/curl-7_54_0-406-gc89eb6d
https://github.com/curl/curl/commit/curl-7_54_0-407-g42a4cd4

curl-7.51.0-8.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-7a6144ef24

curl-7.53.1-8.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-d9de901507

Hi Kamil,

I've hundreds of el6/el7 servers pulling updates from Katello and this issue doesn't affect them. I guess they use different code paths than newer fedora. Only since we introduced Fedora as a Desktop for our developers we stumbled upon this issue.

Also when you push updates from katello server to Fedora client it works as fine. Goferd must be using different code paths than dnf/packagekit.

I would like to personally thank you for investing so much time in getting to the bottom of this issue and responding so promptly.

No problem.  Thanks for the help with debugging it!  And sorry that it took so long since the initial bug report.

curl-7.51.0-8.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-7a6144ef24

curl-7.53.1-8.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-d9de901507

curl-7.53.1-8.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

curl-7.51.0-8.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.