Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 2152903 - 2.57.6 snapd on EL7 breaks certbot
Summary: 2.57.6 snapd on EL7 breaks certbot
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora EPEL
Classification: Fedora
Component: snapd
Version: epel7
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: Zygmunt Krynicki
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 2153040 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-12-13 13:51 UTC by Danie de Jager
Modified: 2022-12-25 02:06 UTC (History)
9 users (show)

Fixed In Version: snapd-2.57.6-2.fc38 snapd-2.57.6-2.el7 snapd-2.57.6-2.el8 snapd-2.57.6-2.el9
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-12-16 08:44:41 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Danie de Jager 2022-12-13 13:51:14 UTC
Description of problem:
The current version and 2.56.2 breaks certbot so that it can no longer run or issue certificates.

Version-Release number of selected component (if applicable):
2.56.2+

How reproducible:
100% of the time

Steps to Reproduce:
1. upgrade snap
2. run certbot, it will crash

Actual results:
list cert:
→ certbot certificates
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x0]

runtime stack:
runtime.throw({0x7257cd, 0x0})
        /usr/lib/golang/src/runtime/panic.go:1198 +0x71 fp=0x7ffe1d6c4f20 sp=0x7ffe1d6c4ef0 pc=0x435291
runtime.sigpanic()
        /usr/lib/golang/src/runtime/signal_unix.go:719 +0x396 fp=0x7ffe1d6c4f70 sp=0x7ffe1d6c4f20 pc=0x44aa16

goroutine 1 [syscall, locked to thread]:
runtime.cgocall(0x5e9a70, 0xc0000d96c0)
        /usr/lib/golang/src/runtime/cgocall.go:156 +0x5c fp=0xc0000d9698 sp=0xc0000d9660 pc=0x40551c
crypto/internal/boring._Cfunc__goboringcrypto_DLOPEN_OPENSSL()
        _cgo_gotypes.go:603 +0x49 fp=0xc0000d96c0 sp=0xc0000d9698 pc=0x5366c9
crypto/internal/boring.init.0()
        /usr/lib/golang/src/crypto/internal/boring/boring.go:46 +0x45 fp=0xc0000d96f8 sp=0xc0000d96c0 pc=0x537445
runtime.doInit(0xad84a0)
        /usr/lib/golang/src/runtime/proc.go:6498 +0x123 fp=0xc0000d9830 sp=0xc0000d96f8 pc=0x4449a3
runtime.doInit(0xad61a0)
        /usr/lib/golang/src/runtime/proc.go:6475 +0x71 fp=0xc0000d9968 sp=0xc0000d9830 pc=0x4448f1
runtime.doInit(0xad9280)
        /usr/lib/golang/src/runtime/proc.go:6475 +0x71 fp=0xc0000d9aa0 sp=0xc0000d9968 pc=0x4448f1
runtime.doInit(0xad8260)
        /usr/lib/golang/src/runtime/proc.go:6475 +0x71 fp=0xc0000d9bd8 sp=0xc0000d9aa0 pc=0x4448f1
runtime.doInit(0xad9bc0)
        /usr/lib/golang/src/runtime/proc.go:6475 +0x71 fp=0xc0000d9d10 sp=0xc0000d9bd8 pc=0x4448f1
runtime.doInit(0xad7220)
        /usr/lib/golang/src/runtime/proc.go:6475 +0x71 fp=0xc0000d9e48 sp=0xc0000d9d10 pc=0x4448f1
runtime.doInit(0xad7fe0)
        /usr/lib/golang/src/runtime/proc.go:6475 +0x71 fp=0xc0000d9f80 sp=0xc0000d9e48 pc=0x4448f1
runtime.main()
        /usr/lib/golang/src/runtime/proc.go:238 +0x1e6 fp=0xc0000d9fe0 sp=0xc0000d9f80 pc=0x437926
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0000d9fe8 sp=0xc0000d9fe0 pc=0x464361

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000036fb0 sp=0xc000036f90 pc=0x437d36
runtime.goparkunlock(...)
        /usr/lib/golang/src/runtime/proc.go:372
runtime.forcegchelper()
        /usr/lib/golang/src/runtime/proc.go:306 +0xad fp=0xc000036fe0 sp=0xc000036fb0 pc=0x437bcd
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000036fe8 sp=0xc000036fe0 pc=0x464361
created by runtime.init.8
        /usr/lib/golang/src/runtime/proc.go:294 +0x25

goroutine 3 [GC sweep wait]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc0000377b0 sp=0xc000037790 pc=0x437d36
runtime.goparkunlock(...)
        /usr/lib/golang/src/runtime/proc.go:372
runtime.bgsweep()
        /usr/lib/golang/src/runtime/mgcsweep.go:163 +0x88 fp=0xc0000377e0 sp=0xc0000377b0 pc=0x4243a8
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0000377e8 sp=0xc0000377e0 pc=0x464361
created by runtime.gcenable
        /usr/lib/golang/src/runtime/mgc.go:181 +0x55

goroutine 4 [GC scavenge wait]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000037f80 sp=0xc000037f60 pc=0x437d36
runtime.goparkunlock(...)
        /usr/lib/golang/src/runtime/proc.go:372
runtime.bgscavenge()
        /usr/lib/golang/src/runtime/mgcscavenge.go:265 +0xcd fp=0xc000037fe0 sp=0xc000037f80 pc=0x4224ad
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000037fe8 sp=0xc000037fe0 pc=0x464361
created by runtime.gcenable
        /usr/lib/golang/src/runtime/mgc.go:182 +0x65

goroutine 18 [finalizer wait]:
runtime.gopark(0xaec640, 0xc000036770, 0xf1, 0x48, 0xad75e0)
        /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000036630 sp=0xc000036610 pc=0x437d36
runtime.goparkunlock(...)
        /usr/lib/golang/src/runtime/proc.go:372
runtime.runfinq()
        /usr/lib/golang/src/runtime/mfinal.go:177 +0xb3 fp=0xc0000367e0 sp=0xc000036630 pc=0x419f33
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0000367e8 sp=0xc0000367e0 pc=0x464361
created by runtime.createfing
        /usr/lib/golang/src/runtime/mfinal.go:157 +0x45
Aborted


Installing certbot
→ snap install certbot --classic
error: cannot perform the following tasks:
- Run hook prepare-plug-plugin of snap "certbot" (run hook "prepare-plug-plugin":
-----
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x0]

runtime stack:
runtime.throw({0x7257cd, 0x0})
        /usr/lib/golang/src/runtime/panic.go:1198 +0x71 fp=0x7ffeb7291e90 sp=0x7ffeb7291e60 pc=0x435291
runtime.sigpanic()
        /usr/lib/golang/src/runtime/signal_unix.go:719 +0x396 fp=0x7ffeb7291ee0 sp=0x7ffeb7291e90 pc=0x44aa16

goroutine 1 [syscall, locked to thread]:
runtime.cgocall(0x5e9a70, 0xc0000db6c0)
        /usr/lib/golang/src/runtime/cgocall.go:156 +0x5c fp=0xc0000db698 sp=0xc0000db660 pc=0x40551c
crypto/internal/boring._Cfunc__goboringcrypto_DLOPEN_OPENSSL()
        _cgo_gotypes.go:603 +0x49 fp=0xc0000db6c0 sp=0xc0000db698 pc=0x5366c9
crypto/internal/boring.init.0()
        /usr/lib/golang/src/crypto/internal/boring/boring.go:46 +0x45 fp=0xc0000db6f8 sp=0xc0000db6c0 pc=0x537445
runtime.doInit(0xad84a0)
        /usr/lib/golang/src/runtime/proc.go:6498 +0x123 fp=0xc0000db830 sp=0xc0000db6f8 pc=0x4449a3
runtime.doInit(0xad61a0)
        /usr/lib/golang/src/runtime/proc.go:6475 +0x71 fp=0xc0000db968 sp=0xc0000db830 pc=0x4448f1
runtime.doInit(0xad9280)
        /usr/lib/golang/src/runtime/proc.go:6475 +0x71 fp=0xc0000dbaa0 sp=0xc0000db968 pc=0x4448f1
runtime.doInit(0xad8260)
        /usr/lib/golang/src/runtime/proc.go:6475 +0x71 fp=0xc0000dbbd8 sp=0xc0000dbaa0 pc=0x4448f1
runtime.doInit(0xad9bc0)
        /usr/lib/golang/src/runtime/proc.go:6475 +0x71 fp=0xc0000dbd10 sp=0xc0000dbbd8 pc=0x4448f1
runtime.doInit(0xad7220)
        /usr/lib/golang/src/runtime/proc.go:6475 +0x71 fp=0xc0000dbe48 sp=0xc0000dbd10 pc=0x4448f1
runtime.doInit(0xad7fe0)
        /usr/lib/golang/src/runtime/proc.go:6475 +0x71 fp=0xc0000dbf80 sp=0xc0000dbe48 pc=0x4448f1
runtime.main()
        /usr/lib/golang/src/runtime/proc.go:238 +0x1e6 fp=0xc0000dbfe0 sp=0xc0000dbf80 pc=0x437926
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0000dbfe8 sp=0xc0000dbfe0 pc=0x464361

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000032fb0 sp=0xc000032f90 pc=0x437d36
runtime.goparkunlock(...)
        /usr/lib/golang/src/runtime/proc.go:372
runtime.forcegchelper()
        /usr/lib/golang/src/runtime/proc.go:306 +0xad fp=0xc000032fe0 sp=0xc000032fb0 pc=0x437bcd
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000032fe8 sp=0xc000032fe0 pc=0x464361
created by runtime.init.8
        /usr/lib/golang/src/runtime/proc.go:294 +0x25

goroutine 3 [GC sweep wait]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc0000337b0 sp=0xc000033790 pc=0x437d36
runtime.goparkunlock(...)
        /usr/lib/golang/src/runtime/proc.go:372
runtime.bgsweep()
        /usr/lib/golang/src/runtime/mgcsweep.go:163 +0x88 fp=0xc0000337e0 sp=0xc0000337b0 pc=0x4243a8
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0000337e8 sp=0xc0000337e0 pc=0x464361
created by runtime.gcenable
        /usr/lib/golang/src/runtime/mgc.go:181 +0x55

goroutine 4 [GC scavenge wait]:
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000033f80 sp=0xc000033f60 pc=0x437d36
runtime.goparkunlock(...)
        /usr/lib/golang/src/runtime/proc.go:372
runtime.bgscavenge()
        /usr/lib/golang/src/runtime/mgcscavenge.go:265 +0xcd fp=0xc000033fe0 sp=0xc000033f80 pc=0x4224ad
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000033fe8 sp=0xc000033fe0 pc=0x464361
created by runtime.gcenable
        /usr/lib/golang/src/runtime/mgc.go:182 +0x65

goroutine 18 [finalizer wait]:
runtime.gopark(0xaec640, 0xc000032770, 0xf1, 0x48, 0xad75e0)
        /usr/lib/golang/src/runtime/proc.go:366 +0xd6 fp=0xc000032630 sp=0xc000032610 pc=0x437d36
runtime.goparkunlock(...)
        /usr/lib/golang/src/runtime/proc.go:372
runtime.runfinq()
        /usr/lib/golang/src/runtime/mfinal.go:177 +0xb3 fp=0xc0000327e0 sp=0xc000032630 pc=0x419f33
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0000327e8 sp=0xc0000327e0 pc=0x464361
created by runtime.createfing
        /usr/lib/golang/src/runtime/mfinal.go:157 +0x45
-----)


Expected results:


Additional info:
I reported the issue here https://github.com/certbot/certbot/issues/9332 for snapd  2.56.2
https://forum.snapcraft.io/t/el7-snap-2-56-2-causes-certbot-to-fail/30669

Comment 1 Maciek Borzecki 2022-12-13 14:19:59 UTC
AFAICT this is caused by boringcrypto, a FIPS compliant build of Go that is used in EPEL. Those binaries are built statically as they are executed inside the snap's mount namespace. Apparently boringcrypto mode still tries to dlopen() openssl, and probably trips over the openssl binary from the base of the snap. I can try to disable boringcrypto for those specific binaries. Ideally, boringcrypto should not break Go.

Comment 2 Danie de Jager 2022-12-13 14:27:40 UTC
Why is this only an issue on EL7 though? I believe I saw mention of boringcrypto in the previous posts I made about this.

Comment 3 Alex Zorin 2022-12-13 21:22:57 UTC
*** Bug 2153040 has been marked as a duplicate of this bug. ***

Comment 4 Alex Zorin 2022-12-16 00:13:05 UTC
Digging a bit, dlopen(libcrypto.so) with default Go linker flags works fine, but it crashes if you build the Go program with -static (like is done for snap-exec, snapctl, snap-update-ns). 

The crash itself is produced inside an `fopen` call in RHEL's `openssl-1.0.2j-deprecate-algos.patch`, but I think even in the absence of that, something else would trigger a crash soon after anyway.

I noticed an "no_openssl" Go build tag in the Go sources, which isn't part of mainline Go but it's in the (now deprecated) FIPS fork. 

I tried this build tag for the static binaries and placing them on top of the current rpm, I am able to install and run Certbot (and other classic snaps) successfully 🎉.

@Maciek Borzecki what are your thoughts about rebuilding the affected packages with these tags and making a release? We have a large number of users affected by this issue at the moment. Thank you!

--- 

One worrying note, I think the programs are still unsafe due to the way that glibc is linked statically. There are warnings in the build about this:

# github.com/snapcore/snapd/cmd/snapctl
/tmp/go-link-605309214/000008.o: In function `mygetgrouplist':
/_/os/user/getgrouplist_unix.go:15: warning: Using 'getgrouplist' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/tmp/go-link-605309214/000007.o: In function `mygetgrgid_r':
/_/os/user/cgo_lookup_unix.go:37: warning: Using 'getgrgid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/tmp/go-link-605309214/000007.o: In function `mygetgrnam_r':
/_/os/user/cgo_lookup_unix.go:42: warning: Using 'getgrnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/tmp/go-link-605309214/000007.o: In function `mygetpwnam_r':
/_/os/user/cgo_lookup_unix.go:32: warning: Using 'getpwnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/tmp/go-link-605309214/000007.o: In function `mygetpwuid_r':
/_/os/user/cgo_lookup_unix.go:27: warning: Using 'getpwuid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/tmp/go-link-605309214/000004.o: In function `_cgo_6cc2654a8ed3_C2func_getaddrinfo':
/tmp/go-build/cgo-gcc-prolog:58: warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking

I suspect that any code path which hits these functions might crash the binaries again. But, at least classic snap support seems to work again.

Comment 5 Maciek Borzecki 2022-12-16 07:11:07 UTC
(In reply to Alex Zorin from comment #4)
> Digging a bit, dlopen(libcrypto.so) with default Go linker flags works fine,
> but it crashes if you build the Go program with -static (like is done for
> snap-exec, snapctl, snap-update-ns). 
> 
> The crash itself is produced inside an `fopen` call in RHEL's
> `openssl-1.0.2j-deprecate-algos.patch`, but I think even in the absence of
> that, something else would trigger a crash soon after anyway.
> 
> I noticed an "no_openssl" Go build tag in the Go sources, which isn't part
> of mainline Go but it's in the (now deprecated) FIPS fork. 
> 
> I tried this build tag for the static binaries and placing them on top of
> the current rpm, I am able to install and run Certbot (and other classic
> snaps) successfully 🎉.
> 
> @Maciek Borzecki what are your thoughts about rebuilding the affected
> packages with these tags and making a release? We have a large number of
> users affected by this issue at the moment. Thank you!

Thank you for looking into this!

I'll try to push out some builds today.

> 
> --- 
> 
> One worrying note, I think the programs are still unsafe due to the way that
> glibc is linked statically. There are warnings in the build about this:
> 
> # github.com/snapcore/snapd/cmd/snapctl
> /tmp/go-link-605309214/000008.o: In function `mygetgrouplist':
> /_/os/user/getgrouplist_unix.go:15: warning: Using 'getgrouplist' in
> statically linked applications requires at runtime the shared libraries from
> the glibc version used for linking
> /tmp/go-link-605309214/000007.o: In function `mygetgrgid_r':
> /_/os/user/cgo_lookup_unix.go:37: warning: Using 'getgrgid_r' in statically
> linked applications requires at runtime the shared libraries from the glibc
> version used for linking
> /tmp/go-link-605309214/000007.o: In function `mygetgrnam_r':
> /_/os/user/cgo_lookup_unix.go:42: warning: Using 'getgrnam_r' in statically
> linked applications requires at runtime the shared libraries from the glibc
> version used for linking
> /tmp/go-link-605309214/000007.o: In function `mygetpwnam_r':
> /_/os/user/cgo_lookup_unix.go:32: warning: Using 'getpwnam_r' in statically
> linked applications requires at runtime the shared libraries from the glibc
> version used for linking
> /tmp/go-link-605309214/000007.o: In function `mygetpwuid_r':
> /_/os/user/cgo_lookup_unix.go:27: warning: Using 'getpwuid_r' in statically
> linked applications requires at runtime the shared libraries from the glibc
> version used for linking
> /tmp/go-link-605309214/000004.o: In function
> `_cgo_6cc2654a8ed3_C2func_getaddrinfo':
> /tmp/go-build/cgo-gcc-prolog:58: warning: Using 'getaddrinfo' in statically
> linked applications requires at runtime the shared libraries from the glibc
> version used for linking
> 
> I suspect that any code path which hits these functions might crash the
> binaries again. But, at least classic snap support seems to work again.

This is fine. AFAIK the binaries in question do not execute any code that would trigger these calls.

Comment 6 Fedora Update System 2022-12-16 08:41:17 UTC
FEDORA-2022-8f35a2f4f2 has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2022-8f35a2f4f2

Comment 7 Fedora Update System 2022-12-16 08:44:41 UTC
FEDORA-2022-8f35a2f4f2 has been pushed to the Fedora 38 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 8 Fedora Update System 2022-12-16 08:48:19 UTC
FEDORA-EPEL-2022-4790972cca has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-4790972cca

Comment 9 Fedora Update System 2022-12-16 08:48:20 UTC
FEDORA-EPEL-2022-80b8c04562 has been submitted as an update to Fedora EPEL 9. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-80b8c04562

Comment 10 Fedora Update System 2022-12-16 08:48:21 UTC
FEDORA-EPEL-2022-548bcdf1ae has been submitted as an update to Fedora EPEL 8. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-548bcdf1ae

Comment 11 Fedora Update System 2022-12-17 00:23:36 UTC
FEDORA-EPEL-2022-548bcdf1ae has been pushed to the Fedora EPEL 8 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-548bcdf1ae

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 12 Fedora Update System 2022-12-17 02:00:43 UTC
FEDORA-EPEL-2022-4790972cca has been pushed to the Fedora EPEL 7 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-4790972cca

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 13 Fedora Update System 2022-12-17 02:36:04 UTC
FEDORA-EPEL-2022-80b8c04562 has been pushed to the Fedora EPEL 9 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-80b8c04562

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 14 Fedora Update System 2022-12-18 02:15:18 UTC
FEDORA-EPEL-2022-4790972cca has been pushed to the Fedora EPEL 7 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 15 Fedora Update System 2022-12-25 00:39:28 UTC
FEDORA-EPEL-2022-548bcdf1ae has been pushed to the Fedora EPEL 8 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 16 Fedora Update System 2022-12-25 02:06:27 UTC
FEDORA-EPEL-2022-80b8c04562 has been pushed to the Fedora EPEL 9 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.