Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1797646 - Fedora 31 - podman, docker and other packages built with Go < 1.13.3 panic when doing TLS/ECDSA on IBM z15
Summary: Fedora 31 - podman, docker and other packages built with Go < 1.13.3 panic wh...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: golang
Version: 31
Hardware: s390x
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Jakub Čajka
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: ZedoraTracker
TreeView+ depends on / blocked
 
Reported: 2020-02-03 14:51 UTC by Hanns-Joachim Uhl
Modified: 2020-08-10 14:33 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-08-10 14:33:02 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
IBM Linux Technology Center 183508 0 None None None 2020-02-03 14:52:39 UTC

Description Hanns-Joachim Uhl 2020-02-03 14:51:19 UTC
Hello Fedora People,

Go based Fedora 31 packages including podman and docker (moby-engine) have been built using 1.13.0 ≤ Go < 1.13.3.

This version of Go is affected by the Go issue 34927 [0]
which results in severely broken functionality and according to the code author[1] also a potential security issue when executed on IBM z15 and later.

In particular this was discovered because

   podman pull hello-world

results in an immediate panic due to stack corruption when run on an IBM z15.

The current version of Go in Fedora 31 (1.13.6) already includes the fix but all packages built using Go need to be rebuilt as this will affect any use of of the Go TLS support on z15.

[0] https://github.com/golang/go/issues/34927
[1] https://github.com/golang/go/issues/34927#issuecomment-542772489

Comment 1 Jakub Čajka 2020-02-03 15:23:46 UTC
(In reply to Hanns-Joachim Uhl from comment #0)
> Hello Fedora People,
> 
> Go based Fedora 31 packages including podman and docker (moby-engine) have
> been built using 1.13.0 ≤ Go < 1.13.3.
> 
> This version of Go is affected by the Go issue 34927 [0]
> which results in severely broken functionality and according to the code
> author[1] also a potential security issue when executed on IBM z15 and later.
> 
> In particular this was discovered because
> 
>    podman pull hello-world
> 
> results in an immediate panic due to stack corruption when run on an IBM z15.
> 
> The current version of Go in Fedora 31 (1.13.6) already includes the fix but
> all packages built using Go need to be rebuilt as this will affect any use
> of of the Go TLS support on z15.
> 
> [0] https://github.com/golang/go/issues/34927
> [1] https://github.com/golang/go/issues/34927#issuecomment-542772489

What version of podman are you using? Did you installed all updates on your system? Looking at the builds of podman there are NVRs that have been built in line with the report and upstream issue, with go1.13.3=>, i.e. you should not see this issue on fully up to date f31 installs. I'm not sure that we can conduct any wide spread rebuild of packages at the moment(but I would assume that most of the high profile Go based packages got already rebuilt too).

Do you have list of packages that you have hit this issue with?

Comment 2 IBM Bug Proxy 2020-02-04 08:40:24 UTC
------- Comment From Niklas.Schnelle 2020-02-04 03:31 EDT-------
I was worried about updated versions too since we have an internal mirror that has a different update schedule. I therefore tried with the version marked for Fedora 31 at

https://src.fedoraproject.org/rpms/podman

Which currently shows podman-1.6.2-2.fc31 I installed that via

https://koji.fedoraproject.org/koji/buildinfo?buildID=1402278

and it still shows the issue. I just grepped 'go1.13' in the build log

https://kojipkgs.fedoraproject.org//packages/podman/1.6.2/2.fc31/data/logs/s390x/build.log

and it looks like that was indeed build with Go 1.13.1. Maybe the builder uses an old Go version?

I also tried with the podman-1.8.0-0.2.dev.gitc4f6d56.fc32.s390x.rpm package that is currently in Fedora 32 and that works.

I don't have a list of packages but since Go is particular suited to network applications I would expect a large percentage to be affected by broken TLS.

Comment 3 Jakub Čajka 2020-02-06 15:53:21 UTC
(In reply to IBM Bug Proxy from comment #2)
> ------- Comment From Niklas.Schnelle 2020-02-04 03:31 EDT-------
> I was worried about updated versions too since we have an internal mirror
> that has a different update schedule. I therefore tried with the version
> marked for Fedora 31 at
> 
> https://src.fedoraproject.org/rpms/podman
> 
> Which currently shows podman-1.6.2-2.fc31 I installed that via
> 
> https://koji.fedoraproject.org/koji/buildinfo?buildID=1402278
> 
> and it still shows the issue. I just grepped 'go1.13' in the build log
> 
> https://kojipkgs.fedoraproject.org//packages/podman/1.6.2/2.fc31/data/logs/
> s390x/build.log
> 
> and it looks like that was indeed build with Go 1.13.1. Maybe the builder
> uses an old Go version?

Currently for me on f31 s390x VM, fully up to date, I have podman-1.7.0-2.fc31 which has been built with go1.13.5(https://kojipkgs.fedoraproject.org//packages/podman/1.7.0/2.fc31/data/logs/s390x/root.log). Could you please verify that? In current Fedora 31 buildroot we have go1.13.6 atm.
My main point is that you shouldn't see this issue with most of the possibly affected Go packages on fully up to date systems(i.e. running `dnf update` should fix your issue), other issue might be systems provisioned from distribution deliverables where you are not running `dnf update` for whatever reason(those might be possible to re-spin, I will need to check with sharkcz about that option, if that is your case).

> 
> I also tried with the podman-1.8.0-0.2.dev.gitc4f6d56.fc32.s390x.rpm package
> that is currently in Fedora 32 and that works.
> 
> I don't have a list of packages but since Go is particular suited to network
> applications I would expect a large percentage to be affected by broken TLS.

AFAIK we currently don't have tools to track/find all packages that might be affected and there is no way we can do limited mass-rebuild of all Go using packages in released Fedora(especially as most of the affected packages have been already rebuilt with Go compiler that has fix for the issue).

Comment 4 IBM Bug Proxy 2020-02-06 16:31:15 UTC
------- Comment From Niklas.Schnelle 2020-02-06 11:24 EDT-------
Oh okay so https://src.fedoraproject.org/rpms/podman just doesn't show the version that would be installed with "dnf update".
I'll have to find a better way to search for the current version used in Fedora, any pointers?

I can confirm that the version you linked to works fine on my z15 LPAR.

So yeah, I guess this really has been resolved already on your end at least for all the packages already rebuild.

Comment 5 IBM Bug Proxy 2020-08-10 14:31:36 UTC
------- Comment From hannsj_uhl.com 2020-08-10 10:22 EDT-------
... closing this bugzilla per the previous comments ...


Note You need to log in before you can comment on or make changes to this bug.