Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1737471 - Cannot pull images from registry.fedoraproject.org
Summary: Cannot pull images from registry.fedoraproject.org
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: openssl
Version: 31
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Tomas Mraz
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: RejectedBlocker AcceptedFreezeException
Depends On:
Blocks: F31BetaFreezeException
TreeView+ depends on / blocked
 
Reported: 2019-08-05 12:58 UTC by Lukas Slebodnik
Modified: 2019-10-02 01:40 UTC (History)
30 users (show)

Fixed In Version: openssl-1.1.1d-1.fc29 openssl-1.1.1d-1.fc30 openssl-1.1.1d-1.fc31
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-10-02 01:40:42 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Lukas Slebodnik 2019-08-05 12:58:51 UTC
Description of problem:
I cannot pull images from the fedora registry with the latest version of podman in rawhide. It works well with docker hub though.
But podman on f30 works as expected

Version-Release number of selected component (if applicable):
sh$ rpm -q podman runc
podman-1.4.5-0.74.dev.git140e08e.fc31.x86_64
runc-1.0.0-97.dev.git80d35c7.fc31.x86_64

How reproducible:
Deterministic

Steps to Reproduce:
1. dnf install -y podman
2. podman pull registry.fedoraproject.org/fedora:30

Actual results:
sh# podman pull registry.fedoraproject.org/fedora:30
Trying to pull registry.fedoraproject.org/fedora:30...
  Get https://registry.fedoraproject.org/v2/: local error: tls: unexpected message
Error: error pulling image "registry.fedoraproject.org/fedora:30": unable to pull registry.fedoraproject.org/fedora:30: unable to pull image: Error initializing source docker://registry.fedoraproject.org/fedora:30: pinging docker registry returned: Get https://registry.fedoraproject.org/v2/: local error: tls: unexpected message

Expected results:
sh# podman pull registry.fedoraproject.org/fedora:30
Trying to pull registry.fedoraproject.org/fedora:30...Getting image source signatures
Copying blob e214a48be34a done
Copying config 1cabdcebde done
Writing manifest to image destination
Storing signatures
1cabdcebde84b73e0f536fabf7ea90f304de806319f3fb848ccfa8e7c5ce2ecc

Additional info:
The same issue is with moby-engine in rawhide

Comment 1 Lukas Slebodnik 2019-08-05 13:04:49 UTC
podman-2:1.4.5-0.78.dev.gite2f38cd.fc31.x86_64 is broken as well.

Comment 2 Lukas Slebodnik 2019-08-05 13:11:57 UTC
I wanted to bisect packages in rawhide but there were bunch of failed builds between release -29 and -59 and podman-2:1.4.5-0.59.dev.git680a383.fc31.x86_64  does not work either.

Comment 3 Brent Baude 2019-08-05 13:13:15 UTC
we think this is the registry, not podman or moby.

Comment 4 Lukas Slebodnik 2019-08-05 13:19:43 UTC
Then could you explain why does it work with podman on f30 ?

Comment 5 Brent Baude 2019-08-05 13:21:34 UTC
we see it randomly working in ci tests, is it possible this is a fluke on F30?

Comment 6 Lukas Slebodnik 2019-08-05 13:24:55 UTC
I've just tried and downgrading podman to 2:1.4.5-0.29.dev.gitd6b41eb.fc31.x86_6
(or moby engine moby-engine-18.09.8-2.ce.git0dd43dd.fc31 -> moby-engine-18.09.8-1.ce.git0dd43dd.fc31)
fixed the probelm.


So maybe it is related to mass rebuild in rawhide and some changes in buildroot.

Comment 7 Brent Baude 2019-08-05 13:25:55 UTC
again, we do see this in our ci tests with PRs though.

Comment 8 Lukas Slebodnik 2019-08-05 13:26:50 UTC
(In reply to Brent Baude from comment #5)
> we see it randomly working in ci tests, is it possible this is a fluke on
> F30?

No.
It works like a magic on f30 or after downgrade (comment 6)

Comment 9 Daniel Walsh 2019-08-05 13:57:44 UTC
Could be the latest version of golang,causing the issue.

Comment 10 Lukas Slebodnik 2019-08-05 14:35:08 UTC
(In reply to Lukas Slebodnik from comment #8)
> (In reply to Brent Baude from comment #5)
> > we see it randomly working in ci tests, is it possible this is a fluke on
> > F30?
> 
> No.
> It works like a magic on f30 or after downgrade (comment 6)

BTW  here is my "test"

podman-1.4.4-4.fc30.x86_64
"""
set -e

for i in {1..100}; do
    echo "------$i-----"
    docker pull registry.fedoraproject.org/fedora:30
    docker pull registry.fedoraproject.org/fedora:29
    docker pull registry.fedoraproject.org/fedora:rawhide

    docker rmi registry.fedoraproject.org/fedora:30 registry.fedoraproject.org/fedora:29 registry.fedoraproject.org/fedora:rawhide
done
"""

And it passed all times (300 pulls from fedora registry)

and negative one on rawhide with podman-1.4.5-0.74.dev.git140e08e.fc31.x86_64
"""
set -e

for i in {1..100}; do
    echo "------$i-----"
    podman pull registry.fedoraproject.org/fedora:30 && exit 1 || true
    podman pull registry.fedoraproject.org/fedora:29 && exit 1 || true
    podman pull registry.fedoraproject.org/fedora:rawhide && exit 1 || true

    podman rmi registry.fedoraproject.org/fedora:30 registry.fedoraproject.org/fedora:29 registry.fedoraproject.org/fedora:rawhide || true 
done
"""
and it failed to pull image 300 times.

Maybe it is something different in your CI but it behaves deterministicaly in my env.

Comment 11 Miloslav Trmač 2019-08-05 16:28:32 UTC
(In reply to Lukas Slebodnik from comment #6)
> I've just tried and downgrading podman to
> 2:1.4.5-0.29.dev.gitd6b41eb.fc31.x86_6 … fixed the probelm.

golang 1.12.7-1.fc31 


(failing)
> podman-1.4.5-0.74.dev.git140e08e.fc31.x86_64
golang 1.13-0.beta1.2.fc31.2



(See also historical https://github.com/containers/buildah/issues/1504 , which was, it seems, never tracked down to the real cause.)

Comment 12 Ben Cotton 2019-08-13 17:12:56 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle.
Changing version to '31'.

Comment 13 Ben Cotton 2019-08-13 18:32:40 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle.
Changing version to 31.

Comment 14 Valentin Rothberg 2019-08-14 08:12:08 UTC
We had a couple of issues with the Fedora registry lately that surfaced also in podman's CI.

Podman v1.5.0 includes fixes for those issues. @Lukas, could you try out Podman v1.5.0 in your environment and check if you still see the pull issues?

Comment 15 Lukas Slebodnik 2019-08-14 09:59:59 UTC
(In reply to Valentin Rothberg from comment #14)
> We had a couple of issues with the Fedora registry lately that surfaced also
> in podman's CI.
> 
> Podman v1.5.0 includes fixes for those issues. @Lukas, could you try out
> Podman v1.5.0 in your environment and check if you still see the pull issues?

I tried the latest version from rawhide repositories podman-1.5.1-0.6.dev.git2348c28.fc31.x86_64
But it did not help

Comment 17 Ed Santiago 2019-08-14 13:25:11 UTC
fwiw --tls-verify=false makes no difference:

   # podman  pull --tls-verify=false  registry.fedoraproject.org/fedora:30
   Trying to pull registry.fedoraproject.org/fedora:30...
     Get https://registry.fedoraproject.org/v2/: local error: tls: unexpected message
   Error: error pulling image "registry.fedoraproject.org/fedora:30": unable to pull registry.fedoraproject.org/fedora:30: unable to pull image: Error initializing source docker://registry.fedoraproject.org/fedora:30: pinging docker registry returned: Get https://registry.fedoraproject.org/v2/: local error: tls: unexpected message

Comment 18 Lukas Slebodnik 2019-08-14 14:49:09 UTC
(In reply to Ed Santiago from comment #17)
> fwiw --tls-verify=false makes no difference:
> 
>    # podman  pull --tls-verify=false  registry.fedoraproject.org/fedora:30
>    Trying to pull registry.fedoraproject.org/fedora:30...
>      Get https://registry.fedoraproject.org/v2/: local error: tls:
> unexpected message
>    Error: error pulling image "registry.fedoraproject.org/fedora:30": unable
> to pull registry.fedoraproject.org/fedora:30: unable to pull image: Error
> initializing source docker://registry.fedoraproject.org/fedora:30: pinging
> docker registry returned: Get https://registry.fedoraproject.org/v2/: local
> error: tls: unexpected message

There are more workarounds :-)
But workaround is not the solution of the problem and as I already mentioned few times it works for me on f30 with tls
so there is not any reason why it should not work work on f31/rawhide

[root@host ~]# podman  pull --tls-verify=true  registry.fedoraproject.org/fedora:30
Trying to pull registry.fedoraproject.org/fedora:30...Getting image source signatures
Copying blob e214a48be34a done
Copying config 1cabdcebde done
Writing manifest to image destination
Storing signatures
1cabdcebde84b73e0f536fabf7ea90f304de806319f3fb848ccfa8e7c5ce2ecc

[root@host ~]# rpm -q podman
podman-1.4.4-4.fc30.x86_64

And Valentin is already trying to debug issue in my env.

Comment 19 Valentin Rothberg 2019-08-21 11:11:06 UTC
Back on the issue. To me it looks like a bug in go1.13beta1 which is currently shipped in Rawhide. It works fine (on the same machine) with a skopeo built with go1.12.7.

The debug logs point to `Err:(*net.OpError)(0xc000425bd0)` (i.e., an error in the net package of go's stdlib).

Can we assign the bug to the go maintainers? I consider it a regression in go.

Comment 20 Daniel Walsh 2019-08-21 11:23:29 UTC
Reassigning to golang to get their input.

Comment 21 Fedora Blocker Bugs Application 2019-08-28 21:21:00 UTC
Proposed as a Blocker for 31-beta by Fedora user dustymabe using the blocker tracking app because:

 Since a lot of tooling now uses containers we want to be able to pull from our registry using our tools (i.e. podman). If we can't it would be a pretty bad review of the distribution. 

I don't see any thing in the existing criteria that mentions containers. That should probably change.

Comment 22 Adam Williamson 2019-08-28 21:35:07 UTC
Well, please go ahead and propose some criteria, then. It's better to do this as soon as possible and let people discuss and reflect on it, than for this to come up in a blocker review meeting without any further prep work and force us to try and invent criteria on the spot or something. Thanks!

Comment 23 Jakub Čajka 2019-08-29 08:37:31 UTC
(In reply to Valentin Rothberg from comment #19)
> Back on the issue. To me it looks like a bug in go1.13beta1 which is
> currently shipped in Rawhide. It works fine (on the same machine) with a
> skopeo built with go1.12.7.
> 

Could you please re-test with recent version of go 1.12.9 in f30(and rc1 in rawhide), that contains the recent https CVE mitigations?

> The debug logs point to `Err:(*net.OpError)(0xc000425bd0)` (i.e., an error
> in the net package of go's stdlib).
> 
> Can we assign the bug to the go maintainers? I consider it a regression in
> go.

Comment 24 Dusty Mabe 2019-08-29 15:05:52 UTC
ok it seems like disabling TLS 1.3 seems to workaround this issue:

```
GODEBUG=tls13=0 podman run --rm -it registry.fedoraproject.org/fedora:30
```

Which was recently enabled in golang 1.13 (now in Fedora 31).

Comment 25 Dusty Mabe 2019-08-29 15:09:16 UTC
Also worth noting that I can pull from quay without the workaround:

```
podman run --rm -it quay.io/fedora/fedora:32-x86_64
```

So it looks like the fedora registry might not support TLS 1.3 ?

Comment 26 Jakub Čajka 2019-08-29 15:34:01 UTC
Opened https://pagure.io/fedora-infrastructure/issue/8154 to ask for update of the registry to rule issues on this side as it is only registry that triggers this. 

For the record this will also trigger with podman build by older go(1.12) if tls1.3 is explicitly enabled via GODEBUG=tls13=1.

Comment 27 Debarshi Ray 2019-09-02 17:50:49 UTC
(In reply to Adam Williamson from comment #22)
> Well, please go ahead and propose some criteria, then. It's better to do
> this as soon as possible and let people discuss and reflect on it, than for
> this to come up in a blocker review meeting without any further prep work
> and force us to try and invent criteria on the spot or something. Thanks!

Currently this breaks Toolbox on Silverblue 31, which makes it quite inconvenient to hack on it.

Comment 28 Lukas Slebodnik 2019-09-02 18:09:41 UTC
(In reply to Debarshi Ray from comment #27)
> (In reply to Adam Williamson from comment #22)
> > Well, please go ahead and propose some criteria, then. It's better to do
> > this as soon as possible and let people discuss and reflect on it, than for
> > this to come up in a blocker review meeting without any further prep work
> > and force us to try and invent criteria on the spot or something. Thanks!
> 
> Currently this breaks Toolbox on Silverblue 31, which makes it quite
> inconvenient to hack on it.

Check comment#24

Comment 29 Michael Catanzaro 2019-09-02 18:39:41 UTC
(In reply to Dusty Mabe from comment #25) 
> So it looks like the fedora registry might not support TLS 1.3 ?

No, that's easy to disprove, just connect with something other than Go:


$ gnutls-cli registry.fedoraproject.org
Processed 149 CA certificate(s).
Resolving 'registry.fedoraproject.org:443'...
Connecting to '152.19.134.142:443'...
- Successfully sent 0 certificate(s) to server.
- Server has requested a certificate.
- Certificate type: X.509
- Got a certificate list of 2 certificates.
- Certificate[0] info:
 - subject `CN=*.fedoraproject.org,O=Red Hat Inc.,L=Raleigh,ST=North Carolina,C=US', issuer `CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US', serial 0x06001d7080298619406b9702bc155525, RSA key 4096 bits, signed using RSA-SHA256, activated `2017-02-01 00:00:00 UTC', expires `2020-05-01 12:00:00 UTC', pin-sha256="Je3KZzxF/iKAGzb7q7z6QyFf2+LvCoUoBqPx0+8FoiI="
	Public Key ID:
		sha1:de906481f661bb04a3df52240d2a219a806ec92a
		sha256:25edca673c45fe22801b36fbabbcfa43215fdbe2ef0a852806a3f1d3ef05a222
	Public Key PIN:
		pin-sha256:Je3KZzxF/iKAGzb7q7z6QyFf2+LvCoUoBqPx0+8FoiI=

- Certificate[1] info:
 - subject `CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US', issuer `CN=DigiCert High Assurance EV Root CA,OU=www.digicert.com,O=DigiCert Inc,C=US', serial 0x04e1e7a4dc5cf2f36dc02b42b85d159f, RSA key 2048 bits, signed using RSA-SHA256, activated `2013-10-22 12:00:00 UTC', expires `2028-10-22 12:00:00 UTC', pin-sha256="k2v657xBsOVe1PQRwOsHsw3bsGT2VzIqz5K+59sNQws="
- Status: The certificate is trusted. 
- Description: (TLS1.3)-(ECDHE-SECP256R1)-(RSA-PSS-RSAE-SHA256)-(AES-256-GCM)
- Options: OCSP status request,
- Handshake was completed


TLS 1.3 working fine. Anyway the server doesn't need to support TLS 1.3. The connection would just use TLS 1.2 if it didn't.

Comment 30 Lukas Slebodnik 2019-09-03 07:34:34 UTC
Could somebody from cryto team help to check TLS 1.3 in the registry and/or in golang?

Comment 31 Nikos Mavrogiannopoulos 2019-09-03 09:01:24 UTC
The TLS 1.3 support in the registry.fedoraproject.org seem to work just fine with openssl and gnutls as Michael writes above. The issue identified here may be an issue in the golang implementation which is not something we cover. Someone from the golang side would have to comment on its TLS1.3 implementation.

My recommendation would be if golang has an in progress TLS1.3 implementation, it will make sense to not enable it by default in fedora.

Comment 32 Hubert Kario 2019-09-03 11:48:53 UTC
after duplicating the ClientHello from the podman (as Go TLS implementation does not support SSLKEYLOGFILE environment variable...) the most likely issue is that the go tls implementation is not able to handle a CertificateRequest message from the server in TLS 1.3.

To answer with certainty I'd need a tcpdump from the podman connection with a corresponding key log file (AFAIK, it requires setting tls.Config KeyLogWriter property, see https://github.com/golang/go/issues/13057, and recompiling podman).

Comment 33 Clement Verna 2019-09-03 13:30:19 UTC
An easy way to reproduce https://paste.fedoraproject.org/paste/-AQrAfoCHJCpnLd0qIjIEw

running the above code with TLS 1.3 returns a nil response with the following error Get https://registry.fedoraproject.org/v2/fedora/manifests/latest: local error: tls: unexpected message

Comment 34 Clement Verna 2019-09-03 13:38:45 UTC
(In reply to Hubert Kario from comment #32)
> after duplicating the ClientHello from the podman (as Go TLS implementation
> does not support SSLKEYLOGFILE environment variable...) the most likely
> issue is that the go tls implementation is not able to handle a
> CertificateRequest message from the server in TLS 1.3.

That would make sense since we are using certificates to authenticate pushes to registry.fp.o.

> 
> To answer with certainty I'd need a tcpdump from the podman connection with
> a corresponding key log file (AFAIK, it requires setting tls.Config
> KeyLogWriter property, see https://github.com/golang/go/issues/13057, and
> recompiling podman).

Comment 35 Jakub Čajka 2019-09-03 17:01:01 UTC
Reported to upstream as https://github.com/golang/go/issues/34040 with all we know so far.

Comment 36 Jakub Čajka 2019-09-03 17:49:18 UTC
I have just built golang-1.13-0.rc2.1.fc31 and golang-1.13-0.rc2.1.fc32 with revert to the 1.12 behavior regarding tls1.3, i.e. it needs to be explicitly enabled at runtime via GODEBUG=tls13=1.

Comment 37 Dusty Mabe 2019-09-03 18:38:43 UTC
(In reply to Jakub Čajka from comment #36)
> I have just built golang-1.13-0.rc2.1.fc31 and golang-1.13-0.rc2.1.fc32 with
> revert to the 1.12 behavior regarding tls1.3, i.e. it needs to be explicitly
> enabled at runtime via GODEBUG=tls13=1.

Should we propose the fc31 as bodhi update and attach it to this bug? I assume podman will also need to be rebuilt? If so we'll need to submit a buildroot override?

Comment 38 Geoffrey Marr 2019-09-03 21:12:20 UTC
Discussed during the 2019-09-03 blocker review meeting: [1]

The decision to classify this bug as a "RejectedBlocker" and an "AcceptedFreezeException" was made because, while the new criterion that could cover this is being discussed, we don't think this absolutely needs to block Beta. We will re-consider it for a Beta or Final blocker if it's still a problem when the criterion is decided. Accepted as a freeze exception issue, however, as it is a significant problem we should try to avoid.

[1] https://meetbot.fedoraproject.org/fedora-blocker-review/2019-09-03/f31-blocker-review.2019-09-03-16.01.txt

Discussed during the 2019-09-03 blocker review meeting: [1]

Comment 39 Filippo Valsorda 2019-09-04 17:10:14 UTC
Thank you for reporting this upstream to us. I investigated and found the server to be violating the TLS 1.3 specification. Let me know if you need any more details to track this down server side and/or to report it to that upstream.

https://github.com/golang/go/issues/34040#issuecomment-527986411

I would ask that you please don't disable TLS 1.3 for all Go users just to accommodate one specific broken server. Rather, please disable it only in the client that needs to connect to that server, or even better fix the server. TLS 1.3 is a major security improvement, and having a distribution disable it by default will cause confusion for users, like any divergence from upstream.

Comment 40 Jakub Čajka 2019-09-04 19:54:16 UTC
I have opened tracking infra issue(when we get fix) https://pagure.io/fedora-infrastructure/issue/8173. 
As this seems to be affecting httpd and based on the comments in the upstream ticket https://github.com/golang/go/issues/34040 reassigning to openssl. I will work on standalone httpd based reproducer as my time allows later this week.

Comment 41 Jakub Čajka 2019-09-04 19:56:34 UTC
(In reply to Filippo Valsorda from comment #39)
> Thank you for reporting this upstream to us. I investigated and found the
> server to be violating the TLS 1.3 specification. Let me know if you need
> any more details to track this down server side and/or to report it to that
> upstream.
> 
> https://github.com/golang/go/issues/34040#issuecomment-527986411
> 
> I would ask that you please don't disable TLS 1.3 for all Go users just to
> accommodate one specific broken server. Rather, please disable it only in
> the client that needs to connect to that server, or even better fix the
> server. TLS 1.3 is a major security improvement, and having a distribution
> disable it by default will cause confusion for users, like any divergence
> from upstream.

I agree on this with you, as this is not a stdlib issue I will revert the patches. FYI the patched build haven't been yet distributed(as we are in the release freeze period). Thanks for all help.

Comment 42 Adam Williamson 2019-09-04 20:49:58 UTC
I see 1.13-1.fc31 and 1.13-1.fc32 were built today, but the TLS 1.3 disabling patch was not removed...

Comment 43 Jakub Čajka 2019-09-05 06:43:53 UTC
(In reply to Adam Williamson from comment #42)
> I see 1.13-1.fc31 and 1.13-1.fc32 were built today, but the TLS 1.3
> disabling patch was not removed...

My "today" time prior all the development of US pacific TZ "today"... Will fix it my "today".

Comment 44 Tomas Mraz 2019-09-05 09:02:06 UTC
Please note that fixing the openssl issue will not fix the problem with podman. Only when the fixed openssl is deployed on registry.fedoraproject.org, it will fix the podman problem.

There is no upstream fix yet, so perhaps there should be some workaround done either in podman or in registry.fedoraproject.org.

Comment 45 Jakub Čajka 2019-09-05 09:15:10 UTC
(In reply to Tomas Mraz from comment #44)
> Please note that fixing the openssl issue will not fix the problem with
> podman. Only when the fixed openssl is deployed on
> registry.fedoraproject.org, it will fix the podman problem.
> 
> There is no upstream fix yet, so perhaps there should be some workaround
> done either in podman or in registry.fedoraproject.org.

Yes that is the reason why I have opened the infra tracking issue #8173.

FYI upstream openssl issue https://github.com/openssl/openssl/issues/9767

Comment 46 Fedora Update System 2019-09-06 16:35:53 UTC
FEDORA-2019-b364562f30 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-b364562f30

Comment 47 Fedora Update System 2019-09-06 16:35:57 UTC
FEDORA-2019-f4a2ced40c has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2019-f4a2ced40c

Comment 48 Fedora Update System 2019-09-07 16:57:32 UTC
openssl-1.1.1c-6.fc31 has been pushed to the Fedora 31 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-f4a2ced40c

Comment 49 Fedora Update System 2019-09-08 03:49:35 UTC
openssl-1.1.1c-6.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-b364562f30

Comment 50 Fedora Update System 2019-09-08 05:46:59 UTC
openssl-1.1.1c-6.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-f5810ab475

Comment 51 Fedora Update System 2019-09-09 02:22:01 UTC
openssl-1.1.1c-6.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.

Comment 52 Kevin Fenzi 2019-09-14 22:24:49 UTC
Hum. So, I checked this issue against our staging server(s):

podman pull registry.stg.fedoraproject.org/fedora:30       
Trying to pull registry.stg.fedoraproject.org/fedora:30...
  Get https://registry.stg.fedoraproject.org/v2/: local error: tls: unexpected message
Error: error pulling image "registry.stg.fedoraproject.org/fedora:30": unable to pull registry.stg.fedoraproject.org/fedora:30: unable to pull image: Error initializing source docker://registry.stg.fedoraproject.org/fedora:30: pinging docker registry returned: Get https://registry.stg.fedoraproject.org/v2/: local error: tls: unexpected message

then I upgraded openssl on the proxy server:
[root@proxy01 ~][STG]# rpm -q openssl
openssl-1.1.1c-6.fc29.x86_64

Rebooted the server. 

But the problem persists:
% podman pull registry.stg.fedoraproject.org/fedora:30       
Trying to pull registry.stg.fedoraproject.org/fedora:30...
  Get https://registry.stg.fedoraproject.org/v2/: local error: tls: unexpected message
Error: error pulling image "registry.stg.fedoraproject.org/fedora:30": unable to pull registry.stg.fedoraproject.org/fedora:30: unable to pull image: Error initializing source docker://registry.stg.fedoraproject.org/fedora:30: pinging docker registry returned: Get https://registry.stg.fedoraproject.org/v2/: local error: tls: unexpected message

What am I missing here? httpd doesn't need to be rebuilt does it?

Comment 53 Fedora Update System 2019-09-15 00:26:05 UTC
openssl-1.1.1c-6.fc31 has been pushed to the Fedora 31 stable repository. If problems still persist, please make note of it in this bug report.

Comment 54 Fedora Update System 2019-09-16 16:15:00 UTC
FEDORA-2019-d51641f152 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-d51641f152

Comment 55 Fedora Update System 2019-09-17 01:23:31 UTC
openssl-1.1.1d-1.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-d51641f152

Comment 56 Adam Williamson 2019-09-17 01:53:59 UTC
Re-opening for now, given Kevin's question in #c52.

Comment 57 Jens Petersen 2019-09-17 03:29:43 UTC
https://pagure.io/fedora-infrastructure/issue/8173

Comment 58 Tomas Mraz 2019-09-17 09:25:01 UTC
I am sorry, the patch was not properly applied in the openssl-1.1.1c-6 build. Please try again with the openssl-1.1.1d-1.fc29.

Comment 59 Kevin Fenzi 2019-09-17 21:33:35 UTC
I can confirm that version works. Thanks!

Comment 60 Fedora Update System 2019-10-02 01:40:42 UTC
openssl-1.1.1d-1.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.