Bug 2170878

Created attachment 1944772 [details]
trying to update system in GNOME Software

The whole update fails because of Chrome. See pkmon output in the back.

Created attachment 1944773 [details]
trying to remove Chrome in GNOME Software

Chrome can't be removed. See pkmon output in the back.

Proposing for a blocker discussion. This problem as it currently stands could be quite a disaster for desktop Fedora.

*** Bug 2170024 has been marked as a duplicate of this bug. ***

Yeah there are multiple duplicates of this, bug 2149762 has been the "master" but we can also treat this one as such, to clear the history a bit.

The short summary is: update-crypto-policies for a longer-term workaround, but to just clean up whatever is failing:

> error: rpmdbNextIterator: skipping h#    2525 
                                           ^^^^
# rpm -q --nosignature --querybynumber 2525
google-chrome-stable-109.0.5414.119-1.x86_64
# rpm -e --nosignature google-chrome-stable-109.0.5414.119-1.x86_64

Or in one line, replacing N with the skipped header number from the message:

# rpm -e --nosignature $(rpm -q --nosignature --querybynumber N)

Thanks, Panu, that helps to simplify my workaround instructions at ask.fp.o.

What can Fedora as a distribution do to make this change more digestible for general users, who are unlikely to find the Common Issue description and not proficient with a command line? The major issue is that this prevents all system updates, and given the current breadth of affected third-party apps, it's likely to affect lots of users.

This seems to be a plain logic bug in rpm. It doesn't make much sense to reject
the signature on an rpm that has already been installed. For query or removal operations
this just isn't useful in any way.

And this would be a reoccurring issue: every time we tighten the security requirements,
or some certificate is revoked or expires, we'd have the same result. This situation
needs to be handled gracefully.

The original idea behind the rpm behavior wrt signature checks on installed packages is, AIUI, protection against tampering. In which case it indeed makes sense to avoid loading a header that doesn't pass signature checking.

The problem is rpm can't tell tampering from denied-by-policy at the moment - the knowledge may be down there in the crypto layer but there's no way to pass it up in the API. So there's no easy fix.

But that doesn't make any sense: the code you are running is from the rpm, and
if the rpm was tempered-with, it's already too late. If somebody can modify files
on disk, they really don't need to "tamper" with the rpm headers if they can change
any other file, incl. /bin/rpm itself. Just rip all the code that tries to verify
installed packages...

That assumes everything is watertight on the fs. All it takes is (accidentally or through some other vulnerability etc) relaxed rpmdb directory ownership/permissions and the story is quite different - rpmdb which may be on a different fs or different host even etc.

There is a case for verifying installed package signatures, but I agree the existing implementation that is faulty.

Eh, the above should've been "I agree the existing implementation doesn't make much sense."
Fixing the whole damn thing isn't something we can do in the scope of this bug though, so this isn't really the place to discuss it all.

There's a QA blocker discussion here:
https://pagure.io/fedora-qa/blocker-review/issue/1039

and a FESCo blocker ticket here:
https://pagure.io/fesco/issue/2960

I was testing this today, with
  rpm-4.18.0-10.fc38.x86_64
  dnf5-5.0.6-2.fc38.x86_64
  google-chrome-stable-109.0.5414.119-1.x86_64
and I get slightly different results:
- with policy DEFAULT:SHA1 I can remove the package using 'dnf remove google-chrome-stable'
- upgrade fails with
  Failed to import public key "https://dl.google.com/linux/linux_signing_key.pub" to rpmdb.
There are two keys in that file and it seems that one is acceptable but the other not:
  0x7FAC5991 is "Google, Inc. Linux Package Signing Key" is rejected.
  0xD38B4796 is "Google Inc. (Linux Packages Signing Authority)" is imported successfully.

Does anyone know what is the difference between the two keys?

There's a big difference between requiring DEFAULT:SHA1 and LEGACY…

_pgpPubKeyLint: -> error: Failure: Certificate A040830F7FAC5991 is unusable
What does this mean?

Setting as a blocker per FESCO: https://meetbot.fedoraproject.org/fedora-meeting/2023-02-28/fesco.2023-02-28-17.00.log.html#l-184

FESCo agrees to block Beta for this issue. In order to unblock, RPM must accept SHA-1 hashes and DSA keys for Fedora 38, ideally with a deprecation warning that it will be disabled in F39. FESCo strongly advises against allowing these algorithms elsewhere, but will accept that for F38 if it's the only achievable, timely solution. (+8, 0, -0)

The decision what algorithms to accept or not is NOT made by rpm, it has no say in this matter.

Reassigning to crypto-policies where the distro defaults are set.

I opened an issue in the fedora-crypto-policies project: https://gitlab.com/redhat-crypto/fedora-crypto-policies/-/issues/45

(In reply to Zbigniew Jędrzejewski-Szmek from comment #13)
> I was testing this today, with
>   rpm-4.18.0-10.fc38.x86_64
>   dnf5-5.0.6-2.fc38.x86_64
>   google-chrome-stable-109.0.5414.119-1.x86_64
> and I get slightly different results:
> - with policy DEFAULT:SHA1 I can remove the package using 'dnf remove
> google-chrome-stable'

That's probably because either the package signature uses SHA-1, or the key used to sign the package uses SHA-1.

> - upgrade fails with
>   Failed to import public key
> "https://dl.google.com/linux/linux_signing_key.pub" to rpmdb.
> There are two keys in that file and it seems that one is acceptable but the
> other not:
>   0x7FAC5991 is "Google, Inc. Linux Package Signing Key" is rejected.
>   0xD38B4796 is "Google Inc. (Linux Packages Signing Authority)" is imported
> successfully.
> 
> Does anyone know what is the difference between the two keys?

One of the keys is a 1024-bit DSA key, the other is a

- 4096-bit RSA main key (key ID 0x7721F63BD38B4796) with
- a SHA-1 Positive certification of a User ID and Public Key packet (this is not an issue, this signature is never verified on import)
- two SHA-1 Generic certification of a User ID and Public Key packets made with DSA keys (probably signed by the old DSA key; I haven't tested this, but I don't think those will be verified on import either)
- two 4096-bit RSA subkeys (key IDs 0x1397BC53640DB551 and 0x6494C6D6997C215E) signed with the main key using SHA-1, both with flags indicating that they can be used to sign data
- three 4096-bit RSA subkeys (key IDs 0x78BD65473CB3BD13, 0x4EB27DB2A3B88B8B, and 0xE88979FB9B30ACF2) signed with the main key using SHA-256, both with flags indicating that they can be used to sign data

0x1397BC53640DB551 expired in 2019, 0x6494C6D6997C215E in 2020
0x78BD65473CB3BD13 expired in 2022, 0x4EB27DB2A3B88B8B is valid until 2024 and 0xE88979FB9B30ACF2 until 2026.

If Google stopped shipping their old DSA and expired keys in this file, the key would likely import just fine.
We could provide a version of the key that has the DSA key, the expired and SHA-1-signed subkeys stripped, but we don't have a way to publish that at https://dl.google.com/linux/linux_signing_key.pub.

It's not just the DSA key.  rpm-sequoia doesn't consider 0x7721F63BD38B4796 to be valid because the self signature (binding signature) uses SHA-1:

```
$ sq inspect 7721F63BD38B4796.pgp
/tmp/linux_signing_key.pub: OpenPGP Certificate.

    Fingerprint: EB4C1BFD4F042F6DDDCCEC917721F63BD38B4796
                 Invalid: No binding signature at time 2023-03-01T09:12:13Z
Public-key algo: RSA
Public-key size: 4096 bits
  Creation time: 2016-04-12 05:51:15 UTC

         UserID: Google Inc. (Linux Packages Signing Authority) <linux-packages-keymaster>
                 Invalid: Policy rejected non-revocation signature (PositiveCertification) requiring second pre-image resistance
                 because: SHA1 is not considered secure since 2023-02-01T00:00:00Z
 Certifications: 2, use --certifications to list

$ sq-keyring-linter 7721F63BD38B4796.pgp
Certificate 7721F63BD38B4796 is not valid under the standard policy: No binding signature at time 2023-03-01T09:53:11Z
Certificate 7721F63BD38B4796 contains a User ID ("Google Inc. (Linux Packages Signing Authority) <linux-packages-keymaster>") protected by SHA-1
Examined 1 certificate.
  0 certificates are invalid and were not linted. (GOOD)
  1 certificate was linted.
  1 of the 1 certificates (100%) has at least one issue. (BAD)
0 of the linted certificates were revoked.
  0 of the 0 certificates has revocation certificates that are weaker than the certificate and should be recreated. (GOOD)
0 of the linted certificates were expired.
1 of the non-revoked linted certificate has at least one non-revoked User ID:
  1 has at least one User ID protected by SHA-1. (BAD)
  1 has all User IDs protected by SHA-1. (BAD)
1 of the non-revoked linted certificates has at least one non-revoked, live subkey:
  0 have at least one non-revoked, live subkey with a binding signature that uses SHA-1. (GOOD)
1 of the non-revoked linted certificates has at least one non-revoked, live, signing-capable subkey:
  0 certificates have at least one non-revoked, live, signing-capable subkey with a strong binding signature, but a backsig that uses SHA-1. (GOOD)

```

(In reply to Panu Matilainen from comment #16)
> The decision what algorithms to accept or not is NOT made by rpm, it has no
> say in this matter.
> 
> Reassigning to crypto-policies where the distro defaults are set.

Panu, the crypto policy changed in Fedora 33 [1], and rpm started to honor it in Fedora 38. The FESCo resolution strongly prefers reverting just the recent change in rpm, instead of reverting the whole policy back to Fedora 32 version for all applications, if I read it correctly (there are also meeting logs [2]). Is it possible to revert just the rpm change, or make Fedora-specific adjustments to let SHA-1 and DSA be accepted on top of the current defaults?

[1] https://fedoraproject.org/wiki/Changes/StrongCryptoSettings2
[2] https://meetbot.fedoraproject.org/fedora-meeting/2023-02-28/fesco.2023-02-28-17.00.log.html

Kamil, the change that brought this issue to the forefront is not a small one.  rpm 4.18 completely replaced the OpenPGP backend.  The new backend respects the system's crypto configuration in `/etc/crypto-policies/back-ends/`; rpm has no say over that.  What is accepted and what is rejected was discussed in this thread: https://gitlab.com/redhat-crypto/fedora-crypto-policies/-/merge_requests/122 .

(In reply to neal from comment #19)
> It's not just the DSA key.  rpm-sequoia doesn't consider 0x7721F63BD38B4796
> to be valid because the self signature (binding signature) uses SHA-1:

Thanks for checking, that's different from the previous backend, which did not verify the self signature.

Clemens and Neal, thank you both. I wasn't aware of 'sq inspect' and 'sq-keyring-linter'.

> If Google stopped shipping their old DSA and expired keys in this file, the key would likely import just fine.
> We could provide a version of the key that has the DSA key, the expired and SHA-1-signed subkeys stripped, but we don't > have a way to publish that at https://dl.google.com/linux/linux_signing_key.pub.

Hmm, I would think that shipping additional keys that are expired or signed
with a weak algorithm should not be a problem.
In fact, shipping expired keys is pretty hard to avoid: one would generally ship
the key before it is expired and it is not possible to retroactively "unpublish"
they expired key afterwards from all places. Those subkeys should not be used
for verification of things (based on the policy), but their presence should not
be reported as an error.

> - 4096-bit RSA main key (key ID 0x7721F63BD38B4796) with
> - two SHA-1 Generic certification of a User ID and Public Key packets made with DSA keys (probably signed by the old DSA key; I haven't tested this, but I don't think those will be verified on import either)

Yes, it's signed by 0xA040830F7FAC5991 and also 0xBC47E92A20D3D1761F389A5231472C3B31B7E803
(whatever that is).

Oh, oh, I think the key *is* imported successfully:
Importing PGP key 0x7FAC5991
Failed to import public key "https://dl.google.com/linux/linux_signing_key.pub" to rpmdb.
Importing PGP key 0xD38B4796
Key imported successfully.

--

The rpm itself is signed by 0xa040830f7fac5991, i.e. the first key in the file, SHA1/DSA.

So even if the second key was imported, this doesn't matter. So instead of changes
to the keys, maybe google needs to just start signing with the newer 0x7721F63BD38B4796?
IIUC, the newer key is essentially unused right now.

(In reply to Kamil Páral from comment #20)
> Panu, the crypto policy changed in Fedora 33 [1], and rpm started to honor
> it in Fedora 38. The FESCo resolution strongly prefers reverting just the
> recent change in rpm, instead of reverting the whole policy back to Fedora
> 32 version for all applications, if I read it correctly (there are also
> meeting logs [2]). Is it possible to revert just the rpm change, or make
> Fedora-specific adjustments to let SHA-1 and DSA be accepted on top of the
> current defaults?

Rpm has been under the crypto policy enforced on openssl all along, it's not like it had any choice on that matter either. What changed here is rpm's OpenPGP and overall crypto backend (see https://fedoraproject.org/wiki/Changes/RpmSequoia), and I guess the policy for that backend is stricter than that of openssl. Somewhere between RHEL-9 specific changes, planned but canceled Fedora changes and this, I lost track.

(In reply to Zbigniew Jędrzejewski-Szmek from comment #23)
> Hmm, I would think that shipping additional keys that are expired or signed
> with a weak algorithm should not be a problem.
> In fact, shipping expired keys is pretty hard to avoid: one would generally
> ship the key before it is expired and it is not possible to retroactively
> "unpublish" they expired key afterwards from all places. Those subkeys should
> not be used for verification of things (based on the policy), but their presence
> should not be reported as an error.

I'm not saying that google should not be shipping expired keys in general, just saying that they should not ship *these* expired keys, since they're signed with SHA-1.
That being said, comment 19 also explains that they would have to re-create their binding signature, or create a completely new key.


> The rpm itself is signed by 0xa040830f7fac5991, i.e. the first key in the
> file, SHA1/DSA.
> 
> So even if the second key was imported, this doesn't matter. So instead of
> changes to the keys, maybe google needs to just start signing with the newer
> 0x7721F63BD38B4796?

I was surprised by this so I double-checked it:

$ rpm --qf "%{NAME}-%{VERSION}-%{RELEASE} %{SIGGPG:pgpsig}\n" -q google-chrome-stable-110.0.5481.177-1.x86_64.rpm
warning: google-chrome-stable-110.0.5481.177-1.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 7fac5991: NOKEY
google-chrome-stable-110.0.5481.177-1 DSA/SHA1, Tue 21 Feb 2023 10:43:00 PM CET, Key ID a040830f7fac5991

Even if it's Google Chrome, do we *really* want to allow installation of software signed with SHA-1 with a 1024-bit DSA key in 2023? Surely that's an oversight by the Chrome team?

> I wasn't aware of 'sq inspect' and 'sq-keyring-linter'.

By the way: sq-keyring-linter can also fixup a certificate by creating new binding signatures:

  $ gpg --export-secret-keys FPR | sq-keyring-linter --fix -p passw0rd -p password123 | gpg --import

To continue on comment #24 a bit:
> Rpm has been under the crypto policy enforced on openssl all along, it's not like it had any choice on that 
> matter either. What changed here is rpm's OpenPGP and overall crypto backend (see 
> https://fedoraproject.org/wiki/Changes/RpmSequoia), and I guess the policy for that backend is stricter 
> than that of openssl.

...and as others have pointed out, some weaknesses have fallen through the cracks and negletting system policy up to now is really just rpm's internal OpenPGP parser not doing it's job very well. Such as failing to check for those binding self-signatures, which is another potential place for weak crypto.

I'm sorry, but reenabling 1024 bit keys, DSA at that, to allow SHA-1 signatures is irresponsible. SHA-1 is completely broken for over 5 years at this point!

What was done to tell Google that they're using cryptography that was last appropriate in the previous millennium?

(In reply to Hubert Kario from comment #28)
> I'm sorry, but reenabling 1024 bit keys, DSA at that, to allow SHA-1
> signatures is irresponsible. SHA-1 is completely broken for over 5 years at
> this point!
> 
> What was done to tell Google that they're using cryptography that was last
> appropriate in the previous millennium?

So far, https://bugs.chromium.org/p/chromium/issues/detail?id=1383526 and https://bugs.chromium.org/p/chromium/issues/detail?id=1398429, but there doesn't seem to be a lot of activity. I also just sent a mail to linux-packages-keymaster, which is the address listed on the keys, which hasn't bounced (yet).

IMHO, *dropping* support for SHA-1 signatures is what is "irresponsible". There are thousands of RPMs out there using them, and backwards compatibility has always been an important selling point of RPMs. Assuming that your software does not depend on outdated library sonames, building it on the oldest possible distribution has always been the way to get it to run everywhere. But that implies that the RPM is going to use old signature technologies. Are SHA-1 signatures not also a part of the LSB spec? As I understand it, RPM is the preferred distribution mechanism in the LSB.

You can always install packages without verifying signatures, dropping SHA-1 does not prevent you from installing software.

It does not prevent *me* from installing software, but it prevents regular users who use GUIs, not CLIs.

So, on the procedural side here...is this really about the new backend in RPM? Or is it about:

https://fedoraproject.org/wiki/Changes/StrongCryptoSettings3

"Cryptographic policies will be tightened in Fedora 38-39, SHA-1 signatures will no longer be trusted by default."

Note that also says:

"The change is an opt-in in Fedora 36-37, a "jump scare" in Fedora 38 where it's reverted before the release, and now, in Fedora 39 it'll finally reach the users."

So does this "jump scare" approach mean this will *already* go away before release? Maybe Alexander can clarify?

* It's not about https://fedoraproject.org/wiki/Changes/StrongCryptoSettings3, because it got cancelled by FeSCo
* It's not really about switching away from the new RPM PGP backend (Sequoia), because there's no point in doing that when we can just tweak defaults instead
* It's not about the introduction of crypto-policies in Fedora 32 either =)

It's about either:
* Chrome maintainers providing a repository protected by 21st century crypto
* Fedora doing a horrible thing of making RPM accept weak cryptography by default

Since the FeSCo decision has been made, I'm working on implementing the 2nd
(see https://gitlab.com/redhat-crypto/fedora-crypto-policies/-/merge_requests/129)
while praying for the 1st to happen instead.
I would've started earlier if somebody warned me earlier than today morning.

(In reply to Kevin Kofler from comment #30)
> Are SHA-1 signatures not also a part of the LSB spec? As I
> understand it, RPM is the preferred distribution mechanism in the LSB.

LSB 3.0.0 Core generic (released July 2005), section 22.2.3 "Signature Section", Table 22-7 "Signature Signing Tag Values" specifies RPMSIGTAG_PGP as "the RSA signature of the combined Header and Payload sections. The data is formatted as a Version 3 Signature Packet as specified in RFC 2440: OpenPGP Message Format". RFC 2440 has been obsoleted by RFC 4880 (which supports SHA-2) since November 2007.

Are you claiming that it is irresponsible to no longer support installing RPMs created before 2008?

As long as no new LSB spec is released:
* explicitly replacing RFC 2440 with RFC 4880 (or a newer RFC), and
* explicitly banning the use of SHA-1 (unless it is already banned by the RFC),

removing SHA-1 support is incompatible with LSB RPMs.

And what really matters is not when exactly RFC 4880 was released, but when RPM started defaulting to something newer than SHA-1, and when this actually reached the oldest supported RHEL branch (probably by means of the older RHEL branches reaching EOL, since I do not think this change was backported), because cross-distribution packages are normally built on the oldest available RHEL/CentOS so that they work on the maximum number of distributions.

> * It's not really about switching away from the new RPM PGP backend (Sequoia), because there's no point in doing that when we can just tweak defaults instead

Does Sequoia support SHA-1 at all? If not, reverting the switch to Sequoia is the only option to revert this change, because the only other option would be to completely disable signature verification, which is not going to happen.

According to:
https://www.redhat.com/en/blog/rhel-security-sha-1-package-signatures-distrusted-rhel-9
RHEL 7 still defaults to SHA-1 signatures, so most third-party packages will be built with SHA-1 signatures until at least 2024-06-30 (standard EOL), possibly 2026-06-30 (ELS EOL) or beyond. So we are not talking about "installing RPMs created before 2008", but about installing RPMs created before 2026!

>> * It's not really about switching away from the new RPM PGP backend (Sequoia), because there's no point in doing that when we can just tweak defaults instead
>
> Does Sequoia support SHA-1 at all? If not, reverting the switch to Sequoia is the only option to revert this change, because the only other option would be to completely disable signature verification, which is not going to happen.

Yes, Sequoia does support SHA-1. Please tone down your rhethorics,
there's no need for anything that drastic
when we have a perfectly working configuration switch to reenable SHA-1.
Note that even the #c0 specifies LEGACY as an already functioning workaround,
albeit one we don't want to be recommending left and right,
but totally proving that SHA-1 can be reenabled with configuration changes alone,
with the packages we already have in Fedora 38 today.

> removing SHA-1 support

We're not removing SHA-1 support from RPM, we are (were?) disabling it by default.

> RHEL 7 still defaults to SHA-1 signatures, so most third-party packages will be built with SHA-1 signatures until at least 2024-06-30

Could you please either bridge the logical gap between "[RHEL 7 is still alive as of today]" and "most third-party packages will be built [on RHEL-7]" or back that with some statistics?

> removing SHA-1 support is incompatible with LSB RPMs.

Even if it is, asking user to opt-in into insecure practices is always better than being insecure by default. Especially since we're talking Fedora, the distro that "always aims to provide the future, first".

> Yes, Sequoia does support SHA-1. Please tone down your rhethorics,
> there's no need for anything that drastic
> when we have a perfectly working configuration switch to reenable SHA-1.

So if it is as easy as that, can we please flip that switch?

> Could you please either bridge the logical gap between "[RHEL 7 is still alive as of today]" and "most third-party packages will be built [on RHEL-7]" or back that with some statistics?

The logic is that, since glibc and other libraries are only backwards, but not forwards, compatible, your best bet to obtain portable binaries is to build on the oldest distribution you can get ahold of.

> So if it is as easy as that, can we please flip that switch?

We can, we're preparing to. That doesn't make this idea any better.
If you check out the attached MR, you'll see how we're trying to limit the damage.

The only silver lining is that FeSCo's decision reads as a very temporary measure to me:
> There is sufficient time between now and Fedora 39 for third-party repos to adapt, and no need to block the change indefinitely on third-party repos that we do not control. (In particular, we expect Google Chrome repo should be updated well before Fedora 39 release.)
> https://pagure.io/fesco/issue/2960#comment-843816

So, if you don't want the default flipped back
as soon as Chrome packagers come to their collective senses and adjust,
I'd suggest you to start collecting arguments against that.

> The logic is that, since glibc and other libraries are only backwards, but not forwards, compatible, your best bet to obtain portable binaries is to build on the oldest distribution you can get ahold of.

BTW, if tomorrow somebody starts a company offering paid support for CentOS 5,
will Fedora magically (re)gain an obligation to be interoperable with RHEL-5?

We'd better have some distribution-wide policy on that,
ideally one that cares about security first.

IMHO, it should already be the case that RPMs built on RHEL 5 are still installable. RHEL 5 ELS ended less than 3 years ago. IMHO, RPMs should remain installable for at least 20 years after they were built, so rejecting SHA-1 signatures is something that can be rediscussed in 2046.

Kevin,
stop bringing up ridiculous arguments.

Most RHEL-5 packages will not work anyway due to missing dependencies and simply won't work in Fedora, and the same is true for any non-trivial or fully statically built package, probably including many RHEL-7 packages given the amount of soname changes there have been in Fedora since then, including most crypto libraries.

Fedora never promised and never will promise backwards compatibility of this kind.

And it was already pointed out that for those rare case someone really wants to install packages that aold, they can do so manually by telling rpm to not check signatures.

SHA1 disabled is perfectly fine as a default, the only reason to defer temporarily is because of a very popular third party package and only because we assume that it will be fixed shortly.

Should Google decide not to fix Chrome's rpm signatures we will end up telling people they have to manually switch their Fedora systems to the LEGACY policy and eventually will block it in there as well.

SHA1 is not sufficiently secure for checking signatures, so we *will* block it going forward.

Allowing insecure signature defeats the whole purpose of checking signatures.
You can go an lobby FeSCO to drop any signature checking if you want.

I have (in my CalcForge RPM repository) packages built in 2009 against Fedora Core 6 that still work fine on current Fedora. There is no need to rebuild packages if they still install fine. This Change breaks them.

Both Alexander and Simo have insinuated that this concession is primarily about the signing keys used by Chrome.  Perhaps an alternate approach to addressing this situation would be to good list these keys and leave the configuration as it was.  In particular, don't allow 1024-bit DSA keys by default.  Adding support for this to rpm-sequoia would not be difficult.  What do you think?

(In reply to neal from comment #46)
> Both Alexander and Simo have insinuated that this concession is primarily
> about the signing keys used by Chrome.  Perhaps an alternate approach to
> addressing this situation would be to good list these keys and leave the
> configuration as it was.  In particular, don't allow 1024-bit DSA keys by
> default.  Adding support for this to rpm-sequoia would not be difficult. 
> What do you think?

It's a comparatively complex and fragile solution for a problem that the Chrome team should be able to fix in an hour or two of work, tops.

It is not that easy if they are deliberately building on an old distribution due to the glibc (and other system library) compatibility concerns.

(and they probably also need to change their signing key, which is also not trivial)

> RPMs should remain installable for at least 20 years after they were built

Why not 50?

Here's a counterproposal that doesn't involve completely arbitrary timespans: RPMs should remain installable by default as long as it's secure. It no longer is.

> This Change breaks them.

Good. In the sense that now users who want to install them have to explicitly opt into no-longer-secure installation practices first. And good that you know it's coming, so you have time to re-sign them or something.

Kevin, please, just stop.

Chrome doesn't support anything older than 5 years:
https://support.google.com/chrome/a/answer/7100626?hl=en

That's RHEL-8 vintage, and definitely not RHEL-5.

The last RHEL that defaults to SHA-1 is RHEL 7, not 5. But if they build on RHEL 8, then they probably need to change their signing key.

Signatures never happen on the builders anyway, they are applied after build and can even be changed after build, there never is a reason to stick to old signatures for modern desktop packages.

I just thought I'd add that this issue doesn't affect just Chrome, it also affects Microsoft Edge (my default browser).  It does not, however, affect Brave, Vivaldi, and Opera (which are also Chromium-based).

$ sudo rpm --import https://repo.vivaldi.com/stable/linux_signing_key.pub
$ sudo rpm --import https://brave-browser-rpm-release.s3.brave.com/brave-core.asc
$ sudo rpm --import https://rpm.opera.com/rpmrepo.key

$ sudo rpm --import https://dl.google.com/linux/linux_signing_key.pub
error: Certificate A040830F7FAC5991:
  Policy rejects A040830F7FAC5991: No binding signature at time 2023-03-02T16:31:48Z
error: https://dl.google.com/linux/linux_signing_key.pub: key 1 import failed.
error: Certificate 7721F63BD38B4796:
  Policy rejects 7721F63BD38B4796: No binding signature at time 2023-03-02T16:31:48Z
error: https://dl.google.com/linux/linux_signing_key.pub: key 2 import failed.

$ sudo rpm --import https://packages.microsoft.com/keys/microsoft.asc
error: Certificate EB3E94ADBE1229CF:
  Policy rejects EB3E94ADBE1229CF: No binding signature at time 2023-03-02T16:32:19Z
error: https://packages.microsoft.com/keys/microsoft.asc: key 1 import failed.

Thanks for the input, Scott.

> $ sudo rpm --import https://packages.microsoft.com/keys/microsoft.asc
> error: Certificate EB3E94ADBE1229CF:
>   Policy rejects EB3E94ADBE1229CF: No binding signature at time 2023-03-02T16:32:19Z
> error: https://packages.microsoft.com/keys/microsoft.asc: key 1 import failed.

This one uses 2048 bit RSA (good) and SHA-1 (bad).

Scott,
you may want to let Microsft Edge people they are out of step with their own corporate messaging:
https://learn.microsoft.com/en-us/lifecycle/announcements/sha-1-signed-content-retired

Pretty sure if you open a bug they should be willing to fix it quickly.

(In reply to Simo Sorce from comment #56)
> Scott,
> you may want to let Microsft Edge people they are out of step with their own
> corporate messaging:
> https://learn.microsoft.com/en-us/lifecycle/announcements/sha-1-signed-
> content-retired
> 
> Pretty sure if you open a bug they should be willing to fix it quickly.


I'm not sure how to go about that. 

I'll try emailing gpgsecurity and see if that gets me anywhere, however I'm not sure what to tell them.  As Alexander pointed out above, their key uses 2048 bit RSA (good) and SHA-1 (bad).

How can it use two keys?

> How can it use two keys?

It doesn't. There are two keys in the file, but only the that matters is the one that is used to sign packages.

> I'll try emailing gpgsecurity and see if that gets me anywhere

Thanks for finding the address

> their key uses 2048 bit RSA (good) and SHA-1 (bad). How can it use two keys?

They're using RSA algorithm for signing the SHA-1 hash of the data.

I think you can just tell them that they're using SHA-1, link to both the key and the policy.

(In reply to Zbigniew Jędrzejewski-Szmek from comment #58)
> > How can it use two keys?
> 
> It doesn't. There are two keys in the file, but only the that matters is the
> one that is used to sign packages.

OK, thanks.

(In reply to Alexander Sosedkin from comment #59)
> > I'll try emailing gpgsecurity and see if that gets me anywhere
> 
> Thanks for finding the address

I found it without looking for it yet.

++++++++++++++++++

[...]

Downloading Packages:
microsoft-edge-beta-111.0.1661.24-1.x86_64.rpm                      29 MB/s | 135 MB     00:04    
---------------------------------------------------------------------------------------------------
Total                                                               29 MB/s | 135 MB     00:04     
microsoft-edge-beta                                                2.8 kB/s | 983  B     00:00    
Importing GPG key 0xBE1229CF:
 Userid     : "Microsoft (Release signing) <gpgsecurity>"
 Fingerprint: BC52 8686 B50D 79E3 39D3 721C EB3E 94AD BE12 29CF
 From       : https://packages.microsoft.com/keys/microsoft.asc
error: Certificate EB3E94ADBE1229CF:
  Policy rejects EB3E94ADBE1229CF: No binding signature at time 2023-03-02T20:58:14Z
Key import failed (code 2). Failing package is: microsoft-edge-beta-111.0.1661.24-1.x86_64
 GPG Keys are configured as: https://packages.microsoft.com/keys/microsoft.asc
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: GPG check FAILED
+++++++++++++++++++++++


> They're using RSA algorithm for signing the SHA-1 hash of the data.
> 
> I think you can just tell them that they're using SHA-1, link to both the
> key and the policy.

OK, thanks.

Email bounced.

(In reply to Scott Beamer from comment #54)
> I just thought I'd add that this issue doesn't affect just Chrome

There's a Fedora Ask link in comment 0, which lists all affected popular applications that I found. If somebody wants to go and try notify all their vendors, there's the list. Thanks.

I filed https://github.com/microsoft/linux-package-repositories/issues/47 for the Microsoft repos.

(In reply to Clemens Lang from comment #64)
> I filed https://github.com/microsoft/linux-package-repositories/issues/47
> for the Microsoft repos.

Thank you.  I had no idea where to go with regard to contacting Microsoft.

I got a bit of free time on my hands, so
I've created a scratch build to aid testing:

1. crypto-policies that
    * provide a separate rpm-sequoia policy and
    * allow SHA-1 and 1024 bit DSA there
   (https://koji.fedoraproject.org/koji/taskinfo?taskID=98247744,
    https://koji.fedoraproject.org/koji/taskinfo?taskID=98248533)

If one then does `update-crypto-policies --set DEFAULT`
(note to self: make sure update proceeds fine w/o crypto-policies-scripts
 and w/o issuing this command)
and

2. rebuilds sequoia-policy-config, so that it
    * consumes the policy from `/etc/crypto-policies/back-ends/rpm-sequoia.config`
      (I've simply replaced the constant in the code)
    * conflicts with crypto-policies < 20230301-1 that doesn't provide one yet
   (https://koji.fedoraproject.org/koji/taskinfo?taskID=98248535)

3. rebuilds rpm-sequoia against new sequoia-policy-config

it seems to work "fine" and
RPM starts trusting the insecure Google&Microsoft keys mentioned above.

So just so everyone is aware: this is an accepted Beta blocker (per FESCo) and the first go/no-go for Beta is Thursday. So we really need whatever is going to get done about this to be done soon, or else we'll be slipping Beta.

Just a note. I noticed that the RpmSequoia F38 Change proposal [1] clearly includes "Contingency mechanism: Revert back to the internal PGP parser" in case of problems. So RPM should be prepared to do that revert if needed. I'm making this bug block that Change tracker. (Of course the solution in comment 66 might be actually a better way to deal with it, I'm not arguing against that, the devs will know better than I do).

[1] https://fedoraproject.org/wiki/Changes/RpmSequoia

Yes reverting the change is listed as a "if nothing else works" option of course, but reverting would be an absurdly bad decision for this case. This is only a matter of (default) system configuration controlled by another component, not a flaw in the Sequoia backend, aka The Change.

(In reply to Alexander Sosedkin from comment #66)
> I got a bit of free time on my hands, so
> I've created a scratch build to aid testing:
> 
> 1. crypto-policies that
>     * provide a separate rpm-sequoia policy and
>     * allow SHA-1 and 1024 bit DSA there
>    (https://koji.fedoraproject.org/koji/taskinfo?taskID=98247744,
>     https://koji.fedoraproject.org/koji/taskinfo?taskID=98248533)

I guess I'll need the rebuilt rpm-sequoia? Installing just the updated crypto-policies-scripts and crypto-policies doesn't solve the issue. Did you scratch-built that too?

I released [0.6.0 of sequoia-policy-config](https://lists.sequoia-pgp.org/hyperkitty/list/announce@lists.sequoia-pgp.org/message/G5MHB7XOQBANTHF5VMBRTGZUVYDNCKSO/), and [1.3.0 of rpm-sequoia](https://lists.sequoia-pgp.org/hyperkitty/list/announce@lists.sequoia-pgp.org/message/QMORPITVRQWR3NNBJ4VTUZ44PMRK2NOA/).

There are two notable changes.

First, when `pgpVerifySignature` verifies a signature, it now
distinguishes between an invalid signature, and one that uses weak
cryptography, or is from a certificate that is expired or has been
revoked.  Specifically, in the case that the signature is okay, but
the cryptography is weak or the certificate is invalid,
`pgpVerifySignature` now returns `RPMRC_NOTTRUSTED` instead of
`RPMRC_FAIL`.

If I understood Panu correctly, RPM does not need to be updated; the change to have rpm-sequoia return `RPMRC_NOTTRUSTED` when using legacy cryptography or out-dated certificates is sufficient to the corresponding packages to be updated or removed, but not installed.

Second, rpm-sequoia now looks for its configuration file by first
checking the environment variable `RPM_SEQUOIA_CRYPTO_POLICY` and the
file `/etc/crypto-policies/back-ends/rpm-sequoia.config`.  Only if
both of those are not set does it fallback to the more generic
`SEQUOIA_CRYPTO_POLICY` environment variable and the file
`/etc/crypto-policies/back-ends/sequoia.config`.
This change means that other Sequoia-based applications won't
automatically accept SHA-1 or 1024-bit DSA keys.

That sounds great! Thanks a lot, neal.

> fallback to ... `/etc/crypto-policies/back-ends/sequoia.config`

Neal, do I understand correctly that I can go ahead and build
https://gitlab.com/redhat-crypto/fedora-crypto-policies/-/merge_requests/129
in rawhide and f38 without introducing any RPM-dependency-level synchronization
between crypto-policies and rpm-sequoia versions?
As in, upgrading either one first is fine,
and upgrading both satisfies FeSCo's request?

Hi Alexander, I hadn't thought about that.  I think there are three relevant cases to consider:

  - Alice has the old rpm-sequoia (1.2) and the old crypto policy packages installed and rpm-sequoia is updated to 1.3.  In that case, the new rpm-sequoia will continue to use the sequoia.policy file and it will be as if nothing had changed.

  - Alice has the old rpm-sequoia (1.2) and the old crypto policy packages installed and the crypto policy package is updated.  In that case, rpm-sequoia will continue to use the sequoia.policy file and it will be as if nothing had change.

  - Alice has rpm 4.17 installed and a very old crypto policy package without a sequoia.policy file, and rpm-sequoia is upgraded to 1.3.  Alice might have a problem, because rpm-sequoia will not see an rpm-sequoia.policy or sequoia.policy file and use its default configuration, which is much more strict.

> Hi Alexander, I hadn't thought about that.
> I think there are three relevant cases to consider:

>  - Alice has the old rpm-sequoia (1.2) and the old crypto policy packages installed
>    and rpm-sequoia is updated to 1.3.
>    In that case, the new rpm-sequoia will continue to use the sequoia.policy file
>    and it will be as if nothing had changed.

>  - Alice has the old rpm-sequoia (1.2) and the old crypto policy packages installed
>    and the crypto policy package is updated.
>    In that case, rpm-sequoia will continue to use the sequoia.policy file
>    and it will be as if nothing had change.

:thumbsup:, will proceed with updating then.

>  - Alice has rpm 4.17 installed and a very old crypto policy package
>    without a sequoia.policy file, and rpm-sequoia is upgraded to 1.3.
>    Alice might have a problem, because rpm-sequoia will not see an rpm-sequoia.policy
>    or sequoia.policy file and use its default configuration, which is much more strict.

Hm, interesting case, but not one to block !129 on right now.
Worst case we'll update crypto-policies in f37,
there's a requirement to be updated before upgrading to 38.

Note we explicitly support and block on upgrades from F36 to F38 as well; we need to ensure whatever is done here will work properly for that upgrade scenario too. So F36 as well as F37 needs to be updated to have whatever appropriate config files it should have (without making the policy there any stricter).

Hi, sorry, I couldn't make it to the Blocker Review meeting earlier today. I am currently preparing the rpm-sequoia update that should fix this issue.
My current plan to push the changes to rawhide first, and assuming that nothing blows up there, I will submit it as an update to Fedora 38 as well.

Uh, so...what's with the crypto-policies update from Alexander? Are you two working together? Do we need changes to both crypto-policies and rpm-sequoia?

As far as I know, both changes are required.
If I understand correctly: The crypto policy will include a separate policy for RPM, and rpm-sequoia needs an update to use that separate policy.

In that case, ideally the builds should be in the same update. Fabio, rather than creating a separate update, can you edit your build into Alexander's update? If you don't have the privileges to do so, let me know when your build is done and I can do it.

The crypto-policies update for f38 has only been built in koji, but not submitted to bodhi:
https://koji.fedoraproject.org/koji/buildinfo?buildID=2165850

If you want rpm-sequoia for f38 ASAP, I can create a side tag, tag the crypto-policies build into it, and then build rpm-sequoia there as well, so both can be submitted as a single bodhi update. (This is assuming that no bodhi update for the build linked above is created in the meantime. if that happens, I will need to use buildroot overrides instead of a side tag).

that would be excellent, yes. Thanks. Sorry, I thought the bug had been set to MODIFIED automatically by an update creation, didn't notice that Alexander set it manually.

Great, I'll keep you posted then.

By the way, the corresponding rawhide update is now stable:
https://bodhi.fedoraproject.org/updates/FEDORA-2023-d1e9212a2c

I'll try to install it in a rawhide VM to do some testing while the f38 builds are in progress. :)

FEDORA-2023-bd9a4614ad has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-bd9a4614ad

As far as I can tell, the builds I just submitted works as expected.
I tried installing google-chrome-stable from the official third-party repo, and it succeeded (on a rawhide VM).
The rpm-sequoia and crypto-policy updates have also already landed in rawhide a few hours ago, and I have not seen any signs of breakage in koschei.

FEDORA-2023-bd9a4614ad has been pushed to the Fedora 38 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-bd9a4614ad

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

(In reply to Fedora Update System from comment #84)
> FEDORA-2023-bd9a4614ad has been submitted as an update to Fedora 38.
> https://bodhi.fedoraproject.org/updates/FEDORA-2023-bd9a4614ad

I tested all third-party apps listed at https://discussion.fedoraproject.org/t/popular-third-party-rpms-fail-to-install-update-remove-due-to-security-policies-verification/70498 (except for MS Teams, their rpm repo is empty at the moment, likely a MS error). The can be installed just fine directly from rpms and also from their repositories. I tested both dnf and gnome-software. I also tested F37 with Chrome installed upgraded to F38, Chrome can be updated as expected. RPM no longer prints "error: rpmdbNextIterator: skipping" for affected rpms, those rpms can be listed and uninstalled as usual. Overall this seems to work as expected. The only thing I'm not able to verify is whether the new policy affects only rpm an no other tools.

I was also testing updates from a previously-updated F38 system:
$ sudo update-crypto-policies --set DEFAULT
$ sudo dnf5 upgrade
...
Importing PGP key 0x7FAC5991
 UserId     : "Google, Inc. Linux Package Signing Key <linux-packages-keymaster>"
 Fingerprint: 4CCA1EAF950CEE4AB83976DCA040830F7FAC5991
 From       : https://dl.google.com/linux/linux_signing_key.pub
Is this ok [y/N]: y
Failed to import public key "https://dl.google.com/linux/linux_signing_key.pub" to rpmdb.
Importing PGP key 0xD38B4796
 UserId     : "Google Inc. (Linux Packages Signing Authority) <linux-packages-keymaster>"
 Fingerprint: EB4C1BFD4F042F6DDDCCEC917721F63BD38B4796
 From       : https://dl.google.com/linux/linux_signing_key.pub
Is this ok [y/N]: y
Key imported successfully.
GPG check for package "google-chrome-stable-110.0.5481.177-1.x86_64" (/var/cache/libdnf5/google-chrome-6ed7e4f336f6863c/packages/google-chrome-stable-110.0.5481.177-1.x86_64.rpm) from repo "google-chrome" has failed: import of the key didn't help, wrong key?
Signature verification failed

OK, this is expected, because the already-installed rpm+crypto-policies does not allow SHA1.

Second attempt:
$ sudo dnf5 upgrade --advisory=FEDORA-2023-bd9a4614ad
$ sudo dnf5 upgrade
(works)

Maybe we should add the above two commands in common bugs advisory.

(In reply to neal from comment #74)
>   - Alice has rpm 4.17 installed and a very old crypto policy package
> without a sequoia.policy file, and rpm-sequoia is upgraded to 1.3.  Alice
> might have a problem, because rpm-sequoia will not see an rpm-sequoia.policy
> or sequoia.policy file and use its default configuration, which is much more
> strict.

That old rpm is not using Sequoia. The upgrade transaction from whatever old version to F38 will happen using the old rpm and policies relevant to it, it's only after the upgrade that Sequoia gets involved. I don't see an issue here.

Oh and BTW, thanks everybody for such a quick resolution on a short notice. I wish FESCo would've ruled on this a bit earlier to avoid an entirely unnecessary rush.

> Oh and BTW, thanks everybody for such a quick resolution on a short notice.

+1, I don't like what we did, but I appreciate all the cooperation.

> I wish FESCo would've ruled on this a bit earlier to avoid an entirely unnecessary rush.

Yeah, this and, maybe, identify and involve the maintainers earlier in the process.
If I knew it's a problem when it became a voting subject,
(I'd argue against it until my metaphorical voice got hoarse, likely, but)
any amount of prep time would ultimately let us resolve it quicker.

/me puts away rpm-sequoia package maintainer hat and puts on FESCo hat

I don't really understand how FESCo could have made a decision earlier?
The ticket with the request to make this issue a blocker bug for F38 Beta was filed on Feb 27, and it was discussed and voted on in the meeting *on the next day* (Feb 28).

Oh, apologies for poor wording and misunderstanding the chain of events. 

Basically, I just wish that the case would've been raised earlier if seen as a blocker. It's not like this change was introduced last week or so, the behavior sat in rawhide for a good while. But no matter, this seems to have come together quite nicely anyhow.

> I don't really understand how FESCo could have made a decision earlier?

I wasn't suggesting to move anything but my awareness of the issue in time.

> The ticket with the request to make this issue a blocker bug for F38 Beta was filed on Feb 27,
> and it was discussed and voted on in the meeting *on the next day* (Feb 28).

Oh, OK, then even if the requester has tagged us back then that'd be just a working day saved.
I have to agree, not much room for improvement here.

I guess one thing I'd note there is this: you were all clearly aware of the underlying issue months ago; it was first raised as a bug in, I think, November.

Speaking for myself, as soon as I saw this issue, it was blindingly obvious that we cannot ship even a Beta this way. Regardless whether we handled it via the release criteria or FESCo, it was just not going to fly. AFAICS, the reaction of every other person who looked at it from a QA or FESCo perspective was the same. There is absolutely no way we can ship a distro which does this to users, it's just not an option.

Apparently it didn't strike you that way? It seems like up until it was accepted as a blocker, you were expecting the resolution to be "we'll just document some cryptic console commands for people to use when their system looks like it's weirdly broken in a way we already knew was going to happen".

This feels weird to me. I'm not sure how we (QA/FESCo on one hand, RPM dev team on the other) can have such differing expectations as to what is an acceptable experience for Fedora users for a change like this. What do you think? Is it not communicated or known clearly enough/widely enough that we will not intentionally ship an OS that behaves like this?

Hi Adam, could you please point me to the issue that you are referring to.

Speaking as the developer of rpm-sequoia, I was not aware of this type of breakage.  I personally expected some packages to not install due to the use of cryptography that we were planning on rejecting.  It's completely normal that these type of infrastructure changes are only addressed when they actually break; warnings are not enough, even for security-minded people.  I expected the maintainers of those repositories to then resign their packages; I did not expect users to have to do anything to fix their systems.

For the record, I was not aware that rpm would refuse to uninstall or upgrade packages signed with an outdated certificate.  And I agree that indeed is a no-go.

Panu mentioned https://bugzilla.redhat.com/show_bug.cgi?id=2149762 in #c5; that was filed on Nov 30th. Mikhail reported the problems with listing packages in the initial report, and problems with upgrades on Dec 6th.

Thanks for linking to that issue.  We're talking about two separate issues: 1) third-parties using weak cryptography, and 2) rpm not being able to uninstall or upgrade a package that was previously installed using cryptography that is now considered to be weak.  I think the focus of the issue you linked to is (1).  I tend to agree with Panu: the third parties need to update the cryptography they are using; there is little that we can do about that.  The bigger issue for Fedora is (2).  I don't think that issue was raised in https://bugzilla.redhat.com/show_bug.cgi?id=2149762.

Comment #2 says "Looks like all 3rd party packages couldn't update." and posts the output of an attempt. Comment #8 shows the log of a `dnf upgrade --refresh` attempt (showing that it fails) and then an attempt to remove the weakly-signed package with `dnf remove sublime-merge` (again showing that it fails).

That, uh, seems like raising the issue to me.

sigh, don't use those hyperlinks. I'm referring to the comments on 2149762 , not on this bug.

> It's completely normal that these type of infrastructure changes are only addressed when they actually break;
> warnings are not enough, even for security-minded people. I expected the maintainers of those repositories to
> then resign their packages; I did not expect users to have to do anything to fix their systems.

There seems to be a historical inconsistency here. You say that things must break to be fixed,
and also that warnings will be ignored. But in fact, there were no warning that could have been
reacted to: nothing in F37 and earlier versions warned about signatures being bad.

It seems that there are two approaches to this kind of transition:
1. drag it out, start warning early, then warning more verbosely, at some point
   "break" things by default but still allow users to revert to status quo ante,
   and finally remove support.
2. compress the timeline, like we did here.

Effectively, we went directly from "full support with nara a warning" in F37 directly
to the last step in F38 branched. After the blocker bugs were filed, we're back to square
one, i.e. full support with no programmatic warnings.

Ideally, if we have to do this or something similar again, we would have warnings
emitted in F(n-2) and F(n-1), with increasingly verbose information about what is
broken and how to fix it, so that users can notify third parties they care about,
and those third parties have time to fix things, and we don't have to scramble at
the last moment.

Or we should just not do this kind of incompatible changes to begin with. The expectation of third-party packagers (and I am also one of those) is that packages that do not depend on some outdated soname remain installable basically forever. Nobody expects RPM to reject previously valid RPM packages.

So F38 will allow some degree of flexibility.

Is RPM or DFN *now* (after the patches that went on) emitting any warnings?

As things stand, none whatsoever.

FEDORA-2023-bd9a4614ad has been pushed to the Fedora 38 stable repository.
If problem still persists, please make note of it in this bug report.

At least when I tested this update on rawhide, I got a few warnings from "dnf install google-chrome-stable" after accepting the GPG signatures. I don't exactly remember which warnings, but there definitely were some.

*** Bug 2170839 has been marked as a duplicate of this bug. ***

Alexander, Panu and others - there are still users who experience this problem [1], even after the updated policy. It seems to be related to official AnyDesk packages. On Fedora 38, these can't be installed:

https://download.anydesk.com/linux/anydesk-6.1.1-1.el7.x86_64.rpm
https://download.anydesk.com/linux/anydesk-6.1.1-1.el8.x86_64.rpm

$ sudo rpm -iv ./anydesk-6.1.1-1.el7.x86_64.rpm 
error: ./anydesk-6.1.1-1.el7.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID cdffde29: BAD
error: ./anydesk-6.1.1-1.el7.x86_64.rpm cannot be installed

But on Fedora 37, they can be installed just fine.

AnyDesk seems to have fixed this issue with their new release, it works on Fedora 38 as well:

https://download.anydesk.com/linux/anydesk-6.2.1-1.el7.x86_64.rpm
https://download.anydesk.com/linux/anydesk-6.2.1-1.el8.x86_64.rpm

But those users, who installed 6.1.1 (or earlier) on Fedora 37, and then upgraded to Fedora 38 before 6.2.1 was available (or perhaps the AnyDesk repo was down at the moment, or they installed it without adding a repo), they are stuck with a broken package in the system once again:

$ sudo update-crypto-policies --set DEFAULT 
Setting system policy to DEFAULT
Note: System-wide crypto policies are applied on application start-up.
It is recommended to restart the system for the change of policies
to fully take place.

$ rpm -qa > /dev/null
error: rpmdbNextIterator: skipping h#      30 
Header V4 RSA/SHA512 Signature, key ID cdffde29: BAD
Header SHA1 digest: OK

$ rpm -q --nosignature --querybynumber 30
anydesk-6.1.1-1.x86_64


Why is anydesk-6.1.1 rpm affected when the policy update fixed the other affected rpms? Is there some crypto we've forgotten to allow? Please look at it, thanks!


(I'm reopening this bug and making it block the Final milestone, so that we don't lose track of this).

[1] https://discussion.fedoraproject.org/t/70379/15

Please provide a link to the GPG key for the Anydesk repository. The question you are asking cannot be answered without the key.

The key is here:

https://keys.anydesk.com/repos/RPM-GPG-KEY

Ref: http://rpm.anydesk.com/howto.html

> $ rpm -qa > /dev/null
> error: rpmdbNextIterator: skipping h#      30 
> Header V4 RSA/SHA512 Signature, key ID cdffde29: BAD
                ^^^^^^
> $ rpm -q --nosignature --querybynumber 30
> anydesk-6.1.1-1.x86_64
                ^

I don't know what this is or where it comes from, but this isn't one of the four packages linked above. All the linked packages have RSA/SHA1 signatures. 

Also note that when upgrading from an older release, if that system doesn't happen to have crypto-policies-scripts package installed there can be some failures until you install that (which runs /usr/bin/update-crypto-policies to get stuff properly updated)

There is no difference in how 6.1.1 and 6.2.1 are signed, they all use SHA1 with the same key ID:

:) cllang@frootmig:/tmp$ for file in anydesk-6.*.rpm; do rpm --qf "%{SIGPGP}\n" -q "$file" | xxd -ps -r | pgpdump; done                                                                                                                  [1/744]
warning: anydesk-6.1.1-1.el7.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID cdffde29: NOKEY
Old: Signature Packet(tag 2)(277 bytes)
        Ver 3 - old
        Hash material(5 bytes):
                Sig type - Signature of a binary document(0x00).
                Creation time - Tue Apr 13 13:08:37 CEST 2021
        Key ID - 0x18DF3741CDFFDE29
        Pub alg - RSA Encrypt or Sign(pub 1)
        Hash alg - SHA1(hash 2)
        Hash left 2 bytes - 63 85
        RSA m^d mod n(2048 bits) - ...
                -> PKCS-1
warning: anydesk-6.1.1-1.el8.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID cdffde29: NOKEY
Old: Signature Packet(tag 2)(277 bytes)
        Ver 3 - old
        Hash material(5 bytes):
                Sig type - Signature of a binary document(0x00).
                Creation time - Tue Apr 13 13:08:37 CEST 2021
        Key ID - 0x18DF3741CDFFDE29
        Pub alg - RSA Encrypt or Sign(pub 1)
        Hash alg - SHA1(hash 2)
        Hash left 2 bytes - 60 51
        RSA m^d mod n(2047 bits) - ...
                -> PKCS-1
warning: anydesk-6.2.1-1.el7.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID cdffde29: NOKEY
Old: Signature Packet(tag 2)(277 bytes)
        Ver 3 - old
        Hash material(5 bytes):
                Sig type - Signature of a binary document(0x00).
                Creation time - Wed Nov  2 12:44:16 CET 2022
        Key ID - 0x18DF3741CDFFDE29
        Pub alg - RSA Encrypt or Sign(pub 1)
        Hash alg - SHA1(hash 2)
        Hash left 2 bytes - 3c fc
        RSA m^d mod n(2047 bits) - ...
                -> PKCS-1
warning: anydesk-6.2.1-1.el8.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID cdffde29: NOKEY
Old: Signature Packet(tag 2)(277 bytes)
        Ver 3 - old
        Hash material(5 bytes):
                Sig type - Signature of a binary document(0x00).
                Creation time - Wed Nov  2 12:44:16 CET 2022
        Key ID - 0x18DF3741CDFFDE29
        Pub alg - RSA Encrypt or Sign(pub 1)
        Hash alg - SHA1(hash 2)
        Hash left 2 bytes - 7b 14
        RSA m^d mod n(2048 bits) - ...
                -> PKCS-1

The public key is RSA 2048 and uses SHA256 hashes. The use of SHA1 in the signatures would prevent the signature from being verified on RHEL 9, but this has never been enforced for Fedora in the default policy.

(In reply to Panu Matilainen from comment #110)
> > $ rpm -qa > /dev/null
> > error: rpmdbNextIterator: skipping h#      30 
> > Header V4 RSA/SHA512 Signature, key ID cdffde29: BAD
>                 ^^^^^^
> > $ rpm -q --nosignature --querybynumber 30
> > anydesk-6.1.1-1.x86_64
>                 ^
> 
> I don't know what this is or where it comes from, but this isn't one of the
> four packages linked above. All the linked packages have RSA/SHA1
> signatures. 

It's an older package that Anydesk had provided (which can't be updated using the AnyDesk Fedora repos because the new 6.2x packages have broken deps. See:
https://discussion.fedoraproject.org/t/cannot-install-anydesk/73854/11 )


> Also note that when upgrading from an older release, if that system doesn't
> happen to have crypto-policies-scripts package installed there can be some
> failures until you install that (which runs /usr/bin/update-crypto-policies
> to get stuff properly updated)


I think I had this installed, certainly have it installed now:

$ rpm -q crypto-policies-scripts
crypto-policies-scripts-20230301-1.gita12f7b2.fc38.noarch

Clemens: Thanks, but what does that mean? I'm still seeing this:

$ rpm -q anydesk
error: rpmdbNextIterator: skipping h#      30 
Header V4 RSA/SHA512 Signature, key ID cdffde29: BAD
Header SHA1 digest: OK
package anydesk is not installed

What should one do in this situation?

Okay, there's a SHA512 signed package at http://rpm.anydesk.com/fedora/x86_64/Packages/anydesk_6.2.1-1_x86_64.rpm which checks out fine. Unfortunately there are no older versions online to check.

Ankur Sinha, can you run the following on the system with the problematic package? The first is just to ensure you have the necessary utilities, its output doesn't matter, it's the rpm -q that we're interested in.

# dnf -y install /usr/bin/xxd pkgdump
# rpm -q --nosignature --qf "%{SIGPGP}\n" anydesk|xxd -ps -r|pgpdump

(In reply to Panu Matilainen from comment #110)
> > $ rpm -q --nosignature --querybynumber 30
> > anydesk-6.1.1-1.x86_64
>                 ^
> 
> I don't know what this is or where it comes from, but this isn't one of the
> four packages linked above.

You already found the link, but in short, I decided to include rpms which don't have dependency problems and they look problematic in the same way:

$ sudo rpm --import 'https://keys.anydesk.com/repos/RPM-GPG-KEY'
$ sudo rpm -i ./anydesk-6.1.1-1.el7.x86_64.rpm 
error: ./anydesk-6.1.1-1.el7.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID cdffde29: BAD
error: ./anydesk-6.1.1-1.el7.x86_64.rpm cannot be installed
$ sudo rpm -i ./anydesk-6.2.1-1.el7.x86_64.rpm 
Created symlink /etc/systemd/system/multi-user.target.wants/anydesk.service → /etc/systemd/system/anydesk.service.
Redirecting to /bin/systemctl start anydesk.service

But I might have confused myself (and others), because this particular problem seems to be related to the gpg key. The older version only fails when the key is present on the system. Otherwise it installs:

$ sudo rpm -e gpg-pubkey-cdffde29-5a38cbae
$ sudo rpm -e anydesk
$ sudo rpm -i ./anydesk-6.1.1-1.el7.x86_64.rpm 
warning: ./anydesk-6.1.1-1.el7.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID cdffde29: NOKEY
Created symlink /etc/systemd/system/multi-user.target.wants/anydesk.service → /etc/systemd/system/anydesk.service.
Redirecting to /bin/systemctl start anydesk.service


So I'm not sure what the problem is here, but the main issue we're discussing is the rpm version that Ankur has on his system, anydesk-6.1.1-1.x86_64, which can't be queried by rpm even after installing the updated policy. Sorry for confusion.

(In reply to Panu Matilainen from comment #113)
> Okay, there's a SHA512 signed package at
> http://rpm.anydesk.com/fedora/x86_64/Packages/anydesk_6.2.1-1_x86_64.rpm
> which checks out fine. Unfortunately there are no older versions online to
> check.
> 
> Ankur Sinha, can you run the following on the system with the problematic
> package? The first is just to ensure you have the necessary utilities, its
> output doesn't matter, it's the rpm -q that we're interested in.
> 
> # dnf -y install /usr/bin/xxd pkgdump
> # rpm -q --nosignature --qf "%{SIGPGP}\n" anydesk|xxd -ps -r|pgpdump

Here it is:


```
$ rpm -q --nosignature --qf "%{SIGPGP}\n" anydesk|xxd -ps -r|pgpdump
Old: Signature Packet(tag 2)(327 bytes)
        Ver 4 - new
        Sig type - Signature of a binary document(0x00).
        Pub alg - RSA Encrypt or Sign(pub 1)
        Hash alg - SHA512(hash 10)
        Hashed Sub: issuer fingerprint(sub 33)(21 bytes)
         v4 -   Fingerprint - d5 63 11 e5 ff 3b 6f 39 d5 a1 6a be 18 df 37 41 cd ff de 29 
        Hashed Sub: signature creation time(sub 2)(4 bytes)
                Time - Thu Apr 15 20:47:54 IST 2021
        Hashed Sub: signer's User ID(sub 28)(18 bytes)
                User ID - info
        Sub: issuer key ID(sub 16)(8 bytes)
                Key ID - 0x18DF3741CDFFDE29
        Hash left 2 bytes - ae e5 
        RSA m^d mod n(2044 bits) - ...
                -> PKCS-1
```

Should I remove the package as noted in https://bugzilla.redhat.com/show_bug.cgi?id=2170878#c5? I can't upgrade at the moment:


$ sudo dnf offline-upgrade download -y
...

Total                                                                                                                                        5.0 MB/s | 494 MB     01:39     
Delta RPMs reduced 538.4 MB of updates to 494.0 MB (8.2% saved)
Running transaction check
error: rpmdbNextIterator: skipping h#      30 
Header V4 RSA/SHA512 Signature, key ID cdffde29: BAD
Header SHA1 digest: OK
error: rpmdbNextIterator: skipping h#      30 
Header V4 RSA/SHA512 Signature, key ID cdffde29: BAD
Header SHA1 digest: OK
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: An rpm exception occurred: package not installed

$ sudo dnf offline-upgrade reboot
Error: system is not ready for upgrade

Ankur, please wait a bit longer. We need to figure what the problem is in your case first, and possibly test a fix.

Panu, Clemens, Alexander, can you please look at this? Thanks!

Would it be possible to try installing the package using rpm directly?  If so, please do: `RPM_TRACE=1 rpm ...`.  rpm will generate a detailed trace, which will probably include additional information about why the signature or certificate is considered invalid.

(In reply to Kamil Páral from comment #117)
> Panu, Clemens, Alexander, can you please look at this? Thanks!

I don't know what's wrong.

We have no concerns from a cryptography point of view with SHA-512 signatures, but they are certainly not very well tested since non of our rpm building tools generate them. Maybe it's just that rpm's old implementation of signature verification did not support SHA-512 signatures; however, if that's the case, I don't know how this package ended up successfully installed at all.

I would expect the new sequoia-based backend to support SHA-512 GPG signatures without problem, but I have not verified this.

(In reply to Kamil Páral from comment #117)
> Ankur, please wait a bit longer. We need to figure what the problem is in
> your case first, and possibly test a fix.
> 
> Panu, Clemens, Alexander, can you please look at this? Thanks!

Sure thing. `--exclude=anydesk` did the trick.

> Would it be possible to try installing the package using rpm directly?  If so, please do: `RPM_TRACE=1 rpm ...`.  rpm will generate a detailed trace, which will probably include additional information about why the signature or certificate is considered invalid.

Oh, indeed. Since this an already installed package, this should give the rpm-sequoia level trace:

    RPM_TRACE=1 rpm -q --querybynumber 30

Both old and new rpm should handle SHA512 without problems but it's indeed not a common path (and the currently available anydesk package with SHA512 validates fine with both pre-sequoia and sequoia using rpm).
Technically, it's also possible the signature is failing "for real". The sequoia trace should tell us.

Created attachment 1953350 [details]
Output of `RPM_TRACE=1 rpm -q --querybynumber 30`

Thanks!

This seems to be the real reason:

> _pgpVerifySignature: certificate 18DF3741CDFFDE29 (philandro Software GmbH <info>) uses legacy cryptography: No binding signature at time 2021-04-15T15:17:54Z
> _pgpVerifySignature: -> error: Failure: No binding signature at time 2021-04-15T15:17:54Z

And sure enough, the old rpm implementation did not verify binding signatures of keys. Perhaps the key has been updated at some point, that would be something neither rpm or dnf handles. In that case, 'rpm -e gpg-pubkey-cdffde29' should sort it out (as dnf will reimport on update), but that's just a theory.

Thanks for posting this.  Panu picked out the relevant bits:

> _pgpVerifySignature: certificate 18DF3741CDFFDE29 (philandro Software GmbH <info>) uses legacy cryptography: No binding signature at time 2021-04-15T15:17:54Z
> _pgpVerifySignature: -> error: Failure: No binding signature at time 2021-04-15T15:17:54Z

We know from earlier:

$ rpm -q --nosignature --qf "%{SIGPGP}\n" anydesk|xxd -ps -r|pgpdump
Old: Signature Packet(tag 2)(327 bytes)
        Ver 4 - new
        Sig type - Signature of a binary document(0x00).
        Pub alg - RSA Encrypt or Sign(pub 1)
        Hash alg - SHA512(hash 10)
        Hashed Sub: issuer fingerprint(sub 33)(21 bytes)
         v4 -   Fingerprint - d5 63 11 e5 ff 3b 6f 39 d5 a1 6a be 18 df 37 41 cd ff de 29 
        Hashed Sub: signature creation time(sub 2)(4 bytes)
                Time - Thu Apr 15 20:47:54 IST 2021

That is the data signature was created on April 15, 2021.  Sequoia checks that the certificate was valid at the time the data signature was made.  Looking at the certificate, we see:

$ wget -q -O- 'https://keys.anydesk.com/repos/RPM-GPG-KEY' | sq packet dump
Public-Key Packet, old CTB, 269 bytes
    Version: 4
    Creation time: 2017-12-19 08:19:58 UTC
    Pk algo: RSA
    Pk size: 2048 bits
    Fingerprint: D56311E5FF3B6F39D5A16ABE18DF3741CDFFDE29
    KeyID: 18DF3741CDFFDE29
  
User ID Packet, old CTB, 44 bytes
    Value: philandro Software GmbH <info>
  
Signature Packet, old CTB, 340 bytes
    Version: 4
    Type: PositiveCertification
    Pk algo: RSA
    Hash algo: SHA256
    Hashed area:
      Key flags: CS
      Symmetric algo preferences: AES256, AES192, AES128, TripleDES
      Hash preferences: SHA256, SHA384, SHA512, SHA224, SHA1
      Compression preferences: Zlib, BZip2, Zip
      Features: MDC
      Keyserver preferences: no modify
      Issuer Fingerprint: D56311E5FF3B6F39D5A16ABE18DF3741CDFFDE29
      Signature creation time: 2021-12-17 17:32:46 UTC
      Key expiration time: P2189DT33168S
    Unhashed area:
      Issuer: 18DF3741CDFFDE29
    Digest prefix: 3C01
    Level: 0 (signature over data)
  
Public-Subkey Packet, old CTB, 269 bytes
    Version: 4
    Creation time: 2017-12-19 08:19:58 UTC
    Pk algo: RSA
    Pk size: 2048 bits
    Fingerprint: CBB3E1703771B5EA0DAF6BF23C1690E043595971
    KeyID: 3C1690E043595971
  
Signature Packet, old CTB, 316 bytes
    Version: 4
    Type: SubkeyBinding
    Pk algo: RSA
    Hash algo: SHA256
    Hashed area:
      Key flags: EtEr
      Issuer Fingerprint: D56311E5FF3B6F39D5A16ABE18DF3741CDFFDE29
      Signature creation time: 2021-12-17 17:33:58 UTC
      Key expiration time: P2189DT33240S
    Unhashed area:
      Issuer: 18DF3741CDFFDE29
    Digest prefix: E530
    Level: 0 (signature over data)

The User ID binding signature was made at 2021-12-17 17:32:46 UTC, which is after the data signature was made.  So based on the knowledge that Sequoia has, the certificate was not valid when the signature was made.  Therefore, Sequoia conservatively determines that the data signature is not valid.

Since the key's creation time is set to 2017-12-19 08:19:58 UTC what likely happened is that old self signatures were stripped from https://keys.anydesk.com/repos/RPM-GPG-KEY .

OK, thank for debugging, everyone. Please help me figure out what to do now in regards to the FESCo statement from comment 15. This is another use case when things looked OK on Fedora 37 and fail on Fedora 38. Is there some approach we should take in F38, in order to ease the transition for our users? Print out warnings but accept the key? Or do you think this case is different, it should just be documented and let users help themselves?

To me the key points here are
1) there's a lot of obsolete/broken crypto out there
2) we need better error messages

Properly dealing with 2) needs an API redesign, but we'll try to work out some sort of bandaid solution.

(In reply to Ankur Sinha (FranciscoD) from comment #116)
> Should I remove the package as noted in
> https://bugzilla.redhat.com/show_bug.cgi?id=2170878#c5? I can't upgrade at the moment:
> 
> $ sudo dnf offline-upgrade download -y
> ...

Using --exclude=anydesk should allow you to update. But OTOH, feel free to remove, the reason is understood now.

> The User ID binding signature was made at 2021-12-17 17:32:46 UTC, which is after the data signature was made.  > So based on the knowledge that Sequoia has, the certificate was not valid when the signature was made.
> Therefore, Sequoia conservatively determines that the data signature is not valid.

This is actually a fine example of what this RpmSequoia change is really about. It's the proper OpenPGP implementation with subtleties like this that we're after, but the largely unrelated legacy crypto algorithms are threatening to steal the show.

If I understand this and the related devel discussion [1] correctly, the general opinion is that we should continue let this Anydesk rpm fail in Fedora 38, only possibly improve the error messaging in rpm (not a blocking change). Which means we can flip this bug back to closed. And I'll create a new specific Common Issue description based on the existing (already resolved) one [2]. Sounds OK?

[1] https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/BWLAW2SFBDYQWUSUCV5EI7CCI72DVUIR/#ZLI5PV6EX3GIW2ZVMETUQXA44EGGPV2Z
[2] https://discussion.fedoraproject.org/t/popular-third-party-rpms-fail-to-install-update-remove-due-to-security-policies-verification/70498

That sounds fine to me (although I don't really have a say in this).  Perhaps you could also point the anydesk team to that issue, and suggest that they update the published certificate to include the old binding signatures that are needed to validate the rpms.

(In reply to Panu Matilainen from comment #128)
> > The User ID binding signature was made at 2021-12-17 17:32:46 UTC, which is after the data signature was made.  > So based on the knowledge that Sequoia has, the certificate was not valid when the signature was made.
> > Therefore, Sequoia conservatively determines that the data signature is not valid.
> 
> This is actually a fine example of what this RpmSequoia change is really
> about. It's the proper OpenPGP implementation with subtleties like this that
> we're after, but the largely unrelated legacy crypto algorithms are
> threatening to steal the show.

But as we can see from the example, "subtleties like this" also break things that have worked before. I do not see what this additional check buys us other than preventing RPMs that previously installed from installing. If the certificate is valid at a later date, there is no reason why it would not have been valid at an earlier date too.

> But as we can see from the example, "subtleties like this" also break things that have worked before.

"Working" is a point of view: from a thief's point of view a locked door is a regression over an open one of course. The anydesk case is hardly malicious, but it's just an example of something rpm failed to check *at all* in the past, and there are any number of such missing validity checks in the old parser, some more severe than others.

Proposal from comment 129 sounds good to me.

Here's the Common Issues entry for the AnyDesk case:
https://discussion.fedoraproject.org/t/common-issues/80077

Kamil, that's an accurate, accessible, and actionable summary of the situation.  Thanks!

Yeah, that's a great description.

I'll close this bug. The original issue (SHA1/DSA signatures) is fixed,
and that was what was approved as a blocker. Signatures that are "wrong" for other
reasons deserve a separate bug. If there's still something to fix, please open
a new bug.

*** Bug 2183489 has been marked as a duplicate of this bug. ***

Some good news: Chrome is now using a RSA signature with SHA512 for its Chrome RPMs:

:) cllang@frootmig:~$ rpm -Kv google-chrome-stable_current_x86_64.rpm
google-chrome-stable_current_x86_64.rpm:
    Header V4 RSA/SHA512 Signature, key ID a3b88b8b: NOKEY
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
    V4 RSA/SHA512 Signature, key ID a3b88b8b: NOKEY
    MD5 digest: OK

The key is a subkey of the 0xEB4C1BFD4F042F6DDDCCEC917721F63BD38B4796 key in https://dl.google.com/linux/linux_signing_key.pub:

         Subkey: 8461EFA0E74ABAE010DE66994EB27DB2A3B88B8B
Public-key algo: RSA
Public-key size: 4096 bits
  Creation time: 2021-10-26 14:11:43 UTC
Expiration time: 2024-10-25 14:11:43 UTC (creation time + P1095D)
      Key flags: signing

The subkey binding signature, primary key binding signature, and positive certification (self-)signature of this key all use SHA256.

> Some good news: Chrome is now using a RSA signature with SHA512 for its Chrome RPMs

That's indeed very good news.

(In reply to Panu Matilainen from comment #123)
> [...trimmed...]
> Perhaps the key has been updated at some point, that
> would be something neither rpm or dnf handles. In that case, 'rpm -e gpg-pubkey-cdffde29' should sort it out (as dnf will reimport on update),
> but that's just a theory.

Thanks! This worked for me at first try.