Based on a few more random samples, it would appear that everything signed by COPR is broken this way now.

OK. Packages built in Copr with rpm-4.18.0-5.fc38 still cannot be installed in Copr with rpm-4.18.0-5.fc38.

See https://copr.fedorainfracloud.org/coprs/churchyard/sequoia/builds/

I'd untag the rpm-4.18.0-5.fc38 build until we've had a chance to investigate, breaking all of COPR is a bit much.

That, as in "I would do so but apparently don't have the permissions to untag".

rpm-4.18.0-5.fc38 also doesn't like the already installed rpmfusion packages here 
e.g. rfpkg-minimal 
Header RSA signature: BAD (header tag 268: invalid OpenPGP signature)

and last but not least google-chrome-stable rpm   from google is 
Header V4 DSA/SHA1 Signature, key ID 7fac5991: BAD

@pmatilai Thanks for cc'ing me.  I'm actively looking into this.

Here's what I'm seeing after adding a bit more instrumentation to rpm-sequoia (note `RPM_TRACE` is compiled in by default):

```
$ RPM_TRACE=1 ~/rpm/_build/rpmkeys -K /tmp/python3-pytest-7.1.3-1.fc38.noarch.rpm
...
_pgpPrtParams: entered
_pgpPrtParams: pkts: &[] <- 0xbcbe40
_pgpPrtParams: paramsp: &mut <- 0x7ffe50716760
_pgpPrtParams: Packet: PublicKey(V4(Key4 { fingerprint: Fingerprint("53DED2CB922D8B8D9E63FD18999F7CBF38AB71F4"), creation_time: 1612983048, pk_algo: RSAEncryptSign, mpis: RSA { e: 17 bits: 0100 01, n: 4096 bits: BE21 02AA A7F0 48D5 522F 451A 9C9D 1A00 C64C EC63 7A71 212D 4BF8 2E20 6968 C831 FD5E 0414 EABF D78A 778C 47B1 6F34 54BF ED97 CD56 6563 BFEC 95CA 9A01 CE0E 25EB BF81 E60C 83A8 E99D 61C5 E863 5511 7722 1E3E B0A1 D15C 2260 C1C7 499C C7EA 4CA9 C78B 2710 8207 83BA 1796 6DF8 D5D8 5826 BF1C B48E 1D0C E4F0 81E6 D68D CF8D 4B98 F39A 5109 A73D 7708 D390 A33F 99AC 2A34 4139 BC39 09DD A869 DEB4 A533 619C 1591 7F6F 3C4B C5B8 C07E 2334 848E 64B6 5339 FE87 5398 09C3 DC14 F516 686B 3F24 64F3 40AE 5263 66C3 7CDD 2D17 2C2C 696B A514 8187 733C 31CE 586B 5C58 6B37 6EF3 4095 362E 69CB A7A0 808E BD31 FF64 953B 9A06 996D B752 3D52 C325 D2C1 2915 601D B753 00AD 09F7 5DC5 1225 3DDC 6251 E5B7 FF5B 914C 37B0 D5EA 1B09 54C8 98FE 49C9 9A44 C513 DC21 0D28 C473 44C0 6214 2E0B 51E7 CEF3 0F18 9BE4 784C FD9D 1AA7 6B94 14BA 8ED5 E8B2 387D 9353 4A5A 85C8 88BF 39AD D9A2 588E C8EB 21C4 9994 DF2D 4768 D707 7AB3 B59C 56D3 B59E ECFD 3281 858F 6816 AE13 E202 55C8 55B2 DF7A 7F24 3C4E B655 63A4 7124 CF05 C674 F7BA 478C 4AF3 11CE 6C45 6B68 E319 31AC 1E20 8FAD 30E1 B055 BC59 BB69 0203 569F 94CF A87B 2535 171F 6AA8 CD3E 758C 873B E929 DCCE 54B5 F334 D3A4 5C35 C75F A06A C0E2 5EB3 F6C5 271C AB18 CCE2 CF46 F2A4 CF18 97DD 3F75 BC7B 5118 25E3 3CE8 0EB0 9273 8E41 6D13 56B9 A056 60DF D7C2 F7D1 D975 B5F4 AD4A 80B5 }, secret: None }))
_pgpPrtParams: PgpDigParams { obj, signid: buffer, userid: userid }: returning 0xb63c30
_pgpPrtParams: -> success
_pgpPrtParamsSubkeys: entered
_pgpPrtParamsSubkeys: pkts: &[] <- 0xbce250
_pgpPrtParamsSubkeys: subkeys: &mut <- 0x7ffe50716750
_pgpPrtParamsSubkeys: subkeys_count: &mut <- 0x7ffe5071674c
_pgpPrtParamsSubkeys: Got 0 subkeys
_pgpPrtParamsSubkeys: -> success
/tmp/python3-pytest-7.1.3-1.fc38.noarch.rpm:_pgpPrtParams: entered
_pgpPrtParams: pkts: &[] <- 0xbd2cb0
_pgpPrtParams: paramsp: &mut <- 0xbce7d8
_pgpPrtParams: Packet: Unknown(Unknown { common: Common { dummy: PhantomData }, tag: Signature, error: Malformed packet: unknown version, container: Container { unprocessed: " (0 bytes)", digest: "2D06800538D394C2" } })
_pgpPrtParams: -> error: Failure: Signature Packet
_pgpPrtParams: entered
_pgpPrtParams: pkts: &[] <- 0xbd2b28
_pgpPrtParams: paramsp: &mut <- 0xbcfb18
_pgpPrtParams: Packet: Unknown(Unknown { common: Common { dummy: PhantomData }, tag: Signature, error: Malformed packet: unknown version, container: Container { unprocessed: " (0 bytes)", digest: "2D06800538D394C2" } })
_pgpPrtParams: -> error: Failure: Signature Packet
...
...

So it seems that the certificate is parsed, but the signature is corrupted/unreadable/who knows ("unknown version").

(Still investigating...)

I extracted the signature and it looks like a version 3 sig.  Could that be?

```
$ sq packet dump /tmp/packets.pgp 
Unknown or Unsupported Packet, old CTB, 277 bytes
    Tag: Signature Packet
    Error: Malformed packet: unknown version
  
[neal@fedora-36 rpm-sequoia]$ sq packet dump --hex /tmp/packets.pgp 
Unknown or Unsupported Packet, old CTB, 3 header bytes + 277 bytes
    Tag: Signature Packet
    Error: Malformed packet: unknown version
  
    00000000  89                                                 CTB
    00000001     01 15                                           length
    00000003           03                                        version
    00000004              05 00 63 6a  70 3c 1f 96 2d 55 63 26       ..cjp<..-Uc&
    00000010  b3 35 01 08 ff 3d 07 ff  49 8d 98 e5 47 5c 7e 76   .5...=..I...G\~v
    00000020  70 3b 8f 3f 34 e4 d6 02  7c 93 0f 02 21 98 44 81   p;.?4...|...!.D.
    00000030  d9 f6 ec 47 d4 e5 cb 85  58 ab 81 b8 88 56 a3 4c   ...G....X....V.L
    00000040  fd fa 72 88 56 5d ef 97  5c 52 b3 a9 48 dc ee 16   ..r.V]..\R..H...
    00000050  a6 26 5a 4e eb dc 9e ea  cc 37 39 ed a6 e8 28 4c   .&ZN.....79...(L
    00000060  e1 5f 2d e5 9d fc 74 ac  3f 19 2f 38 47 51 04 62   ._-...t.?./8GQ.b
    00000070  4e ee b7 1b f7 09 9d 44  9d c2 56 f3 90 65 1d 84   N......D..V..e..
    00000080  5f b7 5b aa 37 67 c7 e3  3d d1 b6 eb 4e 4f 5b 41   _.[.7g..=...NO[A
    00000090  6c f4 d2 3f bd 92 4c ce  83 07 41 f8 73 54 8b 2e   l..?..L...A.sT..
    000000a0  46 2d a7 d4 cf 5f cb ac  3a 92 db 7a f7 30 12 c2   F-..._..:..z.0..
    000000b0  88 2d a9 5f 7a 81 40 10  72 57 dd 3a 61 aa eb 56   .-._z.@.rW.:a..V
    000000c0  9f 0c e6 bb cd 28 84 94  93 c2 cd a2 8d 4b 09 87   .....(.......K..
    000000d0  46 6e 61 4f 98 85 bc c6  c7 4b ca 8c b2 33 6d 30   FnaO.....K...3m0
    000000e0  57 9c 6b 7e ef 5e 0a bc  87 07 99 4a 89 51 b8 6f   W.k~.^.....J.Q.o
    000000f0  10 1e bd c6 bd 89 7f b6  06 2c dd d6 a3 ca 6d 3a   ........,....m:
    00000100  89 64 62 f7 be a5 0e 0d  25 c2 45 74 b3 26 35 88   .db.....%.Et.&5.
    00000110  1f 27 12 6b cd 51 88 f4                            .'.k.Q..
```

The package actually contains two signatures.  Here's the other one.  It's also v3.

```
$ sq packet dump --hex /tmp/packets-1.pgp 
Unknown or Unsupported Packet, old CTB, 3 header bytes + 277 bytes
    Tag: Signature Packet
    Error: Malformed packet: unknown version
  
    00000000  89                                                 CTB
    00000001     01 15                                           length
    00000003           03                                        version
    00000004              05 00 63 6a  70 3c 1f 96 2d 55 63 26       ..cjp<..-Uc&
    00000010  b3 35 01 08 73 1c 07 ff  68 42 ad 7e 62 6f fd d6   .5..s...hB.~bo..
    00000020  02 29 3d 78 c1 98 66 44  1b 62 00 69 22 59 15 f4   .)=x..fD.b.i"Y..
    00000030  e2 9e 4e 86 d6 bc ba 04  95 05 31 85 b2 f1 0b 03   ..N.......1.....
    00000040  67 31 66 56 07 fd 26 b1  6c da 40 a9 5e 9d 38 f7   g1fV..&.l.@.^.8.
    00000050  32 ac f0 86 e7 91 a9 6b  30 a0 3b 40 c4 e7 21 f2   2......k0.;@..!.
    00000060  a5 69 52 7a 7d 22 90 1f  8b c1 10 71 74 cc 30 93   .iRz}".....qt.0.
    00000070  16 c7 14 e6 61 33 af d8  ba 5a 0e 06 14 6c 55 b8   ....a3...Z...lU.
    00000080  86 5c 58 ad ef 71 ac 4a  b0 1f 4d d7 51 63 e1 2f   .\X..q.J..M.Qc./
    00000090  ff 02 08 f0 07 39 7b 19  95 13 70 16 4a d9 52 d0   .....9{...p.J.R.
    000000a0  ca 4b 0f f0 09 59 df e3  f0 d3 04 21 58 9f eb 9b   .K...Y.....!X...
    000000b0  ef dd 58 4e 66 84 71 c5  e2 21 66 91 cc 74 fc d3   ..XNf.q..!f..t..
    000000c0  e4 c2 71 63 99 0b 71 b5  62 68 b0 4f 88 23 bc d6   ..qc..q.bh.O.#..
    000000d0  bf 32 ed ba cc 4b ce 57  1d 72 fe b4 a6 e3 f1 28   .2...K.W.r.....(
    000000e0  43 82 ae 84 53 7a ad 46  b7 cf 8a 3b 3b fe 6e 5e   C...Sz.F...;;.n^
    000000f0  c3 93 fc aa 6b ee b8 ae  a7 a3 05 da 15 7c b3 ae   ....k........|..
    00000100  d0 51 ed 89 87 2d 36 b0  01 80 ff b8 93 09 0b e2   .Q...-6.........
    00000110  bd e6 4d ab 5b d0 73 89                            ..M.[.s.
```

FWIW, rpmkeys is also unhappy with the packages:

```
$ rpmkeys -K /tmp/python3-docutils-0.19-1.fc38.noarch.rpm 
/tmp/python3-docutils-0.19-1.fc38.noarch.rpm: digests SIGNATURES NOT OK
$ rpmkeys --version
RPM version 4.17.0
```

> I extracted the signature and it looks like a version 3 sig.  Could that be?

Righty... the internal parser supports V3 signatures, but not V3 keys. 

> The package actually contains two signatures.  Here's the other one.  It's also v3.

And yeah there can be two signatures, one on the header and one for header+payload, incidentally the latter is known in rpm as "rpm v3 signature" which can get confusing when talking about versions. But that's not the issue here of course.

What are COPR and rpmfusion using to sign packages? Rpm itself doesn't specify any particular version of signatures when calling gpg for signing, and even if it did, --force-v3-sigs option to gpg is long obsolete.

FTR, what Copr does is:
https://pagure.io/copr/copr/blob/21c7baf7bd567fdb70d8bc42d2fe4c6a5b5aede6/f/backend/copr_backend/sign.py#_91

GPG keys for users are generated by:
https://pagure.io/copr/copr/blob/21c7baf7bd567fdb70d8bc42d2fe4c6a5b5aede6/f/keygen/src/copr_keygen/logic.py#_47

Copr keygen is a Fedora 35 machine, with pretty much a default configuration.

Mistyped, sorry: GPG keys for users are generated by:
https://pagure.io/copr/copr/blob/21c7baf7bd567fdb70d8bc42d2fe4c6a5b5aede6/f/keygen/src/copr_keygen/logic.py#_122

> and last but not least google-chrome-stable rpm   from google is 
> Header V4 DSA/SHA1 Signature, key ID 7fac5991: BAD

I'm also concerned by this.  Google's own OpenPGP certificate still has only SHA1 selfsigs:

```
$ sq keyserver get 4CCA1EAF950CEE4AB83976DCA040830F7FAC5991 | sq-keyring-linter 
Certificate A040830F7FAC5991 is not valid under the standard policy + SHA-1: No binding signature at time 2022-11-10T15:29:16Z
Examined 1 certificate.
  1 certificate is invalid and was not linted. (BAD)
$ 
```

I haven't sorted out how to extract the OpenPGP sig from chrome's RPM to inspect it further, but if it's also using SHA1, it is a *good* thing that the update is declining it.  It's 2022, new signatures made over SHA1 should definitely be rejected.

(In reply to Pavel Raiskup from comment #13)
> Mistyped, sorry: GPG keys for users are generated by:
> https://pagure.io/copr/copr/blob/21c7baf7bd567fdb70d8bc42d2fe4c6a5b5aede6/f/
> keygen/src/copr_keygen/logic.py#_122

what seems like it really matters here is how the app's `GPG_BINARY` is configured:

https://pagure.io/copr/copr/blob/21c7baf7bd567fdb70d8bc42d2fe4c6a5b5aede6/f/keygen/src/copr_keygen/gpg.py

Okay, so Copr uses "sign", which is "Wrapper for /bin/sign from obs-sign package", which apparently comes from obs-signd package in Fedora. I didn't read+understand the code at this second but looking at https://github.com/openSUSE/obs-sign/blob/master/signd it talks about V3 quite a bit, including "--force-v3-sigs" to gpg. So that appears to be the thing creating V3 signatures. 

If rpmfusion uses the same, that explains a bit.

FWIW, I distinctively remember coming across this in rfc-4880 back in 2011 when I removed the support for v3 keys in rpm's parser and thinking whether v3 signatures should also be dropped or at least deprecated somehow:

   Implementations SHOULD accept V3 signatures.  Implementations SHOULD generate V4 signatures.

I haven't been looking, but that still stands in the official RFC, and even the draft-in-progress seems to only have deprecated V3 signature creation.

Hm, so it seems like we should fix how we call the /bin/sign:
https://github.com/openSUSE/obs-sign/issues/40

Note that *RPM v3* and *OpenPGP v3* signatures are two entirely different entities and not to be confused (see comment #11). That obs-sign ticket is talking about *rpm* V4 signatures which are twenty years old by now.

Anyway, with the issue now identified: this is not an off-by-one someplace but lacking support for something that is in (unexpectedly) wide use. I'll revert the change now to limit the disruption while we figure what to do with this.

Ok, rpm-4.18.0-6.fc38 reverted back to the internal pgp parser just finished building so the world can continue to revolve while we consider the options.

Thanks Neal (and everybody) for the super-fast response on this.

Created attachment 1923736 [details]
An utility for grabbing signatures from packages

Attached is a simple utility for extracting binary format rpm signatures out of package files and the rpmdb. Eg

[pmatilai🎩︎localhost ~]$ ./grabsig.py /tmp/google-chrome-stable-107.0.5304.110-1.x86_64.rpm 
extracted /tmp/google-chrome-stable-107.0.5304.110-1.x86_64.rpm.dsaheader
extracted /tmp/google-chrome-stable-107.0.5304.110-1.x86_64.rpm.siggpg
[pmatilai🎩︎localhost ~]$ ./grabsig.py libde265
extracted libde265-1.0.9-1.fc36.x86_64.rsaheader
extracted libde265-1.0.9-1.fc36.x86_64.sigpgp

Best enjoyed with a tool capable of dumping the contents, such as 'gpg --list-packets', 'pgpdump' or 'sq packet dump' (but sq fails here because it doesn't support V3 signatures, which is why we have this bug), eg

[pmatilai🎩︎localhost ~]$ gpg --list-packets /tmp/google-chrome-stable-107.0.5304.110-1.x86_64.rpm.dsaheader
# off=0 ctb=88 tag=2 hlen=2 plen=70
:signature packet: algo 17, keyid A040830F7FAC5991
	version 4, created 1667878963, md5len 0, sigclass 0x00
	digest algo 2, begin of digest 4b b4
	hashed subpkt 2 len 4 (sig created 2022-11-08)
	subpkt 16 len 8 (issuer key ID A040830F7FAC5991)
	data: [159 bits]
	data: [159 bits]

So, basically everything signed by obs-signd is affected as it defaults to OpenPGP v3 signatures. And that being used by OBS and multiple other places for signing rpms, this affects at least
- opensuse (and so presumably their enterprise offerings too but can't verify that)
- copr
- rpmfusion

I don't know what RHEL is signed with, but packages in RHEL 7-9 (didn't bother with older) are signed using OpenPGP V3 signatures too. 

So there really is only one conclusion to make: this is a no-go until Sequoia adds support for verifying V3 signatures. Or the world catches up, which is going to be years before all relevant content signed with V3 has gone dropped out of relevance, even if everybody started just now.

FWIW, Fedora itself and also Mageia appear to have V4 signatures, at least as of current versions.

I'm really glad that this was discovered, even if it represents a speedbump in the deployment of better OpenPGP tooling in the short term.

I just reported this to copr, asking them to move to something with more robust testing than obs-sign: https://pagure.io/copr/copr/issue/2360  fwiw, i believe newer versions of obs-sign should be making OpenPGP v4 signatures, so you might be seeing something running an older version of obs-sign, but i'd appreciate verification from folks who know OBS better than i do on that topic.

The upcoming cryptographic refresh of the OpenPGP standard explicitly says that clients MUST NOT generate v3 signatures but MAY verify them.  However, v3 signatures contain significantly less context than v4 signatures.  For example, the data signed by a v3 signature does not include the version number, the choices of hash and asymmetric algorithm involved, or any additional metadata that it is possible to embed in a v4 signature.  The only two pieces of metadata actually covered by a v3 signature are the sigtype (one octet) and a four-byte timestamp (unsigned seconds since the unix epoch).

If we've learned anything from cryptographic signing protocols over the last several decades, it's that one of the risks of signatures is that they can be interpreted out of context.  the simple and obvious fix for this is to ensure that the relevant pieces of context are included in the signed data in an unambiguous way.  v3 signatures cannot do that.

I strongly recommend rpm producing a warning when it encounters a v3 signature, so that we can flush out any remaining legacy signers.

(In reply to Panu Matilainen from comment #22)
> Created attachment 1923736 [details]
> An utility for grabbing signatures from packages
> 
> Attached is a simple utility for extracting binary format rpm signatures out
> of package files and the rpmdb. Eg

thank you for this!  this is a super useful utility.  it would be great to see it exposed directly in the 

> [pmatilai🎩︎localhost ~]$ gpg --list-packets /tmp/google-chrome-stable-107.0.5304.110-1.x86_64.rpm.dsaheader
> # off=0 ctb=88 tag=2 hlen=2 plen=70
> :signature packet: algo 17, keyid A040830F7FAC5991
> 	version 4, created 1667878963, md5len 0, sigclass 0x00
> 	digest algo 2, begin of digest 4b b4
> 	hashed subpkt 2 len 4 (sig created 2022-11-08)
> 	subpkt 16 len 8 (issuer key ID A040830F7FAC5991)
> 	data: [159 bits]
> 	data: [159 bits]

this is an OpenPGPv4 signature, but it is problematic for two different reasons.  I'm glad we're looking at this, because these embedded signatures have been explicitly deprecated for a long time.  These are both 1024-bit DSA signatures, using SHA1 as the digest function.  They were made in 2022-11-08.

I won't rehearse the full history of deprecation of these primitives here, but as one datapoint:

Back in 2015 (7 years ago!) NIST explicitly deprecated DSA keys less than 2048 bits in table 2 of SP 800-131Ar1: https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-131Ar1.pdf  .  The same document deprecates SHA-1 as well (see table 9).   So for the last seven years (if not longer) they have been explicitly marked as "disallowed" for digital signature generation, and "legacy use only" for digital signature validation.

It is really not good for rpm that it accepts such a signature without a warning.  And it's a particularly bad look for Google to be distributing Chrome for GNU/Linux under such a signature in 2022 especially since Google led the way on SHA-1 deprecation for digital signatures in other contexts (https://words.filippo.io/the-unofficial-chrome-sha1-faq/).

It doesn't matter, because OpenPGP v3 signatures will remain in use by the world over for a very long time. If Sequoia isn't going to support it, then we can't use it. No ifs, ands, or buts about it.

obs-signd is probably the only actively maintained signing service that supports a variety of inputs and outputs. Even Fedora's own sigul is basically not maintained. You need to make a case to the OBS people to break backwards compatibility by default.

It is not our responsibility when Sequoia is broken, that's the point of having the separated backend.

Hi all,

Thanks everyone for the quick help and the constructive feedback!  The [MR to add support for verifying v3 signatures](https://gitlab.com/sequoia-pgp/sequoia/-/merge_requests/1377) is done.  Next week we'll do a release.

> (In reply to Neal Gompa from comment #26)
> It is not our responsibility when Sequoia is broken, that's the point of
> having the separated backend.

I'm sorry it seemed like we were unresponsive.  Next time, I'll try harder to respond more quickly.

(In reply to neal from comment #27)
> Hi all,
> 
> Thanks everyone for the quick help and the constructive feedback!  The [MR
> to add support for verifying v3
> signatures](https://gitlab.com/sequoia-pgp/sequoia/-/merge_requests/1377) is
> done.  Next week we'll do a release.
> 

Thanks for resolving this! :)

> > (In reply to Neal Gompa from comment #26)
> > It is not our responsibility when Sequoia is broken, that's the point of
> > having the separated backend.
> 
> I'm sorry it seemed like we were unresponsive.  Next time, I'll try harder
> to respond more quickly.

My problem wasn't the responsiveness, it was the attitude of the response. Comment 24 and Comment 25 basically implied that we're all doing it wrong and it's our fault. That is not an appropriate response. Moreover, from an RPM point of view, we can't be telling people that stuff they have can't be used anymore unless a gun is pointed to our heads to say that (e.g. FIPS policy modes breaking most of the world due to SHA-1 removal).

The OpenPGP RFCs do not outright say that OpenPGP v3 signatures are deprecated and should not be used. Opinions of the Sequoia PGP project members notwithstanding, a compliant OpenPGP implementation should handle them.

obs-sign and other solutions sill use v3 signatures by default because they support building for targets as old as SUSE Linux Enterprise 10 and Red Hat Enterprise Linux 5 today. GPG shipped then used OpenPGP v3 keys/signatures, not v4 ones.

(In reply to Neal Gompa from comment #26)
> obs-signd is probably the only actively maintained signing service that
> supports a variety of inputs and outputs. Even Fedora's own sigul is
> basically not maintained. You need to make a case to the OBS people to break
> backwards compatibility by default.
[...]
> obs-sign and other solutions sill use v3 signatures by default because they support building for targets as old as SUSE Linux Enterprise 10 and Red Hat Enterprise Linux 5 today. GPG shipped then used OpenPGP v3 keys/signatures, not v4 ones.

Sorry but that's just bulls***. Even rpm 4.4 supports OpenPGP v4 signatures, there really is no excuse for anything to use v3 these days, everything that does not support it went EOL something like a decade ago.

And mind you, that's not a criticism to obs-signd in particular, RHEL is equally guilty here. 

> 
> It is not our responsibility when Sequoia is broken, that's the point of
> having the separated backend.

Cut it, right now. We're in this together regardless of whose code it is.

> My problem wasn't the responsiveness, it was the attitude of the response. Comment 24 and Comment 25 basically implied that we're all doing it wrong and it's our fault.

They don't read anything like that to me. There are facts and some recommendations, and it IS us the rpm-ecosystem who has been sleeping here. The good news is that V3 doesn't appear to be heartbleed critically flawed, just that V4 is better and everybody should be using that instead. This actually reminds me a lot about rpm v3 vs v4 transition, certain parties were extremely show in adopting v4 format, probably because they didn't see the flaws in v3 and the somewhat simpler format was just easier for them to support.

Neal Walfield who's the one actually developing rpm-sequoia has been *extremely* responsive, helpful and unopionated towards us throughout. 

This caught us all by surprise. It's just another piece of rusty but vitally important piping in the backyard that everybody had forgotten until it bust open. It's a good thing this was discovered, now lets just move on to fix this up on the various fronts instead of harping at each other.

Status update across the board:

- support for v3 signature verification in Sequoia merged upstream: https://gitlab.com/sequoia-pgp/sequoia/-/merge_requests/1377 and supported in >= 1.11.0 (https://bugzilla.redhat.com/show_bug.cgi?id=2143959)
- rpmfusion has switched to v4 signatures going forward: https://bugzilla.rpmfusion.org/show_bug.cgi?id=6496
- PR to switch COPR to v4 signatures submitted: https://github.com/fedora-copr/copr/pull/2379
- request to have obs-signd create v4 signatures by default submitted: https://github.com/openSUSE/obs-sign/issues/43
- upgrading RHEL signing to v4 raised internally

A couple of interesting bits found during investigating all this:

GnuPG only started creating v4 signatures by default as late as 1.4.8 released in December 2007, and for example RHEL 5 released earlier that year carried 1.4.5 through its lifetime all the way up to 2017 (and 2020 with extended life support). I didn't check, but I'd assume other enterprise distros to be in similar position here. Signatures created on RHEL 6 and newer are v4 by default (both gpg --sign and rpmsign), so that's pretty old news too.

Fedora and EPEL appear to have switched to v4 signatures during 2020, but I haven't found any explicit mention of this. Which makes me suspect this was accidental from Sigil switching to use gnupg2 internally at that time, gnupg2 which has ignored --force-v3-sigs even if passed to it since 2014.

> accidental from Sigil switching

Err, that'd be Sigul. Sigil is an e-book editing application, most certainly unrelated :D

As panu noted, I released sequoia-openpgp 1.11.0 (https://lists.sequoia-pgp.org/hyperkitty/list/devel@lists.sequoia-pgp.org/thread/QYRV5LLHTSTBW57QEBNFAH5VHS3YMG5Q/).  It includes support for parsing and verifying v3 signatures.  By default, the standard policy rejects v3 signatures.

I released sequoia-config-policy 0.5.0 (https://gitlab.com/sequoia-pgp/sequoia-policy-config/-/tags/v0.5.0), which includes support for setting a policy for individual versions of a packet.  That is, it is now possible to set different policies for v3 signature packets and v4 signature packets in a policy configuration file.

I created an MR against fedora-crypto-policies (https://gitlab.com/redhat-crypto/fedora-crypto-policies/-/merge_requests/124) to enable support for v3 signatures by default.  I'm not terribly happy about this MR, because it means that v3 signatures are also enabled for other Sequoia-using programs, which I don't think is desirable.

I released rpm-sequoia 1.2.0 (https://github.com/rpm-software-management/rpm-sequoia/releases/tag/v1.2.0).  It adds support for v3 signature packets.  After talking with panu, we agreed that rpm-sequoia would allow v3 signature packets by default, however, these can be overridden by the crypto policy file (https://github.com/rpm-software-management/rpm-sequoia/commit/85be6f00ec2f9a29bddb8342126b89a496356453#diff-b1a35a68f14e696205874893c07fd24fdb88882b47c23cc0e0c80a30c7d53759R165).  Once v4 signatures are ubiquitous, we'll change rpm-sequoia to not accept v3 signatures by default (https://github.com/rpm-software-management/rpm-sequoia/issues/26).

decathorpe, who has been doing most of the packaging work knows about these releases.  AFAIK, these packages don't depend on any new packages.  Thus, we should be able to try using rpm with sequoia again soon.

Amazing work, decathorpe!  All three packages are already in rawhide:

https://packages.fedoraproject.org/pkgs/rust-sequoia-openpgp/rust-sequoia-openpgp+default-devel/
https://packages.fedoraproject.org/pkgs/rust-sequoia-policy-config/sequoia-policy-config/
https://packages.fedoraproject.org/pkgs/rust-rpm-sequoia/rpm-sequoia/dehttps://packages.fedoraproject.org/pkgs/rust-sequoia-openpgp/rust-sequoia-openpgp+default-devel/https://packages.fedoraproject.org/pkgs/rust-sequoia-policy-config/sequoia-policy-config/https://packages.fedoraproject.org/pkgs/rust-rpm-sequoia/rpm-sequoi

Awesome, thanks guys for all the extra work here! I'll swap rpm back to Sequoia today once I've managed to test this locally first.

Hmm, NOKEY return doesn't seem to work with V3 signatures. With eg https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/30/Everything/x86_64/os/Packages/3/389-admin-console-doc-1.1.12-8.fc30.noarch.rpm I'm getting:

[pmatilai🎩︎localhost _build]$ ./rpmkeys -Kv /tmp/389-admin-console-doc-1.1.12-8.fc30.noarch.rpm
/tmp/389-admin-console-doc-1.1.12-8.fc30.noarch.rpm:
    Header V3 RSA/SHA256 Signature, key ID cfc659b9: BAD
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
    V3 RSA/SHA256 Signature, key ID cfc659b9: BAD
    MD5 digest: OK

...whereas with the internal parser the above BAD's are NOKEY. I'll file a ticket on rpm-sequoia upstream to further sort this out.

Hmm, I see NOKEY:

$ RPM_TRACE=0 ~/rpm-4.18/b/rpmkeys -Kv /tmp/389-admin-console-doc-1.1.12-8.fc30.noarch.rpm 
/tmp/389-admin-console-doc-1.1.12-8.fc30.noarch.rpm:
    Header V3 RSA/SHA256 Signature, key ID cfc659b9: NOKEY
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
    V3 RSA/SHA256 Signature, key ID cfc659b9: NOKEY
    MD5 digest: OK
[neal@fedora-36 rpm-sequoia]$ git describe
v1.1.2-8-g2a0ad53
[neal@fedora-36 rpm-4.18]$ git describe
rpm-4.18.0-release

Can you rerun with RPM_TRACE=1 and send me the trace, please.

Oops, turns out I was testing with rpm-sequoia 1.1.2 from yesterday instead of 1.2.0 with the v3 support. With 1.2.0 it's fine.

Sorry for the false alarm! With that cleared, I'll go ahead and re-enable this now.

Okay, this all should be sorted now in rpm-sequoia >= 1.2.0 and re-enabled in rpm-4.18.0-7.fc38.

Thanks again for all the debugging, development + packaging work neal and decathorpe!

I couldn't have done it without the great collaboration!  Thank you!

This seems like a problem:

[mockbuild@523e7003234241e0ae27a4da6ee0b6e7 copr-backend-git-0.c25b27a]$ rpm -qpi ./test-data-copr-backend-2/build_results/00848963-example/example-1.0.14-1.fc30.x86_64.rpm
error: ./test-data-copr-backend-2/build_results/00848963-example/example-1.0.14-1.fc30.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID ac058ff1: BAD
error: ./test-data-copr-backend-2/build_results/00848963-example/example-1.0.14-1.fc30.x86_64.rpm: not an rpm package (or package manifest)
[mockbuild@523e7003234241e0ae27a4da6ee0b6e7 copr-backend-git-0.c25b27a]$ rpm -q rpm rpm-sequoia
rpm-4.18.0-7.fc38.x86_64
rpm-sequoia-1.2.0-1.fc38.x86_64

The guilty RPM is here:
https://raw.githubusercontent.com/fedora-copr/test-data-copr-backend/main/build_results/00848963-example/example-1.0.14-1.fc30.x86_64.rpm

Per Jakub's report (Copr testsuite failure): https://github.com/fedora-copr/copr/issues/2388

> example/example-1.0.14-1.fc30.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID ac058ff1: BAD
                                                          ^^^^
That's rawhide crypto policy, not rpm.

"update-crypto-policies --set DEFAULT:FEDORA32" to make it accessible, but you really shouldn't. There may also be a smaller hammer to achieve that, this isn't my home turf.

That's right.  Thank you for the fast reply!

```
error: /tmp/example-1.0.14-1.fc30.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID ac058ff1: BAD
error: /tmp/example-1.0.14-1.fc30.x86_64.rpm: not an rpm package (or package manifest)
```

But note the unfortunate spelling ^^^.  That file still _is_ an RPM
package, so I'd expect that /bin/rpm parses the headers and dumps the
requested info out (including a verbose SHA1 error/warning, sure).

I can confirm that `rpm --nosignature -qpi <that rpm>` still works.

Side note, Fedora Copr switched to OpenPGP v4 signatures yesterday.  As I
understand it, RPM stays compatible with OpenPGP v3 headers -> so we don't
plan to mass-resign our (rather large) RPM storage.

Thanks to all for your work on this!

(In reply to Pavel Raiskup from comment #43)
> ```
> error: /tmp/example-1.0.14-1.fc30.x86_64.rpm: Header V3 RSA/SHA1 Signature,
> key ID ac058ff1: BAD
> error: /tmp/example-1.0.14-1.fc30.x86_64.rpm: not an rpm package (or package
> manifest)
> ```
> 
> But note the unfortunate spelling ^^^.  That file still _is_ an RPM
> package, so I'd expect that /bin/rpm parses the headers and dumps the
> requested info out (including a verbose SHA1 error/warning, sure).

As a security measure, rpm does not parse headers with failing signatures (or digests for that matter). But yeah I'm sure the error message could be worded better to cover this particular case.

(In reply to Pavel Raiskup from comment #44)
> Side note, Fedora Copr switched to OpenPGP v4 signatures yesterday.  As I
> understand it, RPM stays compatible with OpenPGP v3 headers -> so we don't
> plan to mass-resign our (rather large) RPM storage.

Excellent news, thanks! The old packages will fall out of fashion over time, and we wont still be in this situation five years from now. This is the best we can ask for, the existing mass of packages out there is such that we can't expect it all to be mass-resigned.

(In reply to Panu Matilainen from comment #30)
> Status update across the board:
[...]
> - upgrading RHEL signing to v4 raised internally

It took us a while, but I'm happy to report that the RH signing server is producing V4 signatures now (since mid-June actually).