Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 2166028 - dnf under mock in bootstrap mode installs packages without /proc mounted
Summary: dnf under mock in bootstrap mode installs packages without /proc mounted
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: mock
Version: 37
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: Copr Team
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: 2080355
TreeView+ depends on / blocked
 
Reported: 2023-01-31 18:20 UTC by Jens Petersen
Modified: 2023-06-01 02:16 UTC (History)
7 users (show)

Fixed In Version: mock-4.0-1.fc38 mock-4.0-1.el9 mock-4.0-1.el8 mock-4.0-1.fc37
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-05-27 01:17:50 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Jens Petersen 2023-01-31 18:20:43 UTC
Description of problem:
It looks to me like something similar to bug 1467299
is reproducible with mock-3.5-1.fc37 when building in koji (for rawhide).

Version-Release number of selected component (if applicable):
mock-3.5-1.fc37

How reproducible:
100%

Steps to Reproduce:
1. Try to rebuild ghc8.10-8.10.7-10.fc38 in mock ie against itself.

Actual results:
root.log contains:
:
DEBUG util.py:445:  Running transaction
DEBUG util.py:445:  /usr/lib64/ghc-8.10.7/bin/ghc-pkg: error while loading shared libraries: libHSterminfo-0.4.1.4-ghc8.10.7.so: cannot open shared object file: No such file or directory
DEBUG util.py:445:  Installed:
:

Expected results:
No error

Additional info:
Note unlike ghc8.10-8.10.7-6.fc38 and ghc-8.10.7-123.fc38,
ghc8.10-8.10.7-10.fc38 uses RPATH's for it's shared libraries:

$ chrpath /usr/lib64/ghc-8.10.7/bin/ghc-pkg 
/usr/lib64/ghc-8.10.7/bin/ghc-pkg: RUNPATH=$ORIGIN/../terminfo-0.4.1.4:$ORIGIN/../ghc-boot-8.10.7:$ORIGIN/../Cabal-3.2.1.0:$ORIGIN/../process-1.6.13.2:$ORIGIN/../parsec-3.1.14.0:$ORIGIN/../text-1.2.4.1:$ORIGIN/../template-haskell-2.16.0.0:$ORIGIN/../pretty-1.1.3.6:$ORIGIN/../ghc-boot-th-8.10.7:$ORIGIN/../mtl-2.2.2:$ORIGIN/../transformers-0.5.6.2:$ORIGIN/../directory-1.3.6.0:$ORIGIN/../unix-2.7.2.2:$ORIGIN/../time-1.9.3:$ORIGIN/../filepath-1.4.2.1:$ORIGIN/../binary-0.8.8.0:$ORIGIN/../containers-0.6.5.1:$ORIGIN/../bytestring-0.10.12.0:$ORIGIN/../deepseq-1.4.4.0:$ORIGIN/../array-0.5.4.0:$ORIGIN/../base-4.14.3.0:$ORIGIN/../integer-gmp-1.0.3.0:$ORIGIN/../ghc-prim-0.6.1:$ORIGIN/../rts

which seems to be causing problems under systemd-nspawn mock again.

ghc8.10-8.10.7-10.fc38 works fine for me locally.

Comment 1 Jens Petersen 2023-01-31 18:35:05 UTC
(Just noting that currently ghc8.10-8.10.7-10.fc38 is only available in a koji sidetag)

Actually same happens when building with ghc9.0 which does not use RPATHs.
Maybe the filetriggers for ghc*-devel are not getting run properly?

I also remember that in eg https://copr.fedorainfracloud.org/coprs/petersen/haskell-language-server/
I had to use simple chroot instead of systemd-nspawn in order to use ghcX.Y, like here.

Comment 2 Jens Petersen 2023-01-31 19:05:34 UTC
Here is another example:
https://koji.fedoraproject.org/koji/taskinfo?taskID=96927426

The first error is in root.log:

DEBUG util.py:445:  /usr/lib64/ghc-9.2.5/bin/ghc-pkg-9.2.5: error while loading shared libraries: libHSterminfo-0.4.1.5-ghc9.2.5.so: cannot open shared object file: No such file or directory

Comment 3 Jens Petersen 2023-02-01 03:57:35 UTC
Unfortunately this is blocking my work on https://fedoraproject.org/wiki/Changes/Haskell_GHC_9.2_and_Stackage_20 :

Though I realised though this problem has existed for some time: likely since well before https://fedoraproject.org/wiki/Changes/GHC_parallel_version_installs (F36)

Comment 4 Zbigniew Jędrzejewski-Szmek 2023-02-01 12:42:03 UTC
This seems to be a problem with installation order. Locally, I get the following:

$ koji download-build -a x86_64 -a noarch ghc8.10-8.10.7-10.fc38
$ mock --clean && mock -i ghc8.10-*
...
Running transaction
  Preparing        :                                                                                                                                                              1/1 
  Installing       : ghc8.10-base-4.14.3.0-10.fc38.x86_64                                                                                                                       1/151 
  Installing       : ghc8.10-array-0.5.4.0-10.fc38.x86_64                                                                                                                       2/151 
  Installing       : ghc8.10-deepseq-1.4.4.0-10.fc38.x86_64                                                                                                                     3/151 
  Installing       : ghc8.10-filesystem-8.10.7-10.fc38.noarch                                                                                                                   4/151 
  Installing       : ghc8.10-bytestring-0.10.12.0-10.fc38.x86_64                                                                                                                5/151 
  Installing       : ghc8.10-containers-0.6.5.1-10.fc38.x86_64                                                                                                                  6/151 
  Installing       : ghc8.10-time-1.9.3-10.fc38.x86_64                                                                                                                          7/151 
  Installing       : ghc8.10-ghc-boot-th-8.10.7-10.fc38.x86_64                                                                                                                  8/151 
  Installing       : ghc8.10-unix-2.7.2.2-10.fc38.x86_64                                                                                                                        9/151 
  Installing       : ghc8.10-pretty-1.1.3.6-10.fc38.x86_64                                                                                                                     10/151 
  Installing       : ghc8.10-filepath-1.4.2.1-10.fc38.x86_64                                                                                                                   11/151 
  Installing       : ghc8.10-directory-1.3.6.0-10.fc38.x86_64                                                                                                                  12/151 
  Installing       : ghc8.10-template-haskell-2.16.0.0-10.fc38.x86_64                                                                                                          13/151 
  Installing       : ghc8.10-transformers-0.5.6.2-10.fc38.x86_64                                                                                                               14/151 
  Installing       : ghc8.10-binary-0.8.8.0-10.fc38.x86_64                                                                                                                     15/151 
  Installing       : ghc8.10-mtl-2.2.2-10.fc38.x86_64                                                                                                                          16/151 
  Installing       : ghc8.10-ghc-boot-8.10.7-10.fc38.x86_64                                                                                                                    17/151 
  Installing       : ghc8.10-process-1.6.13.2-10.fc38.x86_64                                                                                                                   18/151 
  Installing       : ghc8.10-ghc-heap-8.10.7-10.fc38.x86_64                                                                                                                    19/151 
  Installing       : ghc8.10-ghci-8.10.7-10.fc38.x86_64                                                                                                                        20/151 
  Installing       : ghc8.10-text-1.2.4.1-10.fc38.x86_64                                                                                                                       21/151 
  Installing       : ghc8.10-stm-2.5.0.1-10.fc38.x86_64                                                                                                                        22/151 
  Installing       : ghc8.10-terminfo-0.4.1.4-10.fc38.x86_64                                                                                                                   23/151 
  Installing       : ghc8.10-exceptions-0.10.4-10.fc38.x86_64                                                                                                                  24/151 
  Installing       : ghc8.10-parsec-3.1.14.0-10.fc38.x86_64                                                                                                                    25/151 
  Installing       : ghc8.10-hpc-0.6.1.0-10.fc38.x86_64                                                                                                                        26/151 
  Installing       : ghc8.10-ghc-8.10.7-10.fc38.x86_64                                                                                                                         27/151 
  Installing       : ghc8.10-Cabal-3.2.1.0-10.fc38.x86_64                                                                                                                      28/151 
  Installing       : ghc8.10-haskeline-0.8.2-10.fc38.x86_64                                                                                                                    29/151 
  Installing       : ghc8.10-libiserv-8.10.7-10.fc38.x86_64                                                                                                                    30/151 
  Installing       : ghc8.10-array-doc-0.5.4.0-10.fc38.noarch                                                                                                                  31/151 
  Installing       : ghc8.10-base-doc-4.14.3.0-10.fc38.noarch                                                                                                                  32/151 
  Installing       : ghc8.10-binary-doc-0.8.8.0-10.fc38.noarch                                                                                                                 33/151 
  Installing       : ghc8.10-bytestring-doc-0.10.12.0-10.fc38.noarch                                                                                                           34/151 
  Installing       : ghc8.10-Cabal-doc-3.2.1.0-10.fc38.noarch                                                                                                                  35/151 
  Installing       : ghc8.10-containers-doc-0.6.5.1-10.fc38.noarch                                                                                                             36/151 
  Installing       : ghc8.10-deepseq-doc-1.4.4.0-10.fc38.noarch                                                                                                                37/151 
  Installing       : ghc8.10-directory-doc-1.3.6.0-10.fc38.noarch                                                                                                              38/151 
  Installing       : ghc8.10-exceptions-doc-0.10.4-10.fc38.noarch                                                                                                              39/151 
  Installing       : ghc8.10-filepath-doc-1.4.2.1-10.fc38.noarch                                                                                                               40/151 
  Installing       : ghc8.10-ghc-boot-doc-8.10.7-10.fc38.noarch                                                                                                                41/151 
  Installing       : ghc8.10-ghc-boot-th-doc-8.10.7-10.fc38.noarch                                                                                                             42/151 
  Installing       : ghc8.10-ghc-compact-doc-0.1.0.0-10.fc38.noarch                                                                                                            43/151 
  Installing       : ghc8.10-ghc-doc-8.10.7-10.fc38.noarch                                                                                                                     44/151 
  Installing       : ghc8.10-ghc-heap-doc-8.10.7-10.fc38.noarch                                                                                                                45/151 
  Installing       : ghc8.10-ghci-doc-8.10.7-10.fc38.noarch                                                                                                                    46/151 
  Installing       : ghc8.10-haskeline-doc-0.8.2-10.fc38.noarch                                                                                                                47/151 
  Installing       : ghc8.10-hpc-doc-0.6.1.0-10.fc38.noarch                                                                                                                    48/151 
  Installing       : ghc8.10-libiserv-doc-8.10.7-10.fc38.noarch                                                                                                                49/151 
  Installing       : ghc8.10-mtl-doc-2.2.2-10.fc38.noarch                                                                                                                      50/151 
  Installing       : ghc8.10-parsec-doc-3.1.14.0-10.fc38.noarch                                                                                                                51/151 
  Installing       : ghc8.10-pretty-doc-1.1.3.6-10.fc38.noarch                                                                                                                 52/151 
  Installing       : ghc8.10-process-doc-1.6.13.2-10.fc38.noarch                                                                                                               53/151 
  Installing       : ghc8.10-stm-doc-2.5.0.1-10.fc38.noarch                                                                                                                    54/151 
  Installing       : ghc8.10-template-haskell-doc-2.16.0.0-10.fc38.noarch                                                                                                      55/151 
  Installing       : ghc8.10-terminfo-doc-0.4.1.4-10.fc38.noarch                                                                                                               56/151 
  Installing       : ghc8.10-text-doc-1.2.4.1-10.fc38.noarch                                                                                                                   57/151 
  Installing       : ghc8.10-time-doc-1.9.3-10.fc38.noarch                                                                                                                     58/151 
  Installing       : ghc8.10-transformers-doc-0.5.6.2-10.fc38.noarch                                                                                                           59/151 
  Installing       : ghc8.10-unix-doc-2.7.2.2-10.fc38.noarch                                                                                                                   60/151 
  Installing       : ghc8.10-xhtml-doc-3000.2.2.1-10.fc38.noarch                                                                                                               61/151 
  Installing       : ghc8.10-xhtml-3000.2.2.1-10.fc38.x86_64                                                                                                                   62/151 
  Installing       : libmpc-1.2.1-5.fc37.x86_64                                                                                                                                63/151 
  Installing       : cpp-12.2.1-4.fc37.x86_64                                                                                                                                  64/151 
  Installing       : ghc8.10-doc-8.10.7-10.fc38.x86_64                                                                                                                         65/151 
  Installing       : ghc8.10-ghc-compact-0.1.0.0-10.fc38.x86_64                                                                                                                66/151 
  Installing       : ghc8.10-manual-8.10.7-10.fc38.noarch                                                                                                                      67/151 
  Installing       : ncurses-c++-libs-6.3-4.20220501.fc37.x86_64                                                                                                               68/151 
  Installing       : ncurses-devel-6.3-4.20220501.fc37.x86_64                                                                                                                  69/151 
  Installing       : kernel-headers-6.1.5-200.fc37.x86_64                                                                                                                      70/151 
  Installing       : glibc-headers-x86-2.36-9.fc37.noarch                                                                                                                      71/151 
  Installing       : libxcrypt-devel-4.4.33-4.fc37.x86_64                                                                                                                      72/151 
  Installing       : glibc-devel-2.36-9.fc37.x86_64                                                                                                                            73/151 
  Installing       : annobin-docs-10.92-1.fc37.noarch                                                                                                                          74/151 
  Installing       : libtool-ltdl-2.4.7-2.fc37.x86_64                                                                                                                          75/151 
  Installing       : gmp-c++-1:6.2.1-3.fc37.x86_64                                                                                                                             76/151 
  Installing       : gmp-devel-1:6.2.1-3.fc37.x86_64                                                                                                                           77/151 
  Installing       : gc-8.0.6-4.fc37.x86_64                                                                                                                                    78/151 
  Installing       : guile22-2.2.7-6.fc37.x86_64                                                                                                                               79/151 
  Installing       : make-1:4.3-11.fc37.x86_64                                                                                                                                 80/151 
  Installing       : gcc-12.2.1-4.fc37.x86_64                                                                                                                                  81/151 
  Running scriptlet: gcc-12.2.1-4.fc37.x86_64                                                                                                                                  81/151 
  Installing       : ghc8.10-base-devel-4.14.3.0-10.fc38.x86_64                                                                                                                82/151 
  Installing       : ghc8.10-compiler-8.10.7-10.fc38.x86_64                                                                                                                    83/151 
  Installing       : ghc8.10-base-prof-4.14.3.0-10.fc38.x86_64                                                                                                                 84/151 
  Installing       : ghc8.10-array-devel-0.5.4.0-10.fc38.x86_64                                                                                                                85/151 
  Installing       : ghc8.10-deepseq-devel-1.4.4.0-10.fc38.x86_64                                                                                                              86/151 
  Installing       : ghc8.10-bytestring-devel-0.10.12.0-10.fc38.x86_64                                                                                                         87/151 
  Installing       : ghc8.10-containers-devel-0.6.5.1-10.fc38.x86_64                                                                                                           88/151 
  Installing       : ghc8.10-filepath-devel-1.4.2.1-10.fc38.x86_64                                                                                                             89/151 
  Installing       : ghc8.10-filepath-prof-1.4.2.1-10.fc38.x86_64                                                                                                              90/151 
  Installing       : ghc8.10-array-prof-0.5.4.0-10.fc38.x86_64                                                                                                                 91/151 
  Installing       : ghc8.10-deepseq-prof-1.4.4.0-10.fc38.x86_64                                                                                                               92/151 
  Installing       : ghc8.10-bytestring-prof-0.10.12.0-10.fc38.x86_64                                                                                                          93/151 
  Installing       : ghc8.10-containers-prof-0.6.5.1-10.fc38.x86_64                                                                                                            94/151 
  Installing       : ghc8.10-binary-devel-0.8.8.0-10.fc38.x86_64                                                                                                               95/151 
  Installing       : ghc8.10-transformers-devel-0.5.6.2-10.fc38.x86_64                                                                                                         96/151 
  Installing       : ghc8.10-transformers-prof-0.5.6.2-10.fc38.x86_64                                                                                                          97/151 
  Installing       : ghc8.10-binary-prof-0.8.8.0-10.fc38.x86_64                                                                                                                98/151 
  Installing       : ghc8.10-time-devel-1.9.3-10.fc38.x86_64                                                                                                                   99/151 
  Installing       : ghc8.10-unix-devel-2.7.2.2-10.fc38.x86_64                                                                                                                100/151 
  Installing       : ghc8.10-directory-devel-1.3.6.0-10.fc38.x86_64                                                                                                           101/151 
  Installing       : ghc8.10-time-prof-1.9.3-10.fc38.x86_64                                                                                                                   102/151 
  Installing       : ghc8.10-unix-prof-2.7.2.2-10.fc38.x86_64                                                                                                                 103/151 
  Installing       : ghc8.10-directory-prof-1.3.6.0-10.fc38.x86_64                                                                                                            104/151 
  Installing       : ghc8.10-ghc-boot-th-devel-8.10.7-10.fc38.x86_64                                                                                                          105/151 
  Installing       : ghc8.10-ghc-boot-th-prof-8.10.7-10.fc38.x86_64                                                                                                           106/151 
  Installing       : ghc8.10-process-devel-1.6.13.2-10.fc38.x86_64                                                                                                            107/151 
  Installing       : ghc8.10-mtl-devel-2.2.2-10.fc38.x86_64                                                                                                                   108/151 
  Installing       : ghc8.10-mtl-prof-2.2.2-10.fc38.x86_64                                                                                                                    109/151 
  Installing       : ghc8.10-process-prof-1.6.13.2-10.fc38.x86_64                                                                                                             110/151 
  Installing       : ghc8.10-pretty-devel-1.1.3.6-10.fc38.x86_64                                                                                                              111/151 
  Installing       : ghc8.10-template-haskell-devel-2.16.0.0-10.fc38.x86_64                                                                                                   112/151 
  Installing       : ghc8.10-text-devel-1.2.4.1-10.fc38.x86_64                                                                                                                113/151 
  Installing       : ghc8.10-stm-devel-2.5.0.1-10.fc38.x86_64                                                                                                                 114/151 
  Installing       : ghc8.10-ghc-heap-devel-8.10.7-10.fc38.x86_64                                                                                                             115/151 
  Installing       : ghc8.10-terminfo-devel-0.4.1.4-10.fc38.x86_64                                                                                                            116/151 
  Installing       : ghc8.10-terminfo-prof-0.4.1.4-10.fc38.x86_64                                                                                                             117/151 
  Installing       : ghc8.10-ghc-heap-prof-8.10.7-10.fc38.x86_64                                                                                                              118/151 
  Installing       : ghc8.10-exceptions-devel-0.10.4-10.fc38.x86_64                                                                                                           119/151 
  Installing       : ghc8.10-stm-prof-2.5.0.1-10.fc38.x86_64                                                                                                                  120/151 
  Installing       : ghc8.10-parsec-devel-3.1.14.0-10.fc38.x86_64                                                                                                             121/151 
  Installing       : ghc8.10-pretty-prof-1.1.3.6-10.fc38.x86_64                                                                                                               122/151 
  Installing       : ghc8.10-template-haskell-prof-2.16.0.0-10.fc38.x86_64                                                                                                    123/151 
  Installing       : ghc8.10-text-prof-1.2.4.1-10.fc38.x86_64                                                                                                                 124/151 
  Installing       : ghc8.10-ghc-boot-devel-8.10.7-10.fc38.x86_64                                                                                                             125/151 
  Installing       : ghc8.10-ghc-boot-prof-8.10.7-10.fc38.x86_64                                                                                                              126/151 
  Installing       : ghc8.10-ghci-devel-8.10.7-10.fc38.x86_64                                                                                                                 127/151 
  Installing       : ghc8.10-ghci-prof-8.10.7-10.fc38.x86_64                                                                                                                  128/151 
  Installing       : ghc8.10-hpc-devel-0.6.1.0-10.fc38.x86_64                                                                                                                 129/151 
  Installing       : ghc8.10-ghc-devel-8.10.7-10.fc38.x86_64                                                                                                                  130/151 
  Installing       : ghc8.10-hpc-prof-0.6.1.0-10.fc38.x86_64                                                                                                                  131/151 
  Installing       : ghc8.10-libiserv-devel-8.10.7-10.fc38.x86_64                                                                                                             132/151 
  Installing       : ghc8.10-parsec-prof-3.1.14.0-10.fc38.x86_64                                                                                                              133/151 
  Installing       : ghc8.10-exceptions-prof-0.10.4-10.fc38.x86_64                                                                                                            134/151 
  Installing       : ghc8.10-Cabal-devel-3.2.1.0-10.fc38.x86_64                                                                                                               135/151 
  Installing       : ghc8.10-haskeline-devel-0.8.2-10.fc38.x86_64                                                                                                             136/151 
  Installing       : ghc8.10-ghc-compact-devel-0.1.0.0-10.fc38.x86_64                                                                                                         137/151 
  Installing       : ghc8.10-xhtml-devel-3000.2.2.1-10.fc38.x86_64                                                                                                            138/151 
  Installing       : ghc8.10-devel-8.10.7-10.fc38.x86_64                                                                                                                      139/151 
  Installing       : ghc8.10-xhtml-prof-3000.2.2.1-10.fc38.x86_64                                                                                                             140/151 
  Installing       : ghc8.10-ghc-compact-prof-0.1.0.0-10.fc38.x86_64                                                                                                          141/151 
  Installing       : ghc8.10-haskeline-prof-0.8.2-10.fc38.x86_64                                                                                                              142/151 
  Installing       : ghc8.10-Cabal-prof-3.2.1.0-10.fc38.x86_64                                                                                                                143/151 
  Installing       : ghc8.10-libiserv-prof-8.10.7-10.fc38.x86_64                                                                                                              144/151 
  Installing       : ghc8.10-ghc-prof-8.10.7-10.fc38.x86_64                                                                                                                   145/151 
  Installing       : ghc8.10-prof-8.10.7-10.fc38.x86_64                                                                                                                       146/151 
  Installing       : ghc8.10-compiler-default-8.10.7-10.fc38.x86_64                                                                                                           147/151 
  Installing       : ghc8.10-doc-index-8.10.7-10.fc38.noarch                                                                                                                  148/151 
  Installing       : ghc8.10-8.10.7-10.fc38.x86_64                                                                                                                            149/151 
  Installing       : annobin-plugin-gcc-10.92-1.fc37.x86_64                                                                                                                   150/151 
  Running scriptlet: annobin-plugin-gcc-10.92-1.fc37.x86_64                                                                                                                   150/151 
  Installing       : gcc-plugin-annobin-12.2.1-4.fc37.x86_64                                                                                                                  151/151 
  Running scriptlet: gcc-plugin-annobin-12.2.1-4.fc37.x86_64                                                                                                                  151/151 
/usr/lib64/ghc-8.10.7/bin/ghc-pkg: error while loading shared libraries: libHSterminfo-0.4.1.4-ghc8.10.7.so: cannot open shared object file: No such file or directory

/usr/lib64/ghc-8.10.7/bin/haddock: error while loading shared libraries: libHSxhtml-3000.2.2.1-ghc8.10.7.so: cannot open shared object file: No such file or directory
warning: %transfiletriggerin(ghc8.10-doc-index-8.10.7-10.fc38.noarch) scriptlet failed, exit status 127

Error in <unknown> scriptlet in rpm package gcc-plugin-annobin
  Verifying        : gc-8.0.6-4.fc37.x86_64                                                                                                                                     1/151 
...

If I do 'mock --shell' afterwards, /usr/lib64/ghc-8.10.7/bin/ghc-pkg works fine.
So it seems some scriptlet does something which makes ghc-pgk work again.

execsnoop shows:
...
ghc-pkg-8.10.7   3790103 3790102   0 /usr/bin/ghc-pkg-8.10.7 recache --no-user-package-db
ghc-pkg          3790103 3790102   0 /usr/lib64/ghc-8.10.7/bin/ghc-pkg --global-package-db /usr/lib64/ghc-8.10.7/package.conf.d recache --no-user-package-db
sh               3790104 3790000   0 /bin/sh /var/tmp/rpm-tmp.t4raco 0
env              3790105 3790104   0 /bin/env -C /usr/share/doc/ghc8.10/html/libraries ./gen_contents_index
gen_contents_in  3790105 3790104   0 ./gen_contents_index
ls               3790106 3790105   0 /bin/ls Cabal-3.2.1.0/Cabal.haddock array-0.5.4.0/array.haddock base-4.14.3.0/base.haddock binary-0.8.8.0/binary.haddock bytestring-0.10.12.0/bytestring.haddock containers-0.6.5.1/containers.haddock deepseq-1.4.4.0/deepseq.haddock directory-1.3.6.0/directory.haddock exceptions-0.10.4/exceptions.haddock filepath-1.4.2.1/filepath.haddock ghc-8.10.7/ghc.haddock ghc-boot-8.10.7/ghc-boot.haddock ghc-boot-th-8.10.7/ghc-boot-th.haddock ghc-compact-0.1.0.0/ghc-compact.haddock ghc-heap-8.10.7/ghc-heap.haddock ghc-prim-0.6.1/ghc-prim.haddock ghci-8.10.7/ghci.haddock haskeline-0.8.2/haskeline.haddock hpc-0.6.1.0/hpc.haddock 
grep             3790109 3790107   0 /bin/grep -v /ghc\.haddock
sort             3790110 3790107   0 /bin/sort
ls               3790108 3790107   0 /bin/ls -1 Cabal-3.2.1.0/Cabal.haddock array-0.5.4.0/array.haddock base-4.14.3.0/base.haddock binary-0.8.8.0/binary.haddock bytestring-0.10.12.0/bytestring.haddock containers-0.6.5.1/containers.haddock deepseq-1.4.4.0/deepseq.haddock directory-1.3.6.0/directory.haddock exceptions-0.10.4/exceptions.haddock filepath-1.4.2.1/filepath.haddock ghc-8.10.7/ghc.haddock ghc-boot-8.10.7/ghc-boot.haddock ghc-boot-th-8.10.7/ghc-boot-th.haddock ghc-compact-0.1.0.0/ghc-compact.haddock ghc-heap-8.10.7/ghc-heap.haddock ghc-prim-0.6.1/ghc-prim.haddock ghci-8.10.7/ghci.haddock haskeline-0.8.2/haskeline.haddock 
sed              3790113 3790111   0 /bin/sed s#/.*##
sed              3790116 3790114   0 /bin/sed s#/.*##
sed              3790119 3790117   0 /bin/sed s#/.*##
sed              3790122 3790120   0 /bin/sed s#/.*##
sed              3790125 3790123   0 /bin/sed s#/.*##
sed              3790128 3790126   0 /bin/sed s#/.*##
sed              3790131 3790129   0 /bin/sed s#/.*##
sed              3790134 3790132   0 /bin/sed s#/.*##
sed              3790137 3790135   0 /bin/sed s#/.*##
sed              3790140 3790138   0 /bin/sed s#/.*##
sed              3790143 3790141   0 /bin/sed s#/.*##
sed              3790146 3790144   0 /bin/sed s#/.*##
sed              3790149 3790147   0 /bin/sed s#/.*##
sed              3790152 3790150   0 /bin/sed s#/.*##
sed              3790155 3790153   0 /bin/sed s#/.*##
sed              3790158 3790156   0 /bin/sed s#/.*##
sed              3790161 3790159   0 /bin/sed s#/.*##
sed              3790164 3790162   0 /bin/sed s#/.*##
sed              3790167 3790165   0 /bin/sed s#/.*##
sed              3790170 3790168   0 /bin/sed s#/.*##
sed              3790173 3790171   0 /bin/sed s#/.*##
sed              3790176 3790174   0 /bin/sed s#/.*##
sed              3790179 3790177   0 /bin/sed s#/.*##
sed              3790182 3790180   0 /bin/sed s#/.*##
sed              3790185 3790183   0 /bin/sed s#/.*##
sed              3790188 3790186   0 /bin/sed s#/.*##
sed              3790191 3790189   0 /bin/sed s#/.*##
sed              3790194 3790192   0 /bin/sed s#/.*##
sed              3790197 3790195   0 /bin/sed s#/.*##
sed              3790200 3790198   0 /bin/sed s#/.*##
sed              3790203 3790201   0 /bin/sed s#/.*##
sed              3790206 3790204   0 /bin/sed s#/.*##
haddock          3790207 3790105   0 /usr/bin/haddock --gen-index --gen-contents -o . -t Haskell Hierarchical Libraries -p prologue.txt --read-interface=Cabal-3.2.1.0,Cabal-3.2.1.0/Cabal.haddock --read-interface=array-0.5.4.0,array-0.5.4.0/array.haddock --read-interface=base-4.14.3.0,base-4.14.3.0/base.haddock --read-interface=binary-0.8.8.0,binary-0.8.8.0/binary.haddock --read-interface=bytestring-0.10.12.0,bytestring-0.10.12.0/bytestring.haddock --read-interface=containers-0.6.5.1,containers-0.6.5.1/containers.haddock --read-interface=deepseq-1.4.4.0,deepseq-1.4.4.0/deepseq.haddock --read-interface=directory-1.3.6.0,directory-1.3.6.0/directory.haddock --read-interface=exceptions-0.10.4,exceptions-0.10.4/exceptions.haddock --read-interface=filepath-1.4.2.1,filepath-1.4.2.1/filepath.haddock --read-interface=ghc-boot-8.10.7,ghc-boot-8.10.7/ghc-boot.haddock 
haddock          3790207 3790105   0 /usr/lib64/ghc-8.10.7/bin/haddock -B/usr/lib64/ghc-8.10.7 -l/usr/lib64/ghc-8.10.7 --gen-index --gen-contents -o . -t Haskell Hierarchical Libraries -p prologue.txt --read-interface=Cabal-3.2.1.0,Cabal-3.2.1.0/Cabal.haddock --read-interface=array-0.5.4.0,array-0.5.4.0/array.haddock --read-interface=base-4.14.3.0,base-4.14.3.0/base.haddock --read-interface=binary-0.8.8.0,binary-0.8.8.0/binary.haddock --read-interface=bytestring-0.10.12.0,bytestring-0.10.12.0/bytestring.haddock --read-interface=containers-0.6.5.1,containers-0.6.5.1/containers.haddock --read-interface=deepseq-1.4.4.0,deepseq-1.4.4.0/deepseq.haddock --read-interface=directory-1.3.6.0,directory-1.3.6.0/directory.haddock --read-interface=exceptions-0.10.4,exceptions-0.10.4/exceptions.haddock 
...

Comment 5 Zbigniew Jędrzejewski-Szmek 2023-02-01 13:23:34 UTC
Hmm, nvm. You are right, this seems to be a repeat of #1467299.
'sudo strace -f -o/tmp/log -p …' on the dnf process shows:

3801559 execve("/usr/lib64/ghc-8.10.7/bin/ghc-pkg", ["/usr/lib64/ghc-8.10.7/bin/ghc-pk"..., "--global-package-db", "/usr/lib64/ghc-8.10.7/package.co"..., "recache", "--no-user-package-db"], 0x563473a64da0 /* 18 vars */) = 0
3801559 brk(NULL)                       = 0x1b2e000
3801559 arch_prctl(0x3001 /* ARCH_??? */, 0x7ffca6c8aa50) = -1 EINVAL (Invalid argument)
3801559 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1664d1c000
3801559 readlink("/proc/self/exe", 0x7ffca6c89680, 4096) = -1 ENOENT (No such file or directory)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
3801559 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
3801559 openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
3801559 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=7627, ...}, AT_EMPTY_PATH) = 0
3801559 mmap(NULL, 7627, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f1664d1a000
3801559 close(3)                        = 0
3801559 openat(AT_FDCWD, "/lib64/glibc-hwcaps/x86-64-v4/libHSterminfo-0.4.1.4-ghc8.10.7.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
3801559 newfstatat(AT_FDCWD, "/lib64/glibc-hwcaps/x86-64-v4", 0x7ffca6c89c80, 0) = -1 ENOENT (No such file or directory)
3801559 openat(AT_FDCWD, "/lib64/glibc-hwcaps/x86-64-v3/libHSterminfo-0.4.1.4-ghc8.10.7.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
3801559 newfstatat(AT_FDCWD, "/lib64/glibc-hwcaps/x86-64-v3", 0x7ffca6c89c80, 0) = -1 ENOENT (No such file or directory)
3801559 openat(AT_FDCWD, "/lib64/glibc-hwcaps/x86-64-v2/libHSterminfo-0.4.1.4-ghc8.10.7.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
3801559 newfstatat(AT_FDCWD, "/lib64/glibc-hwcaps/x86-64-v2", 0x7ffca6c89c80, 0) = -1 ENOENT (No such file or directory)
3801559 openat(AT_FDCWD, "/lib64/tls/haswell/avx512_1/x86_64/libHSterminfo-0.4.1.4-ghc8.10.7.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
3801559 newfstatat(AT_FDCWD, "/lib64/tls/haswell/avx512_1/x86_64", 0x7ffca6c89c80, 0) = -1 ENOENT (No such file or directory)
3801559 openat(AT_FDCWD, "/lib64/tls/haswell/avx512_1/libHSterminfo-0.4.1.4-ghc8.10.7.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
3801559 newfstatat(AT_FDCWD, "/lib64/tls/haswell/avx512_1", 0x7ffca6c89c80, 0) = -1 ENOENT (No such file or directory)
3801559 openat(AT_FDCWD, "/lib64/tls/haswell/x86_64/libHSterminfo-0.4.1.4-ghc8.10.7.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
3801559 newfstatat(AT_FDCWD, "/lib64/tls/haswell/x86_64", 0x7ffca6c89c80, 0) = -1 ENOENT (No such file or directory)
3801559 openat(AT_FDCWD, "/lib64/tls/haswell/libHSterminfo-0.4.1.4-ghc8.10.7.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
3801559 newfstatat(AT_FDCWD, "/lib64/tls/haswell", 0x7ffca6c89c80, 0) = -1 ENOENT (No such file or directory)
3801559 openat(AT_FDCWD, "/lib64/tls/avx512_1/x86_64/libHSterminfo-0.4.1.4-ghc8.10.7.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
3801559 newfstatat(AT_FDCWD, "/lib64/tls/avx512_1/x86_64", 0x7ffca6c89c80, 0) = -1 ENOENT (No such file or directory)
3801559 openat(AT_FDCWD, "/lib64/tls/avx512_1/libHSterminfo-0.4.1.4-ghc8.10.7.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
3801559 newfstatat(AT_FDCWD, "/lib64/tls/avx512_1", 0x7ffca6c89c80, 0) = -1 ENOENT (No such file or directory)
3801559 openat(AT_FDCWD, "/lib64/tls/x86_64/libHSterminfo-0.4.1.4-ghc8.10.7.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
3801559 newfstatat(AT_FDCWD, "/lib64/tls/x86_64", 0x7ffca6c89c80, 0) = -1 ENOENT (No such file or directory)
3801559 openat(AT_FDCWD, "/lib64/tls/libHSterminfo-0.4.1.4-ghc8.10.7.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
...
3801559 openat(AT_FDCWD, "/usr/lib64/libHSterminfo-0.4.1.4-ghc8.10.7.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
3801559 newfstatat(AT_FDCWD, "/usr/lib64", {st_mode=S_IFDIR|0555, st_size=9536, ...}, 0) = 0
3801559 writev(2, [{iov_base="/usr/lib64/ghc-8.10.7/bin/ghc-pk"..., iov_len=33}, {iov_base=": ", iov_len=2}, {iov_base="error while loading shared libra"..., iov_len=36}, {iov_base=": ", iov_len=2}, {iov_base="libHSterminfo-0.4.1.4-ghc8.10.7."..., iov_len=34}, {iov_base=": ", iov_len=2}, {iov_base="cannot open shared object file", iov_len=30}, {iov_base=": ", iov_len=2}, {iov_base="No such file or directory", iov_len=25}, {iov_base="\n", iov_len=1}], 10) = 167

Oh, and this dnf is invoked in the following way:
/usr/bin/systemd-nspawn -q -M 661de85990a24862b7e886e0caba252c -D /var/lib/mock/fedora-37-x86_64-bootstrap/root -a --capability=cap_ipc_lock --bind=/tmp/mock-resolv.x7l587j0:/etc/resolv.conf --console=pipe --setenv=TERM=vt100 --setenv=SHELL=/bin/bash --setenv=HOME=/var/lib/mock/fedora-37-x86_64/root/installation-homedir --setenv=HOSTNAME=mock --setenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin --setenv=PROMPT_COMMAND=printf "\033]0;<mock-chroot>\007" --setenv=PS1=<mock-chroot> \s-\v\$  --setenv=LANG=C.UTF-8 --setenv=LC_MESSAGES=C.UTF-8 --resolv-conf=off /usr/bin/dnf --installroot /var/lib/mock/fedora-37-x86_64/root/ --releasever 37 --setopt=deltarpm=False --allowerasing --disableplugin=local --disableplugin=spacewalk --disableplugin=versionlock install /builddir/ghc8.10-8.10.7-10.fc38.x86_64.rpm ...

/usr/bin/python3 /usr/bin/dnf --installroot /var/lib/mock/fedora-37-x86_64/root/ --releasever 37 --setopt=deltarpm=False --allowerasing --disableplugin=local --disableplugin=spacewalk --disableplugin=versionlock install /builddir/ghc8.10-8.10.7-10.fc38.x86_64.rpm ...

And dnf apparently does rpm installation by itself, it doesn't invoke rpm, and does not set up /var/lib/mock/fedora-37-x86_64/root/proc/.

I would reassign this to dnf, but I don't know who should set up the chroot — dnf or mock.

Comment 6 Zbigniew Jędrzejewski-Szmek 2023-02-01 13:30:31 UTC
In case this wasn't clear: systemd-nspawn sets up /var/lib/mock/fedora-37-x86_64-bootstrap/root,
but dnf is called with --installroot=/var/lib/mock/fedora-37-x86_64/root/.

Comment 7 Pavel Raiskup 2023-02-02 10:04:20 UTC
Thank you for the detailed analysis.  This is weird, we have
https://github.com/rpm-software-management/mock/blob/bf9b051b7dc74ad70c60165660da7db6a834ee9e/mock/py/mockbuild/package_manager.py#L186
and that should make the /proc mounted.  I'll have a look.

Comment 8 Pavel Raiskup 2023-02-02 10:31:25 UTC
On host there are four dirs:

a) /proc
b) /var/lib/mock/fedora-37-x86_64/root/proc
c) /var/lib/mock/fedora-37-x86_64-bootstrap/root/proc
d) /var/lib/mock/fedora-37-x86_64-bootstrap/root/var/lib/mock/fedora-37-x86_64/root/proc

We seem to bind-mount "a -> b" and "a -> c".  Then we switch
using chroot() to "c" and call DNF there, we expect that "a" -> "d"
mountpoint exists.  It doesn't.  Note that from within the "c" directory,
"d" appears to be "/var/lib/mock/fedora-37-x86_64/root".
Good catch Zbyszek.

The fix from the previous experience won't be trivial.

If you are in hurry, I'd propose disabling the bootstrap chroot in Koji
for the time being (or at least for the Haskell stack).

Comment 9 Jens Petersen 2023-02-03 03:55:22 UTC
Thank you, Pavel

(In reply to Pavel Raiskup from comment #8)
> We seem to bind-mount "a -> b" and "a -> c".  Then we switch
> using chroot() to "c" and call DNF there, we expect that "a" -> "d"
> mountpoint exists.  It doesn't.  Note that from within the "c" directory,
> "d" appears to be "/var/lib/mock/fedora-37-x86_64/root".

I see

> If you are in hurry, I'd propose disabling the bootstrap chroot in Koji
> for the time being (or at least for the Haskell stack).

How to do that?  By patching the koji's mock?

I was in a hurry but I doubt I will be able to start building before branching, let alone finishing.

Comment 10 Jens Petersen 2023-02-03 05:19:19 UTC
I might even suggest that "simple chroot" should actually to removed from copr now to match koji.
The reason I didn't report this issue earlier is really because I could use simple chroot as a workaround in copr.
Of course this may not be practical for all repos I dunno.

Comment 11 Miroslav Suchý 2023-02-03 06:12:08 UTC
(In reply to Jens Petersen from comment #10)
> I might even suggest that "simple chroot" should actually to removed from
> copr now to match koji.

Copr is not intended to match 100% Koji. 

Copr is intended for easy builds, often experimental builds. Quite often with non-standard setup, which can be easily configured with few clicks. And not by admin setup upon ticket request. And this issue shows that "simple chroot" is still sometimes needed.

Comment 12 Pavel Raiskup 2023-02-03 07:50:26 UTC
> How to do that?  By patching the koji's mock?

I believe Koji has some config options.
I that would be a Fedora Infra/Releng task.

I'm quite surprised --chroot=simple works-around this problem.
The mountpoint patterns shouldn't differ, I'd guess it works
by some weird accident.

But then temporarily using --chroot=simple in Koji might be
a less intrusive work-around.

Comment 13 Jens Petersen 2023-02-03 09:02:00 UTC
(In reply to Miroslav Suchý from comment #11)
> (In reply to Jens Petersen from comment #10)
> > I might even suggest that "simple chroot" should actually to removed from
> > copr now to match koji.
> 
> Copr is not intended to match 100% Koji. 

Sure I get it - I simply meant that by forcing wider usage of systemd-nspawn
more problems like this one could be sorted out perhaps for good.
(That may not be realistic as I said, but quite some time passed now since
mock/copr started defaulting to systemd-nspawn.)

(In reply to Pavel Raiskup from comment #12)
> I believe Koji has some config options.
> I that would be a Fedora Infra/Releng task.

Okay I may ask releng for advice.

> I'm quite surprised --chroot=simple works-around this problem.
> The mountpoint patterns shouldn't differ, I'd guess it works
> by some weird accident.

mock simple chroot uses a bootstrap chroot?

Comment 14 Pavel Raiskup 2023-02-03 09:11:22 UTC
Correct, `--chroot=simple` doesn't imply --no-bootstrap-chroot.

Comment 15 Jens Petersen 2023-02-07 15:25:30 UTC
I added ld.so.conf.d conf in ghc-9.2.5-126.fc38 last week which help a little,
but nevertheless other package failed to build apparently due to ghc-pkg
not setting up the ghc package package db correctly:
<https://koji.fedoraproject.org/koji/taskinfo?taskID=97121910>.

However I couldn't reproduce this locally with mock,
so this new failure is still mysterious to me.

Comment 16 Jens Petersen 2023-02-10 16:09:58 UTC
(In reply to Jens Petersen from comment #15)
> I added ld.so.conf.d conf in ghc-9.2.5-126.fc38 last week which help a
> little,
> but nevertheless other package failed to build apparently due to ghc-pkg
> not setting up the ghc package package db correctly:
> <https://koji.fedoraproject.org/koji/taskinfo?taskID=97121910>.

Erm oops, actually this new failure seems due to packaging dependency issue actually...

Comment 17 Jens Petersen 2023-02-10 16:12:19 UTC
(In reply to Jens Petersen from comment #16)
> (In reply to Jens Petersen from comment #15)
> > I added ld.so.conf.d conf in ghc-9.2.5-126.fc38 last week which help a
> > little,
> > but nevertheless other package failed to build apparently due to ghc-pkg
> > not setting up the ghc package package db correctly:
> > <https://koji.fedoraproject.org/koji/taskinfo?taskID=97121910>.
> 
> Erm oops, actually this new failure seems due to packaging dependency issue
> actually...

Well it may be somewhat related since the dependency generation uses ghc-pkg,
but I hope restoring the RUNPATH in ghc may help fix/workaround it.

Comment 18 Pavel Raiskup 2023-02-17 15:38:23 UTC
Draft PR: https://github.com/rpm-software-management/mock/pull/1040

Comment 19 Fedora Update System 2023-05-22 13:40:37 UTC
FEDORA-2023-90a03d8d62 has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-90a03d8d62

Comment 20 Fedora Update System 2023-05-22 13:40:40 UTC
FEDORA-2023-5c8d906435 has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2023-5c8d906435

Comment 21 Fedora Update System 2023-05-22 13:40:43 UTC
FEDORA-EPEL-2023-1652f46b68 has been submitted as an update to Fedora EPEL 8. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2023-1652f46b68

Comment 22 Fedora Update System 2023-05-22 13:40:46 UTC
FEDORA-EPEL-2023-74629814f7 has been submitted as an update to Fedora EPEL 9. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2023-74629814f7

Comment 23 Fedora Update System 2023-05-24 00:49:45 UTC
FEDORA-EPEL-2023-74629814f7 has been pushed to the Fedora EPEL 9 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2023-74629814f7

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 24 Fedora Update System 2023-05-24 01:54:41 UTC
FEDORA-2023-5c8d906435 has been pushed to the Fedora 37 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-5c8d906435`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-5c8d906435

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 25 Fedora Update System 2023-05-24 02:05:14 UTC
FEDORA-EPEL-2023-1652f46b68 has been pushed to the Fedora EPEL 8 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2023-1652f46b68

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 26 Fedora Update System 2023-05-24 14:18:00 UTC
FEDORA-2023-90a03d8d62 has been pushed to the Fedora 38 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-90a03d8d62`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-90a03d8d62

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 27 Fedora Update System 2023-05-27 01:17:50 UTC
FEDORA-2023-90a03d8d62 has been pushed to the Fedora 38 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 28 Jens Petersen 2023-05-28 10:14:52 UTC
Thank you!

At least I am able to build current ghc8.10 with mock-4.0 in my Rawhide toolbox locally now.
(I think that wasn't possible earlier, though I didn't write it explicitly above.)
So once Fedora Koji moves to mock-4.0, I should be able to build it again (and also in copr).

Comment 29 Fedora Update System 2023-06-01 00:47:53 UTC
FEDORA-EPEL-2023-74629814f7 has been pushed to the Fedora EPEL 9 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 30 Fedora Update System 2023-06-01 00:47:57 UTC
FEDORA-EPEL-2023-1652f46b68 has been pushed to the Fedora EPEL 8 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 31 Fedora Update System 2023-06-01 02:16:28 UTC
FEDORA-2023-5c8d906435 has been pushed to the Fedora 37 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.