I am sorry, but I am not using rpm-ostree. Without further information, I can not see how this is a rpki-client specific issue. So please provide more details.

I am so sorry. It is needed to be rpm-ostree.

> error: Checkout filesystem-3.14-2.fc32.x86_64: opendir(local): No such file or directory

Can you give a bit more information:

1) The output of `rpm-ostree status`

2) The exact command you're running

rpm-ostree status
State: busy
AutomaticUpdates: disabled
Transaction: 
  Initiator: client(id:gnome-software dbus:1.62 unit:gnome-launched-gnome-software-service.desktop-1791.scope uid:1000)
Deployments:
● ostree://fedora:fedora/rawhide/x86_64/silverblue
                   Version: Rawhide.20200507.n.0 (2020-05-07T06:32:47Z)
                BaseCommit: f15c5fb30eaa3028c73e68a99a8bb676b6fd38a2c9355b82f57757aa84a5e13e
              GPGSignature: Valid signature by 963A2BEB02009608FE67EA4249FD77499570FF31
           LayeredPackages: dtv-scan-tables gstreamer1-plugins-ugly-free v4l-utils
             LocalPackages: patchutils-0.3.4-15.fc32.x86_64
###

rpm-ostree upgrade

### 

It seems today that it hangs to loop and busy also do not respond cancel command.

OK that's concerning, it's possible that something broke in rawhide related to this; handling of the `filesystem` package is special in rpm-ostree.

*** Bug 1838841 has been marked as a duplicate of this bug. ***

Seeing this in iot composes as well when trying to add layered packages:

rpm-ostree install wget

Checking out tree 9ed1421... done
Enabled rpm-md repositories: rawhide-modular rawhide fedora-cisco-openh264
rpm-md repo 'rawhide-modular' (cached); generated: 2020-05-17T20:12:17Z
rpm-md repo 'rawhide' (cached); generated: 2020-05-17T21:32:12Z
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2020-03-17T20:11:20Z
Importing rpm-md... done
Resolving dependencies... done
Checking out packages... done
error: Checkout filesystem-3.14-2.fc32.x86_64: opendir(local): No such file or directory

From the logs:

May 22 10:44:17 localhost.localdomain rpm-ostree[1043]: Initiated txn PkgChange for client(id:cli dbus:1.21 unit:session-1.scope uid:1000): /org/projectatomic/rpmostree1/fedora_iot
May 22 10:44:24 localhost.localdomain rpm-ostree[1043]: Librepo version: 1.11.3 with CURL_GLOBAL_ACK_EINTR support (libcurl/7.70.0 OpenSSL/1.1.1g-fips zlib/1.2.11 brotli/1.0.7 libidn2/2.3.0 libpsl/0.21.0 (+libidn2/2.3.0) libssh/0.9.4/openssl/zlib nghttp2/1.40.0)
May 22 10:44:28 localhost.localdomain rpm-ostree[1043]: Preparing pkg txn; enabled repos: ['rawhide-modular', 'rawhide', 'fedora-cisco-openh264'] solvables: 61738
May 22 10:44:28 localhost.localdomain rpm-ostree[1043]: Txn PkgChange on /org/projectatomic/rpmostree1/fedora_iot failed: Checkout filesystem-3.14-2.fc32.x86_64: opendir(local): No such file or directory

OK I think what's going on here probably has something to do with the rpm switch to use the sqlite backend.

Depsolving isn't finding the "base" packages - one can see this with e.g.
`rpm-ostree install --dry-run wget`
and notice that depsolving includes `filesystem` and `bash` etc.

Trying to re-layer `filesystem` failed because it tries to unpack `/usr/local` but that's really `/var/usrlocal` which the unpacking process can't see.
(This is part of how rpm-ostree protects user data, it's very hard even if there are bugs for us to walk off and affect the real /var where e.g. the homedir is)

Going to need to add some debugging prints.

https://github.com/openSUSE/libsolv/pull/386

https://src.fedoraproject.org/rpms/libsolv/c/c6160aa95a361d9ac498e1586b9227e55753314d?branch=master
https://koji.fedoraproject.org/koji/taskinfo?taskID=44965714

Moving to rpm to start a discussion:

Let's use opportunity of switching the backend to sqlite to also move the rpm database to `/usr/lib/sysimage/rpm` across the board?

And also per that PR, add an API to librpm to discover the database location, preferring in order:
 - `/usr/lib/sysimage/rpm`
 - `/usr/share/rpm`
 - `/var/lib/rpm`
?

I'd love if we could stop messing with the global `_dbpath`, which isn't even used by libsolv apparently.

*** Bug 1841206 has been marked as a duplicate of this bug. ***

Proposed as a Blocker for 33-beta by Fedora user lruzicka using the blocker tracking app because:

 The installed system must be able appropriately to install, remove, and update software with the default console tool for the relevant software type (e.g. default console package manager). This includes downloading of packages to be installed/updated.

Colin,
could you provide some details on why do you need the paths?

Many tools use the paths because they detect rpmdb changes,
but maybe we should provide another way as part of librpm API.
The database should remain an implementation detail, a private API.

For rpm-ostree, the database must be in /usr - this has been extensively discussed before, see e.g.
http://lists.rpm.org/pipermail/rpm-maint/2017-October/006681.html

I'd also like to use this opportunity of changing the backend to sqlite to *also* move the rpm database to /usr/lib/sysimage/rpm for all editions/spins.

Ideally as long as librpm supports probing per above, we wouldn't need to know the specific
path.

That said, in order for us to support transitioning from /usr/share/rpm we'd probably need to make it configurable where to place the rpmdb, which might require some support for both configuring and probing the path explicitly.

Oh' no solution yet. And how many years it takes to solve this. What can be workaround. I want to stick to Rawhide and not to rebase. How to force update?

*** Bug 1843320 has been marked as a duplicate of this bug. ***

> What can be workaround. I want to stick to Rawhide and not to rebase. How to force update?

You can use `rpm-ostree override replace` with e.g. the builds from https://koji.fedoraproject.org/koji/buildinfo?buildID=1519658
or temporarily remove your layered packages, upgrade, then re-add them.

Right, that was not helping too much. Can you open it a little bit. I am not that level yet. I would also report workaround to to Fedora discussion 19780 if you do not mind.

Try:

```
$ rpm-ostree override replace https://kojipkgs.fedoraproject.org//packages/libsolv/0.7.14/2.fc33/x86_64/libsolv-0.7.14-2.fc33.x86_64.rpm
$ reboot
---
$ rpm-ostree upgrade
```

All right, I was dummy. I needed to understand to point correct version of it. Forgive me I am not any-more young and dynamic also I hear that I am not also flexible any more. However it does not allow replace base pkg and error die command, and do not give wanted result. Is there option -f to brutally force to replace base packages.

To rebase from 32 to Rawhide is necessary to override replace too?

(In reply to Colin Walters from comment #10)
> https://src.fedoraproject.org/rpms/libsolv/c/
> c6160aa95a361d9ac498e1586b9227e55753314d?branch=master
> https://koji.fedoraproject.org/koji/taskinfo?taskID=44965714

Did you actually talk to either Igor or myself (the libsolv package maintainers) before doing this? I was pretty surprised to see this change committed and pushed like this...

(In reply to Colin Walters from comment #11)
> Moving to rpm to start a discussion:
> 
> Let's use opportunity of switching the backend to sqlite to also move the
> rpm database to `/usr/lib/sysimage/rpm` across the board?
> 
> And also per that PR, add an API to librpm to discover the database
> location, preferring in order:
>  - `/usr/lib/sysimage/rpm`
>  - `/usr/share/rpm`
>  - `/var/lib/rpm`
> ?
> 
> I'd love if we could stop messing with the global `_dbpath`, which isn't
> even used by libsolv apparently.

If we're even talking about rpmdb relocation, I don't want to bother with /usr/share/rpm at all. If you want to move the rpmdb, make change proposal for it. System-wide changes for Fedora 33 are still able to be submitted, so go ahead and do it. I certainly won't object to moving it to /usr/lib/sysimage/rpm.

(In reply to Colin Walters from comment #11)
> Moving to rpm to start a discussion:
> 
> Let's use opportunity of switching the backend to sqlite to also move the
> rpm database to `/usr/lib/sysimage/rpm` across the board?

No, absolutely not. 

The move to sqlite is hard enough as it is, our infrastructure *still* isn't running with sqlite enabled because of buildsystem foo and datacenter bar and builder this and that.

> I'd love if we could stop messing with the global `_dbpath`, which isn't
> even used by libsolv apparently.

Setting %_dbpath to the location of the rpmdb is not "messing", it's the documented way of telling librpm where your rpmdb is.

How would the move make it more difficult?

There's something for you to think about.

OMG, it was about month ago i report and no solution for this. Oh' and no workaround. How it can be? Our systems may fall down, and no updates for it. This is not fair.

A side-track of a rpm-ostree bug is not the place to discuss fundamental rpm changes.

Sure, filed as https://bugzilla.redhat.com/show_bug.cgi?id=1847619

So, this bug is still proposed as a blocker for F33 Beta, but I'm confused as to the current status at this point. Is there still a serious problem with ostree here, Lukas?

Yes there is. No updates in five weeks now. No work around what work. Still expecting instructions what work.

This should have been fixed in libsolv as of https://src.fedoraproject.org/rpms/libsolv/c/c6160aa95a361d9ac498e1586b9227e55753314d?branch=master

I just tested the latest rpm-ostree+libsolv in 

$ rpm-ostree status -b
State: idle
BootedDeployment:
● ostree://fedora:fedora/rawhide/x86_64/silverblue
                   Version: Rawhide.20200622.n.0 (2020-06-22T06:03:56Z)
                    Commit: d44b3db4d47b448adb6a19bb5cfe4f58de02e4d3960829ef695e3f3288f9630f
              GPGSignature: Valid signature by 963A2BEB02009608FE67EA4249FD77499570FF31
$

and things look fine.

I am stuck version what was in use when this bug appears. How I do update?

● ostree://fedora:fedora/rawhide/x86_64/silverblue
                   Version: Rawhide.20200507.n.0 (2020-05-07T06:32:47Z)

> I am stuck version what was in use when this bug appears. How I do update?

Probably the easiest thing to do is:

- `rpm-ostree reset` (this will remove all your overlays and overrides)
- reboot
- `rpm-ostree upgrade`
- reboot
- `rpm-ostree install` back whatever overlays you want

All right. I got Version: Rawhide.20200623.n.0 (2020-06-23T06:05:59Z) but yea it does any-more find Mobile Broadband M2 device. And need to stick Rawhide.20200507.n.0 (2020-05-07T06:32:47Z) ancient version. I do not know guys. How it can be so difficult to maintain simple OS? The main issue with any Linux is that information is so distributed. You need to read tons of pages to have simple solution for some issue. I like so much Silverblue as it has way to go earlier version. It was always such a disaster with some earlier version to keep it running. Here most of the people have mobile broadband and you out of game fast w/ connection where Ethernet works always.

If I understand it correctly, this got fixed in F33 but not in F32 (see bug 1876194). So I'm removing the F33 proposed blocker.

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days