Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1838691 - rpm-ostree update fails in filesystem checkout
Summary: rpm-ostree update fails in filesystem checkout
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: rpm-ostree
Version: rawhide
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: Colin Walters
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1838841 1841206 1843320 (view as bug list)
Depends On:
Blocks: IoT 1876194
TreeView+ depends on / blocked
 
Reported: 2020-05-21 15:19 UTC by Juha Koskiniemi
Modified: 2023-09-14 06:00 UTC (History)
28 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1876194 (view as bug list)
Environment:
Last Closed: 2020-06-22 22:52:40 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Juha Koskiniemi 2020-05-21 15:19:50 UTC
rpm-ostree fails 

error: Checkout filesystem-3.14-2.fc32.x86_64: opendir(local): No such file or directory

Comment 1 Robert Scheck 2020-05-21 15:55:51 UTC
I am sorry, but I am not using rpm-ostree. Without further information, I can not see how this is a rpki-client specific issue. So please provide more details.

Comment 2 Juha Koskiniemi 2020-05-21 20:09:55 UTC
I am so sorry. It is needed to be rpm-ostree.

Comment 3 Colin Walters 2020-05-21 20:12:46 UTC
> error: Checkout filesystem-3.14-2.fc32.x86_64: opendir(local): No such file or directory

Can you give a bit more information:

1) The output of `rpm-ostree status`

2) The exact command you're running

Comment 4 Juha Koskiniemi 2020-05-22 08:05:54 UTC
rpm-ostree status
State: busy
AutomaticUpdates: disabled
Transaction: 
  Initiator: client(id:gnome-software dbus:1.62 unit:gnome-launched-gnome-software-service.desktop-1791.scope uid:1000)
Deployments:
● ostree://fedora:fedora/rawhide/x86_64/silverblue
                   Version: Rawhide.20200507.n.0 (2020-05-07T06:32:47Z)
                BaseCommit: f15c5fb30eaa3028c73e68a99a8bb676b6fd38a2c9355b82f57757aa84a5e13e
              GPGSignature: Valid signature by 963A2BEB02009608FE67EA4249FD77499570FF31
           LayeredPackages: dtv-scan-tables gstreamer1-plugins-ugly-free v4l-utils
             LocalPackages: patchutils-0.3.4-15.fc32.x86_64
###

rpm-ostree upgrade

### 

It seems today that it hangs to loop and busy also do not respond cancel command.

Comment 5 Colin Walters 2020-05-22 14:23:14 UTC
OK that's concerning, it's possible that something broke in rawhide related to this; handling of the `filesystem` package is special in rpm-ostree.

Comment 6 Paul Whalen 2020-05-22 15:38:37 UTC
*** Bug 1838841 has been marked as a duplicate of this bug. ***

Comment 7 Paul Whalen 2020-05-22 15:47:47 UTC
Seeing this in iot composes as well when trying to add layered packages:

rpm-ostree install wget

Checking out tree 9ed1421... done
Enabled rpm-md repositories: rawhide-modular rawhide fedora-cisco-openh264
rpm-md repo 'rawhide-modular' (cached); generated: 2020-05-17T20:12:17Z
rpm-md repo 'rawhide' (cached); generated: 2020-05-17T21:32:12Z
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2020-03-17T20:11:20Z
Importing rpm-md... done
Resolving dependencies... done
Checking out packages... done
error: Checkout filesystem-3.14-2.fc32.x86_64: opendir(local): No such file or directory

From the logs:

May 22 10:44:17 localhost.localdomain rpm-ostree[1043]: Initiated txn PkgChange for client(id:cli dbus:1.21 unit:session-1.scope uid:1000): /org/projectatomic/rpmostree1/fedora_iot
May 22 10:44:24 localhost.localdomain rpm-ostree[1043]: Librepo version: 1.11.3 with CURL_GLOBAL_ACK_EINTR support (libcurl/7.70.0 OpenSSL/1.1.1g-fips zlib/1.2.11 brotli/1.0.7 libidn2/2.3.0 libpsl/0.21.0 (+libidn2/2.3.0) libssh/0.9.4/openssl/zlib nghttp2/1.40.0)
May 22 10:44:28 localhost.localdomain rpm-ostree[1043]: Preparing pkg txn; enabled repos: ['rawhide-modular', 'rawhide', 'fedora-cisco-openh264'] solvables: 61738
May 22 10:44:28 localhost.localdomain rpm-ostree[1043]: Txn PkgChange on /org/projectatomic/rpmostree1/fedora_iot failed: Checkout filesystem-3.14-2.fc32.x86_64: opendir(local): No such file or directory

Comment 8 Colin Walters 2020-05-24 23:37:10 UTC
OK I think what's going on here probably has something to do with the rpm switch to use the sqlite backend.

Depsolving isn't finding the "base" packages - one can see this with e.g.
`rpm-ostree install --dry-run wget`
and notice that depsolving includes `filesystem` and `bash` etc.

Trying to re-layer `filesystem` failed because it tries to unpack `/usr/local` but that's really `/var/usrlocal` which the unpacking process can't see.
(This is part of how rpm-ostree protects user data, it's very hard even if there are bugs for us to walk off and affect the real /var where e.g. the homedir is)

Going to need to add some debugging prints.

Comment 9 Colin Walters 2020-05-25 20:46:34 UTC
https://github.com/openSUSE/libsolv/pull/386

Comment 11 Colin Walters 2020-05-26 13:05:07 UTC
Moving to rpm to start a discussion:

Let's use opportunity of switching the backend to sqlite to also move the rpm database to `/usr/lib/sysimage/rpm` across the board?

And also per that PR, add an API to librpm to discover the database location, preferring in order:
 - `/usr/lib/sysimage/rpm`
 - `/usr/share/rpm`
 - `/var/lib/rpm`
?

I'd love if we could stop messing with the global `_dbpath`, which isn't even used by libsolv apparently.

Comment 12 Lukas Ruzicka 2020-05-28 21:09:47 UTC
*** Bug 1841206 has been marked as a duplicate of this bug. ***

Comment 13 Fedora Blocker Bugs Application 2020-05-28 21:10:45 UTC
Proposed as a Blocker for 33-beta by Fedora user lruzicka using the blocker tracking app because:

 The installed system must be able appropriately to install, remove, and update software with the default console tool for the relevant software type (e.g. default console package manager). This includes downloading of packages to be installed/updated.

Comment 14 Daniel Mach 2020-06-01 11:43:05 UTC
Colin,
could you provide some details on why do you need the paths?

Many tools use the paths because they detect rpmdb changes,
but maybe we should provide another way as part of librpm API.
The database should remain an implementation detail, a private API.

Comment 15 Colin Walters 2020-06-01 14:10:54 UTC
For rpm-ostree, the database must be in /usr - this has been extensively discussed before, see e.g.
http://lists.rpm.org/pipermail/rpm-maint/2017-October/006681.html

I'd also like to use this opportunity of changing the backend to sqlite to *also* move the rpm database to /usr/lib/sysimage/rpm for all editions/spins.

Ideally as long as librpm supports probing per above, we wouldn't need to know the specific
path.

That said, in order for us to support transitioning from /usr/share/rpm we'd probably need to make it configurable where to place the rpmdb, which might require some support for both configuring and probing the path explicitly.

Comment 16 Juha Koskiniemi 2020-06-03 08:21:00 UTC
Oh' no solution yet. And how many years it takes to solve this. What can be workaround. I want to stick to Rawhide and not to rebase. How to force update?

Comment 17 Jonathan Lebon 2020-06-03 13:23:44 UTC
*** Bug 1843320 has been marked as a duplicate of this bug. ***

Comment 18 Colin Walters 2020-06-03 14:07:19 UTC
> What can be workaround. I want to stick to Rawhide and not to rebase. How to force update?

You can use `rpm-ostree override replace` with e.g. the builds from https://koji.fedoraproject.org/koji/buildinfo?buildID=1519658
or temporarily remove your layered packages, upgrade, then re-add them.

Comment 19 Juha Koskiniemi 2020-06-05 15:21:07 UTC
Right, that was not helping too much. Can you open it a little bit. I am not that level yet. I would also report workaround to to Fedora discussion 19780 if you do not mind.

Comment 20 Jonathan Lebon 2020-06-05 19:36:06 UTC
Try:

```
$ rpm-ostree override replace https://kojipkgs.fedoraproject.org//packages/libsolv/0.7.14/2.fc33/x86_64/libsolv-0.7.14-2.fc33.x86_64.rpm
$ reboot
---
$ rpm-ostree upgrade
```

Comment 21 Juha Koskiniemi 2020-06-06 05:07:43 UTC
All right, I was dummy. I needed to understand to point correct version of it. Forgive me I am not any-more young and dynamic also I hear that I am not also flexible any more. However it does not allow replace base pkg and error die command, and do not give wanted result. Is there option -f to brutally force to replace base packages.

Comment 22 thiago.frmoraes 2020-06-06 21:28:12 UTC
To rebase from 32 to Rawhide is necessary to override replace too?

Comment 23 Neal Gompa 2020-06-07 05:44:38 UTC
(In reply to Colin Walters from comment #10)
> https://src.fedoraproject.org/rpms/libsolv/c/
> c6160aa95a361d9ac498e1586b9227e55753314d?branch=master
> https://koji.fedoraproject.org/koji/taskinfo?taskID=44965714

Did you actually talk to either Igor or myself (the libsolv package maintainers) before doing this? I was pretty surprised to see this change committed and pushed like this...

(In reply to Colin Walters from comment #11)
> Moving to rpm to start a discussion:
> 
> Let's use opportunity of switching the backend to sqlite to also move the
> rpm database to `/usr/lib/sysimage/rpm` across the board?
> 
> And also per that PR, add an API to librpm to discover the database
> location, preferring in order:
>  - `/usr/lib/sysimage/rpm`
>  - `/usr/share/rpm`
>  - `/var/lib/rpm`
> ?
> 
> I'd love if we could stop messing with the global `_dbpath`, which isn't
> even used by libsolv apparently.

If we're even talking about rpmdb relocation, I don't want to bother with /usr/share/rpm at all. If you want to move the rpmdb, make change proposal for it. System-wide changes for Fedora 33 are still able to be submitted, so go ahead and do it. I certainly won't object to moving it to /usr/lib/sysimage/rpm.

Comment 24 Panu Matilainen 2020-06-08 07:02:20 UTC
(In reply to Colin Walters from comment #11)
> Moving to rpm to start a discussion:
> 
> Let's use opportunity of switching the backend to sqlite to also move the
> rpm database to `/usr/lib/sysimage/rpm` across the board?

No, absolutely not. 

The move to sqlite is hard enough as it is, our infrastructure *still* isn't running with sqlite enabled because of buildsystem foo and datacenter bar and builder this and that.

> I'd love if we could stop messing with the global `_dbpath`, which isn't
> even used by libsolv apparently.

Setting %_dbpath to the location of the rpmdb is not "messing", it's the documented way of telling librpm where your rpmdb is.

Comment 25 Colin Walters 2020-06-08 12:38:45 UTC
How would the move make it more difficult?

Comment 26 Panu Matilainen 2020-06-09 08:52:12 UTC
There's something for you to think about.

Comment 27 Juha Koskiniemi 2020-06-14 18:01:37 UTC
OMG, it was about month ago i report and no solution for this. Oh' and no workaround. How it can be? Our systems may fall down, and no updates for it. This is not fair.

Comment 28 Panu Matilainen 2020-06-15 05:24:41 UTC
A side-track of a rpm-ostree bug is not the place to discuss fundamental rpm changes.

Comment 29 Colin Walters 2020-06-16 17:15:16 UTC
Sure, filed as https://bugzilla.redhat.com/show_bug.cgi?id=1847619

Comment 30 Adam Williamson 2020-06-22 20:19:21 UTC
So, this bug is still proposed as a blocker for F33 Beta, but I'm confused as to the current status at this point. Is there still a serious problem with ostree here, Lukas?

Comment 31 Juha Koskiniemi 2020-06-22 20:39:07 UTC
Yes there is. No updates in five weeks now. No work around what work. Still expecting instructions what work.

Comment 32 Colin Walters 2020-06-22 22:52:28 UTC
This should have been fixed in libsolv as of https://src.fedoraproject.org/rpms/libsolv/c/c6160aa95a361d9ac498e1586b9227e55753314d?branch=master

I just tested the latest rpm-ostree+libsolv in 

$ rpm-ostree status -b
State: idle
BootedDeployment:
● ostree://fedora:fedora/rawhide/x86_64/silverblue
                   Version: Rawhide.20200622.n.0 (2020-06-22T06:03:56Z)
                    Commit: d44b3db4d47b448adb6a19bb5cfe4f58de02e4d3960829ef695e3f3288f9630f
              GPGSignature: Valid signature by 963A2BEB02009608FE67EA4249FD77499570FF31
$

and things look fine.

Comment 33 Juha Koskiniemi 2020-06-23 06:27:24 UTC
I am stuck version what was in use when this bug appears. How I do update?

● ostree://fedora:fedora/rawhide/x86_64/silverblue
                   Version: Rawhide.20200507.n.0 (2020-05-07T06:32:47Z)

Comment 34 Jonathan Lebon 2020-06-23 13:08:13 UTC
> I am stuck version what was in use when this bug appears. How I do update?

Probably the easiest thing to do is:

- `rpm-ostree reset` (this will remove all your overlays and overrides)
- reboot
- `rpm-ostree upgrade`
- reboot
- `rpm-ostree install` back whatever overlays you want

Comment 35 Juha Koskiniemi 2020-06-23 17:46:41 UTC
All right. I got Version: Rawhide.20200623.n.0 (2020-06-23T06:05:59Z) but yea it does any-more find Mobile Broadband M2 device. And need to stick Rawhide.20200507.n.0 (2020-05-07T06:32:47Z) ancient version. I do not know guys. How it can be so difficult to maintain simple OS? The main issue with any Linux is that information is so distributed. You need to read tons of pages to have simple solution for some issue. I like so much Silverblue as it has way to go earlier version. It was always such a disaster with some earlier version to keep it running. Here most of the people have mobile broadband and you out of game fast w/ connection where Ethernet works always.

Comment 36 Kamil Páral 2020-09-07 14:04:43 UTC
If I understand it correctly, this got fixed in F33 but not in F32 (see bug 1876194). So I'm removing the F33 proposed blocker.

Comment 37 Red Hat Bugzilla 2023-09-14 06:00:59 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days


Note You need to log in before you can comment on or make changes to this bug.