Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1880499 - rpm-ostree rebase fails when using aarch64 disk images
Summary: rpm-ostree rebase fails when using aarch64 disk images
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: ostree
Version: 32
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Jonathan Lebon
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: RejectedBlocker
Depends On:
Blocks: IoT
TreeView+ depends on / blocked
 
Reported: 2020-09-18 17:03 UTC by Paul Whalen
Modified: 2020-10-23 21:26 UTC (History)
13 users (show)

Fixed In Version: ostree-2020.6-4.fc33
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-09-25 17:03:06 UTC
Type: Bug


Attachments (Terms of Use)
Fedora-IoT-33-20200923.0 (713.96 KB, text/plain)
2020-09-24 02:36 UTC, Paul Whalen
no flags Details

Description Paul Whalen 2020-09-18 17:03:47 UTC
Description of problem:

Attempting to rebase an F32 IoT disk image to F33 reboots back into F32:

[root@rpi3-2 ~]# rpm-ostree rebase fedora/devel/$(uname -m)/iot
..
  libibverbs-31.0-1.fc33.aarch64
  mozjs78-78.2.0-1.fc33.aarch64
  parsec-0.4.0-5.fc33.aarch64
  pciutils-3.6.4-2.fc33.aarch64
  rdma-core-31.0-1.fc33.aarch64
  rtl-sdr-0.6.0-8.fc33.aarch64
  tpm2-pkcs11-1.4.0-1.fc33.aarch64
Run "systemctl reboot" to start a reboot
[root@rpi3-2 ~]# rpm-ostree status
State: idle
Deployments:
  ostree://fedora-iot:fedora/devel/aarch64/iot
                   Version: 33.20200918.0 (2020-09-18T14:33:12Z)
                    Commit: 14ec850c3bc3e369bf21db65795f345a3e8cb6f8c31c7028c62a988f38f018b4
              GPGSignature: Valid signature by 963A2BEB02009608FE67EA4249FD77499570FF31
                      Diff: 329 upgraded, 31 downgraded, 7 removed, 12 added

* ostree://fedora-iot:fedora/stable/aarch64/iot
                   Version: 32.20200916.0 (2020-09-16T09:46:43Z)
                    Commit: 71b75cb76c95aa815113dcebc00f308bbdca053df9d6871e915e6ae0a9a7d328
              GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0

  ostree://fedora-iot:fedora/stable/aarch64/iot
                   Version: 32.20200603.0 (2020-06-03T10:45:43Z)
                    Commit: c02bd26925b4e849fd0e53f3645e97b5cb22f47d7614c5a047d6200c64b3421b
              GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4

After rebooting:

[root@rpi3-2 ~]# rpm-ostree status
State: idle
Deployments:
* ostree://fedora-iot:fedora/stable/aarch64/iot
                   Version: 32.20200916.0 (2020-09-16T09:46:43Z)
                    Commit: 71b75cb76c95aa815113dcebc00f308bbdca053df9d6871e915e6ae0a9a7d328
              GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0

  ostree://fedora-iot:fedora/stable/aarch64/iot
                   Version: 32.20200603.0 (2020-06-03T10:45:43Z)
                    Commit: c02bd26925b4e849fd0e53f3645e97b5cb22f47d7614c5a047d6200c64b3421b
              GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4


[root@rpi3-2 ~]# ls -l /boot

drwx------. 4 root root 16384 Jan  1  1970 efi
drwx------. 2 root root  4096 Jun  3 11:19 grub2
lrwxrwxrwx. 1 root root     8 Sep 17 19:54 loader -> loader.1
drwxr-xr-x. 3 root root  4096 Sep 18 16:34 loader.0
drwxr-xr-x. 3 root root  4096 Sep 17 19:54 loader.1
drwx------. 2 root root 16384 Jun  3 11:17 lost+found
drwxr-xr-x. 5 root root  4096 Sep 18 16:01 ostree

[root@rpi3-2 boot]# cat loader.0/entries/ostree-2-fedora-iot.conf 
title Fedora 33.20200918.0 (IoT Edition Prerelease) (ostree:0)
version 2
options net.ifnames=0 modprobe.blacklist=vc4 root=UUID=b25cda92-50c9-438d-b468-dd7e7737a5df ostree=/ostree/boot.0/fedora-iot/8504d31e0477aa45af19593abfcd03ad807dc45917e675043ef9d71466e4ec54/0
linux /ostree/fedora-iot-8504d31e0477aa45af19593abfcd03ad807dc45917e675043ef9d71466e4ec54/vmlinuz-5.8.6-301.fc33.aarch64
initrd /ostree/fedora-iot-8504d31e0477aa45af19593abfcd03ad807dc45917e675043ef9d71466e4ec54/initramfs-5.8.6-301.fc33.aarch64.img
fdtdir /ostree/fedora-iot-8504d31e0477aa45af19593abfcd03ad807dc45917e675043ef9d71466e4ec54/dtb

Comment 1 Jonathan Lebon 2020-09-18 17:35:26 UTC
After the reboot, can you post the output of `journalctl -u ostree-finalize-staged.service -b -1` ?

Comment 2 Paul Whalen 2020-09-18 18:34:00 UTC
-- Logs begin at Mon 2020-07-27 00:00:01 UTC, end at Fri 2020-09-18 18:30:37 UTC. --
Sep 18 18:26:17 rpi3-2 systemd[1]: Finished OSTree Finalize Staged Deployment.
Sep 18 18:28:03 rpi3-2 systemd[1]: Stopping OSTree Finalize Staged Deployment...
Sep 18 18:28:03 rpi3-2 ostree[5006]: Finalizing staged deployment
Sep 18 18:28:17 rpi3-2 ostree[5006]: Copying /etc changes: 4 modified, 0 removed, 30 added
Sep 18 18:28:17 rpi3-2 ostree[5006]: Copying /etc changes: 4 modified, 0 removed, 30 added
Sep 18 18:28:35 rpi3-2 ostree[5006]: **
Sep 18 18:28:35 rpi3-2 ostree[5006]: OSTree:ERROR:src/libostree/ostree-sysroot-deploy.c:1774:install_deployment_kernel: assertion failed (kernel_layout->bootcsum == bootcsum): ("ec7e4bda983014ad30a8b1684e4bfe78a018b0879bec7c28871c06eec22ea69a" == "0d89fa6172a6382d2d77fda37f7c41d3811888ec38222069d6591a48681d7ced")
Sep 18 18:28:35 rpi3-2 ostree[5006]: Bail out! OSTree:ERROR:src/libostree/ostree-sysroot-deploy.c:1774:install_deployment_kernel: assertion failed (kernel_layout->bootcsum == bootcsum): ("ec7e4bda983014ad30a8b1684e4bfe78a018b0879bec7c28871c06eec22ea69a" == "0d89fa6172a6382d2d77fda37f7c41d3811888ec38222069d6591a48681d7ced")
Sep 18 18:28:35 rpi3-2 systemd[1]: ostree-finalize-staged.service: Control process exited, code=dumped, status=6/ABRT
Sep 18 18:28:35 rpi3-2 systemd[1]: ostree-finalize-staged.service: Failed with result 'core-dump'.
Sep 18 18:28:35 rpi3-2 systemd[1]: Stopped OSTree Finalize Staged Deployment.
Sep 18 18:28:35 rpi3-2 systemd[1]: ostree-finalize-staged.service: Consumed 9.340s CPU time.

Comment 3 Colin Walters 2020-09-18 18:50:17 UTC
OK this is https://github.com/ostreedev/ostree/issues/2154#

Comment 4 Paul Whalen 2020-09-18 20:03:03 UTC
Proposing as a blocker for F33 Beta, this breaks upgrades from F32 IoT on AArch64 SBC's.

Comment 5 Jonathan Lebon 2020-09-18 21:59:50 UTC
@Paul, would you be able to test the RPMs at https://koji.fedoraproject.org/koji/taskinfo?taskID=51770009 (this is from the CI output of https://src.fedoraproject.org/rpms/ostree/pull-request/18) using the instructions at https://src.fedoraproject.org/rpms/ostree/pull-request/18#comment-56631?

Comment 6 Paul Whalen 2020-09-21 14:11:49 UTC
(In reply to Jonathan Lebon from comment #5)
> @Paul, would you be able to test the RPMs at
> https://koji.fedoraproject.org/koji/taskinfo?taskID=51770009 (this is from
> the CI output of https://src.fedoraproject.org/rpms/ostree/pull-request/18)
> using the instructions at
> https://src.fedoraproject.org/rpms/ostree/pull-request/18#comment-56631?

Rebuilt for F32. Booting into the pre-2020.4 deployment, 'cleanup -p, then upgrade && override replace' worked and I was able to upgrade to f33.

[root@rpi3-2 ~]# rpm-ostree status
State: idle
Deployments:
* ostree://fedora-iot:fedora/devel/aarch64/iot
Version: 33.20200918.0 (2020-09-18T14:33:12Z)
Commit: 14ec850c3bc3e369bf21db65795f345a3e8cb6f8c31c7028c62a988f38f018b4
GPGSignature: Valid signature by 963A2BEB02009608FE67EA4249FD77499570FF31

ostree://fedora-iot:fedora/stable/aarch64/iot
Version: 32.20200916.0 (2020-09-16T09:46:43Z)
BaseCommit: 71b75cb76c95aa815113dcebc00f308bbdca053df9d6871e915e6ae0a9a7d328
GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0
ReplacedBasePackages: ostree ostree-libs 2020.6-2.fc32 -> 2020.6-3.fc32

Comment 7 Adam Williamson 2020-09-21 18:57:26 UTC
Discussed at 2020-09-21 blocker review meeting: https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2020-09-21/f33-blocker-review.2020-09-21-16.00.html . We agreed that this seems like a previous release blocker, if anything - it seems like the change needed to fix this would go into F32, not F33, so it does not block F33 composes. We're waiting for an F32 update to be available to make a decision on blocker status.

Comment 8 Jonathan Lebon 2020-09-22 20:53:32 UTC
Bodhi updates:
https://bodhi.fedoraproject.org/updates/FEDORA-2020-d7eaa87aed
https://bodhi.fedoraproject.org/updates/FEDORA-2020-905cb23b8b

(I can't seem to attach the RHBZ to the errata via the Bodhi interface for some reason.)

> We agreed that this seems like a previous release blocker, if anything - it seems like the change needed to fix this would go into F32, not F33, so it does not block F33 composes. We're waiting for an F32 update to be available to make a decision on blocker status.

I think we want this in both f32 and f33. In the former so that IoT users can upgrade to f33, and in the latter so that the bug isn't immediately re-introduced after upgrading and preventing further updates.

Comment 9 František Zatloukal 2020-09-23 06:21:31 UTC
Changing to ON_QA as there are fixes to test.

Comment 10 Paul Whalen 2020-09-24 02:18:00 UTC
The update breaks F33 IoT composes, the aarch64 disk image fails:

..
Deployment starting: fedora/devel/aarch64/iot
Deployment complete: fedora/devel/aarch64/iot
.
Configuring storage
.
Installing boot loader
.
Performing post-installation setup tasks
================================================================================
================================================================================
Error
 
  The following error occurred while installing.  This is a fatal error and
  installation will be aborted.
 
  ostree ['admin', 'instutil', 'set-kargs', 'net.ifnames=0',
  'modprobe.blacklist=vc4', 'root=UUID=4f9cf357-f1cd-43e4-9259-b8d73823bbce']
  exited with code -6


Checking the logs:
 
Sep 24 00:17:05 fedora anaconda[1643]: anaconda: ui.tui.spokes.storage: Partitioning has been applied: ValidationReport(error_messages=[], warning_messages=[])
Sep 24 00:23:44 fedora anaconda[1643]: program: OSTree:ERROR:src/libostree/ostree-sysroot-deploy.c:1681:install_deployment_kernel: assertion failed (kernel_layout->bootcsum == bootcsum): ("c9170badf0790324f14b8569d005d4b0e18daa878a4f890b622cc4cb12f66f83" == "606b2d809bc64043e2b8dfb952f0306500aac7e51cd9b9aeab34e2e68d2f791b")
 
 
Sep 24 00:23:44 fedora anaconda[1643]: program: Bail out! OSTree:ERROR:src/libostree/ostree-sysroot-deploy.c:1681:install_deployment_kernel: assertion failed (kernel_layout->bootcsum == bootcsum): ("c9170badf0790324f14b8569d005d4b0e18daa878a4f890b622cc4cb12f66f83" == "606b2d809bc64043e2b8dfb952f0306500aac7e51cd9b9aeab34e2e68d2f791b")
Sep 24 00:23:44 fedora anaconda[1643]: simpleline: Pushing modal screen IpmiErrorDialog to stack
Sep 24 00:23:44 fedora anaconda[1643]: simpleline: Processing screen ScreenData(IpmiErrorDialog,None,True)
Sep 24 00:23:44 fedora anaconda[1643]: simpleline: Input is required by ScreenData(IpmiErrorDialog,None,True) screen

Comment 11 Paul Whalen 2020-09-24 02:36:23 UTC
Created attachment 1716195 [details]
Fedora-IoT-33-20200923.0

Comment 12 Jonathan Lebon 2020-09-24 13:57:52 UTC
This looks like it could be a mismatch between the OSTree in the install media vs the OSTree in the target commit. Can you verify that they're the same latest one? If not, we'll need to recompose the install media.

Comment 13 Peter Robinson 2020-09-24 14:10:32 UTC
(In reply to Jonathan Lebon from comment #12)
> This looks like it could be a mismatch between the OSTree in the install
> media vs the OSTree in the target commit. Can you verify that they're the
> same latest one? If not, we'll need to recompose the install media.

Should be the same, each compose does both.

Comment 14 Jonathan Lebon 2020-09-24 14:40:12 UTC
I'm looking at this Koji task: https://koji.fedoraproject.org/koji/taskinfo?taskID=52099135. Is this the compose we're referring to?

Sorry, I don't remember too well how all this stuff is wired together, though there I see:

Install Tree: https://kojipkgs.fedoraproject.org/compose/branched/latest-Fedora-33/compose/Everything/aarch64/os/

And querying that tree:

$ dnf repoquery ostree --repofrompath=tmp,https://kojipkgs.fedoraproject.org/compose/branched/latest-Fedora-33/compose/Everything/aarch64/os/ --disablerepo '*' --enablerepo tmp --refresh
Added tmp repo from https://kojipkgs.fedoraproject.org/compose/branched/latest-Fedora-33/compose/Everything/aarch64/os/
Last metadata expiration check: 0:00:34 ago on Thu Sep 24 10:04:43 2020.
ostree-0:2020.5-2.fc33.aarch64

To confirm, I also booted the x86_64 ISO of that same compose:

$ curl -LO https://kojipkgs.fedoraproject.org/compose/branched/latest-Fedora-33/compose/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-33-20200923.n.0.iso

In the shell tmux pane:

$ ostree --version | grep Version
 Version: '2020.5'

Comment 15 Paul Whalen 2020-09-24 14:51:16 UTC
Compose that failed is here:
https://kojipkgs.fedoraproject.org/compose/iot/Fedora-IoT-33-20200923.0/

The "latest" symlinks point to the last successful compose.

Comment 16 Peter Robinson 2020-09-24 14:54:33 UTC
We fail the entire compose if the ostree part of it fails so the installer will always gets the latest there else it'll fail

Comment 17 Jonathan Lebon 2020-09-24 15:53:35 UTC
Looking at the logs again:

> 16:12:20,145 INFO anaconda:program: Bail out! OSTree:ERROR:src/libostree/ostree-sysroot-deploy.c:1681:install_deployment_kernel: assertion failed (kernel_layout->bootcsum == bootcsum): ("c9170badf0790324f14b8569d005d4b0e18daa878a4f890b622cc4cb12f66f83" == "606b2d809bc64043e2b8dfb952f0306500aac7e51cd9b9aeab34e2e68d2f791b")

Are we 100% sure the install media being used *isn't* from https://kojipkgs.fedoraproject.org/compose/branched/latest-Fedora-33/compose/Everything/aarch64/os/?

In that assertion the left side is from the freshly calculated tree, and since this runs in a chroot, we can confirm that it's 2020.6-3. I've confirmed this by pulling down the same commit as well:

$ rpm-ostree db list --repo=. fedora-iot:fedora/devel/aarch64/iot^ | grep ostree
ostree commit: fedora-iot:fedora/devel/aarch64/iot^ (3c66894e8841dd673ef271dbafaf98917e430e5318f7302e1d1aa38b85494575)
 ...
 ostree-2020.6-3.fc33.aarch64
 ostree-libs-2020.6-3.fc33.aarch64
 ...

And reproducing the checksum:

$ cat vmlinuz initramfs.img | sha256sum
c9170badf0790324f14b8569d005d4b0e18daa878a4f890b622cc4cb12f66f83  -

The right side is the one that was calculated from the ostree of the install image. And the only possible way it could've gotten that checksum is if it included the device tree:

$ find dtb/ -type f | sort | xargs cat > all-dtbs
$ cat vmlinuz initramfs.img all-dtbs | sha256sum
606b2d809bc64043e2b8dfb952f0306500aac7e51cd9b9aeab34e2e68d2f791b  -

And the only possible way it could've included dtb/ is if it's *not* 2020.6-3. The code for recursing into dtb/ literally does not exist there.

Comment 18 Geoffrey Marr 2020-09-24 19:27:18 UTC
Discussed during the 2020-09-24 Fedora 33 Go/No-Go meeting: [0]

The decision to classify this bug as a "RejectedBlocker (Beta)" was made as it affects a small group of users, is in a beta, is an edge case, and we expect it to be fixed soon.

[0] https://meetbot-raw.fedoraproject.org/fedora-meeting-1/2020-09-24/f33-beta-go_no_go-meeting.2020-09-24-17.00.txt

Comment 19 Fedora Update System 2020-09-24 22:15:25 UTC
FEDORA-2020-a499a7386f has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2020-a499a7386f

Comment 20 Colin Walters 2020-09-24 22:16:26 UTC
OK backported https://github.com/ostreedev/ostree/pull/2202 to https://bodhi.fedoraproject.org/updates/FEDORA-2020-a499a7386f

Any additional testing for that would be appreciated!  But it's ultimately just removing the assertion, we had to convince ourselves that was safe, and we're pretty sure it is.

Comment 21 Fedora Update System 2020-09-25 17:03:06 UTC
FEDORA-2020-a499a7386f has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 22 Adam Williamson 2020-10-23 21:26:31 UTC
Bug fixed, commonbugs not needed.


Note You need to log in before you can comment on or make changes to this bug.