After the reboot, can you post the output of `journalctl -u ostree-finalize-staged.service -b -1` ?

-- Logs begin at Mon 2020-07-27 00:00:01 UTC, end at Fri 2020-09-18 18:30:37 UTC. --
Sep 18 18:26:17 rpi3-2 systemd[1]: Finished OSTree Finalize Staged Deployment.
Sep 18 18:28:03 rpi3-2 systemd[1]: Stopping OSTree Finalize Staged Deployment...
Sep 18 18:28:03 rpi3-2 ostree[5006]: Finalizing staged deployment
Sep 18 18:28:17 rpi3-2 ostree[5006]: Copying /etc changes: 4 modified, 0 removed, 30 added
Sep 18 18:28:17 rpi3-2 ostree[5006]: Copying /etc changes: 4 modified, 0 removed, 30 added
Sep 18 18:28:35 rpi3-2 ostree[5006]: **
Sep 18 18:28:35 rpi3-2 ostree[5006]: OSTree:ERROR:src/libostree/ostree-sysroot-deploy.c:1774:install_deployment_kernel: assertion failed (kernel_layout->bootcsum == bootcsum): ("ec7e4bda983014ad30a8b1684e4bfe78a018b0879bec7c28871c06eec22ea69a" == "0d89fa6172a6382d2d77fda37f7c41d3811888ec38222069d6591a48681d7ced")
Sep 18 18:28:35 rpi3-2 ostree[5006]: Bail out! OSTree:ERROR:src/libostree/ostree-sysroot-deploy.c:1774:install_deployment_kernel: assertion failed (kernel_layout->bootcsum == bootcsum): ("ec7e4bda983014ad30a8b1684e4bfe78a018b0879bec7c28871c06eec22ea69a" == "0d89fa6172a6382d2d77fda37f7c41d3811888ec38222069d6591a48681d7ced")
Sep 18 18:28:35 rpi3-2 systemd[1]: ostree-finalize-staged.service: Control process exited, code=dumped, status=6/ABRT
Sep 18 18:28:35 rpi3-2 systemd[1]: ostree-finalize-staged.service: Failed with result 'core-dump'.
Sep 18 18:28:35 rpi3-2 systemd[1]: Stopped OSTree Finalize Staged Deployment.
Sep 18 18:28:35 rpi3-2 systemd[1]: ostree-finalize-staged.service: Consumed 9.340s CPU time.

OK this is https://github.com/ostreedev/ostree/issues/2154#

Proposing as a blocker for F33 Beta, this breaks upgrades from F32 IoT on AArch64 SBC's.

@Paul, would you be able to test the RPMs at https://koji.fedoraproject.org/koji/taskinfo?taskID=51770009 (this is from the CI output of https://src.fedoraproject.org/rpms/ostree/pull-request/18) using the instructions at https://src.fedoraproject.org/rpms/ostree/pull-request/18#comment-56631?

(In reply to Jonathan Lebon from comment #5)
> @Paul, would you be able to test the RPMs at
> https://koji.fedoraproject.org/koji/taskinfo?taskID=51770009 (this is from
> the CI output of https://src.fedoraproject.org/rpms/ostree/pull-request/18)
> using the instructions at
> https://src.fedoraproject.org/rpms/ostree/pull-request/18#comment-56631?

Rebuilt for F32. Booting into the pre-2020.4 deployment, 'cleanup -p, then upgrade && override replace' worked and I was able to upgrade to f33.

[root@rpi3-2 ~]# rpm-ostree status
State: idle
Deployments:
* ostree://fedora-iot:fedora/devel/aarch64/iot
Version: 33.20200918.0 (2020-09-18T14:33:12Z)
Commit: 14ec850c3bc3e369bf21db65795f345a3e8cb6f8c31c7028c62a988f38f018b4
GPGSignature: Valid signature by 963A2BEB02009608FE67EA4249FD77499570FF31

ostree://fedora-iot:fedora/stable/aarch64/iot
Version: 32.20200916.0 (2020-09-16T09:46:43Z)
BaseCommit: 71b75cb76c95aa815113dcebc00f308bbdca053df9d6871e915e6ae0a9a7d328
GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0
ReplacedBasePackages: ostree ostree-libs 2020.6-2.fc32 -> 2020.6-3.fc32

Discussed at 2020-09-21 blocker review meeting: https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2020-09-21/f33-blocker-review.2020-09-21-16.00.html . We agreed that this seems like a previous release blocker, if anything - it seems like the change needed to fix this would go into F32, not F33, so it does not block F33 composes. We're waiting for an F32 update to be available to make a decision on blocker status.

Bodhi updates:
https://bodhi.fedoraproject.org/updates/FEDORA-2020-d7eaa87aed
https://bodhi.fedoraproject.org/updates/FEDORA-2020-905cb23b8b

(I can't seem to attach the RHBZ to the errata via the Bodhi interface for some reason.)

> We agreed that this seems like a previous release blocker, if anything - it seems like the change needed to fix this would go into F32, not F33, so it does not block F33 composes. We're waiting for an F32 update to be available to make a decision on blocker status.

I think we want this in both f32 and f33. In the former so that IoT users can upgrade to f33, and in the latter so that the bug isn't immediately re-introduced after upgrading and preventing further updates.

Changing to ON_QA as there are fixes to test.

The update breaks F33 IoT composes, the aarch64 disk image fails:

..
Deployment starting: fedora/devel/aarch64/iot
Deployment complete: fedora/devel/aarch64/iot
.
Configuring storage
.
Installing boot loader
.
Performing post-installation setup tasks
================================================================================
================================================================================
Error
 
  The following error occurred while installing.  This is a fatal error and
  installation will be aborted.
 
  ostree ['admin', 'instutil', 'set-kargs', 'net.ifnames=0',
  'modprobe.blacklist=vc4', 'root=UUID=4f9cf357-f1cd-43e4-9259-b8d73823bbce']
  exited with code -6


Checking the logs:
 
Sep 24 00:17:05 fedora anaconda[1643]: anaconda: ui.tui.spokes.storage: Partitioning has been applied: ValidationReport(error_messages=[], warning_messages=[])
Sep 24 00:23:44 fedora anaconda[1643]: program: OSTree:ERROR:src/libostree/ostree-sysroot-deploy.c:1681:install_deployment_kernel: assertion failed (kernel_layout->bootcsum == bootcsum): ("c9170badf0790324f14b8569d005d4b0e18daa878a4f890b622cc4cb12f66f83" == "606b2d809bc64043e2b8dfb952f0306500aac7e51cd9b9aeab34e2e68d2f791b")
 
 
Sep 24 00:23:44 fedora anaconda[1643]: program: Bail out! OSTree:ERROR:src/libostree/ostree-sysroot-deploy.c:1681:install_deployment_kernel: assertion failed (kernel_layout->bootcsum == bootcsum): ("c9170badf0790324f14b8569d005d4b0e18daa878a4f890b622cc4cb12f66f83" == "606b2d809bc64043e2b8dfb952f0306500aac7e51cd9b9aeab34e2e68d2f791b")
Sep 24 00:23:44 fedora anaconda[1643]: simpleline: Pushing modal screen IpmiErrorDialog to stack
Sep 24 00:23:44 fedora anaconda[1643]: simpleline: Processing screen ScreenData(IpmiErrorDialog,None,True)
Sep 24 00:23:44 fedora anaconda[1643]: simpleline: Input is required by ScreenData(IpmiErrorDialog,None,True) screen

Created attachment 1716195 [details]
Fedora-IoT-33-20200923.0

This looks like it could be a mismatch between the OSTree in the install media vs the OSTree in the target commit. Can you verify that they're the same latest one? If not, we'll need to recompose the install media.

(In reply to Jonathan Lebon from comment #12)
> This looks like it could be a mismatch between the OSTree in the install
> media vs the OSTree in the target commit. Can you verify that they're the
> same latest one? If not, we'll need to recompose the install media.

Should be the same, each compose does both.

I'm looking at this Koji task: https://koji.fedoraproject.org/koji/taskinfo?taskID=52099135. Is this the compose we're referring to?

Sorry, I don't remember too well how all this stuff is wired together, though there I see:

Install Tree: https://kojipkgs.fedoraproject.org/compose/branched/latest-Fedora-33/compose/Everything/aarch64/os/

And querying that tree:

$ dnf repoquery ostree --repofrompath=tmp,https://kojipkgs.fedoraproject.org/compose/branched/latest-Fedora-33/compose/Everything/aarch64/os/ --disablerepo '*' --enablerepo tmp --refresh
Added tmp repo from https://kojipkgs.fedoraproject.org/compose/branched/latest-Fedora-33/compose/Everything/aarch64/os/
Last metadata expiration check: 0:00:34 ago on Thu Sep 24 10:04:43 2020.
ostree-0:2020.5-2.fc33.aarch64

To confirm, I also booted the x86_64 ISO of that same compose:

$ curl -LO https://kojipkgs.fedoraproject.org/compose/branched/latest-Fedora-33/compose/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-33-20200923.n.0.iso

In the shell tmux pane:

$ ostree --version | grep Version
 Version: '2020.5'

Compose that failed is here:
https://kojipkgs.fedoraproject.org/compose/iot/Fedora-IoT-33-20200923.0/

The "latest" symlinks point to the last successful compose.

We fail the entire compose if the ostree part of it fails so the installer will always gets the latest there else it'll fail

Looking at the logs again:

> 16:12:20,145 INFO anaconda:program: Bail out! OSTree:ERROR:src/libostree/ostree-sysroot-deploy.c:1681:install_deployment_kernel: assertion failed (kernel_layout->bootcsum == bootcsum): ("c9170badf0790324f14b8569d005d4b0e18daa878a4f890b622cc4cb12f66f83" == "606b2d809bc64043e2b8dfb952f0306500aac7e51cd9b9aeab34e2e68d2f791b")

Are we 100% sure the install media being used *isn't* from https://kojipkgs.fedoraproject.org/compose/branched/latest-Fedora-33/compose/Everything/aarch64/os/?

In that assertion the left side is from the freshly calculated tree, and since this runs in a chroot, we can confirm that it's 2020.6-3. I've confirmed this by pulling down the same commit as well:

$ rpm-ostree db list --repo=. fedora-iot:fedora/devel/aarch64/iot^ | grep ostree
ostree commit: fedora-iot:fedora/devel/aarch64/iot^ (3c66894e8841dd673ef271dbafaf98917e430e5318f7302e1d1aa38b85494575)
 ...
 ostree-2020.6-3.fc33.aarch64
 ostree-libs-2020.6-3.fc33.aarch64
 ...

And reproducing the checksum:

$ cat vmlinuz initramfs.img | sha256sum
c9170badf0790324f14b8569d005d4b0e18daa878a4f890b622cc4cb12f66f83  -

The right side is the one that was calculated from the ostree of the install image. And the only possible way it could've gotten that checksum is if it included the device tree:

$ find dtb/ -type f | sort | xargs cat > all-dtbs
$ cat vmlinuz initramfs.img all-dtbs | sha256sum
606b2d809bc64043e2b8dfb952f0306500aac7e51cd9b9aeab34e2e68d2f791b  -

And the only possible way it could've included dtb/ is if it's *not* 2020.6-3. The code for recursing into dtb/ literally does not exist there.

Discussed during the 2020-09-24 Fedora 33 Go/No-Go meeting: [0]

The decision to classify this bug as a "RejectedBlocker (Beta)" was made as it affects a small group of users, is in a beta, is an edge case, and we expect it to be fixed soon.

[0] https://meetbot-raw.fedoraproject.org/fedora-meeting-1/2020-09-24/f33-beta-go_no_go-meeting.2020-09-24-17.00.txt

FEDORA-2020-a499a7386f has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2020-a499a7386f

OK backported https://github.com/ostreedev/ostree/pull/2202 to https://bodhi.fedoraproject.org/updates/FEDORA-2020-a499a7386f

Any additional testing for that would be appreciated!  But it's ultimately just removing the assertion, we had to convince ourselves that was safe, and we're pretty sure it is.

FEDORA-2020-a499a7386f has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.

Bug fixed, commonbugs not needed.