Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1349721 - Upgrade from Fedora 24 to Fedora Rawhide with dnf-plugin-system-upgrade fails, system stuck in boot loop until booted with enforcing=0
Summary: Upgrade from Fedora 24 to Fedora Rawhide with dnf-plugin-system-upgrade fails...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: selinux-policy
Version: 25
Hardware: All
OS: Linux
urgent
urgent
Target Milestone: ---
Assignee: Lukas Vrabec
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: AcceptedBlocker
: 1398696 (view as bug list)
Depends On:
Blocks: F25BetaBlocker
TreeView+ depends on / blocked
 
Reported: 2016-06-24 05:33 UTC by Adam Williamson
Modified: 2016-12-13 00:31 UTC (History)
10 users (show)

Fixed In Version: selinux-policy-3.13.1-208.fc25
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-08-17 03:04:37 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Adam Williamson 2016-06-24 05:33:00 UTC
To reproduce, just try to upgrade from Fedora 24 to current Rawhide with dnf-plugin-system-upgrade. When you run `dnf system-upgrade reboot`, the system will get stuck in an eternal boot loop - it boots, fails to reach the upgrade target, reboots, tries again, ad infinitum. You can break out by booting with `enforcing=0`, which allows the system to reach the upgrade target and successfully upgrade.

Strangely, this doesn't seem to affect upgrades from F23 to Rawhide, those work fine.

Proposing as a Beta blocker: "For each one of the release-blocking package sets, it must be possible to successfully complete a direct upgrade from fully updated installations of the last two stable Fedora releases with that package set installed."

I saw this when upgrading my desktop to Rawhide, openQA is running into it - see https://openqa.fedoraproject.org/tests/23677 , for e.g. - and nivag reported the same thing today in IRC.

Comment 1 Adam Williamson 2016-06-24 19:32:49 UTC
So a bit more detail here...to debug this, you can copy /usr/lib/systemd/system/dnf-system-upgrade.service to /etc/systemd/system and drop the FailureAction=reboot line from the copy; this will prevent the system going into a boot loop. You can also do `systemctl enable debug-shell.service` , which gets you a shell on tty9.

Armed with that, and systemd.log_level=debug, I can see this:

dnf-system-upgrade.service: ConditionPathExists=/system-update/.dnf-system-upgrade succeeded.
dnf-system-upgrade.service: Failed to load environment files: Permission denied
dnf-system-upgrade.service: Failed to run 'start' task: Permission denied
dnf-system-upgrade.service: Changed dead -> failed
dnf-system-upgrade.service: Job dnf-system-upgrade.service/start finished, result=failed

I can't find any trace of an AVC, but it's certainly an SELinux issue, since it works fine with enforcing=0.

/system-updates/.dnf-system-upgrade itself seems to be the 'EnvironmentFile' that systemd wants to open - it's listed as EnvironmentFile in the unit - and its permissions are:

-rw-r--r--. 1 root.root unconfined_u:object_r:rpm_var_lib_t:s0

It would be systemd itself which would try to read the file, I believe, and which is presumably being denied.

Comment 2 Adam Williamson 2016-06-24 20:39:40 UTC
I checked the F23 case; /system-update/.dnf-system-upgrade has the exact same permissions and label there, so I guess the difference is that in F23 systemd is permitted to read that file but in F24 it is not...

Comment 3 Zbigniew Jędrzejewski-Szmek 2016-07-19 20:40:55 UTC
I think that the solution is the same as in the (unfinished) .autorelabel discussion: we should boot selinux in permissive mode for the upgrade.

Comment 4 Petr Schindler 2016-07-21 09:39:14 UTC
Discussed at 2016-07-20 blocker review meeting: [1]. 

This bug was accepted as Beta blocker: Failing upgrade violates the beta criterion: "For each one of the release-blocking package sets, it must be possible to successfully complete a direct upgrade from fully updated installations of the last two stable Fedora releases with that package set installed."

[1] https://meetbot.fedoraproject.org/fedora-blocker-review/2016-07-20/f25-blocker-review.2016-07-20-16.00.html

Comment 5 Petr Lautrbach 2016-07-21 14:35:36 UTC
The reason why it fails is that systemd running as init_t tries to read EnvironmentFile - /system-update/.dnf-system-upgrade which is labeled with rpm_var_lib_t type and it's not allowed in the policy:


# sesearch -A -s init_t -t rpm_var_lib_t 
Found 2 semantic av rules:
   allow init_t file_type : dir { getattr search open } ; 
   allow init_t file_type : filesystem { unmount getattr } ; 


/system-update/ is symlink to /var/lib/dnf/systemd-upgrade which according to matchpathcon has a correct label:

# matchpathcon /var/lib/dnf/systemd-upgrade/.dnf-system-upgrade
/var/lib/dnf/systemd-upgrade/.dnf-system-upgrade	system_u:object_r:rpm_var_lib_t:s0

So the fix could be to add a policy which allows init_t read rpm_var_lib_t.

You can try the following example local policy module:

# cat > localdnfupgradefix.cil <<EOF
(allow init_t rpm_var_lib_t (file (getattr open read)))
EOF
# semodule -i localdnfupgradefix.cil
# dnf system-upgrade reboot

Comment 6 Lukas Vrabec 2016-07-21 14:53:19 UTC
It make sense. init_t domain was unconfined domain till F23. You can output from F23:
$ sesearch -A -s init_t -t rpm_var_lib_t -c file -p read 
Found 1 semantic av rules:
   allow files_unconfined_type file_type : file { ioctl read write create getattr setattr lock relabelfrom relabelto append unlink link rename execute swapon quotaon mounton execute_no_trans open audit_access } ;


This is reason why it was allowed in >F23. 

From F24, init_t is not unconfined_domain and there is no allow rule for this:

# sesearch -A -s init_t -t rpm_var_lib_t -c file -p read
#

I believe we should add this rule to make it working. 

Petr, 
Thanks for testing.

Comment 7 Jan Kurik 2016-07-26 04:37:27 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 25 development cycle.
Changing version to '25'.

Comment 8 Fedora Update System 2016-08-12 15:57:50 UTC
selinux-policy-3.13.1-208.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-662487f8f1

Comment 9 Fedora Update System 2016-08-17 03:02:58 UTC
selinux-policy-3.13.1-208.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.

Comment 10 Lou Hafer 2016-11-24 16:31:57 UTC
It's still possible to find oneself stuck in this loop using the --datadir option offered by dnf-plugin-system-upgrade. Just upgraded from Fed24 to Fed25 and relocated datadir under /home because it had lots of space. Result was SEL type of home_root_t and failure as described above. The description above is very clear and I was able to use a Fed24 Live CD to boot the system, mount the relevant logical volume, and use chcon to change the SEL type to rpm_var_lib_t. The system upgrade completed with no further issues.

This is a bit of a trap --- there's no hint that using --datadir will get you this sort of grief. From a user perspective the ideal fix would be for dnf-plugin-system-upgrade to force the SEL context it requires, but this might not be ideal from a security standpoint. Next best would be a warning, something like `You've moved the data directory; ensure that the SEL context is rpm_var_lib_t before proceeding with reboot.'

Comment 11 Zbigniew Jędrzejewski-Szmek 2016-12-13 00:31:51 UTC
*** Bug 1398696 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.