Created attachment 1055095 [details]
fedup.log

Confirmed that this also occurs with F21.

Short answer: fedup is abandoned. Docs/criteria/tests/etc will need to be adapted to the New Upgrade Method.. once we figure out exactly what that is.

(dnf-plugin-fedup is the current prototype but it might get integrated into upstream DNF; we should have a clearer plan for it in a couple of days.)

(In reply to Adam Williamson from comment #0)
> Testing upgrade from F21 to F23 (I don't think the source release matters in
> this case), after downloading the updates and rebooting, the system crashes
> early in boot of the upgrade environment. A couple of suspicious errors are
> visible:
> 
> systemd[1]: [/etc/systemd/system/system-upgrade-shell.service:8] Couldn't
> unescape assignment, ignoring: TERM=linux PS1=system-update:\w\$\x20
> systemd[1]: Caught <ABRT>, dumped core as pid 132.
> systemd:[1]: Freezing execution.
> 
> and that's about it.

Hey look, a systemd crash.

So here's the deal: fedup's boot mechanism is using systemd in an unsupported (and, honestly, inherently broken) way, which inevitably causes these kind of problems.

The systemd maintainers have explicitly rejected supporting it[1], and I completely agree with their reasoning. It's just not fixable.

This is the main reason fedup is deprecated and abandoned for F23 and onward.

[1] For reference, here's the discussion in systemd-devel:
  http://lists.freedesktop.org/archives/systemd-devel/2015-March/029030.html
  http://lists.freedesktop.org/archives/systemd-devel/2015-April/031013.html

> Per https://fedorahosted.org/fesco/ticket/1463 there seem to be moves afoot
> to replace fedup with a dnf plugin, but this seems terribly late for F23.

There's really no discussion to be had, here. fedup is unfixably broken; upstream can't (and won't) support it. There's no choice but to abandon and replace it.

I *have* been telling people this since May:

  ​https://lists.fedoraproject.org/pipermail/devel/2015-May/210905.html

I also wrote a prototype replacement, and I've been working to get it integrated into upstream DNF. systemd upstream is also helping ensure this works:

  http://lists.freedesktop.org/archives/systemd-devel/2015-July/033605.html

As far as I'm concerned, that's the way forward.

> Per all current policies / procedures / etc, fedup is the official upgrade
> mechanism and still blocks the release, so nominating this as a Beta blocker:

Yeah, that's going to need to be changed. Fedup should not be the official upgrade mechanism. I'm sorry, but it's just not supportable. Consider it abandoned by upstream.

"telling people this since May" is fine, but there's a reason we have a Change process. QA looks at the Changes to know what to test. docs looks at the Changes to know what to document. Users look at the Changes to know what's coming. The reason we have a process for that is that experience tells us it doesn't work very well to try and plan our product releases by remembering everything anyone sent to any mailing list we have.

We can deal with testing a new mechanism, but it has to *be* there in some sense. A proof of concept in a github repo is a long way from a thing that's integrated into our product and properly communicated to our users, and we need to get from point A to point B in three months, and I'd be more confident about us achieving that if we were using the process that was put in place to do it.

Okay: https://fedoraproject.org/wiki/Changes/DNF_System_Upgrades

Discussed at 2015-07-27 blocker review meeting: http://meetbot-raw.fedoraproject.org/fedora-blocker-review/2015-07-27/f23-blocker-review.2015-07-27-16.00.log.txt . Accepted as a Beta blocker. As things stand, fedup is still the 'recommended upgrade mechanism' for upgrades to Fedora 23. If it stops being that, this particular bug will no longer merit blocker status (but whatever replaces it as the 'recommended upgrade mechanism' will have to work).

As the Change to DNF upgrades was accepted by FESCo and is well underway, I'm re-proposing this for discussion, with the expectation it will be rendered not-a-blocker. I've been updating wiki pages and stuff, so the Upgrading page now lists DNF as the 'supported' method for upgrades to F23.

FWIW, the code to build upgrade.img has been removed from lorax upstream, so after the next F23 build of lorax it won't even be possible to use fedup for upgrades to F23.

(See https://github.com/rhinstaller/lorax/commit/d9c23d1)

*** Bug 1260319 has been marked as a duplicate of this bug. ***

As fedup does not work from F22 to F23, and fedup's only task is to upgrade the system (and it fails at that), shouldn't it be removed from the F22 repository?

The new tool obsoletes it. Things are not removed from the frozen release repository, ever, by policy (except maybe in case of some sort of terrible legal issue).

Right now the packages are in updates-testing, so:

[wwoods@metroid ~]$ sudo dnf --enablerepo=updates-testing update
Last metadata expiration check performed 1:32:45 ago on Tue Sep  8 11:08:47 2015.
Dependencies resolved.
================================================================================
 Package                      Arch   Version              Repository       Size
================================================================================
Installing:
 dnf-plugin-system-upgrade    noarch 0.4.0-1.fc22         updates-testing  41 k
     replacing  fedup.noarch 0.9.2-1.fc22
[etc]

So, yes, fedup will be removed *from your system* once dnf-plugin-system-upgrade is installed.

I plan to officially retire the fedup and fedup-dracut packages from the distribution once dnf-plugin-system-upgrade makes it to the main repos.

Why there is not some fedup compatibility layer? IOW fedup was greatly marketed, so why not keep using this command whatever runs in background?

[adamw@adam openqa_fedora_tools (dnf-update)]$ rpm -qf /usr/bin/fedup
dnf-plugin-system-upgrade-0.4.0-1.fc23.noarch

(In reply to awilliam from comment #14)
> [adamw@adam openqa_fedora_tools (dnf-update)]$ rpm -qf /usr/bin/fedup
> dnf-plugin-system-upgrade-0.4.0-1.fc23.noarch

Ah, ok ... no worries then ;)

Discussed at 2015-09-10 blocker review meeting: https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2015-09-10/f23-blocker-review.2015-09-10-16.00.log.txt . This is now demoted from blocker status, as fedup is no longer the 'supported' method for upgrades to Fedora 23, per https://fedoraproject.org/wiki/Changes/DNF_System_Upgrades and https://fedoraproject.org/wiki/Upgrading (which I've updated to reflect the accepted Change).

wwoods may wish to close the bug, we'd have no objection to him doing so.

So last night I tried to use fedup to upgrade from F22-F23 and ended up stumbling upon this very bug.

If we are going to present something new to our users for F22->F23, shouldn't fedup help the user upgrade even if it doesn't work anymore?

It could just ask you to install the dnf plugin and run dnf system-upgrade (I understand that rework fedup as a wrap of the new dnf functionality could be a bit of a challenge as the semantics of the command line are different).

At the very least it should show you a warning that fedup is deprecated and where to go from there so that the user can still upgrade which is why the run fedup in the first place.

As is, users unaware of fedup being deprecated will end up in users wasting quite a bit of time (namely hours) downloading all the upgrades and not realizing that the thing will not work until they reboot, with no obvious way to know how to go ahead.

(In reply to Alberto Ruiz from comment #17)
> So last night I tried to use fedup to upgrade from F22-F23 and ended up
> stumbling upon this very bug.
> 
> If we are going to present something new to our users for F22->F23,
> shouldn't fedup help the user upgrade even if it doesn't work anymore?

The replacement package was just pushed to the stable repos today.
So if you update your system after today, you get the new package:

  ========================================================================
   Package                           Arch   Version        Repository Size
  ========================================================================
  Installing:
   dnf-plugin-system-upgrade         noarch 0.4.1-1.fc22   updates    44 k
       replacing  fedup.noarch 0.9.2-1.fc22
   python2-dnf-plugin-system-upgrade noarch 0.4.1-1.fc22   updates    25 k

> It could just ask you to install the dnf plugin and run dnf system-upgrade
> (I understand that rework fedup as a wrap of the new dnf functionality could
> be a bit of a challenge as the semantics of the command line are different).
> 
> At the very least it should show you a warning that fedup is deprecated and
> where to go from there so that the user can still upgrade which is why the
> run fedup in the first place.

The new package provides a new /usr/bin/fedup which does exactly that:

  [wwoods@metroid ~]$ sudo fedup --network 23
  fedup has been replaced by 'dnf system-upgrade'. Use that instead.
  Redirecting to 'dnf system-upgrade download --releasever 23':
  [etc...]


  [wwoods@metroid ~]$ sudo fedup --network 23 --product workstation
  fedup has been replaced by 'dnf system-upgrade'. Use that instead.
  fedup: warning: '--product' is not used anymore. ignoring.
  Redirecting to 'dnf system-upgrade download --releasever 23':
  [etc...]

> As is, users unaware of fedup being deprecated will end up in users wasting
> quite a bit of time (namely hours) downloading all the upgrades and not
> realizing that the thing will not work until they reboot, with no obvious
> way to know how to go ahead.

Well. The replacement package is now live, so anyone who updates won't be using the old fedup anymore. For anyone who tries using the old fedup, post-Beta builds will not have upgrade.img anymore, so it won't even be possible to hit this bug after that.

(That was *supposed* to happen for Beta, but since this bug was closed nobody noticed that upgrade.img was still getting built. Alas; I'm reopening this bug until upgrade.img is dropped from the repo that fedup tries to use by default.)

lorax-23.18-1.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-15675

*** Bug 1264942 has been marked as a duplicate of this bug. ***

Discussed at 2015-09-22 blocker review meeting: https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2015-09-22/f23-blocker-review.2015-09-22-16.00.html . Rejected as a blocker: we agreed that so long as fedup has been obsoleted in all stable releases, the presence of the upgrade.img's in the trees would be unfortunate but not severe enough to block the release.

Sorry, Will, I just realized we forgot to discuss this as an FE. I'll throw an FE nomination on here now just in case it doesn't get pushed before freeze.

Discussed at 2015-09-28 freeze exception review meeting: https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2015-09-28/f23-blocker-review.2015-09-28-16.01.html . Accepted as a freeze exception issue: dropping upgrade.img images from the tree would avoid people with old fedup installs trying upgrades and wasting time and bandwidth, cannot cleanly be done post-release, and should be a safe fix.

lorax-23.18-1.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.