I am reluctant about adding complete-transaction again, because DNF can not control what happens with the RPMDB until the complete-transaction is run again and faces applying outdated transaction to the system (dependencies have gone missing, packages depending on now duplicate versions might have been installed etc.) Closign this as WONTFIX with a vote, i.e. once enough people are CCed in the bug, we will see about adding this.

Note that the problem of unfinished transaction generally means two different things:

a) transaction was terminated during the install phase and some packages are left to be installed. In this case, running 'dnf upgrade' again will do the right thing and install the remaining packages.

b) transaction was terminated during the erase phase. Some packages are left to be installed. In this case the duplicate versions of packages will be removed on their next upgrade. Alternatively, the autoremove plugin should be eventually capable of removing duplicate, non-installonly packages from the system (bug 963345).

*** Bug 1242135 has been marked as a duplicate of this bug. ***

> a) transaction was terminated during the install phase and some packages
> are left to be installed. In this case, running 'dnf upgrade' again will
> do the right thing and install the remaining packages.

Really?

Running "dnf upgrade" will install the remaining packages... even if the transaction which had failed was a "dnf install foo bar baz"?

If that's the case, then at the very least, please add a message about that (similarly to how yum used to ask us to run yum-complete-transaction), because I (and I suspect others) would have never figured that I could finish an **install** transaction by running a new **upgrade**.

(In reply to Ales Kozumplik from comment #1)
> b) transaction was terminated during the erase phase. Some packages are left
> to be installed. In this case the duplicate versions of packages will be
> removed on their next upgrade. Alternatively, the autoremove plugin should
> be eventually capable of removing duplicate, non-installonly packages from
> the system (bug 963345).

In this case I'm still forced to use package-cleanup or clean by hands because autoremove won't help.

I have been upgrading from F23 -> F24 and my notebook run out of power and switched off. The transaction was not finished.
"package-cleanup --cleandupes" done the job. But ordinary user has very little chance to identify whether he fit in a) or b) and what is the right action to do.

This is a vital feature to recover from crashes during the upgrade process. And of course people will only notice that it is missing when they actually have a messed up system! So user demand is almost certainly much higher than what you can see in this bug.

In the unified DNF 2 world, the accounting needs to be handled at libdnf level so that PackageKit-dnf transactions will also be protected.

I agree that it would be a sensible improvement to have this feature. The issue with completing 'outdated' transactions doesn't seem significant enough to block this feature, to me.

It's really a big problem if you manage to get an incomplete update transaction. There's no very clean way for a user to recover. package-cleanup doesn't really do it; it can probably get your database back to a 'clean' state but it can't do anything about the missed scriptlets (and maybe there are other things dnf does in the missed phases that package-cleanup can't cover, I dunno).

Obviously we could refuse to (or warn hard about) completing a transaction which isn't the most recent transaction. As for non-DNF transactions, well, I don't know enough about the internals to make concrete suggestions, but some thoughts:

1. If we can tell the *time* of the last modification to the RPM database, can't we just check if it's after the aborted transaction failed?

2. For each package in the aborted transaction, we could check if the RPM database reports any EVR other than one that was involved in the aborted transaction, and abort / warn in that case?

The need for this feature is of course correspondingly greater so long as https://bugzilla.redhat.com/show_bug.cgi?id=1382063 and/or https://bugzilla.redhat.com/show_bug.cgi?id=1382224 (or similar) are not implemented.

There is always the possibility of loss of power (or other hardware failure) during dnf operations. So, having complete-transaction is always needed.

Could the DNF team please re-evaluate this, in the light of the comments above? Thanks!

ok, reconsidered. The new plugin may call passively removal of duplicates (bug 1284981) + fix other easily detected problems. The solution will not be 100% bullet proof though.

Thanks! I don't think bulletproof is expected, but hopefully we can provide something in at least some situations which would otherwise be painful.

It's best to take into account what the last, failed, transaction was.

(In reply to Honza Silhan from comment #11)
> ok, reconsidered. The new plugin may call passively removal of duplicates
> (bug 1284981) + fix other easily detected problems. The solution will not be
> 100% bullet proof though.

How would you remove duplicates? When I use 'advised' methods (like the one in #1284981), it doesn't work as it tries to remove lots of packages as 'dependencies'. So, each time I was in such a situation, the only way out of it for me was to 'rpm -e --nodeps OLD_VER_PACKAGES'. I didn't find ANY way to deal with such cases using dnf alone. I agree with Kevin that it should try to finish the last transaction.

So - yesterday during  mass-rebuild upgrade of rawhide I've 'witnessed' again kill of 'dnf' in the middle of upgrade  of  4000 packages  (thankfully in cleanup phase)

But I've been left with 1000 duplicate packages.

It's quite annoying manual work to clean-up this mess left after 'dnf' being aborted.

I'm amazed why there is not at least a SIMPLE reporting tool showing me older packages for removal in year 2017....

It should be even built-in  'rpm' functionality - since  RPM simply HAS to know which of the duplicated package is the file owner - eventually marking packages that needs full reinstall.

Something as simple as    'rpm --resolve-duplicates' should be there for years - rpm DB has ALL the info....

Also note:

dnf  repoquery  --duplicated  --latest-limit -1 

No longer works!  '-1' is not an accepted argument - even though it's even printed in --help....

Last I heard, Lennart Poettering was working on systemd-rpm, which should take care of it. :D

If you on Rawhide you can remove all duplicate packages with command:
# dnf remove --duplicates --setopt=protected_packages=

I too got the same problem when I updated f26 to rawhide last week, I managed to remove many duplicates with 
dnf distro-sync --releasever=rawhide

but now left with sudo and systemd-* duplicate packages. I am not able to find yet a way to remove these duplicate packages.

Anyone knows how to remove duplicates of sudo and systemd* packages?

(In reply to Mikhail from comment #17)
> If you on Rawhide you can remove all duplicate packages with command:
> # dnf remove --duplicates --setopt=protected_packages=

ah I missed to see this comment. I just used this command and it removed sudo and systemd duplicate packages.

Thanks Mikhail.

(In reply to Mikhail from comment #17)
> If you on Rawhide you can remove all duplicate packages with command:
> # dnf remove --duplicates --setopt=protected_packages=

I just tried this on a f26 system which had the transaction interrupted because it couldn't find a package which was on an NFS share.

Now both rpm and dnf have been removed so I'm basically screwed.

Ouch. You should be able to repair that by booting with a rescue CD and using `rpm --root`. 

But: that shouldn't happen anyway, as "--duplicates" should only remove the older versions and leave newer versions intact. There might be a (separate) bug there. :(

(In reply to Matthew Miller from comment #21)
> Ouch. You should be able to repair that by booting with a rescue CD and
> using `rpm --root`. 

That's pretty much the plan but I was trying to fix it remotely, now I have to wait until I can get in front of it. dnf has the --installroot option so that should work I'm guessing. About 500+ packages were removed. Should I try group installing "Workstation" to make sure I have a basic functioning system?

 
> But: that shouldn't happen anyway, as "--duplicates" should only remove the
> older versions and leave newer versions intact. There might be a (separate)
> bug there. :(

Not sure I want to reproduce the problem to verify it :)

Fedora 29 changed to end-of-life (EOL) status on 2019-11-26. Fedora 29 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

This bug was accidentally closed due to a query error. Reopening.

I think this can be closed now?

       "dnf history replay [--ignore-installed] [--ignore-extras] [--skip-unavailable] <filename>
              Replay  a  transaction  stored in file <filename> by History Store Command. The replay will perform
              the exact same operations on the packages as in the original transaction and will  return  with  an
              error  if case of any differences in installed packages or their versions. See also the Transaction
              JSON Format specification of the file format.

              --ignore-installed
                     Don't check for the installed packages being in the same state  as  those  recorded  in  the
                     transaction.  E.g. in case there is an upgrade foo-1.0 -> foo-2.0 stored in the transaction,
                     but there is foo-1.1 installed on the target system.

              --ignore-extras
                     Don't check for extra packages pulled into the transaction on the target  system.  E.g.  the
                     target  system  may  not have some dependency, which was installed on the source system. The
                     replay errors out on this by default, as the transaction would not be the same.

              --skip-unavailable
                     In case some packages stored in the transaction are not available on the target system, skip
                     them instead of erroring out."

No, replaying is not the same as completing an interrupted transaction. As hinted by the description ("and will return with an error if case of any differences in installed packages or their versions"), replaying expects to find the same initial state as before the transaction, whereas completing needs to handle any intermediate states. Sure, you can skip the main sanity check with --ignore-installed, but I doubt that "dnf history replay" will know what to do with an interrupted transaction that left more than one version of the same package installed (or at least, marked as such in the RPM database), which is the main error for whose resolution we need complete-transaction support. (It is the most common case when a transaction gets interrupted in the middle.)

Furthermore, how would I know what the <filename> is? Where does that come from? I started a "dnf upgrade", and for whatever reason there was a crash or a reboot. And I want to clean that up. I didn't have any <filename> to specify, to "dnf upgrade", and I don't have one now.

At least some time ago there were some reports (including from me) sporadic segfaults after dnf downloaded what it was going to install. In all cases the segfault happened before the actual transaction started, but something in the download code was corrupting memory. Haven't happened in a while, so the bug is probably fixed, I guess. But in any case, something blew up, and I just want to restart it and complete what wasn't completed, without knowing where some <filename> is.

(In reply to Kevin Kofler from comment #26)
> No, replaying is not the same as completing an interrupted transaction. As
> hinted by the description ("and will return with an error if case of any
> differences in installed packages or their versions"), replaying expects to
> find the same initial state as before the transaction, whereas completing
> needs to handle any intermediate states. Sure, you can skip the main sanity
> check with --ignore-installed, but I doubt that "dnf history replay" will
> know what to do with an interrupted transaction that left more than one
> version of the same package installed (or at least, marked as such in the
> RPM database), which is the main error for whose resolution we need
> complete-transaction support. (It is the most common case when a transaction
> gets interrupted in the middle.)

You doubt, but haven't tested? ok. I do think it will handle the case, but I also haven't tested it. 

(In reply to Sam Varshavchik from comment #27)
> Furthermore, how would I know what the <filename> is? Where does that come
> from? I started a "dnf upgrade", and for whatever reason there was a crash
> or a reboot. And I want to clean that up. I didn't have any <filename> to
> specify, to "dnf upgrade", and I don't have one now.

You need to create it with 'dnf history store N > filename' specifying the history entry you want to store into a firename.
> 
> At least some time ago there were some reports (including from me) sporadic
> segfaults after dnf downloaded what it was going to install. In all cases
> the segfault happened before the actual transaction started, but something
> in the download code was corrupting memory. Haven't happened in a while, so
> the bug is probably fixed, I guess. But in any case, something blew up, and
> I just want to restart it and complete what wasn't completed, without
> knowing where some <filename> is.

Yyou can do 'dnf history store N > filename' and then replay that, possibly editing the json history file to adjust with what you want to do. 

In any case, likely this will be my last reply for now... I am not a dnf maintainer, I simply found this and thought it might meet the needs requested in this bug. 
If not, oh well.

Well, the replay command clearly was not designed for this use case. If it works at all, it would be purely by accident, and you have to jump through hoops to use it: it should not be necessary to store the history entry to a file and then to replay it, that's what "dnf history redo last" is supposed to be for, but "redo" does not have the flags to skip the sanity checks.

It is still an absolute requirement to have a complete-transaction command designed for this use case and tested (!) with this use case. I do not understand why such an essential requirement gets ignored for more than 6 years, whereas DNF developers get tasked by Red Hat management to work on unnecessary features such as modularity instead.

Oh, and does an interrupted command even show up in the history at all, so you can store and replay it from there? If the history entry is created only after the transaction is complete, then there is nothing you can replay.

I may try setting up a VM to simulate a reboot/power outage but I agree with Kevin, even if it works it's extremely cumbersome and unintuitive.