Hi, you should set `keepcache=1` in dnf.conf. You don't want to have packages from unsuccessful transaction kept on system everytime. You could have downloaded a package with lot of dependencies then you pressed ctrl+c because you have changed your mind and rather plan to install a more lightweight package providing the same functionality with fewer dependencies. The downloaded packages will not be "ever" used then.

*** This bug has been marked as a duplicate of bug 1218415 ***

I have tried using the keepcache=1 option but that causes all downloaded data to remain available on the disk even after the transaction was completed successfully. This is expected behavior for the keepcache parameter, but the behavior I'm looking for is that all downloaded data remains on disk when the transaction fails (keepcache=1), but automatically gets removed when the transaction succeeds (keepcache=0).

+1
Just had to spend many extra hours re-downloading packages because I control-C dnf when it got stuck at 99%.
And it is doing it again right now, 116 packages downloaded, 00:00 ETA, stuck.
keepcache is not the same thing.

PR: https://github.com/rpm-software-management/dnf/pull/295

*** Bug 1223779 has been marked as a duplicate of this bug. ***

*** Bug 1225349 has been marked as a duplicate of this bug. ***

*** Bug 1228960 has been marked as a duplicate of this bug. ***

*** Bug 1225221 has been marked as a duplicate of this bug. ***

*** Bug 1225942 has been marked as a duplicate of this bug. ***

*** Bug 1225342 has been marked as a duplicate of this bug. ***

last example of tis when dnf failed to complete the download of one package, than it exited and removed all almost 1GB of downloaded files. dnf should not remove the dowqnloaded files unless the transaction is completed successfully!

I've just hit this issue too. I ran
# dnf upgrade
and it downloaded 600MBytes only to say
warning: /var/cache/dnf/x86_64/22/google-chrome/packages/google-chrome-stable-43.0.2357.130-1.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 7fac5991: NOKEY
Error: Public key for google-chrome-stable-43.0.2357.130-1.x86_64.rpm is not installed

At which point all 600MBytes of downloads were deleted.

You can blow away used files after a successful transaction but deleting them after failure is extraordinarily painful. Perhaps you can delete them after the next successful transaction so people have the opportunity to retry with different options?

Neither OS X, Windows nor Debian's apt throw away update downloads just because the update procedure failed so I think dnf is the odd one out...

*** Bug 1243043 has been marked as a duplicate of this bug. ***

Nearly 300GB to be re-downloaded today after failed trx chech due to lack of free space in root filesystem.

Please consider this seriously.

dnf should probably act just like yum does in this area. Plus, we really need --downloadonly.

Brgds,

Ed

I agree that keepcache is not the correct approach for this issue.  It makes no sense to flush downloaded cache after a failed transaction.

Yesterday I had to restart upgrade of 331 packages several times because downloads were stalling (hitting Ctrl-C) and one package was failing a transaction because of a packaging issue (polkit).  It wasn't until the 4th or 5th attempt that I realized that it was downloading EVERY package EVERY time I retried.  Then I spent a 30 minutes researching the issue and updating all my machines with keepcache=1.

I'm a linux user with over 20 years experience, but the average newbie user would not figure out this kind of problem, not easily at least.

The default should be to retain cache until the transaction is successful and only then flush the cache, if keepcache is set to 0.

This is also a bandwidth issue for both clients and servers.

Now I have to ensure that keepcache is configured to 1, and adjust my scripts to run 'dnf clean' after a successful transaction.

If the issue is unwanted packages in the cache after changing your mind (the dependencies scenario from comment #1 above) this is resolved if 'dnf clean' is performed immediately after any successful upgrade/install.  Default to run 'dnf clean' after a successful transaction, unless keepcache=1, never clean after a failed transaction.

Here in China the following behaviour is the norm:

------------------------------
# dnf update 
Last metadata expiration check performed 2:39:53 a[...]
Install    7 Packages
Upgrade  337 Packages

Total download size: 561 M
Is this ok [y/N]: y
Downloading Packages:
(1/344): kernel-4.0.7-300.fc22.x86_64.rpm        73 kB/s |  67 kB     00:00    
(2/344): kernel-devel-4.0.7-300.fc22.x86_64.rpm 150 kB/s | 9.4 MB     01:04    
(3/344): kernel-core-4.0.7-300.fc22.x86_64.rpm  263 kB/s |  19 MB     01:14 
[...]

[MIRROR] autocorr-en-4.4.4.3-3.fc22.noarch.rpm: Curl error (78): Remote file not found for ftp://mirror.rise.ph/fedora/linux//updates/22/x86_64/a/autocorr-en-4.4.4.3-3.fc22.noarch.rpm [RETR response: 550]
[MIRROR] autocorr-en-4.4.4.3-3.fc22.noarch.rpm: Status code: 404 for http://ftp.cuhk.edu.hk/pub/Linux/fedora/updates/22/x86_64/a/autocorr-en-4.4.4.3-3.fc22.noarch.rpm
[FAILED] autocorr-en-4.4.4.3-3.fc22.noarch.rpm: No more mirrors to try - All mirrors were already tried without success
[DRPM] autoconf-2.69-18.fc22_2.69-20.fc22.noarch.drpm: done                    
Error: Error downloading packages:            ] 590 kB/s |  65 MB     07:04 ETA
  Cannot download a/autocorr-en-4.4.4.3-3.fc22.noarch.rpm: All mirrors were tried
------------------------------

It have been a few weeks since update didn't choke on some of the mirrors. At least with yum you could re-run the update until you eventually had all the packages downloaded.

Forgot to mention that "keepcache=1" doesn't solve the [MIRROR] error.

dnf can be happily downloading packages, but when dnf is filling the queue with new packages to download, it will hit the 'maximum connection limit' on good mirrors or mirrors that is not up-to-date and fail.

This is now it looks:
[...]
[SKIPPED] autoconf-2.69-20.fc22.noarch.rpm: Already downloaded 
(15-17/338): bash-4.  5% [=                   ] 591 kB/s |  15 MB     07:07 ETA
[...]
All mirrors were already tried without success
(15-17/338): bash-4.  5% [=                   ] 608 kB/s |  20 MB     06:47 ETA
Error: Error downloading packages:
  Cannot download drpms/bash-4.3.39-1.fc22_4.3.39-3.fc22.x86_64.drpm: All mirrors were tried

*** Bug 1245402 has been marked as a duplicate of this bug. ***

Good news everyone! Looks like this "issue" will be fixed in DNF 1.0.2 http://dnf.readthedocs.org/en/latest/release_notes.html#release-notes :

"1.0.2 Release Notes

When a transaction is not successfully finished, DNF preserves downloaded packages until the next successful transaction even if keepcache option is set to False."

Curious that a link to this issue didn't make the release notes though...

Excellent news.  Now I guess I get to go around removing keepcache from all my servers :(

;)

dnf-1.0.2-3.fc22, hawkey-0.5.9-3.fc22 has been submitted as an update for Fedora 22.
https://admin.fedoraproject.org/updates/dnf-1.0.2-3.fc22,hawkey-0.5.9-3.fc22

Package dnf-1.0.2-3.fc22, hawkey-0.5.9-3.fc22:
* should fix your issue,
* was pushed to the Fedora 22 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing dnf-1.0.2-3.fc22 hawkey-0.5.9-3.fc22'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-12577/dnf-1.0.2-3.fc22,hawkey-0.5.9-3.fc22
then log in and leave karma (feedback).

Interrupted the update and after restarting it SKIPPED the packages that were already cached, as expected.  Much better.

BTW, you should probably update your "Fedora Update System" message to use "dnf update --enablerepo=..." instead of yum.

dnf-1.0.2-3.fc22, hawkey-0.5.9-3.fc22 has been pushed to the Fedora 22 stable repository.  If problems still persist, please make note of it in this bug report.

I'm afraid there is still a problem but it might be appropriate to spin it off into a new ticket. Basically dnf's cache management is far too brutal.

I have been following this problem due to it's relevance for 1048433.

Having installed the updates to dnf I ran dnf-automatic. This now behaves and does not wipe the cache when it exits.

To prove the cache was valid I updated just one package from those downloaded. This worked with the package being called from cache.

Buoyed I tried updating the remaining packages. This did not work. It pulled the first package from cache then proceeded to download the rest.

Checking the log I see that the first run of "dnf update" cleaned the cache on exit.

The above scenario is one used to update low bandwidth or data capped sites. Losing the download is bad news as there are times where picking off updates and applying them is more appropriate than updating the entire system in one session.

I agree with you that in your case, it would make sense to remove just the one installed. On the other hand, if we don't want to force users to regularly run "dnf clean packages" and if we don't want to eat all the users disk space (in the worst case), we need to remove the unneeded downloads at some point. Since it is still very hard to read the user's mind and since we want the code as simple as possible (which is beneficial for you as well as for us), the best moment seems to be the first successful transaction. Do you have a better idea when and how should we recognize that a downloaded package will never be needed any more and thus that we can remove it? I mean, people often try to install something but then they cancel the transaction and don't repeat it later any more, or they don't upgrade the system as often as dnf-automatic downloads the updates etc. If you have an (simple) idea, please, file a new report.

I agree too but I hope we can write this one off to brain-fade.
I run an update regime where I will fix a high risk flaw as soon as it is available but leave others to a planned cycle.
Been running this with yum for years so much so that I'd forgotten the keepcache=1 hack needed in yum.conf. I had been looking for a command line option in dnf and only just seen dnf.conf also has keepcache=true as an option - OOPS.

I believe it's rare a user treis to install a package (and fails) and later changes its mind. What i expect to happen usually is that either: 1) the update is retried and after the update is installed, the rpm from cache can be removed or 2) the update is installed and a NEWER version of the package is installed, and then the older version from cache can be also removed together with the newer version.

Right, users probably do not often change their mind. On the other hand, what should we do with those who do?
Anyway, I think that your use case is very valid. I think that it deserves to be tracked in its own issue report even if we don't have a proper solution so far.

(BTW, I, for example, in order to diagnose a bug, often do "dnf install foo" just to find out what dependencies would "foo" add to my system but I never intend to install "foo". Right, there might be a better way - a combination of repoquery arguments, --assumeno etc., but you can never stop people doing this no matter how wrong it is.)

ok, so let's see if John Trickey will create a ticket for this. Then i'll propose a solution.

(Oh, right, it does not download packages in this case. Ignore me, sorry.)

Hello. How about one of the following two strategies:

Do just like yum does.

        - OR -

1) Any successful transaction should remove just the processed packages.

2) Unlimited update transaction ("dnf update") should in addition remove packages downoaded earlier than N (configurable parameter) time units ago.

Of course, no strategy will be good enough for all people and situations.


Also please read with great care about metadata_expire and metadata_expire_filter in yum's doc. Especially, description of yum's metadata_expire says: "So that if the current metadata downloaded is less than this many seconds old then yum  will not  update  the  metadata against the repository." -- It does not say that yum should redownload older metadata again there are no newer metadata available. It says that yum should not download metadata too frequently, even when newer metadata are possibly available.

It is my opinion that metadata_expire in dnf has completely different meaning than in yum, causing dnf to download the same metadata again and again each time they local copy is older than metadata_expire value, even when the matadata do not change. Example of this repeated downloading of the "fedora" repo metadata by dnf in contrast to how yum behaves.

Brgds,

Ed

(In reply to Eduard Vopicka from comment #33)
> Hello. How about one of the following two strategies:
> 
> Do just like yum does.
> 
>         - OR -
> 
> 1) Any successful transaction should remove just the processed packages.
> 
> 2) Unlimited update transaction ("dnf update") should in addition remove
> packages downoaded earlier than N (configurable parameter) time units ago.
> 
> Of course, no strategy will be good enough for all people and situations.
> 

The latter, especially #1, seems to me to be the correct solution.  If I run 'dnf update foo' the transaction should only apply to cached package foo and its dependences. Although I would prefer to never remove unused cached packages since disk is cheap, and bandwidth is often not, I could live without the implementation of option #2 :)

What i would like to see in any application writing data to disk is this:

Before writing data to disk, check if there's enough available disk space, so that after data is written there's still 10% free disk space left for user (10% shall be change by user, if needed).
If this is not fulfilled, decline to perform the operation, giving user details so that he has to create more free space.

In case of dnf, when there's not enough disk space and cleaning the cache would provide the required disk space, we can ask the user for approval to proceed with disk cache cleanup. (if this is not adding enough free disk spcae, provide user with option to stil lcleanup cache, but make it clear this is not enough).

(In reply to cornel panceac from comment #31)

It is highly unlikely I will raise a ticket which will require a solution other than a bugfix. I was too busy proving the original bug had been fixed to notice I had a configuration error. As soon as I have enough updates to create a test case, I will be able to see if "keepcache=true" works and, unless I find anything else, that's all which worries me about dnf.

If you can see a different problem I suggest you raise the ticket and propose the solution.

I agree with Radek in that the user should not have to worry about cache management but I add the caveat "unless he/she chooses otherwise". As long as it all works as described on the tin, I don't think there's a problem.

Strategy 1 in Eduard's is as far as dnf should go. However such behaviour should be configurable. Strategy 2 would be a step too far but acceptable as an option to "dnf clean ..."

The two strategies mentioned in #33 are:

The first, behave like yum does.

The second, 1) and 2) together. 1) cleans sucessfully processed packages only. And 2) performs delayed general cleanup of zombie downloaded packages. Thus, you do not lose downloaded packages that may be used by the next or repeated dnf command, but with some delay the will also be no zombie packages in download directories and no need for explicit maintenance.

And I expect 1) and 2) to work together in harmony.

And those requiring total control are supported via keep_cache=1.

Ed