Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.
Bug 1319786 - [rfe] Proxy setting per repository
Summary: [rfe] Proxy setting per repository
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: dnf
Version: rawhide
Hardware: Unspecified
OS: Unspecified
medium
unspecified
Target Milestone: ---
Assignee: rpm-software-management
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-03-21 14:17 UTC by Jetchko Jekov
Modified: 2019-04-29 02:15 UTC (History)
8 users (show)

Fixed In Version: dnf-4.2.5-1.fc29
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-04-29 02:15:02 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Jetchko Jekov 2016-03-21 14:17:41 UTC
Description of problem:
Back in the 'yum' days it was possible to set 'global' proxy in /etc/yum.conf and then disable proxy for specific repository by setting proxy=_none_ in the relevant repo file.

Now with dnf this is not possible anymore.
Setting proxy = _none_ in .repo file is ignored and dnf is using proxy set globally in /etc/dnf/dnf.conf.
Setting it to None (proxy = None) as it is suggested as default by  http://dnf.readthedocs.org/en/latest/api_conf.html#dnf.conf.Conf.proxy  gives parsing error:
Repository 'test-repo': Error parsing config: Error parsing "proxy = 'None'": URL must be http, ftp, https, socks5, socks5h, socks4 or socks4a not ""

If I try to set it via config manager to 'None' I am getting this:
dnf config-manager --setopt proxy=None test-repo --save
Config error: Error parsing "proxy = 'None'": URL must be http, ftp, https, socks5, socks5h, socks4 or socks4a not ""

Strange enough if I try the following:
dnf config-manager --setopt proxy=_none_ test-repo --save
This command succeeds and leads to line 'proxy = None' in the repo file, but next run of dnf update for example gives again parsing error:
Repository 'test-repo': Error parsing config: Error parsing "proxy = 'None'": URL must be http, ftp, https, socks5, socks5h, socks4 or socks4a not ""

Version-Release number of selected component (if applicable):
dnf-1.1.7-2.fc23.noarch
dnf-conf-1.1.7-2.fc23.noarch
dnf-plugins-core-0.1.17-1.fc23.noarch

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Alexey Pakseykin 2016-07-04 04:22:41 UTC
I experience the same issue with `dnf`.

Using `yum-deprecated` works with the same repo configuration files:

```
yum-deprecated -d 10 update
Yum command has been deprecated, use dnf instead.
See 'man dnf' and 'man yum2dnf' for more information.

Loading "langpacks" plugin
Adding en_US to language list
Config time: 0.006
Adding en_US to language list
Yum version: 3.4.3
rpmdb time: 0.000
Updating Everything
Building updates object
Setting up Package Sacks
base                                                                                                                                                                                        | 3.8 kB  00:00:00     
jenkins                                                                                                                                                                                     | 2.9 kB  00:00:00     
sonar_qube                                                                                                                                                                                  | 2.5 kB  00:00:00     
updates                                                                                                                                                                                     | 4.7 kB  00:00:00     
pkgsack time: 1.384
up:Obs Init time: 0.173
putting fedora-release-workstation in simple update
...
```

While `dnf` fails to download `repodata/repomd.xml` which is clearly available:

```
dnf -d 10 update                                                                                                                                                                              
timer: config: 4 ms
cachedir: /var/cache/dnf
Loaded plugins: generate_completion_cache, copr, download, Query, noroot, builddep, langpacks, config-manager, system-upgrade, protected_packages, needs-restarting, debuginfo-install, reposync, playground
langpacks: No languages are enabled
Adding en_US to language list
initialized Langpacks plugin
DNF version: 1.1.9
Command: dnf -d 10 update
Installroot: /
Releasever: 24
Base command: upgrade
Extra commands: []
repo: downloading from remote: updates, _Handle: metalnk: None, mlist: None, urls ['http://local-yum-mirrors-role-host/fedora/linux/updates/24/x86_64'].
Cannot download 'http://local-yum-mirrors-role-host/fedora/linux/updates/24/x86_64': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried.
repo: using cache for: sonar_qube
not found deltainfo for: sonar_qube - fc24
not found updateinfo for: sonar_qube - fc24
repo: using cache for: base
not found deltainfo for: base - fc24
not found updateinfo for: base - fc24
repo: using cache for: jenkins
not found deltainfo for: jenkins - fc24
not found updateinfo for: jenkins - fc24
Failed to synchronize cache for repo 'updates', disabling.
timer: sack setup: 1132 ms
sonar_qube: using metadata from Fri Jun 10 02:07:28 2016.
base: using metadata from Wed Jun 15 00:07:53 2016.
jenkins: using metadata from Mon Jun 27 03:12:31 2016.
Last metadata expiration check: 0:11:24 ago on Mon Jul  4 11:49:07 2016.
--> Starting dependency resolution
--> Finished dependency resolution
timer: depsolve: 380 ms
Dependencies resolved.
langpacks: enabled languages are ['en', 'en_US']
Nothing to do.
Completion plugin: Generating completion cache...
Complete!
Cleaning up.
```


Proxy settings per repository is very important in proxied (enerprise) environments where access to repositories is split - some of them are only available via proxy on the Internet and others are only available on internal host which cannot be accessed via proxy.

The workaround with commandline options is also not acceptable when updates are done automatically (e.g. in continuous integration platforms) - they go around standard automated configuration tools and require special treatment.

Comment 2 Alexey Pakseykin 2016-07-04 04:33:10 UTC
Additional info for my last https://bugzilla.redhat.com/show_bug.cgi?id=1319786#c1 comment 1...

The test results are from the latest Fedora 24:

```
uname -r
4.5.7-300.fc24.x86_64
```

Version of packages with `dnf` keyword:

```
rpm -qa | grep dnf
dnf-langpacks-0.15.1-4.fc24.noarch
dnf-plugins-core-0.1.21-2.fc24.noarch
dnf-plugin-system-upgrade-0.7.1-2.fc24.noarch
python-dnf-plugins-extras-common-0.0.12-2.fc24.noarch
python2-dnf-1.1.9-2.fc24.noarch
dnf-langpacks-conf-0.15.1-4.fc24.noarch
python3-dnf-1.1.9-2.fc24.noarch
python3-dnf-plugin-system-upgrade-0.7.1-2.fc24.noarch
python2-dnf-plugin-system-upgrade-0.7.1-2.fc24.noarch
python3-dnf-plugins-core-0.1.21-2.fc24.noarch
dnf-1.1.9-2.fc24.noarch
dnf-conf-1.1.9-2.fc24.noarch
dnf-yum-1.1.9-2.fc24.noarch
python-dnf-langpacks-0.15.1-4.fc24.noarch
python2-dnf-plugins-core-0.1.21-2.fc24.noarch
python3-dnf-langpacks-0.15.1-4.fc24.noarch
python-dnf-plugins-extras-migrate-0.0.12-2.fc24.noarch
```

Version of packages with `yum` keyword:

```
rpm -qa | grep yum
yum-3.4.3-509.fc24.noarch
yum-utils-1.1.31-509.fc24.noarch
yum-langpacks-0.4.5-3.fc24.noarch
yum-metadata-parser-1.1.4-16.fc24.x86_64
dnf-yum-1.1.9-2.fc24.noarch
```

As in the original description, I use `proxy=_none_` configuration option for `updates` repository (`local-yum-mirrors-role-host` - hostname accessible only without proxy):

```
[updates]                                                                       
name=updates - fc24                                                             
baseurl=http://local-yum-mirrors-role-host/fedora/linux/updates/24/x86_64       
# The host must be accessible via LAN (without proxy).                          
proxy=_none_                                                                    
enabled=1                                                                       
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-24-x86_64                     
gpgcheck=1                                                                      
skip_if_unavailable=1                                                           
```

Comment 3 Fedora Admin XMLRPC Client 2016-07-08 09:36:41 UTC
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 4 Alexey Pakseykin 2016-07-15 09:16:29 UTC
Some important observation regarding the behavior when `http_proxy` and `https_proxy` environment variables are set.

I noticed that issue does not reproduce when `sudo` is used (`dnf update` fails while `sudo dnf update` does not).

Again, both both `/etc/dnf/dnf.conf` and `/etc/yum.conf` contain proper and identical proxy settings (`dnf` has to contact YUM repository via proxy by default) while few YUM repositories in `/etc/yum.repos.d/*` specify `proxy=_none_` (overriding default to contact the host with YUM repositories directly avoiding proxy).

In addition to config files, I also have `http_proxy` abd `https_proxy` env variables set in shell by default and pointing to the same proxy.

What happens is that `sudo` does NOT propagate these environment variables by default:

```
env | grep http
http_proxy=http://...
https_proxy=http://...

sudo env | grep http
# NO OUTPUT
```

The original description does not indicate anything about `http_proxy` and `https_proxy` environment variables:
https://bugzilla.redhat.com/show_bug.cgi?id=1319786#c0
While I cannot speak about exact test conditions used by author of the original description, what I can currently see is that `dnf` _CORRECTLY_ handles overwrites of proxy settings in configuration file by `proxy=_none_` setting per individual repository. My tests mentioned in the previous comments were apparently run in shell as root directly (without `sudo` - it is my habit for long running setup work):
https://bugzilla.redhat.com/show_bug.cgi?id=1319786#c1

PROBLEM:
What `dnf` does _NOT_ handle correctly (at least in my latest tests) are those `http_proxy` abd `https_proxy` environment variables. I assume the logic:
(1) first overrides setting per repository by `proxy=_none_`;
(2) then setting per repository is overridden again by environment variables.

So, to demonstrate working and non-working condition, `dnf` can be run in sub-shell (by resetting only env vars without touching any configuration files):

```
# Non-working:
(dnf install gimp)

# Working:
(unset https_proxy http_proxy ; dnf install gimp)
```

Normally (in many tools), configuration file settings are overridden by environment variables and then overridden by command line parameters.
In case of `proxy=_none_` I can argue that such configuration file setting should not be overridden by environment variable setting. Why? Because `proxy=_none_` is not a mode for entire `dnf` runtime, instead, it is configuration date per repository. For example, it is OK to override global proxy settings in `/etc/dnf/dnf.conf` while it is NOT OK to override individual repository settings in `/etc/yum.repos.d/*`.

In fact, `yum-depricated` also works as expected - it never overrides individual repoisotory proxy setting by environment variables.

Comment 5 Jetchko Jekov 2016-07-15 09:53:11 UTC
I am sorry but I cant understand what are you talking about.
I guess your problem (if there is any) warrants another bug.

The problem I reported is not related to environment variables at all.
And it doesn't depend on presence or absence of *_proxy/*_PROXY environment variables. 

I just re-checked the status again in F24
dnf-1.1.9-2.fc24.noarch

$ dnf config-manager --setopt proxy=None fedora --save
Config error: Error parsing "proxy = None": URL must be http, ftp, https, socks5, socks5h, socks4 or socks4a not ""

$ grep proxy /etc/yum.repos.d/fedora.repo
<no output>

$ dnf config-manager --setopt proxy=_none_ fedora --save
$ grep proxy /etc/yum.repos.d/fedora.repo
proxy = None 
 ^^- thats is set at the right place and the end of [fedora] section

$ dnf update
Repository 'fedora': Error parsing config: Error parsing "proxy = None": URL must be http, ftp, https, socks5, socks5h, socks4 or socks4a not ""

Comment 6 Alexey Pakseykin 2016-07-15 13:07:01 UTC
Let's separate the cases...

Before doing this, I'll refer to the beginning of the original description (indent added for visual clarity):

    Back in the 'yum' days it was possible
    to set 'global' proxy in /etc/yum.conf
    and then disable proxy for specific
    repository by setting proxy=_none_ in
    the relevant repo file.

    Now with dnf this is not possible anymore.
    Setting proxy = _none_ in .repo file is
    ignored and dnf is using proxy set
    globally in /etc/dnf/dnf.conf.

    ...

I noticed exactly the same behaviour - I had `proxy=_none_` for specific repo and it was ignored. What we end up with are different additional information.


---


### CASE 1: `dnf config-manager` sets `proxy = None` which `dnf` cannot understand ###

Now, the symptoms in your last comment 5 are actually related to the way `dnf config-manager` updates the configuration:
https://bugzilla.redhat.com/show_bug.cgi?id=1319786#c5
Specifically, it sets `proxy = None` (instead of `proxy=_none_`) which `dnf` itself cannot parse when it reads the YUM repository configuration file.

I'm using the same `dnf` RPM version (list of relevant packages in comment 2 above). And if you try to set `proxy=_none_` manually as it was for `yum` in the past, I guess it will start working (unless CASE 2).

This is an issue, but more of `dnf config-manager` command rather than `dnf` because `proxy=_none_` works. Correct me if I'm wrong.


---


### CASE 2: `http_proxy` and `https_proxy` env var override `proxy=_none_` setting per YUM repository in config files ###

In this case, I don't really focus on how `dnf config-manager` command works (and whether it sets proxy config for specific repository correctly or not). Instead, I'm highlighting that `proxy=_none_` set manually in configuration file for specific repository actually works (in my tests) UNTIL there is also `http_proxy` and `https_proxy` environment variables. These env vars override the proxy settings per repository and `dnf` goes through proxy when he does not.


---


If I'm not mistaken and things in the cases 1 and 2 are the way I re-phrased, then they both have to be addressed. In fact, I only realized root cause of my issue when I made at comment 4.

It could also be split into separate bugs under more specific title.

Comment 7 Jetchko Jekov 2016-07-18 14:47:29 UTC
Sorry for the later reply.

Yes you are right of course ... usually I have proxy set in environment which is same as the proxy set in /etc/dnf/dnf.conf. And I didnt realize that maybe env vars are in play here too :/

I have feeling that env var will override proxy=_none_ even in main conf file but I dont have enough free time to test my suspicions.

but I still think that this should be split in 2 bugs:

1.for dnf config-manager brokenness.(not that this is the only broken thing with dnf config-manager)

2.for broken proxy [override] handling

Comment 8 Fedora End Of Life 2016-11-24 16:10:28 UTC
This message is a reminder that Fedora 23 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 23. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '23'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 23 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 9 Jaroslav Mracek 2017-05-26 08:53:56 UTC
I checked it in the code and it looks like that the problem was solved with new conf parser, that is part of dnf-2.4 released into rawhide and Fedora 26. We also provide testing repository for Fedora 24+ (dnf copr enable rpmsoftwaremanagement/dnf-nightly). If you will experience the problem with dnf-2.4+, please don't hesitate to reopen the bug report.

Comment 10 Eva Mrakova 2018-02-14 11:00:56 UTC
Fedora 27
dnf-2.7.5-2.fc27.noarch

There is still the problem addressed as "CASE 2" in comment 6:
`http_proxy` and `https_proxy` env vars override `proxy=_none_` setting per repo.

Also, there's nothing about 'proxy=_none_' in the dnf documentation.


Actual results:
# dnf --disablerepo=\* --enablerepo=testrepo list testpkg\*
127.0.0.1 - - [14/Feb/2018 05:41:32] "GET /repo/repodata/repomd.xml HTTP/1.1" 200 -
Initiating transfer of repo/repodata/repomd.xml with bandwidth=0 blocksize=1024 sentmax=3069
127.0.0.1 - - [14/Feb/2018 05:41:32] "GET /repo/repodata/14d5bfb573cce2c3a7af84d0227d6ff631fa2ca9148a6798c350b707156111a2-primary.xml.gz HTTP/1.1" 200 -
Initiating transfer of repo/repodata/14d5bfb573cce2c3a7af84d0227d6ff631fa2ca9148a6798c350b707156111a2-primary.xml.gz with bandwidth=0 blocksize=1024 sentmax=716
127.0.0.1 - - [14/Feb/2018 05:41:32] "GET /repo/repodata/cad133d4c796562c30b9b335a37727f6a2ca9f928014fb05c24432907c4b8173-filelists.xml.gz HTTP/1.1" 200 -
Initiating transfer of repo/repodata/cad133d4c796562c30b9b335a37727f6a2ca9f928014fb05c24432907c4b8173-filelists.xml.gz with bandwidth=0 blocksize=1024 sentmax=314
testrepo                                    793 kB/s | 1.0 kB     00:00        
Last metadata expiration check: 0:00:00 ago on Wed 14 Feb 2018 05:41:32 AM EST.
Available Packages
testpkgOrig.noarch                    1.0-1                     testrepo
testpkgOrig.src                       1.0-1                     testrepo
# dnf clean all
5 files removed
# http_proxy='127.0.0.1:3128' dnf --disablerepo=\* --enablerepo=testrepo list testpkg\*
127.0.0.1 - - [14/Feb/2018 05:42:41] "GET /repo/repodata/repomd.xml HTTP/1.1" 200 -
Initiating transfer of repo/repodata/repomd.xml with bandwidth=0 blocksize=1024 sentmax=3069
127.0.0.1 - - [14/Feb/2018 05:42:41] "GET /repo/repodata/af0c3882d302061b0f5ab10d193b6fdb9a1bdfab6523d44d35e004c53e2d56a7-primary.xml.gz HTTP/1.1" 200 -
Initiating transfer of repo/repodata/af0c3882d302061b0f5ab10d193b6fdb9a1bdfab6523d44d35e004c53e2d56a7-primary.xml.gz with bandwidth=0 blocksize=1024 sentmax=719
127.0.0.1 - - [14/Feb/2018 05:42:41] "GET /repo/repodata/bc59709f583b39366b423bf1b664e62cab8fa0d62fdde427ccdc9e663be515c2-filelists.xml.gz HTTP/1.1" 200 -
Initiating transfer of repo/repodata/bc59709f583b39366b423bf1b664e62cab8fa0d62fdde427ccdc9e663be515c2-filelists.xml.gz with bandwidth=0 blocksize=1024 sentmax=317
testrepo                                    413 kB/s | 1.0 kB     00:00        
Last metadata expiration check: 0:00:00 ago on Wed 14 Feb 2018 05:42:41 AM EST.
Available Packages
testpkgProxy.noarch                    1.0-1                    testrepo
testpkgProxy.src                       1.0-1                    testrepo
# cat  /etc/yum.repos.d/test.repo
[testrepo]
name=testrepo
baseurl=http://127.0.0.1/repo
gpgcheck=0
enabled=0
proxy=_none_
#

Note: the results are the same when in the .repo file there is proxy=None or
proxy= (as say the docs)

Expected results:
there should be listed *Orig* packages instead of *Proxy* ones when http_proxy
env var is set

Comment 11 Dave Close 2019-01-04 20:27:50 UTC
Recently upgraded five systems from F28 to F29 via system-upgrade. Now using dnf-4.0.9-2.fc29.noarch. Using a local mirror of the Fedora repos that does not require use of the company proxy. /etc/dnf/dnf.conf specifies the company proxy settings but /etc/yum.repos.d/fedora.repo (and others) specify proxy=_none_. This works correctly when "dnf upgrade" is started from the command line as root. When run from cron as root, it reports, "[MIRROR] NetworkManager-libnm-1.12.6-3.fc29_1.12.6-4.fc29.x86_64.drpm: Curl error (5): Couldn't resolve proxy name for http://(local mirror)/pub/fedora/linux/updates/29/Everything/x86_64/os/drpms/NetworkManager-libnm-1.12.6-3.fc29_1.12.6-4.fc29.x86_64.drpm [Could not resolve proxy: _none_]" and similar errors for other packages. I have tried prefacing the cron command with "http_proxy= https_proxy= no_proxy=" to no avail. I have found no relevant environment differences between cron and the command line invocations. This worked perfectly in F28.

Comment 12 Daniel Mach 2019-03-02 15:21:15 UTC
Related PR: https://github.com/rpm-software-management/dnf/pull/1300

Comment 13 Fedora Update System 2019-04-25 16:43:24 UTC
dnf-4.2.5-1.fc29 libdnf-0.31.0-2.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-2612a121ba

Comment 14 Fedora Update System 2019-04-26 22:09:58 UTC
dnf-4.2.5-1.fc29, libdnf-0.31.0-2.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-2612a121ba

Comment 15 Fedora Update System 2019-04-29 02:15:02 UTC
dnf-4.2.5-1.fc29, libdnf-0.31.0-2.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.