Bug 548269

Thanks for the report! curl was switched to c-ares for name resolving in Fedora 13 and this feature hasn't been implemented in c-ares yet. I'll clone this bug for that package.

> If I explicitly add "--ipv4", then it works fine.  With the previous version,
> that wasn't necessary, of course.  And I have no way of forcing other libcurl
> apps in this way.

We can of course implement a way to do the same for applications in the meantime. What about adding a new environment variable (e.g. CURL_DISABLE_IPV6) to achieve this? Which applications are you actually focused on? 

> According to yum history, my previously working curl/libcurl was
> 7.19.7-4.fc13.i686.  I can't find that version at the moment to try rolling

The above mentioned change was done in 7.19.6-11.fc13. If the same worked for you in 7.19.7-4.fc13, it must have been a bug in your network ;-)

Workaround for libcurl applications like yum that fail on a limited number of domains:

For each domain name,

1. look up an IPv4 address using the "host" command, e.g.

host mirrors.fedoraproject.org

2. hard-code that IPv4 address in /etc/hosts, e.g. by adding the following line

66.35.62.166	mirrors.fedoraproject.org wildcard.fedoraproject.org

3. ensure that /etc/host contains the line

order hosts,bind

(it does by default)

Remember that this workaround will break if and when the server's IPv4 address changes, so undo it as soon as you don't need it any more!

Alternatively, use DNS servers that do not provide IPv6 addresses.

Oops, that should be /etc/host.conf not /etc/host, in step 3 above.

Thanks for heads up!

However I don't think we need to change all applications using libcurl. The proposed environment variable should solve this transient problem fairly well without changing anything beyond the curl package.

> The proposed environment variable ....
Would not be saner for curl to try an IPv4 address first in case both are available; or at least retry with it if IPv6 failed?

> ... this transient problem ....
Judging from experiences so far this "transient" problem is about to stay for many years to come.  I am afraid that I have a zero control over various intermediaries relaying my net traffic and the situation may vary if I am travelling with my laptop.

(In reply to comment #1)
> We can of course implement a way to do the same for applications in the
> meantime. What about adding a new environment variable (e.g. CURL_DISABLE_IPV6)
> to achieve this? Which applications are you actually focused on? 

Well, yum in particular.  It's quite a showstopper when I can't reach mirrors.fedoraproject.org. :)  Robin's workaround helps though, to hardcode the IPv4 addresses in /etc/hosts.  That or an environment variable are both kludges though, so I hope we can fix it for real before F13 is released...

> > According to yum history, my previously working curl/libcurl was
> > 7.19.7-4.fc13.i686.  I can't find that version at the moment to try rolling
> 
> The above mentioned change was done in 7.19.6-11.fc13. If the same worked for
> you in 7.19.7-4.fc13, it must have been a bug in your network ;-)  

I thought that was strange too, but I got the 7.19.7-4 packages from koji now and I confirmed it.  Toggling back and forth, it definitely works with 7.19.7-4 and not with 7.19.7-8.

Comparing the build logs, the most obvious difference is that -4 was built against c-ares-1.6.0-3 and -8 against c-ares-1.7.0-1.  However, I still had c-ares-1.7 installed while testing that -4 works, so something in the headers at build time must materially affect how libcurl comes out.

> Robin's workaround helps though ...

Not really.  At least not here. 'dig mirrors.fedoraproject.org' produces for me:
mirrors.fedoraproject.org. 3600 IN      CNAME   wildcard.fedoraproject.org.
wildcard.fedoraproject.org. 60  IN      A       66.35.62.166
wildcard.fedoraproject.org. 60  IN      A       80.239.156.215
wildcard.fedoraproject.org. 60  IN      A       152.46.7.222

and using values from different A records, and also sticking these into 'mirrorlist=...' instead of host names in /etc/yum.repos.d/fedora-rawhide.repo was not effective.  Backing off to 7.19.7-4.fc13 versions of curl packages resolved yum troubles immediately.

(In reply to comment #5)
> > The proposed environment variable ....
> Would not be saner for curl to try an IPv4 address first in case both are
> available; or at least retry with it if IPv6 failed?

Definitely. This way it works in Fedora 11 and 12 now. Feel free to report a bug if that's not the case.

> > ... this transient problem ....
> Judging from experiences so far this "transient" problem is about to stay for
> many years to come.  I am afraid that I have a zero control over various
> intermediaries relaying my net traffic and the situation may vary if I am
> travelling with my laptop.  

I was talking about transient problem in c-ares, not about transient problem in networking :-)

(In reply to comment #6)
> That or an environment variable are both kludges
> though, so I hope we can fix it for real before F13 is released...

Then subscribe yourself to bug 548396 and vote for it. Patches are always welcome ;-)

> Comparing the build logs, the most obvious difference is that -4 was built
> against c-ares-1.6.0-3 and -8 against c-ares-1.7.0-1.  However, I still had
> c-ares-1.7 installed while testing that -4 works, so something in the headers
> at build time must materially affect how libcurl comes out.  

Now I see your point. However I have sort of bad news for you - it had been yet another bug, definitely not the correct behavior:

http://permalink.gmane.org/gmane.comp.web.curl.library/26067

Finally if you wonder why we decided for c-ares to break what used to work before, look at bug 539809.

(In reply to comment #8)
> Then subscribe yourself to bug 548396 and vote for it. Patches are always
> welcome ;-)

Sure, if I get a chance I will look at it, though I imagine if it were simple it would have been fixed already...

> > Comparing the build logs, the most obvious difference is that -4 was built
> > against c-ares-1.6.0-3 and -8 against c-ares-1.7.0-1.  However, I still had
> > c-ares-1.7 installed while testing that -4 works, so something in the headers
> > at build time must materially affect how libcurl comes out.  
> 
> Now I see your point. However I have sort of bad news for you - it had been yet
> another bug, definitely not the correct behavior:
> 
> http://permalink.gmane.org/gmane.comp.web.curl.library/26067

I can sympathize; it's painful when restoring correct behavior in one place produces regressions elsewhere (of the sort that it "should" have been broken before too).

> Finally if you wonder why we decided for c-ares to break what used to work
> before, look at bug 539809.  

IMO, that bug is less serious than having yum unable to talk to fedoraproject.org, so maybe we should hold off on c-ares for a bit?  We're a long way from F13 release though, so hopefully it can come together soon...

Re: comment #8
>> Would not be saner for curl to try an IPv4 address first in case both are
>> available; or at least retry with it if IPv6 failed?

> Definitely. This way it works in Fedora 11 and 12 now. Feel free to report a
> bug if that's not the case.

AFAICS this is already done. Bug 548269 filed by Josh Stone (and cloned as bug 548396); or you are talking about something else?  If curl would work in F13 the way it supposedly does in F11 or 12 then it is hard to imagine how you could get such reports.

Please see bug 548396 comment 1 -- I believe the current c-ares API cannot support mixed protocol results.

Perhaps on the curl side, instead of asking c-ares for PF_UNSPEC, it could make separate calls for PF_INET6 and PF_INET, and then merge the results?

(In reply to comment #11)
> Please see bug 548396 comment 1 -- I believe the current c-ares API cannot
> support mixed protocol results.
> 
> Perhaps on the curl side, instead of asking c-ares for PF_UNSPEC, it could make
> separate calls for PF_INET6 and PF_INET, and then merge the results?  

Right, fixing it that way in libcurl is indeed doable and will make everything work with existing c-ares versions.

I think fixing it in the c-ares API to allow that to get both kinds is desirable too, but...

(In reply to comment #12)
> Right, fixing it that way in libcurl is indeed doable and will make everything
> work with existing c-ares versions.

I also like that idea. Unfortunately I don't have enough time to write the patch reasonable soon and Fedora seems to be pretty broken right now. Would anybody object if I add the environment variable to disable IPv6 completely?

Even if we implement the fallback, the first attempt will still take some time for no reason. I think such a variable could be handy if you know your IPv6 network is broken.

I am raising severity to high as broken yum on Fedora could be pretty annoying.

Do you have any proposal for the name of such a variable? Are there already any equivalents elsewhere?

Wouldnt it make sense to just drop the AAAA record from the DNS temporarily? Should be the easiest workaround, if one can get hold of the admins.

(In reply to comment #14)
> Wouldnt it make sense to just drop the AAAA record from the DNS temporarily?
> Should be the easiest workaround, if one can get hold of the admins.  

Not really. It would break all hosts connected to working IPv6, but broken IPv4 network.

> It would break all hosts connected to working IPv6, but broken IPv4
> network.

If you insist on an environment variable then it seems to me that you should use to turn on IPv6 if a need arise.  Otherwise you will have a widespread damage on hands; especially if something like this will seep into a Fedora release.  "A general user population" is not in a habit of reading bugzilla.

So far curl-7.19.7-4.fc13 does something like this

$ curl -v fedoraproject.org
* About to connect() to fedoraproject.org port 80 (#0)
*   Trying 66.35.62.162... connected
* Connected to fedoraproject.org (66.35.62.162) port 80 (#0)
> GET / HTTP/1.1
.......

Did I miss some bugzilla reports saying that this is broken in practice?  You indeed may have, say, CURL_USE_IPv6 for such cases.

> Even if we implement the fallback, the first attempt will still take some time
> for no reason.

In such situation I would suggest that for a foreasable time you should try IPv4 first and only if this fails go for IPv6.  You are not likely to see many delays and only when they start to prevail this order can be switched around.

> I think such a variable could be handy if you know your IPv6
> network is broken.

As a matter of practicality you can generally assume that unless you specifically know otherwise in a given case and that why I think that such variable should be set only if you know that your IPv6 network, and that includes all steps between you and your target, is NOT broken.

I've just restored the behavior of curl-7.19.7-4.fc13 to unblock yum for now:

--- a/lib/hostares.c
+++ b/lib/hostares.c
@@ -327,7 +327,7 @@ Curl_addrinfo *Curl_getaddrinfo(struct connectdata *conn,

   switch(data->set.ip_version) {
   default:
-#if ARES_VERSION >= 0x010601
+#if 0
     family = PF_UNSPEC; /* supported by c-ares since 1.6.1, so for older
                            c-ares versions this just falls through and defaults
                            to PF_INET */

*** Bug 549088 has been marked as a duplicate of this bug. ***

Installation of these RPMs should unblock yum on broken IPv6 network:

http://koji.fedoraproject.org/koji/buildinfo?buildID=147869

Leave here a note if the problem persists.

I'll try to prepare the proper fix in time to get it into f13-dist.

(In reply to comment #15)
> (In reply to comment #14)
> > Wouldnt it make sense to just drop the AAAA record from the DNS temporarily?
> > Should be the easiest workaround, if one can get hold of the admins.  
> 
> Not really. It would break all hosts connected to working IPv6, but broken IPv4
> network.  

That's kinda theoretical, you must admit...

(In reply to comment #20)
> That's kinda theoretical, you must admit...  

-1 from me for dropping the AAAA records from DNS.

You're free to discuss it with administrators, though it should be no more relevant for this bug.

*** Bug 539563 has been marked as a duplicate of this bug. ***

(In reply to comment #8)
> (In reply to comment #5)
> > > The proposed environment variable ....
> > Would not be saner for curl to try an IPv4 address first in case both are
> > available; or at least retry with it if IPv6 failed?
> 
> Definitely. This way it works in Fedora 11 and 12 now. Feel free to report a
> bug if that's not the case.

The retrying part is indeed how it works in Fedora 11 (curl-7.19.7-3.fc11.i586), but it still tries IPv6 first, causing unnecessary delay:

curl -v fedoraproject.org
* About to connect() to fedoraproject.org port 80 (#0)
*   Trying 2610:28:200:1::fed0:1... Timeout
*   Trying 66.35.62.162... connected

which takes about a minute.  Guess I'll file a bug report, but I notice that firefox has the same issue and doesn't seem to use curl; is there another package the bug should be against?

> but it still tries IPv6 first, causing unnecessary delay:

Like I suggested in comment #16 the order should be IPv4 first and if failed followed by IPv6.  At least for quite a while that should work without deleays
in (nearly?) all cases.

> is there another package the bug should be against?

Not sure.  Maybe c-ares?  Some comments seem to indicate that the real breakage lies there.

(In reply to comment #23)

Without c-ares, libcurl defaults to using getnameinfo() for name resolving (if
available) and that's what most apps will do as well.

That function is provided by libc and gai.conf is what needs to be altered to modify the order it should return addresses (and thus which order libcurl will try them).

Thanks, filed bug 549892.

*** Bug 548500 has been marked as a duplicate of this bug. ***

This bug appears to have been reported against 'rawhide' during the Fedora 13 development cycle.
Changing version to '13'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

I've experimentally built (lib)curl based on threaded DNS resolver in order to replace c-ares.  We'll see if the approach fits well into Fedora.

http://koji.fedoraproject.org/koji/buildinfo?buildID=168222

(In reply to comment #29)
> I've experimentally built (lib)curl based on threaded DNS resolver in order to
> replace c-ares.  We'll see if the approach fits well into Fedora.
> 
> http://koji.fedoraproject.org/koji/buildinfo?buildID=168222    

I don't have an IPv6 network to test, but my original reproducer "curl -v fedoraproject.org" works fine for me on the new build.

(In reply to comment #30)
> I don't have an IPv6 network to test, but my original reproducer "curl -v
> fedoraproject.org" works fine for me on the new build.    

Thank you for the feedback.  Generally the behavior should be equal to F-12 in this aspect as it uses the glibc based name resolving again.  The only difference is that alarm(2) is no more utilized.  In contrast to that we use the pthread library to implement the timeouts.  Once the threaded resolver becomes stable enough, I'll get it to F-13 as well.  For more details about the progress, see the corresponding thread on upstream mailing list:

http://thread.gmane.org/gmane.comp.web.curl.library/27709

curl-7.20.1-1.fc13 has been submitted as an update for Fedora 13.
http://admin.fedoraproject.org/updates/curl-7.20.1-1.fc13

curl-7.20.1-1.fc13 has been pushed to the Fedora 13 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update curl'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/curl-7.20.1-1.fc13

curl-7.20.1-1.fc13 has been pushed to the Fedora 13 stable repository.  If problems still persist, please make note of it in this bug report.

This update appears to have introduced some issues, see bug #599340.

*** Bug 560135 has been marked as a duplicate of this bug. ***