What version of curl do you have installed?

# rpm -q curl libcurl
curl-7.19.7-9.fc13.x86_64
libcurl-7.19.7-9.fc13.x86_64

curl doesn't work, while wget has no problem:

# curl -v -S www.kernel.org; wget www.kernel.org

* Could not resolve host: www.kernel.org (Domain name not found)
* Closing connection #0
curl: (6) Could not resolve host: www.kernel.org (Domain name not found)


--2010-01-11 16:57:50--  http://www.kernel.org/
Translacja sabretooth.local... 192.168.6.9
Łączenie się z sabretooth.local|192.168.6.9|:8080... połączono.
Żądanie Proxy wysłano, oczekiwanie na odpowiedź... 200 OK

Could you please capture the following strace and attach?

$ strace -e trace=open -o strace.txt curl -svo/dev/null http://www.kernel.org

Created attachment 383018 [details]
stracde output

Thanks for the strace, it works as expected.  Unfortunately mdns is not supported by c-ares for now. Only "dns" and "files" resources are supported for now.  Reassigning to c-ares as it is the place where this bug should be addressed.  Patches are welcome!

It works in curl-7.19.7-6.fc12.x86_64 in F12. Shouldn't this bug be treated as a regression?

I think so.

The initial work to support mdns as an option has been done here:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=476735

However, that only adds mdns as an option, it doesn't actually enable c-ares to do mdns lookups/resolves. For that, you'd either need to add code to c-ares to let NSS do the work (which upstream has already said it does not want to do), or leverage avahi to handle mdns. I suspect the latter would be acceptable to upstream, but I'm very far from an expert in this space.

Adding Lennart Poettering to the CC, as he _is_ an expert in this space. :)

Uh. c-ares' approach is very questionnable. You really should not bypass the NSS stack when doing name resolving. Implementing your own DNS client stack adds a bunch of redundant code, adds a whole new set of security issues and makes it impossible to replace the name resolving protocol backend as this case shows.

There are better solutions, like using libasyncns, which is my own solution to the problem and which simply makes the NSS usable asynchronously without headaches (http://0pointer.de/lennart/projects/libasyncns/). libasyncns has been imported into the glib tree recently, since it is about the only solution for somewhat clean name resolution, if you don't want to run into issues like this.

So yepp, my recommendation is to drop c-ares, and adopt libasyncns for all asynchronous name resolution. But then, I wrote libasyncns. But I did so for a reason. And the glib folks adopted libasyncns for a reason, too.

The other options would be to implement your own mDNS stack in addition to the classic unicast DNS stack. However I can only strongly recommend not to do this since mDNS is highly optimized for reducing network traffic and does so by employing quite sophisticated cache coherency protocols. And you cannot implement that properly unless you are a constantly running mDNS daemon, which is Avahi for us.

The last option would be to talk directly to the /var/run/avahi-daemon/socket= socket. It speaks a very simple text based protocol for doing mDNS name resolution via Avahi. It's not really documented, but nss-mdns should give you an idea how to do this. If you really want to stick with c-ares then this is certainly the best solution.

> There are better solutions, like using libasyncns, which is my own solution to
> the problem and which simply makes the NSS usable asynchronously without
> headaches (http://0pointer.de/lennart/projects/libasyncns/). libasyncns has

I've never heard about libasyncns.  Thanks for the hint!  It looks pretty compact to me.  Do you think we can replace c-ares by libasyncns completely in libcurl?

> been imported into the glib tree recently, since it is about the only solution
> for somewhat clean name resolution, if you don't want to run into issues like
> this.

Does it mean libasyncns is now maintained at two places?

Looking into asyncns-test.c I am a bit scared by the signal(SIGCHLD, SIG_IGN) as the first command :-)  Is it really safe to do something like that in a library?

(In reply to comment #10)
> > There are better solutions, like using libasyncns, which is my own solution to
> > the problem and which simply makes the NSS usable asynchronously without
> > headaches (http://0pointer.de/lennart/projects/libasyncns/). libasyncns has
> 
> I've never heard about libasyncns.  Thanks for the hint!  It looks pretty
> compact to me.  Do you think we can replace c-ares by libasyncns completely in
> libcurl?

Yes, definitely.

You may either link against libasyncns or import asyncns.c and .h into your tree. If you do the latter, then you'll of course have to make sure to update it from time to time, however you don't have this additional dependency on a tiny library. I recommend to link instead of import.

I think all distros these days ship libasyncns these days anyway, so that dependency angst shouldn't matter too much.

> > been imported into the glib tree recently, since it is about the only solution
> > for somewhat clean name resolution, if you don't want to run into issues like
> > this.
> 
> Does it mean libasyncns is now maintained at two places?

Not really. The upstream version is the one that matters. I work with the glib folks to make sure the versions stay in sync and patches flow both ways. It works quite well in this case I guess, but of course if everyone imports asyncns.c into their trees this won't scale. 
 
> Looking into asyncns-test.c I am a bit scared by the signal(SIGCHLD, SIG_IGN)
> as the first command :-)  Is it really safe to do something like that in a
> library?    

No it certainly is not. However, you don't really need to put this there.

You can compile asyncns.c in two ways: either it runs the name resolving code in a seperate thread (default and how it is build in the distros) or in a seperate process. Only if you use the forking version the SIG_IGN matters. Because only then the other side of the internally used communication socket might die (maybe because someone killed the process explicitly or so) and thus if all bad things come together trigger a SIGPIPE because asyncns wrote to the disconnected socket right at that time.

Also, on Linux (i.e. where MSG_NOSIGNAL is available) the SIGPIPE madness isn't necessary at all.

So basically the SIG_IGN blocking matters only to be completely correct if you compile libasyncns in a non-default way, and only if you do so on non-Linux systems.

> I think all distros these days ship libasyncns these days anyway, so that
> dependency angst shouldn't matter too much.

I don't see any problems in the new dependency, too.

> You can compile asyncns.c in two ways: either it runs the name resolving code
> in a seperate thread (default and how it is build in the distros) or in a
> seperate process. Only if you use the forking version the SIG_IGN matters.
> Because only then the other side of the internally used communication socket
> might die (maybe because someone killed the process explicitly or so) and thus
> if all bad things come together trigger a SIGPIPE because asyncns wrote to the
> disconnected socket right at that time.

So is the library based on threading then?  The main advantage of c-ares over other libraries is that it is thread-safe, but does not use threading.  We have already thread-based name resolver in libcurl, but it's kind of not recommended.  So that I need to compare libasyncns with the built-in thread resolver at first.  Then I'll probably raise the issue at libcurl upstream since that's the place where the eventual enhancement should go.

The built-in thread resolver seems to be built only on Windows.  So there is some extra work needed in both cases.  I think the c-ares solution is a bit saner for a library in the first place since it does not require threading for itself.

I am only afraid the 'mdns' keyword is not the only one ignored by c-ares now.  We can write a patch for c-ares and close the bug ... and a new bug can be just opened for yet another glibc/nss plug-in.

FWIW I've just CC'd Daniel Stenberg as the upstream maintainer of libcurl and c-ares.

(In reply to comment #12)

> So is the library based on threading then?  The main advantage of c-ares over
> other libraries is that it is thread-safe, but does not use threading.  We have
> already thread-based name resolver in libcurl, but it's kind of not
> recommended.  So that I need to compare libasyncns with the built-in thread
> resolver at first.  Then I'll probably raise the issue at libcurl upstream
> since that's the place where the eventual enhancement should go.    

The library does use threading internally, but that is not seen on the outside. It is completely abstracted away, in a way, that -- as mentioned -- you can even replace that threading with a proper fork()ed off process if you wish.

Basically, from the outside you just have one opaque object (asyncns_t*) that allows you to get an fd (a single one) you can stick in your poll() loop.

This is the entire API:

http://0pointer.de/lennart/projects/libasyncns/doxygen/asyncns_8h.html

(In reply to comment #13)
> The built-in thread resolver seems to be built only on Windows.  So there is
> some extra work needed in both cases.  I think the c-ares solution is a bit
> saner for a library in the first place since it does not require threading for
> itself.

but it implements DNS natively and bypasses the NSS. That means it is unable to make use of any non-DNS name resolvers (mdns, ldap, yadda yadda), nor of any caching services available in the NSS or those resolvers (nscd, avahi, yadda yadda).

Really, c-ares is not a good choice.

Due to c-ares' limitations in being a feature-complete replacement to the libc based resolver functions, we're working on a native threaded resolver that is already in the main code targeted for (lib)curl 7.20.0 but it is currently disabled by default and there actually isn't a current easy way to enable it.

The plan is to work on it and make it possible to enable it with configure starting sometime post-7.20.0.

To clarify: this is meant for POSIX systems. libcurl already has and builds with a threaded resolver for win32 systems by default.

Daniel, just out of curiosity, can you elaborate on that a little? i.e. I hope and assume you will base it on NSS, right? Are you planning to run those functions in a child process? or thread? What will this resolver code do that libasyncns doesn't do?

(In reply to comment #17)
> Daniel, just out of curiosity, can you elaborate on that a little? i.e. I hope
> and assume you will base it on NSS, right? Are you planning to run those
> functions in a child process? or thread? What will this resolver code do that
> libasyncns doesn't do?    

The code is already there - look into lib/hostthre.c in the curl sources.  It's only not supported on Linux for now because of some portability issues.

(In reply to comment #17)
> Daniel, just out of curiosity, can you elaborate on that a little? i.e. I hope
> and assume you will base it on NSS, right? Are you planning to run those
> functions in a child process? or thread? What will this resolver code do that
> libasyncns doesn't do?    

I've not checked libasyncns at all (I didn't know it existed until this bug entry was brought to my attention), but I can only assume that if it is based on the regular name resolver functions and it uses threads, then the functionality should be equivalent.

The code for threaded resolving already in libcurl should work but it hasn't yet undergone extensive testing for POSIX systems and it won't do that until after the 7.20.0 release that is due to get shipped next week.

7.20.1-2.fc14 seem to work with http_proxy=http://something.local:8080. Can we expect backport to F13?

(In reply to comment #20)
> 7.20.1-2.fc14 seem to work with http_proxy=http://something.local:8080. Can we
> expect backport to F13?    

Tomasz, thank you for your interest and the testing!

We can backport it to F13 at some point, but definitely _not_ now.  The threaded DNS resolver is even not supported by upstream beyond WIN32 yet.  It was experimentally enabled in rawhide two days ago and I don't consider the packages stable for now.  There are still some open questions which should be addressed before it hits stable Fedora.

For more details see the upstream thread:
http://curl.haxx.se/mail/lib-2010-04/0285.html

Starting by 7.20.1-1.fc13, libcurl no longer uses c-ares for name resolving.  It means that up2date libcurl is no more affected by this bug.

This bug is against c-ares and there has been nothing changed.  The support for mdns is still missing.  It should be closed as WONTFIX.