If we replaced that libtool with a symlink to /usr/bin/libtool would that make a difference, or would it have (potentially) the same problem?

I'm guessing the answer is no.  Which flags changes are you concerned with here specifically?  

In general those rebuild ordering dep chains are by design, the build systems of apr/apr-util (and httpd and dependencies thereof) are tied together to ensure the same set of CFLAGS/CPPFLAGS/... are used all the time. So e.g. if you build apr with -D_FILE_OFFSET_BITS=... then you need to ensure you build apr-util (and httpd, and httpd modules) exactly the same way, so you get the right ABI.

(In reply to Joe Orton from comment #1)
> If we replaced that libtool with a symlink to /usr/bin/libtool would that
> make a difference, or would it have (potentially) the same problem?

You are right, it does not make a difference.  I will have to file a bug against libtool as well.  Not good.

> In general those rebuild ordering dep chains are by design, the build
> systems of apr/apr-util (and httpd and dependencies thereof) are tied
> together to ensure the same set of CFLAGS/CPPFLAGS/... are used all the
> time. So e.g. if you build apr with -D_FILE_OFFSET_BITS=... then you need to
> ensure you build apr-util (and httpd, and httpd modules) exactly the same
> way, so you get the right ABI.

Understood, but the redhat-rpm-config build flags are carefully selected *not* to change ABI (which is why they do not touch _FILE_OFFSET_BITS).  And in fact, /usr/lib/apr-1/build/libtool does *not* contain the string “OFFSET”, and I don't see that it sources other scripts which provide this information.  (And if you don't use off_t/ino_t/struct stat in headers, _FILE_OFFSET_BITS does not matter to you anyway.)

So ABI consistency isn't a valid reason to do this kind of build flags capture on Fedora.

Hmm.  Could it be that /usr/lib*/apr-1/build/libtool isn't the culprit here?  It seems to me that it captures the flags only for some Cygwin/Mingw stuff, but not for native compilation.  In fact, I don't see the problem with /usr/bin/libtool.

However, there is also /usr/lib*/apr-1/build/apr_rules.mk, which contains (on i686, as of apr-devel-1.6.3-1.fc28.i686):

CFLAGS=-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m32 -march=i686 -fasynchronous-unwind-tables

Still no _FILE_OFFSET_BITS, though, so no ABI concerns there.

Whatever causes this, I do think we need to fix it.

(In reply to Florian Weimer from comment #3)
> Understood, but the redhat-rpm-config build flags are carefully selected
> *not* to change ABI (which is why they do not touch _FILE_OFFSET_BITS).  And
> in fact, /usr/lib/apr-1/build/libtool does *not* contain the string
> “OFFSET”, and I don't see that it sources other scripts which provide this
> information.  (And if you don't use off_t/ino_t/struct stat in headers,
> _FILE_OFFSET_BITS does not matter to you anyway.)
> 
> So ABI consistency isn't a valid reason to do this kind of build flags
> capture on Fedora.

It's why the *upstream* projects have designed buildsystems to be dependent, and we can't simply throw that away.  The build flags are threaded down to tools like apxs used when building RPMs and by end-users to build stuff against API/ABI we ship. I'm not sure we could (or should try to) successfully disentangle the two cases.

> However, there is also /usr/lib*/apr-1/build/apr_rules.mk, which contains
> (on i686, as of apr-devel-1.6.3-1.fc28.i686):
> 
> CFLAGS=-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2
> -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4
> -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m32
> -march=i686 -fasynchronous-unwind-tables

Yes that's probably the root cause.

> Whatever causes this, I do think we need to fix it.

I would argue this should be fixed at RCM level by ensuring mass rebuilds are ordered correctly.  I'm sure most of the extensible scripting languages do similar things to APR here, e.g. stuff using `perl -V:ccflags` to build; PHP extensions, this is not a new problem and definitely not unique to apr/httpd.

This bug appears to have been reported against 'rawhide' during the Fedora 28 development cycle.
Changing version to '28'.

Restated:

It's a design goal of upstream interfaces to preserve and expose build flags, and the build systems of apr/httpd/mod_* are based around this. To the extent this is a problem for Fedora, AFAICT it's fixable by ordering a mass rebuild correctly. I don't see any other viable option.

Agreed in principle.  We don't need the ordering, actually—I think things converge pretty quickly, so we can fix up things with targeted rebuilds as necessary.

The situation is certainly not ideal, but apr isn't going to be the only package with this issue.