I'm not sure what's the difference between the database in GeoIP and the database in geoip-geolite, but what I do know is that some apps (such as the GeoIP c library) requires the former, and some apps (like python-pygeoip) requires the latter. I also know that geoip-geolite is more complete, containing the region and ASN databases as well.

I suggest either changing GeoIP.dat's file name in geoip-geolite to geolite-GeoIP.dat

Hi, sorry for not acknowledging this ticket sooner.

FWIW, the less-complete GeoIP package existed first.  I submitted geoip-geolite for review without doing my homework.  This coulda/shoulda been caught before that.

Now that we're here, Elad's rationale makes sense.  geoip-geolite is in a little
better situation in that it is *just* the data files which we can update independently.  GeoIP contains both some code and some of the data files.

My first suggestion is that we remove GeoIP.dat from the GeoIP package and have it instead depend on and point to this geoip-geolite data package instead.

If the GeoIP maintainers are not amenable to that solution, my fallback suggestion is to, as I think Elad is suggesting, rename geoip-geolite's data files to geolite-*.dat so there is no conflict.

I'll go find and notify the GeoIP package maintainers of this ticket.

If the GeoIP package included the v6 databases, would python-geoip be able to run against this instead?

That seems to me to be the extent of the differences.  Oh, and that GeoIP calls one file "GeoLiteASNum.dat" and geoip-geolite calls it "GeoLiteIPASNum.dat".

(In reply to Philip Prindeville from comment #3)
> That seems to me to be the extent of the differences.
according to rpm -ql, the only GeoIP database contained in the GeoIP package is /usr/share/GeoIP/GeoIP.dat (which is the "Country Edition" database) while geoip-geolite contains the (much useful) ASNum and Location databases in addition to the country database.
Furthermore, geoip-geolite does not contain perl scripts to update the databases, which is good because this, like any other software in Fedora, should be updated by the package management system. I suggest that the GeoIP package will remain only for the C library and that it will use the data in geoip-geolite.

(In reply to Elad Alfassa from comment #4)
> (In reply to Philip Prindeville from comment #3)
> > That seems to me to be the extent of the differences.
> according to rpm -ql, the only GeoIP database contained in the GeoIP package
> is /usr/share/GeoIP/GeoIP.dat (which is the "Country Edition" database)
> while geoip-geolite contains the (much useful) ASNum and Location databases
> in addition to the country database.
> Furthermore, geoip-geolite does not contain perl scripts to update the
> databases, which is good because this, like any other software in Fedora,
> should be updated by the package management system. I suggest that the GeoIP
> package will remain only for the C library and that it will use the data in
> geoip-geolite.

I don't think your information is current. Looking at the 1.5.0 release, I see:

-rw-r--r--  root     root     /usr/share/GeoIP/GeoIP-initial.dat
lrwxrwxrwx  root     root     /usr/share/GeoIP/GeoIP.dat
-rw-r--r--  root     root     /usr/share/GeoIP/GeoLiteASNum.dat
-rw-r--r--  root     root     /usr/share/GeoIP/GeoLiteCity.dat
-rw-r--r--  root     root     /usr/share/GeoIP/GeoLiteCountry.dat

(In reply to Philip Prindeville from comment #5)
> I don't think your information is current. Looking at the 1.5.0 release

Both Fedora 18 and Fedora 19 have 1.4.8, is 1.5.0 rawhide only?

(In reply to Elad Alfassa from comment #6)
> (In reply to Philip Prindeville from comment #5)
> > I don't think your information is current. Looking at the 1.5.0 release
> 
> Both Fedora 18 and Fedora 19 have 1.4.8, is 1.5.0 rawhide only?

It has not yet been pushed to updates, but it is in git. We're still getting the issues all worked out for a hopefully seamless update.

Patching python-geoip to use the data files from the GeoIP package is a possibility.  It is somewhat frustrating though that a python-geoip user would have to also install the libGeoIP code, which is not needed by python-geoip.

It seems to me we have different minds about the way this should go and the most practical solution I can see is to re-namespace the geoip-geolite package to keep its files in /usr/share/geoip-geolite/ folder (separate from the GeoIP package and its updater).  python-geoip can then be patched to look in this new location.

(In reply to Ralph Bean from comment #8)
> Patching python-geoip to use the data files from the GeoIP package is a
> possibility.  It is somewhat frustrating though that a python-geoip user
> would have to also install the libGeoIP code, which is not needed by
> python-geoip.

So what's to stop us from putting the libraries in a GeoIP-libs subpackage?

But more to the point, what exactly about it is frustrating about 200KB library in an era when computers come with disk drives measured in hundreds of gigabytes?

In any case, geoipupdate is going to have a dependency on libGeoIPUpdate.so and the size of the data files is 22MB, so 200KB in so's represents noise (less than 1% of the total).

> It seems to me we have different minds about the way this should go and the
> most practical solution I can see is to re-namespace the geoip-geolite
> package to keep its files in /usr/share/geoip-geolite/ folder (separate from
> the GeoIP package and its updater).  python-geoip can then be patched to
> look in this new location.

I'm not sure I understand this: having the libraries laying around and taking up 200KB is "somewhat frustrating", but you're ok with having a 22MB database duplicated a 2nd time?

Clearly it's not the wasted space that frustrates you, so what exactly is your objection to having the libraries being present but unused?

(In reply to Philip Prindeville from comment #9)
> (In reply to Ralph Bean from comment #8)
> > Patching python-geoip to use the data files from the GeoIP package is a
> > possibility.  It is somewhat frustrating though that a python-geoip user
> > would have to also install the libGeoIP code, which is not needed by
> > python-geoip.
> 
> So what's to stop us from putting the libraries in a GeoIP-libs subpackage?

Nothing.  That is a good idea.  Let's do that.

> But more to the point, what exactly about it is frustrating about 200KB
> library in an era when computers come with disk drives measured in hundreds
> of gigabytes?

It is not about size.  It is about code.

When administering a system, it is good practice to only install code required
to run whatever service(s) you intend to run.

(In reply to Ralph Bean from comment #10)
> > But more to the point, what exactly about it is frustrating about 200KB
> > library in an era when computers come with disk drives measured in hundreds
> > of gigabytes?
> 
> It is not about size.  It is about code.
> 
> When administering a system, it is good practice to only install code
> required
> to run whatever service(s) you intend to run.

Good practice is limiting available attack surfaces, by not having potential exploits.

The mere presence of a .so is not an attack surface, since the .so itself (a) needs to be insecure, and (b) exploitable from a calling process.

But I think you're misunderstanding the wider principle of Fedora being an expansive distro. It is most definitely not an minimalist environment.

If you want minimalism, I'd recommend PuppyLinux.

Fedora is a "full bells & whistles" distribution.

(In reply to Philip Prindeville from comment #11)
> If you want minimalism, I'd recommend PuppyLinux.

If I want to resolve the conflict between our packages, what do you recommend?

(In reply to Ralph Bean from comment #12)
> (In reply to Philip Prindeville from comment #11)
> > If you want minimalism, I'd recommend PuppyLinux.
> 
> If I want to resolve the conflict between our packages, what do you
> recommend?

Well, let's start by asking if you could live with GeoIP containing ALL of the databases, but also including the libraries.

(In reply to Philip Prindeville from comment #13)
> Well, let's start by asking if you could live with GeoIP containing ALL of
> the databases, but also including the libraries.

Live?  Of course.  Please let me know once that's in place and I'll ship a new release of python-pygeoip that depends on it.

Can you also add an Obsoletes: geoip-geolite to GeoIP?  That should make upgrades more smooth, correct?

Philip, can you tell me why in GeoIP we replace GeoIP.dat with a symlink to GeoLiteCountry.dat?

Upstream's README in 1.5.0 says:

   geoipupdate may be used to download our free databases.

   Put this into the config file /usr/local/etc/GeoIP.conf
   to download the free GeoLite databases GeoLiteCountry, GeoLiteCity and
   GeoLiteASNum

   LicenseKey 000000000000
   UserId 999999
   ProductIds 506 533 517

   Free users should create symlinks for the GeoIP databases.
   For example:

   cd /usr/local/share/GeoIP
   ln -s GeoLiteCity.dat GeoIPCity.dat
   ln -s GeoLiteCountry.dat GeoIPCountry.dat
   ln -s GeoLiteASNum.dat GeoIPASNum.dat


No mention there of doing anything with GeoIP.dat. Is it actually necessary to clobber it? If we did what upstream says and left GeoIP.dat alone, would that work for both GeoIP and geoip-geolite users?

(In reply to Ralph Bean from comment #14)
> 
> Can you also add an Obsoletes: geoip-geolite to GeoIP?  That should make
> upgrades more smooth, correct?

Shouldn't be a problem.

(In reply to Paul Howarth from comment #15)
> Philip, can you tell me why in GeoIP we replace GeoIP.dat with a symlink to
> GeoLiteCountry.dat?
> 
> Upstream's README in 1.5.0 says:
> 
>    geoipupdate may be used to download our free databases.
> 
>    Put this into the config file /usr/local/etc/GeoIP.conf
>    to download the free GeoLite databases GeoLiteCountry, GeoLiteCity and
>    GeoLiteASNum
> 
>    LicenseKey 000000000000
>    UserId 999999
>    ProductIds 506 533 517
> 
>    Free users should create symlinks for the GeoIP databases.
>    For example:
> 
>    cd /usr/local/share/GeoIP
>    ln -s GeoLiteCity.dat GeoIPCity.dat
>    ln -s GeoLiteCountry.dat GeoIPCountry.dat
>    ln -s GeoLiteASNum.dat GeoIPASNum.dat
> 
> 
> No mention there of doing anything with GeoIP.dat. Is it actually necessary
> to clobber it? If we did what upstream says and left GeoIP.dat alone, would
> that work for both GeoIP and geoip-geolite users?

Well, in that case, GeoIP.dat would be whatever version was released with the tarball, whereas the "Lite" versions of the files would be whatever version was available at the time of install.

So in theory the data might not match up (and from an empirical point-of-view, I see CIDR blocks getting reassigned all of the time).

The idea being that fresher data is better than stale data.

Especially if you're using that data, for instance, to make significant policy decisions about whether to accept network connections, issue DRM licenses, etc.

Dumping the ::database_info() for each file from a simple Perl script, I get:

GeoIP-initial	GEO-106FREE 20130219 Build 1 Copyright (c) 2012 MaxMind Inc All Rights Reserved
GeoLiteCountry	GEO-106FREE 20130604 Build 1 Copyright (c) 2013 MaxMind Inc All Rights Reserved
GeoLiteASNum	GEO-117 20130603 Build 1 Copyright (c) 2013 MaxMind Inc All Rights Reserved
GeoLiteCity	GEO-533LITE 20130604 Build 1 Copyright (c) 2013 MaxMind Inc All Rights Reserved

and we see that the first two databases are very similar (despite the size discrepancy), but months apart in terms of their data.

Also, an end-of-life machine like my F16 mail server (which I've not had time to update) continues to get weekly updated databases, despite having an GeoIP-1.4.8 install on it.

OK, but which of the data files gets used? I'm not familiar with the GeoIP API myself, so I don't know whether the answer is "all of them", "just GeoIP.dat", "whatever is specified by the API caller", or "what's configured in the GeoIP.conf file".

And if multiple files are used, which data takes precedence when they disagree with each other?

(In reply to Paul Howarth from comment #18)
> OK, but which of the data files gets used? I'm not familiar with the GeoIP
> API myself, so I don't know whether the answer is "all of them", "just
> GeoIP.dat", "whatever is specified by the API caller", or "what's configured
> in the GeoIP.conf file".

The short answer is: "It depends."

It depends on which API one is using, and how the database is opened.

In Perl, for instance, Geo::IP is typically used by MIMEDefang or SpamAssassin to identify the country of origin of the peer on an SMTP connection.

In that usage case, one does:

  my $gi = Geo::IP->new(GEOIP_STANDARD);

which opens GeoIP.dat by default (which in this case is an alias for GeoLiteCountry.dat) and contains equivalent data, is in Comment #17.

Other Perl clients might use the Geo::IP::open() method and explicitly open one of the other databases (such as the GeoIPASNum database for routing inheritance, or web server CGI's might use GeoLiteCountry or GeoLiteCity to guess the country or region to customize content for the user's estimated location: I've seen this go badly wrong when users are tunneling via corporate VPN).

FYI: the GeoIP.conf file just tells geoipupdate what databases you have licenses for (and hence it should attempt to update). It doesn't normally get used for actually querying the databases themselves (and indeed this file isn't even visible from the Perl Geo::IP API... nor the nominal C API from what I can tell).

> And if multiple files are used, which data takes precedence when they
> disagree with each other?

If they disagree with each other, then something is wrong.

Different databases contain (in theory) different data. The 'lite' databases contain less granular data than the commercially licensed databases. The database you open depends on how you open the database.

The Perl method Geo::IP::new() opens the "GeoIP.dat" database unless you explicitly specify a path, for instance.

If you had the commercial and free-to-use versions of the databases, you'd want to explicitly open the commercial version in your code.

OK, so it makes sense for GeoIP.dat to be a symlink to GeoLiteCountry.dat as the latter is basically a more up to date version of the former.

I think that rather than downloading the current versions of the free databases in %post, we should take a snapshot of them and include them as sources in the repo and the distributed packages themselves. Whilst this would mean that Internet-connected systems would have older copies of the data by default, it would also mean that non-Internet connected machines (or ones that needed a proxy for this access etc.) would at least have a copy of the free databases. I think that this is a pre-requisite for obsoleting/providing geoip-geolite. Admins installing the GeoIP-update package would get regular updates as before. Not doing the update in %post would also mean that every user of GeoIP that did not install GeoIP-update would have the same data, which would at least be consistent.

Regarding database file naming, is it true that:

 * GeoLite*.dat are the free versions
 * GeoIP*.dat are the commercial versions

If so, I don't think we should implement the symlinks suggested by upstream as we would be overwriting the commercial databases (of users that had them) at package update time.

Is the difference in naming of the ASNum database between the GeoIP and geoip-geolite packages likely to be a problem to anyone?

(In reply to Paul Howarth from comment #20)
> I think that rather than downloading the current versions of the free
> databases in %post, we should take a snapshot of them and include them as
> sources in the repo and the distributed packages themselves.

Alas, when the builder servers run in Koji, they have been sandboxed to not have Internet access as a security measure to stop them from being compromised.

> Whilst this
> would mean that Internet-connected systems would have older copies of the
> data by default, it would also mean that non-Internet connected machines (or
> ones that needed a proxy for this access etc.) would at least have a copy of
> the free databases. I think that this is a pre-requisite for
> obsoleting/providing geoip-geolite. Admins installing the GeoIP-update
> package would get regular updates as before. Not doing the update in %post
> would also mean that every user of GeoIP that did not install GeoIP-update
> would have the same data, which would at least be consistent.
> 
> Regarding database file naming, is it true that:
> 
>  * GeoLite*.dat are the free versions
>  * GeoIP*.dat are the commercial versions
> 
> If so, I don't think we should implement the symlinks suggested by upstream
> as we would be overwriting the commercial databases (of users that had them)
> at package update time.

Well, users that had commercial licenses would already need to modify their /etc/GeoIP.conf... but we can add a test to see if "geoipupdate" ended up creating a GeoLiteCountry.dat before creating the symlink. That's a good idea.

Then again, if they get a license *after* installing, they would have to manually delete GeoLiteCountry.dat to stop us from linking to it.

> Is the difference in naming of the ASNum database between the GeoIP and
> geoip-geolite packages likely to be a problem to anyone?

Again, it depends on the API invoked by the user or by 3rd party packages (like python-geoip) if they explicitly open a database or not.

(In reply to Philip Prindeville from comment #21)
> (In reply to Paul Howarth from comment #20)
> > I think that rather than downloading the current versions of the free
> > databases in %post, we should take a snapshot of them and include them as
> > sources in the repo and the distributed packages themselves.
> 
> Alas, when the builder servers run in Koji, they have been sandboxed to not
> have Internet access as a security measure to stop them from being
> compromised.

That's not what I meant; I was suggesting that we, as maintainers, download those files and use them in the package build process just like the GeoIP tarball itself (which is presumably what is currently being done for the geoip-geolite package itself).

> > Regarding database file naming, is it true that:
> > 
> >  * GeoLite*.dat are the free versions
> >  * GeoIP*.dat are the commercial versions
> > 
> > If so, I don't think we should implement the symlinks suggested by upstream
> > as we would be overwriting the commercial databases (of users that had them)
> > at package update time.
> 
> Well, users that had commercial licenses would already need to modify their
> /etc/GeoIP.conf... but we can add a test to see if "geoipupdate" ended up
> creating a GeoLiteCountry.dat before creating the symlink. That's a good
> idea.
> 
> Then again, if they get a license *after* installing, they would have to
> manually delete GeoLiteCountry.dat to stop us from linking to it.

If they get a commercial license, will the data they have be in GeoIP.dat or some other named file, or both?

> > Is the difference in naming of the ASNum database between the GeoIP and
> > geoip-geolite packages likely to be a problem to anyone?
> 
> Again, it depends on the API invoked by the user or by 3rd party packages
> (like python-geoip) if they explicitly open a database or not.

That's probably one for Ralph to answer then.

(In reply to Paul Howarth from comment #22)

> That's not what I meant; I was suggesting that we, as maintainers, download
> those files and use them in the package build process just like the GeoIP
> tarball itself (which is presumably what is currently being done for the
> geoip-geolite package itself).

We could... don't like the idea of having binary data be in git, but... I guess it's unavoidable.

> > Then again, if they get a license *after* installing, they would have to
> > manually delete GeoLiteCountry.dat to stop us from linking to it.
> 
> If they get a commercial license, will the data they have be in GeoIP.dat or
> some other named file, or both?

Depends on the license. It could be any of GeoIP.dat, GeoIPCountry.dat, GeoIPCity.dat, GeoIPISP.dat, GeoIPASNum.dat, etc.

> > > Is the difference in naming of the ASNum database between the GeoIP and
> > > geoip-geolite packages likely to be a problem to anyone?
> > 
> > Again, it depends on the API invoked by the user or by 3rd party packages
> > (like python-geoip) if they explicitly open a database or not.
> 
> That's probably one for Ralph to answer then.

Yup.

(In reply to Philip Prindeville from comment #23)
> (In reply to Paul Howarth from comment #22)
> 
> > That's not what I meant; I was suggesting that we, as maintainers, download
> > those files and use them in the package build process just like the GeoIP
> > tarball itself (which is presumably what is currently being done for the
> > geoip-geolite package itself).
> 
> We could... don't like the idea of having binary data be in git, but... I
> guess it's unavoidable.

It's avoidable - treat the data files as sources and upload them to the lookaside cache rather than putting them in git.

> > > Then again, if they get a license *after* installing, they would have to
> > > manually delete GeoLiteCountry.dat to stop us from linking to it.
> > 
> > If they get a commercial license, will the data they have be in GeoIP.dat or
> > some other named file, or both?
> 
> Depends on the license. It could be any of GeoIP.dat, GeoIPCountry.dat,
> GeoIPCity.dat, GeoIPISP.dat, GeoIPASNum.dat, etc.

It's a problem if it's GeoIP.dat - we could overwrite it on package update. Perhaps we should check that GeoIP.dat is already a symlink to GeoIP-initial.dat before clobbering it?

(In reply to Paul Howarth from comment #24)
 
> It's a problem if it's GeoIP.dat - we could overwrite it on package update.
> Perhaps we should check that GeoIP.dat is already a symlink to
> GeoIP-initial.dat before clobbering it?

Done.

(In reply to Philip Prindeville from comment #25)
> (In reply to Paul Howarth from comment #24)
>  
> > It's a problem if it's GeoIP.dat - we could overwrite it on package update.
> > Perhaps we should check that GeoIP.dat is already a symlink to
> > GeoIP-initial.dat before clobbering it?
> 
> Done.

Is there any new version for FC18 which supports this softlinking already? I'm interested in testing. Thank you.

(In reply to Peter Bieringer from comment #26)
> (In reply to Philip Prindeville from comment #25)
> > (In reply to Paul Howarth from comment #24)
> >  
> > > It's a problem if it's GeoIP.dat - we could overwrite it on package update.
> > > Perhaps we should check that GeoIP.dat is already a symlink to
> > > GeoIP-initial.dat before clobbering it?
> > 
> > Done.
> 
> Is there any new version for FC18 which supports this softlinking already?
> I'm interested in testing. Thank you.

Here's a scratch build you can try:

http://koji.fedoraproject.org/koji/taskinfo?taskID=5520594

I'm not happy with the script for updating the IPv6 databases in the GeoIP-update6 package but otherwise this should provide everything that GeoIP and geoip-geolite provided.

> Here's a scratch build you can try:
> 
> http://koji.fedoraproject.org/koji/taskinfo?taskID=5520594

Thank you, some suggestions:

1. two softlinks are still missing:

# ln -s GeoLiteCity.dat GeoCity.dat
# ln -s GeoLiteCityv6.dat GeoCityv6.dat


> I'm not happy with the script for updating the IPv6 databases in the
> GeoIP-update6 package

Yeah, it's bad that GeoIP currently does not support all files via geoipupdate.

A scan shows only following

# Filename via http://updates.maxmind.com/app/update_getfilename?product_id=ID

# 106 GeoIP.dat
# 111 GeoIPOrg.dat
# 112 GeoIPRegion.dat
# 115 GeoIPRegion.dat
# 117 GeoIPASNum.dat
# 119 GeoIPUserType.dat
# 121 GeoIPISP.dat
# 122 GeoIPISP.dat
# 132 GeoIPCity.dat
# 133 GeoIPCity.dat
# 135 GeoIPAreaCode.dat
# 137 GeoIPDMACode.dat

# 506 GeoLiteCountry.dat
# 517 GeoLiteASNum.dat
# 532 GeoLiteCity.dat
# 533 GeoLiteCity.dat

The update mechanism is a little bit strange at all (requesting first filename, then download, extract and look into file and analyzing the DB-ID), also geoipupdate is not failsafe...specifying an non-existent ProductID, it returns in an strange result:

Downloading gzipped GeoIP Database...
Done
Updating /usr/share/GeoIP/
Saving gzip file to /usr/share/GeoIP/.gz ... download data to a gz file named /usr/share/GeoIP/.gz 
Done
Uncompressing gzip file ... Done
Performing sanity checks ... Database type is 1
database_info  FAIL null
Received Error -21 (Sanity check database_info string failed) when attempting to update GeoIP Database

".gz" -> this should not happen, geoipupdate should stop earlier, when update_getfilename request return empty string.


Perhaps one has a contact to Maxmind and can inform them that they should update their update_getfilename with additional file names at least related to IPv6 and fix geoipupdate.

Perhaps as an workaround, geoipupdate can be enhanced using new ProductID range (e.g. 1xxx) with hardcoded filenames for the meantime (skipping update_getfilename) until Maxmind extend their update_getfilename service.

(In reply to Peter Bieringer from comment #28)
> 1. two softlinks are still missing:
> 
> # ln -s GeoLiteCity.dat GeoCity.dat
> # ln -s GeoLiteCityv6.dat GeoCityv6.dat

Mentioned softlinks were not correct, proper ones are:

# ln -s GeoLiteCity.dat GeoIPCity.dat
# ln -s GeoLiteCityv6.dat GeoIPCityv6.dat

 
> Perhaps as an workaround, geoipupdate can be enhanced using new ProductID
> range (e.g. 1xxx) with hardcoded filenames for the meantime (skipping
> update_getfilename) until Maxmind extend their update_getfilename service.

Investigation of the current update mechanism shows that this is not possible, therefore the current workaround must stay until MaxMind supports IPv6 database files via their update "API".

geoip-geolite has been retired from rawhide.

The issue is still present in f19:

Transaction check error:
  file /usr/share/GeoIP/GeoIP.dat from install of GeoIP-1.4.8-6.fc19.x86_64 conflicts with file from package geoip-geolite-2013.04-1.fc19.noarch

(In reply to Ralph Bean from comment #30)
> geoip-geolite has been retired from rawhide.

It might make sense to retire it from f19 and f20, and push out builds for GeoIP for the same.

*** Bug 1019164 has been marked as a duplicate of this bug. ***

The bug is closed with errata, but this issue is still present in epel6. Will there be similar errata pushed for epel6? 

If not, please consider again going the easy way out, that is on epel6, to remove GeoIP.dat from the GeoIP package, and make the package depend on geoip-geolite.

Ingvar

GeoIP-1.5.1-5.el6 has been submitted as an update for Fedora EPEL 6.
https://admin.fedoraproject.org/updates/GeoIP-1.5.1-5.el6

If you're using EPEL (EL6), please give karma points to this build so I can move it from updates-testing to updates.  Thanks.

GeoIP-1.5.1-5.el6 has been pushed to the Fedora EPEL 6 stable repository.  If problems still persist, please make note of it in this bug report.