1166321 – generating rhel-x86_64-server-5 repodata: java.lang.OutOfMemoryError: GC overhead limit exceeded

Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1166321 - generating rhel-x86_64-server-5 repodata: java.lang.OutOfMemoryError: GC overhead limit exceeded

Summary: generating rhel-x86_64-server-5 repodata: java.lang.OutOfMemoryError: GC over...

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	None
Product:	Spacewalk
Classification:	Community
Component:	Server
Sub Component:
Version:	2.2
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	---
Assignee:	Stephen Herr
QA Contact:	Red Hat Satellite QA List
Docs Contact:
URL:
Whiteboard:
Depends On:	1132398
Blocks:	space23
TreeView+	depends on / blocked

Reported:	2014-11-20 20:06 UTC by Stephen Herr
Modified:	2015-04-14 19:03 UTC (History)
CC List:	11 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:	1132398
Environment:
Last Closed:	2015-04-14 19:03:26 UTC
Embargoed:

Attachments	(Terms of Use)

Description Stephen Herr 2014-11-20 20:06:25 UTC

+++ This bug was initially created as a clone of Bug #1132398 +++

Description of problem:
Generating rhel-x86_64-server-5 repodata causes "java.lang.OutOfMemoryError: GC overhead limit exceeded"


Version-Release number of selected component (if applicable):
Red Hat Satellite 5.6.0


How reproducible:
sometimes


Steps to Reproduce:
1. Sync rhel-x86_64-server-6 and rhel-x86_64-server-5 channels
   (I had a bunch of other channels on the system as well)


Actual results:
When watching /var/log/rhn/rhn_taskomatic_daemon.log, Taskomatic failed with "java.lang.OutOfMemoryError: GC overhead limit exceeded" while generating repodata.


Expected results:
Taskomatic should be able to generate repodata for RHEL5 and RHEL6 channels in default setup.


Additional info:
When I have followed https://access.redhat.com/solutions/43122, Taskomatic was able to generate repodata.

INFO   | jvm 1    | 2014/08/20 05:19:45 | 2014-08-20 05:19:45,262 [Thread-42] WARN  com.redhat.rhn.taskomatic.core.SchedulerKernel - Number of interrupted runs: 1
INFO   | jvm 1    | 2014/08/20 05:19:47 | 2014-08-20 05:19:47,627 [DefaultQuartzScheduler_Worker-3] INFO  com.redhat.rhn.taskomatic.task.ChannelRepodata - Resetting 2199 unfinished channel repodata tasks
INFO   | jvm 1    | 2014/08/20 05:19:47 | 2014-08-20 05:19:47,899 [DefaultQuartzScheduler_Worker-3] INFO  com.redhat.rhn.taskomatic.task.ChannelRepodata - In the queue: 2
INFO   | jvm 1    | 2014/08/20 05:19:49 | 2014-08-20 05:19:49,451 [Thread-53] INFO  com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - File Modified Date:1969-12-31 19:00:00 EST
INFO   | jvm 1    | 2014/08/20 05:19:49 | 2014-08-20 05:19:49,451 [Thread-53] INFO  com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Channel Modified Date:2014-08-19 08:53:57 EDT
INFO   | jvm 1    | 2014/08/20 05:19:56 | 2014-08-20 05:19:56,254 [Thread-53] INFO  com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Generating new repository metadata for channel 'rhel-x86_64-server-6'(sha256) 12831 packages, 2442 errata
INFO   | jvm 1    | 2014/08/20 05:21:14 | Exception in thread "Thread-53" 
INFO   | jvm 1    | 2014/08/20 05:21:15 | java.lang.OutOfMemoryError: GC overhead limit exceeded
INFO   | jvm 1    | 2014/08/20 05:21:15 | 	at org.postgresql.util.PGbytea.toBytes(PGbytea.java:91)
INFO   | jvm 1    | 2014/08/20 05:21:15 | 	at org.postgresql.jdbc2.AbstractJdbc2ResultSet.getBytes(AbstractJdbc2ResultSet.java:2271)
INFO   | jvm 1    | 2014/08/20 05:21:15 | 	at org.postgresql.jdbc2.AbstractJdbc2ResultSet.internalGetObject(AbstractJdbc2ResultSet.java:150)
INFO   | jvm 1    | 2014/08/20 05:21:15 | 	at org.postgresql.jdbc3.AbstractJdbc3ResultSet.internalGetObject(AbstractJdbc3ResultSet.java:38)
INFO   | jvm 1    | 2014/08/20 05:21:15 | 	at org.postgresql.jdbc4.AbstractJdbc4ResultSet.internalGetObject(AbstractJdbc4ResultSet.java:298)
INFO   | jvm 1    | 2014/08/20 05:21:15 | 	at org.postgresql.jdbc2.AbstractJdbc2ResultSet.getObject(AbstractJdbc2ResultSet.java:2541)
INFO   | jvm 1    | 2014/08/20 05:21:15 | 	at org.postgresql.jdbc2.AbstractJdbc2ResultSet.getObject(AbstractJdbc2ResultSet.java:2550)
INFO   | jvm 1    | 2014/08/20 05:21:15 | 	at com.mchange.v2.c3p0.impl.NewProxyResultSet.getObject(NewProxyResultSet.java:1102)
INFO   | jvm 1    | 2014/08/20 05:21:15 | 	at com.redhat.rhn.common.db.datasource.CachedStatement.getObject(CachedStatement.java:793)
INFO   | jvm 1    | 2014/08/20 05:21:15 | 	at com.redhat.rhn.common.db.datasource.CachedStatement.addToObject(CachedStatement.java:776)
INFO   | jvm 1    | 2014/08/20 05:21:15 | 	at com.redhat.rhn.common.db.datasource.CachedStatement.processResultSet(CachedStatement.java:626)
INFO   | jvm 1    | 2014/08/20 05:21:15 | 	at com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:474)
INFO   | jvm 1    | 2014/08/20 05:21:15 | 	at com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:443)
INFO   | jvm 1    | 2014/08/20 05:21:15 | 	at com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:345)
INFO   | jvm 1    | 2014/08/20 05:21:15 | 	at com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:351)
INFO   | jvm 1    | 2014/08/20 05:21:15 | 	at com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:287)
INFO   | jvm 1    | 2014/08/20 05:21:15 | 	at com.redhat.rhn.common.db.datasource.SelectMode.execute(SelectMode.java:110)
INFO   | jvm 1    | 2014/08/20 05:21:15 | 	at com.redhat.rhn.manager.task.TaskManager.getChannelPackageDtos(TaskManager.java:52)
INFO   | jvm 1    | 2014/08/20 05:21:15 | 	at com.redhat.rhn.taskomatic.task.repomd.RpmRepositoryWriter.writeRepomdFiles(RpmRepositoryWriter.java:170)
INFO   | jvm 1    | 2014/08/20 05:21:15 | 	at com.redhat.rhn.taskomatic.task.repomd.ChannelRepodataWorker.run(ChannelRepodataWorker.java:104)
INFO   | jvm 1    | 2014/08/20 05:21:15 | 	at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:761)
INFO   | jvm 1    | 2014/08/20 05:21:15 | 	at java.lang.Thread.run(Thread.java:745)
INFO   | jvm 1    | 2014/08/20 05:21:15 | 2014-08-20 05:21:15,366 [Thread-60] INFO  com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - File Modified Date:1969-12-31 19:00:00 EST
INFO   | jvm 1    | 2014/08/20 05:21:15 | 2014-08-20 05:21:15,366 [Thread-60] INFO  com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Channel Modified Date:2014-08-18 09:24:30 EDT
INFO   | jvm 1    | 2014/08/20 05:21:18 | 2014-08-20 05:21:18,254 [Thread-60] INFO  com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Generating new repository metadata for channel 'rhel-x86_64-server-5'(sha1) 16133 packages, 2941 errata
INFO   | jvm 1    | 2014/08/20 05:22:11 | Exception in thread "Thread-60" java.lang.OutOfMemoryError: GC overhead limit exceeded
INFO   | jvm 1    | 2014/08/20 05:22:11 | 	at java.lang.reflect.Method.copy(Method.java:151)
INFO   | jvm 1    | 2014/08/20 05:22:11 | 	at java.lang.reflect.ReflectAccess.copyMethod(ReflectAccess.java:136)
INFO   | jvm 1    | 2014/08/20 05:22:11 | 	at sun.reflect.ReflectionFactory.copyMethod(ReflectionFactory.java:300)
INFO   | jvm 1    | 2014/08/20 05:22:11 | 	at java.lang.Class.copyMethods(Class.java:2891)
INFO   | jvm 1    | 2014/08/20 05:22:11 | 	at java.lang.Class.getMethods(Class.java:1467)
INFO   | jvm 1    | 2014/08/20 05:22:11 | 	at com.redhat.rhn.common.db.datasource.CachedStatement.addToObject(CachedStatement.java:739)
INFO   | jvm 1    | 2014/08/20 05:22:11 | 	at com.redhat.rhn.common.db.datasource.CachedStatement.processResultSet(CachedStatement.java:626)
INFO   | jvm 1    | 2014/08/20 05:22:11 | 	at com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:474)
INFO   | jvm 1    | 2014/08/20 05:22:11 | 	at com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:443)
INFO   | jvm 1    | 2014/08/20 05:22:11 | 	at com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:345)
INFO   | jvm 1    | 2014/08/20 05:22:11 | 	at com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:351)
INFO   | jvm 1    | 2014/08/20 05:22:11 | 	at com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:287)
INFO   | jvm 1    | 2014/08/20 05:22:11 | 	at com.redhat.rhn.common.db.datasource.SelectMode.execute(SelectMode.java:110)
INFO   | jvm 1    | 2014/08/20 05:22:11 | 	at com.redhat.rhn.manager.task.TaskManager.getChannelPackageDtos(TaskManager.java:52)
INFO   | jvm 1    | 2014/08/20 05:22:11 | 	at com.redhat.rhn.taskomatic.task.repomd.RpmRepositoryWriter.writeRepomdFiles(RpmRepositoryWriter.java:170)
INFO   | jvm 1    | 2014/08/20 05:22:11 | 	at com.redhat.rhn.taskomatic.task.repomd.ChannelRepodataWorker.run(ChannelRepodataWorker.java:104)
INFO   | jvm 1    | 2014/08/20 05:22:11 | 	at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
INFO   | jvm 1    | 2014/08/20 05:22:11 | 	at java.lang.Thread.run(Thread.java:745)

--- Additional comment from Stephen Herr on 2014-08-27 13:52:41 EDT ---

In the process of fixing this I also discovered (and fixed) another problem:
In the updateinfo.xml.gz files the href entry for bugs in an erratum will be empty if the erratum doesn't have a url stored in the db. The correct behavior is for us to generate a bugzilla url based on the bug id.

Example:
Old (broken) xml:
<update from="" status="final" type="bugfix" version="1"><id>CL-BA-2013:0408</id><title>cifs-utils bug fix and enhancement update</title>...<references><reference href="" id="838606" type="bugzilla">enable PIE and RELRO in cifs-utils binaries</reference></references>...</update>

New (correct) xml:
<update from="" status="final" type="bugfix" version="1"><id>CL-BA-2013:0408</id><title>cifs-utils bug fix and enhancement update</title>...<references><reference href="http://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=838606" id="838606" type="bugzilla">enable PIE and RELRO in cifs-utils binaries</reference></references>...</update>

--- Additional comment from Stephen Herr on 2014-08-28 15:23:17 EDT ---

Committing to spacewalk master:
877f330803e62c7327bb852d9aee44f021ec9ab9

The new updateinfo.xml.gz will be different both because of the fix for 1135073 and because of a different ordering of the errata (hibernate's old ordering was non-specific, so I can't really recreate it), but in other respects the repomd files *should* be identical.

--- Additional comment from Stephen Herr on 2014-08-28 15:25:56 EDT ---

Oh, I forgot to mention what this fix actually does.

Now instead of loading all packages into memory to write the repodata we only load 1000 at a time, so taskomatic's required memory size should no longer grow with channel size. I can successfully regenerate the repodata for rhel-x86_64-server-5 with taskomatic given a maxmemory of 512.

--- Additional comment from Stephen Herr on 2014-10-13 15:34:16 EDT ---

Also needed (for debian repos), committing to Spacewalk master:
ef2946bcbc92012fbd8ea31368615d178bedb462

Comment 1 Grant Gainey 2015-03-23 16:59:14 UTC

Moving bugs to ON_QA as we move to release Spacewalk 2.3

Comment 2 Grant Gainey 2015-04-14 19:03:26 UTC

Spacewalk 2.3 has been released. See

https://fedorahosted.org/spacewalk/wiki/ReleaseNotes23

Note You need to log in before you can comment on or make changes to this bug.