Some ideas on debugging:

1. Does any python script work?
2. Can you run 'python -m pdb /usr/bin/cnucnu'
   - enter 'c' to continue the execution at the Pdb-prompt
3. Can you strace and/or ltrace the /usr/bin/cnucnu

(In reply to comment #1)
> Some ideas on debugging:
> 
> 1. Does any python script work?

Some invocations work:
/usr/bin/cnucnu --help

Also yum worked at least enough to allow to install cnucnu. Installed packages are:
Jan 09 17:31:14 Installed: python-magic-5.11-4.fc18.armv5tel
Jan 09 17:31:15 Installed: python-bugzilla-0.7.0-2.fc18.noarch
Jan 09 17:31:19 Installed: pyOpenSSL-0.13-4.fc18.armv5tel
Jan 09 17:31:21 Installed: python-simplejson-2.6.0-2.fc18.armv5tel
Jan 09 17:31:22 Installed: python-zope-event-3.5.1-4.fc18.noarch
Jan 09 17:31:23 Installed: python-zope-interface-4.0.2-3.fc18.armv5tel
Jan 09 17:31:24 Installed: python-bunch-1.0.1-3.fc18.noarch
Jan 09 17:31:26 Installed: python-fedora-0.3.29-2.fc18.noarch
Jan 09 17:31:29 Installed: pyserial-2.6-3.fc18.noarch
Jan 09 17:31:41 Installed: python-babel-0.9.6-5.fc18.noarch
Jan 09 17:31:46 Installed: m2crypto-0.21.1-9.fc18.armv5tel
Jan 09 17:31:47 Installed: python-fpconst-0.7.3-10.fc18.noarch
Jan 09 17:32:17 Installed: python-twisted-core-12.1.0-2.fc18.armv5tel
Jan 09 17:32:19 Installed: SOAPpy-0.11.6-15.fc18.noarch
Jan 09 17:32:26 Installed: python-twisted-web-12.1.0-2.fc18.armv5tel
Jan 09 17:32:39 Installed: python-genshi-0.6-4.fc17.armv5tel
Jan 09 17:32:43 Installed: libyaml-0.1.4-3.fc18.armv5tel
Jan 09 17:32:44 Installed: PyYAML-3.10-6.fc18.armv5tel
Jan 09 17:32:45 Installed: cnucnu-0-0.11.20121004git618ed580.fc18.noarch


> 2. Can you run 'python -m pdb /usr/bin/cnucnu'
>    - enter 'c' to continue the execution at the Pdb-prompt

python -m pdb /usr/bin/yum
> /usr/bin/yum(2)<module>()
-> import sys
(Pdb) c
Ungültiger Maschinenbefehl

# python -m pdb /usr/bin/cnucnu report-outdated                                                                                                                                       
> /usr/bin/cnucnu(20)<module>()
-> import logging
(Pdb) c
Ungültiger Maschinenbefehl

Just running "import sys" in an interactive python prompt worked without problems.

> 3. Can you strace and/or ltrace the /usr/bin/cnucnu

The system just crashed completly during tests, I will see if the programs are installed after it rebooted.

Actually rpm does not work as well:
not working:
rpm -qa
rpm -Uhv strace-4.7-2.fc18.armv5tel.rpm

working:
rpm --help

There is no trace command that I could use to debug further.

Unpack the rpm with rpm2cpio and run the command there? (Or unpack it on a separate box and scp it and/or gdb over).

There does not seem to exist ltrace for arm:
http://ftp-stud.hs-esslingen.de/pub/fedora-secondary/development/18/arm/os/Packages/l/

strace for rpm -ql:
munmap(0xb6fe9000, 27378)               = 0
open("/pkcs11.txt", O_RDONLY)           = -1 ENOENT (No such file or directory)
--- SIGILL {si_signo=SIGILL, si_code=ILL_ILLOPC, si_addr=0x49327724} ---
+++ killed by SIGILL +++
Illegal instruction


strace for cnucnu:
brk(0xb75000)                           = 0xb75000
gettimeofday({1357856683, 804006}, NULL) = 0
--- SIGILL {si_signo=SIGILL, si_code=ILL_ILLOPC, si_addr=0xb6b4b700} ---
+++ killed by SIGILL +++
Illegal instruction

Created attachment 676573 [details]
core dump from running rpm -ql

At that low a level... throwing at glibc.

My best guess is an architecture mismatch between the built files and your hardware. Alternatively you have flaky DRAM.

I started to look into this, but the easiest way is to use an x86/x86_64 cross-gdb to load the core file and inspect the instruction at the time of failure. Unfortunately from FC17 x86_64 I don't see a cross-gdb (just cross gcc, and binutils). We need a cross-gdb.

The next easiest thing is for me to just run a simulated FC18 ARM environment. I've started downloading the FC18 Versatile (QEMU) image, but I figured I might as well ask some questions.

(1) Can someone with a cross-gdb or native ARM gdb determine what the faulting instruction was?

(2) Run `readelf -a /usr/bin/rpm` and attach it. I'm most interested in the GNU attributes section of the output (at the end) since it describes what ARM architecture the file was built for.

(3) Run `ldd /usr/bin/rpm` and provide the output.

(4) Run `objdump -ldr /usr/bin/rpm >& dump.asm` and attach dump.asm.

Thanks.

In addition:

(5) Run `cat /proc/cpuinfo` and provide the output.

(6) Run `uname -a` and provide the output.

(7) Did you build and boot a custom kernel for this system? If yes, provide the .config.

Thanks.

Core was generated by `rpm -ql'.
Program terminated with signal 4, Illegal instruction.
#0  pt_TestAbort () at ../../../mozilla/nsprpub/pr/src/pthreads/ptio.c:1219
1219            PR_SetError(PR_PENDING_INTERRUPT_ERROR, 0);
(gdb) l
1214    static PRBool pt_TestAbort(void)
1215    {
1216        PRThread *me = PR_GetCurrentThread();
1217        if(_PT_THREAD_INTERRUPTED(me))
1218        {
1219            PR_SetError(PR_PENDING_INTERRUPT_ERROR, 0);
1220            me->state &= ~PT_THREAD_ABORTED;
1221            return PR_TRUE;
1222        }
1223        return PR_FALSE;
(gdb) disassemble 
Dump of assembler code for function pt_TestAbort:
   0x49327700 <+0>:     push    {r4, lr}
   0x49327704 <+4>:     bl      0x4930f8e4
   0x49327708 <+8>:     ldr     r1, [r0, #3279928]      ; 0xa8
   0x4932770c <+12>:    mov     r4, r0
   0x49327710 <+16>:    cmp     r1, #0
   0x49327714 <+20>:    bne     0x49327740 <pt_TestAbort+64>
   0x49327718 <+24>:    ldr     r0, [r0]
   0x4932771c <+28>:    ands    r0, r0, #16
   0x49327720 <+32>:    popeq   {r4, pc}
=> 0x49327724 <+36>:    ldr     r0, [pc, #40882336]     ; 0x49327748 <pt_TestAbort+72>
   0x49327728 <+40>:    bl      0x49310034
   0x4932772c <+44>:    ldr     r3, [r4]
   0x49327730 <+48>:    mov     r0, #1
   0x49327734 <+52>:    bic     r3, r3, #16
   0x49327738 <+56>:    str     r3, [r4]
   0x4932773c <+60>:    pop     {r4, pc}
   0x49327740 <+64>:    mov     r0, #0
   0x49327744 <+68>:    pop     {r4, pc}
   0x49327748 <+72>:                    ; <UNDEFINED> instruction: 0xffffe897
End of assembler dump.

Created attachment 679810 [details]
readelf -a /usr/bin/rpm

# ldd /usr/bin/rpm
        librpm.so.3 => /lib/librpm.so.3 (0xb6f2a000)
        librpmio.so.3 => /lib/librpmio.so.3 (0xb6eff000)
        libselinux.so.1 => /lib/libselinux.so.1 (0xb6ed9000)
        libcap.so.2 => /lib/libcap.so.2 (0xb6ecd000)
        libacl.so.1 => /lib/libacl.so.1 (0xb6ebe000)
        libdb-5.3.so => /lib/libdb-5.3.so (0xb6d4b000)
        libbz2.so.1 => /lib/libbz2.so.1 (0xb6d31000)
        libelf.so.1 => /lib/libelf.so.1 (0xb6d12000)
        liblzma.so.5 => /lib/liblzma.so.5 (0xb6ce9000)
        liblua-5.1.so => /lib/liblua-5.1.so (0xb6cbd000)
        libm.so.6 => /lib/libm.so.6 (0xb6c12000)
        libnss3.so => /lib/libnss3.so (0xb6b15000)
        libpopt.so.0 => /lib/libpopt.so.0 (0xb6b04000)
        libz.so.1 => /lib/libz.so.1 (0xb6ae8000)
        libdl.so.2 => /lib/libdl.so.2 (0xb6adb000)
        libpthread.so.0 => /lib/libpthread.so.0 (0xb6abb000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xb6a94000)
        libc.so.6 => /lib/libc.so.6 (0xb694d000)
        /lib/ld-linux.so.3 (0xb6f89000)
        libpcre.so.1 => /lib/libpcre.so.1 (0xb68ef000)
        libattr.so.1 => /lib/libattr.so.1 (0xb68e2000)
        librt.so.1 => /lib/librt.so.1 (0xb68d2000)
        libnssutil3.so => /lib/libnssutil3.so (0xb68ab000)
        libplc4.so => /lib/libplc4.so (0xb689f000)
        libplds4.so => /lib/libplds4.so (0xb6894000)
        libnspr4.so => /lib/libnspr4.so (0xb685a000)

Created attachment 679813 [details]
objdump -ldr /usr/bin/rpm >& dump.asm

I've gathered these files and output from a Fedora-18-Beta-kirkwood-arm chroot on
a armv7hl. The Kirkwood machines are armv5tel.

Till, can you provide the info that was requested in comment #9?

(In reply to comment #10)
> Core was generated by `rpm -ql'.
> Program terminated with signal 4, Illegal instruction.
> #0  pt_TestAbort () at ../../../mozilla/nsprpub/pr/src/pthreads/ptio.c:1219
> 1219            PR_SetError(PR_PENDING_INTERRUPT_ERROR, 0);
> (gdb) l
> 1214    static PRBool pt_TestAbort(void)
> 1215    {
> 1216        PRThread *me = PR_GetCurrentThread();
> 1217        if(_PT_THREAD_INTERRUPTED(me))
> 1218        {
> 1219            PR_SetError(PR_PENDING_INTERRUPT_ERROR, 0);
> 1220            me->state &= ~PT_THREAD_ABORTED;
> 1221            return PR_TRUE;
> 1222        }
> 1223        return PR_FALSE;
> (gdb) disassemble 
> Dump of assembler code for function pt_TestAbort:
>    0x49327700 <+0>:     push    {r4, lr}
>    0x49327704 <+4>:     bl      0x4930f8e4
>    0x49327708 <+8>:     ldr     r1, [r0, #3279928]      ; 0xa8

The assembler or gdb has incorrectly assembled or disassembled a immediate-offset register relative load instruction.

>    0x4932770c <+12>:    mov     r4, r0
>    0x49327710 <+16>:    cmp     r1, #0
>    0x49327714 <+20>:    bne     0x49327740 <pt_TestAbort+64>
>    0x49327718 <+24>:    ldr     r0, [r0]
>    0x4932771c <+28>:    ands    r0, r0, #16
>    0x49327720 <+32>:    popeq   {r4, pc}
> => 0x49327724 <+36>:    ldr     r0, [pc, #40882336]     ; 0x49327748
> <pt_TestAbort+72>

This is a PC-relative load, but the the offset of the PC-relative load is too big. The offset should be *very* small and no more than ~4K at most. The "illegal instruction" is because this is likely an invalid encoding. This is starting to look like a binutils bug.

>    0x49327728 <+40>:    bl      0x49310034
>    0x4932772c <+44>:    ldr     r3, [r4]
>    0x49327730 <+48>:    mov     r0, #1
>    0x49327734 <+52>:    bic     r3, r3, #16
>    0x49327738 <+56>:    str     r3, [r4]
>    0x4932773c <+60>:    pop     {r4, pc}
>    0x49327740 <+64>:    mov     r0, #0
>    0x49327744 <+68>:    pop     {r4, pc}
>    0x49327748 <+72>:                    ; <UNDEFINED> instruction: 0xffffe897
> End of assembler dump.

The PC-relative load is trying to load the value in the constant pool at the *end* of the function e.g. 0xffffe897, but instead it looks like the assembler encoded the value incorrectly or gdb disassembled it incorrectly.

Can you get an `objdump -ldr <file>` of the dynamic library that contains the function `pt_TestAbort'? I want to see what the bfd disassembler says about the function above and compare it to gdb.

Notes:
- Unfortunately the binutils used to build /usr/bin/rpm didn't use gnu attributes to mark the architectures supported by the binary.

Created attachment 680127 [details]
output of "objdump -ldr /lib/libnspr4.so", the library that I expect to have pt_TestAbort inlined

Hmm, the only files that reference pt_TestAbort are:
- /lib/debug/usr/lib/libnspr4.so.debug
- /usr/src/debug/nspr-4.9.2/mozilla/nsprpub/pr/src/pthreads/ptio.c

My guess is that the calling of pt_TestAbort has been optimized and pt_TestAbort
was inlined. Trying to figure out what called pt_TestAbort results in this:

(gdb) bt
#0  pt_TestAbort () at ../../../mozilla/nsprpub/pr/src/pthreads/ptio.c:1219
#1  0x01135900 in ?? ()
#2  0x01135900 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

I can not reproduce the "Illegal Instruction" error (using a chroot on a
different ARM architecture), but setting a gdb-breakpoiint on pr_TestAbort
results in the following backtrace:

Breakpoint 1, pt_TestAbort () at ../../../mozilla/nsprpub/pr/src/pthreads/ptio.c:1215
1215    {
(gdb) bt
#0  pt_TestAbort () at ../../../mozilla/nsprpub/pr/src/pthreads/ptio.c:1215
#1  0xb68d1614 in PR_Access (name=name@entry=0x18700 "/secmod.db", how=how@entry=PR_ACCESS_EXISTS) at ../../../mozilla/nsprpub/pr/src/pthreads/ptio.c:3611
#2  0xb691187c in nssutil_ReadSecmodDB (filename=0x17268 "secmod.db", dbname=0xb6ffeeb8 <__stack_chk_guard> "", dbname@entry=0x186a8 "/pkcs11.txt", 
    params=0xa <Address 0xa out of bounds>, 
    params@entry=0x17ed0 "configdir='' certPrefix='' keyPrefix='' secmod='' flags=readOnly,noCertDB,noModDB,forceOpen,optimizeSpace updatedir='' updateCertPrefix='' updateKeyPrefix='' updateid='' updateTokenDescription='' ", rw=-1090523340, appName=<optimized out>, dbType=<optimized out>) at utilmod.c:358
#3  0xb69119f8 in NSSUTIL_DoModuleDBFunction (function=function@entry=0, 
    parameters=parameters@entry=0x17ed0 "configdir='' certPrefix='' keyPrefix='' secmod='' flags=readOnly,noCertDB,noModDB,forceOpen,optimizeSpace updatedir='' updateCertPrefix='' updateKeyPrefix='' updateid='' updateTokenDescription='' ", args=args@entry=0x0) at utilmod.c:669
#4  0xb6530ce8 in NSC_ModuleDBFunc (function=0, 
    parameters=0x17ed0 "configdir='' certPrefix='' keyPrefix='' secmod='' flags=readOnly,noCertDB,noModDB,forceOpen,optimizeSpace updatedir='' updateCertPrefix='' updateKeyPrefix='' updateid='' updateTokenDescription='' ", args=0x0) at pkcs11.c:2652
#5  0xb6ba6b1c in SECMOD_GetModuleSpecList (module=module@entry=0x17e48) at pk11pars.c:914
#6  0xb6ba6da8 in SECMOD_LoadModule (
    modulespec=modulespec@entry=0x17278 "name=\"NSS Internal Module\" parameters=\"configdir='' certPrefix='' keyPrefix='' secmod='' flags=readOnly,noCertDB,noModDB,forceOpen,optimizeSpace updatedir='' updateCertPrefix='' updateKeyPrefix='' upd"..., parent=parent@entry=0x0, recurse=recurse@entry=1) at pk11pars.c:1028
#7  0xb6b7b540 in nss_InitModules (isContextInit=0, optimizeSpace=1, forceOpen=1, noModDB=1, noCertDB=0, readOnly=0, pwRequired=<optimized out>, configStrings=0x0, 
    configName=0xb6c4f02c "", updateName=0xb6c4f02c "", updateID=0xb6c4f02c "", updKeyPrefix=0xb6c4f02c "", 
    updCertPrefix=0xb6b7bd70 <NSS_NoDB_Init+104> "D", <incomplete sequence \342>, updateDir=0x0, secmodName=0x17238 "Hr\001", keyPrefix=<optimized out>, 
    certPrefix=<optimized out>, configdir=0xb6c4f02c "") at nssinit.c:438
#8  nss_Init (configdir=0xb6c4f02c "", certPrefix=<optimized out>, keyPrefix=<optimized out>, secmodName=0x17238 "Hr\001", updateDir=0xb6c4f02c "", 
    updCertPrefix=0xb6c4f02c "", updKeyPrefix=0xb6c4f02c "", updateID=0xb6c4f02c "", updateName=0xb6c4f02c "", initContextPtr=initContextPtr@entry=0x0, 
    initParams=initParams@entry=0x0, readOnly=readOnly@entry=1, noCertDB=noCertDB@entry=1, noModDB=noModDB@entry=1, forceOpen=forceOpen@entry=1, 
    noRootInit=noRootInit@entry=1, optimizeSpace=optimizeSpace@entry=1, noSingleThreadedModules=noSingleThreadedModules@entry=0, 
    allowAlreadyInitializedModules=allowAlreadyInitializedModules@entry=0, dontFinalizeModules=dontFinalizeModules@entry=0) at nssinit.c:643
#9  0xb6b7bd70 in NSS_NoDB_Init (configdir=configdir@entry=0x0) at nssinit.c:878
#10 0xb6f63718 in rpmInitCrypto () at digest_nss.c:46
#11 0xb6fafc50 in rpmReadConfigFiles (file=0x0, target=0x1 <Address 0x1 out of bounds>, target@entry=0x0) at rpmrc.c:1617
#12 0xb6fa00b4 in rpmcliConfigured () at poptALL.c:66
#13 0xb6fa05ac in rpmcliInit (argc=argc@entry=2, argv=0xbefff814, optionsTable=<optimized out>) at poptALL.c:290
#14 0x00008fb4 in main (argc=2, argv=<optimized out>) at rpmqv.c:91

PR_Access is available in libnspr4.so, attaching its "objdump -ldr".

(In reply to comment #8)
> My best guess is an architecture mismatch between the built files and your
> hardware. Alternatively you have flaky DRAM.

I tried to reproduce this on a second Dockstar with a new image. There the error did not occur during initial testing but I then broke the initramfs and did not finish with testing, because I ran out of time. I also did not get to try to reproduce the error on the Dockstar that showed the error initially.

(In reply to comment #9)
> In addition:
> 
> (5) Run `cat /proc/cpuinfo` and provide the output.
> 
> (6) Run `uname -a` and provide the output.
> 
> (7) Did you build and boot a custom kernel for this system? If yes, provide
> the .config.

I do not have the original USB image anymore, but it was the standard default kirkwood image with the default kernel. I can provide the cpuinfo information after I got one Dockstar to boot again..

The "objdump -ldr /lib/libnspr4.so" contains the sequence that leads up to the crash in the function:
0001f4b8 <PRP_NakedBroadcast>:
   ...
   1f700:	e92d4010 	push	{r4, lr}
   1f704:	ebffa076 	bl	78e4 <_init+0x308>
   1f708:	e59010a8 	ldr	r1, [r0, #168]	; 0xa8
   1f70c:	e1a04000 	mov	r4, r0
   1f710:	e3510000 	cmp	r1, #0
   1f714:	1a000009 	bne	1f740 <PRP_NakedBroadcast+0x288>
   1f718:	e5900000 	ldr	r0, [r0]
   1f71c:	e2100010 	ands	r0, r0, #16
   1f720:	08bd8010 	popeq	{r4, pc}
   1f724:	e59f001c 	ldr	r0, [pc, #28]	; 1f748 <PRP_NakedBroadcast+0x290>
   1f728:	ebffa241 	bl	8034 <_init+0xa58>
   1f72c:	e5943000 	ldr	r3, [r4]
   1f730:	e3a00001 	mov	r0, #1
   1f734:	e3c33010 	bic	r3, r3, #16
   1f738:	e5843000 	str	r3, [r4]
   1f73c:	e8bd8010 	pop	{r4, pc}
   1f740:	e3a00000 	mov	r0, #0
   1f744:	e8bd8010 	pop	{r4, pc}
   1f748:	ffffe897 			; <UNDEFINED> instruction: 0xffffe897
   ...

This is the only occurrence of that sequence.

You'll note that PR_Access does call PRP_NakedBoardcast.

Here the offsets are reasonable, and the load address is sufficiently (8-byte) aligned. It might just have been a defect in gdb that the target of the load is printed in the disassembly instead of the actual offset (as it should be).

There is no reason for this instruction fault the CPU.

Can I get a confirmation from the reporter that this happens consistently?

Were there multiple threads present? `info threads' from gdb after loading the core?

This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

This message is a reminder that Fedora 18 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 18. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '18'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 18's end of life.

Thank you for reporting this issue and we are sorry that we may not be 
able to fix it before Fedora 18 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior to Fedora 18's end of life.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.