Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1174037

Summary: python 2.7.9 is FTBFS on aarch64: test_ctypes tests fail
Product: [Fedora] Fedora Reporter: Peter Robinson <pbrobinson>
Component: libffiAssignee: Anthony Green <green>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: bkabrda, c.david86, dmalcolm, fweimer, green, ivazqueznet, jonathansteffan, lkundrak, matti.picus, mjuszkie, mstuchli, ncoghlan, negativo17, rkuska, rth, tomspur, tradej
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libffi-3.1-7.fc22 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-01-29 17:43:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 922257, 1287815    
Attachments:
Description Flags
some check
none
fix to get aarch64 a bit further
none
Reproducer for libffi testsuite
none
libffi fix none

Description Peter Robinson 2014-12-15 00:42:36 UTC
http://arm.koji.fedoraproject.org/koji/taskinfo?taskID=2825638

----------------------------------------------------------------------
Ran 39 tests in 3.240s
OK
360 tests OK.
1 test failed:
    test_ctypes
39 tests skipped:
    test_aepack test_al test_applesingle test_bsddb185 test_bsddb3
    test_cd test_cl test_codecmaps_cn test_codecmaps_hk
    test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_curses
    test_dl test_gdb test_gl test_imgfile test_ioctl test_kqueue
    test_linuxaudiodev test_macos test_macostools test_msilib
    test_ossaudiodev test_pep277 test_scriptpackages test_smtpnet
    test_socketserver test_startfile test_sunaudiodev test_timeout
    test_tk test_ttk_guionly test_unicode_file test_urllib2net
    test_urllibnet test_winreg test_winsound test_zipfile64
2 skips unexpected on linux2:
    test_gdb test_ioctl
[1337952 refs]

Comment 1 Peter Robinson 2015-01-04 12:19:11 UTC
Any update Robert? This is now blocking all aarch64 builds

Comment 2 Robert Kuska 2015-01-05 08:50:24 UTC
I've tried to work on this before Christmas but I didn't get access to aarch64 machine with fedora image. I tried to contact several people (also #arm) without success. Any ideas?

Comment 3 Peter Robinson 2015-01-05 09:10:35 UTC
(In reply to Robert Kuska from comment #2)
> I've tried to work on this before Christmas but I didn't get access to
> aarch64 machine with fedora image. I tried to contact several people (also
> #arm) without success. Any ideas?

You didn't ask here or on #fedora-arm. Did you try beaker? There's aarch64 machines in there too

Comment 4 Dave Malcolm 2015-01-08 17:26:06 UTC
The error appears to be:

======================================================================
FAIL: test_struct_by_value (ctypes.test.test_win32.Structures)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/Python-2.7.9/Lib/ctypes/test/test_win32.py", line 112, in test_struct_by_value
    self.assertEqual(ret.left, left.value)
AssertionError: -200 != 10
----------------------------------------------------------------------
Ran 438 tests in 1.894s
FAILED (failures=1, skipped=89)

Comment 5 Dave Malcolm 2015-01-08 17:45:47 UTC
Notes:

(A) every other suite within test_win32.py is wrapped with:

    @unittest.skipUnless(sys.platform == "win32", 'Windows-specific test')

(B) the failing test is calling into _ctypes_test module, which is built from Python-2.7.9/Modules/_ctypes/_ctypes_test.c

Comment 6 Dave Malcolm 2015-01-08 17:51:03 UTC
Notes: the function is question has signature:
RECT ReturnRect(int i, RECT ar, RECT* br, POINT cp, RECT dr,
                RECT *er, POINT fp, RECT gr);

where:

typedef struct {
    long x;
    long y;
} POINT;

typedef struct {
    long left;
    long top;
    long right;
    long bottom;
} RECT;

mstuchli reports that gdb reports:
  (gdb) print sizeof(long)
  8
  (gdb) print sizeof(RECT)
  32

Comment 7 Matej Stuchlik 2015-01-09 11:17:10 UTC
Note:
The direct cause of ReturnRect returning the incorrect value is that POINT fp is {x = 4396722784816, y = 5773424} while is should be equal to {x = 15, y = 25}.

Comment 8 mattip 2015-01-09 13:31:11 UTC
ctypes test committer here. 
While this test is in the test_win32.py file, all platforms should successfully pass it, the failure would seem to indicate a possible problem with libffi and arm64. I constructed the test to overload the stack-heap argument passing mechanism on win64, does arm64 have a similar issue?

Comment 9 Robert Kuska 2015-01-09 20:41:30 UTC
Yes, it looks like the stack is corrupted.
Also http://bugs.python.org/issue11835 looks similiar but it's arm64 specific.

I have tried to rebuild python with the latest (upstream released) libffi but it didn't help.

Comment 10 Robert Kuska 2015-01-12 10:03:22 UTC
Note: I have just finished python build with bundled libffi with the same result.

Comment 11 Robert Kuska 2015-01-12 11:20:06 UTC
cc libffi maintainer

Anthony, would it be possible for you to help with debugging this issue?

Comment 12 Marcin Juszkiewicz 2015-01-13 11:12:00 UTC
Created attachment 979569 [details]
some check

Something is weird. Applied that patch on x86-64:

------------------------------------------------------------------
test test_ctypes produced unexpected output:
**********************************************************************
hrw1: 10 20 30 40 
**********************************************************************
------------------------------------------------------------------

But on aarch64:

------------------------------------------------------------------
test_struct_by_value (ctypes.test.test_win32.Structures) ... 
hrw1: 721554505738 42949672980 85899345950 128849018920 
FAIL
------------------------------------------------------------------

Looks like structs are given wrong on aarch64.

Comment 13 Marcin Juszkiewicz 2015-01-13 12:00:41 UTC
Created attachment 979585 [details]
fix to get aarch64 a bit further

With this patch applied at least values of c_long variables are correct.

Now the problem is with ReturnRect() calls.

Comment 14 Matej Stuchlik 2015-01-13 12:11:41 UTC
(In reply to Marcin Juszkiewicz from comment #13)
> Created attachment 979585 [details]
> fix to get aarch64 a bit further
> 
> With this patch applied at least values of c_long variables are correct.
> 
> Now the problem is with ReturnRect() calls.

Which Python are you testing this on, Marcin? The 2.7.9 in Fedora rawhide already includes this patch.

Comment 15 Marcin Juszkiewicz 2015-01-13 12:51:19 UTC
Matej: I played with upstream one as issue existed as well and was easier to hack around.

Comment 16 Marcin Juszkiewicz 2015-01-13 12:55:16 UTC
Heh. I was few commits before during my tests ;(

Comment 17 Anthony Green 2015-01-13 15:30:59 UTC
I can have a quick look at this this afternoon if somebody can point me at an aarch64 box.

Comment 18 Anthony Green 2015-01-13 20:20:57 UTC
Created attachment 979728 [details]
Reproducer for libffi testsuite

Here's a reproducer.  Drop it in libffi's libffi.call testsuite directory and run make check.  Passes on x86, but fails on aarch64.

Comment 19 Robert Kuska 2015-01-14 15:54:49 UTC
I have renamed the attachment as python_ctypes.c.

make check output:

FAIL: libffi.call/python_ctypes.c -W -Wall -Wno-psabi -O0 execution test
FAIL: libffi.call/python_ctypes.c -W -Wall -Wno-psabi -O2 execution test
FAIL: libffi.call/python_ctypes.c -W -Wall -Wno-psabi -O3 execution test
FAIL: libffi.call/python_ctypes.c -W -Wall -Wno-psabi -Os execution test
FAIL: libffi.call/python_ctypes.c -W -Wall -Wno-psabi -O2 -fomit-frame-pointer execution test


Great, thank you for your help. I am changing the component to libffi (hope you don't mind).

Comment 20 Richard Henderson 2015-01-14 19:05:08 UTC
Created attachment 980183 [details]
libffi fix

For any 1-16 byte structure not passed in registers,
we weren't copying from the object to be passed, but
from the array of objects.  To obvious ill effect.

Comment 21 Peter Robinson 2015-01-15 03:08:17 UTC
Patch added and built on rawhide (both mainline and aarch64), testing python build now