Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1458960

Summary: aarch64 firefox-53.0.3-2 crashes due to unsaved x28 in baseline JIT code
Product: [Fedora] Fedora Reporter: Jeremy Linton <jeremy.linton>
Component: firefoxAssignee: Jan Horak <jhorak>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 26CC: gecko-bugs-nobody, jhorak, kengert, pjasicek, stransky
Target Milestone: ---   
Target Release: ---   
Hardware: aarch64   
OS: Linux   
URL: https://bugzilla.mozilla.org/show_bug.cgi?id=1375074
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-05-03 11:59:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 245418, 1564204    
Attachments:
Description Flags
fedora build fix for aarch64 gcc7 register spilling none

Description Jeremy Linton 2017-06-05 23:40:27 UTC
Description of problem: Firefox is crashing again. This time with:

[Child 2343] WARNING: pipe error (25): Connection reset by peer: file /builddir/build/BUILD/firefox-53.0.3/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 346
Crash Annotation GraphicsCriticalError: |[C0][GFX1-]: Receive IPC close with reason=AbnormalShutdown (t=7.21357) Segmentation fault (core dumped)


Version-Release number of selected component (if applicable): 53.0.3-2


How reproducible: 100% of the time on aarch64


Steps to Reproduce:
1. start firefox
2. wait about 5 seconds.
3.

Actual results:

[  590.442657] Chrome_ChildThr[2445]: unhandled level 0 translation fault (11) at 0x00000000, esr 0x92000044
[  590.453164] pgd = ffff80034278d000
[  590.457711] [00000000] *pgd=0000000000000000

[  590.465193] CPU: 0 PID: 2445 Comm: Chrome_ChildThr Tainted: G        W I     4.11.0-2.fc26.aarch64 #1
[  590.475278] Hardware name: AMD Overdrive/Supercharger/Default string, BIOS ROD1002C 04/08/2016
[  590.484751] task: ffff80034295d700 task.stack: ffff800342a24000
[  590.491483] PC is at 0xffff92a59e54
[  590.495814] LR is at 0xffff92a59c44
[  590.500129] pc : [<0000ffff92a59e54>] lr : [<0000ffff92a59c44>] pstate: 80000000
[  590.508377] sp : 0000ffff892fe5e0
[  590.512523] x29: 0000ffff892fe9d0 x28: 0000ffff98b9a700 
[  590.518648] x27: 0000ffff893641a0 x26: 00000000000f4240 
[  590.524785] x25: 0000ffff9844b950 x24: 0000000000000000 
[  590.530927] x23: 0000ffff9650f0d0 x22: 0000ffff984f8058 
[  590.537055] x21: 0000ffff964d1000 x20: 0000ffff964d1910 
[  590.543160] x19: 0000ffff984c8100 x18: 0000000000000000 
[  590.549240] x17: 0000ffff98b51e50 x16: 0000ffff964cc208 
[  590.555322] x15: 00000d51b2ae6aa8 x14: 001a18d51781bc2a 
[  590.561402] x13: 00000001f4000000 x12: 0000000000000017 
[  590.567485] x11: 0000ffff98500541 x10: 0000ffff98bb7c68 
[  590.573569] x9 : 0000ffff98500540 x8 : 0000000000000039 
[  590.579631] x7 : 0000ffff98500540 x6 : 0000000000000000 
[  590.585695] x5 : 0000ffff984f8118 x4 : 0000ffff984b1000 
[  590.591757] x3 : 00000000000008a1 x2 : 0000000000000000 
[  590.597837] x1 : 0000ffff953c2e88 x0 : 0000aaaab6493110 



Expected results:
It runs..

Additional info:
I will attach a better backtrace tomorrow, abrt is refusing to upload it because it says that the backtrace is bad.

Comment 1 Jeremy Linton 2017-06-05 23:49:12 UTC
Hmm odd, I force installed pbrobinsons koji build https://koji.fedoraproject.org/koji/taskinfo?taskID=19707766 of 53.0.2 which was working, and now that is crashing.

Comment 2 Jeremy Linton 2017-06-07 22:11:21 UTC
Built a local version and here is the backtrace

Thread 1 "firefox" received signal SIGSEGV, Segmentation fault.
0x0000ffffb4202094 in js::detail::HashTable<js::HashMapEntry<js::CrossCompartmentKey, js::detail::UnsafeBareReadBarriered<JS::Value> >, js::HashMap<js::CrossCompartmentKey, js::detail::UnsafeBareReadBarriered<JS::Value>, js::CrossCompartmentKey::Hasher, js::SystemAllocPolicy>::MapHashPolicy, js::SystemAllocPolicy>::lookup (collisionBit=0, keyHash=2402884294, l=..., this=<optimized out>)
    at /root/firefox_fedpkg_sucks/firefox-53.0.3/objdir/dist/include/js/HashTable.h:1382
1382            Entry* entry = &table[h1];
(gdb) bt
#0  0x0000ffffb4202094 in js::detail::HashTable<js::HashMapEntry<js::CrossCompartmentKey, js::detail::UnsafeBareReadBarriered<JS::Value> >, js::HashMap<js::CrossCompartmentKey, js::detail::UnsafeBareReadBarriered<JS::Value>, js::CrossCompartmentKey::Hasher, js::SystemAllocPolicy>::MapHashPolicy, js::SystemAllocPolicy>::lookup(js::CrossCompartmentKey const&, unsigned int, unsigned int) const (collisionBit=0, keyHash=2402884294, l=..., this=<optimized out>) at /root/firefox_fedpkg_sucks/firefox-53.0.3/objdir/dist/include/js/HashTable.h:1382
#1  0x0000ffffb4202094 in js::detail::HashTable<js::HashMapEntry<js::CrossCompartmentKey, js::detail::UnsafeBareReadBarriered<JS::Value> >, js::HashMap<js::CrossCompartmentKey, js::detail::UnsafeBareReadBarriered<JS::Value>, js::CrossCompartmentKey::Hasher, js::SystemAllocPolicy>::MapHashPolicy, js::SystemAllocPolicy>::lookup(js::CrossCompartmentKey const&) const (l=..., this=<optimized out>)
    at /root/firefox_fedpkg_sucks/firefox-53.0.3/objdir/dist/include/js/HashTable.h:1736
#2  0x0000ffffb4202094 in js::HashMap<js::CrossCompartmentKey, js::detail::UnsafeBareReadBarriered<JS::Value>, js::CrossCompartmentKey::Hasher, js::SystemAllocPolicy>::lookup(js::CrossCompartmentKey const&) const (l=..., this=<optimized out>) at /root/firefox_fedpkg_sucks/firefox-53.0.3/objdir/dist/include/js/HashTable.h:106
#3  0x0000ffffb4202094 in js::NurseryAwareHashMap<js::CrossCompartmentKey, JS::Value, js::CrossCompartmentKey::Hasher, js::SystemAllocPolicy>::lookup(js::CrossCompartmentKey const&) const (l=..., this=<optimized out>) at /root/firefox_fedpkg_sucks/firefox-53.0.3/js/src/gc/NurseryAwareHashMap.h:90
#4  0x0000ffffb4202094 in JSCompartment::wrap(JSContext*, JS::MutableHandle<JS::Value>) (this=0xffffffffd8a8, cx=cx@entry=0xffffffffd3a0, vp=vp@entry=$jsval((JSObject *) 0x7000005d8b0 [object Function ""])) at /root/firefox_fedpkg_sucks/firefox-53.0.3/js/src/jscompartmentinlines.h:109
#5  0x0000ffffb42023a0 in js::PromiseObject::resolve(JSContext*, JS::Handle<JS::Value>) (this=<optimized out>, cx=cx@entry=0xffffffffd3a0, resolutionValue=$jsval((JSObject *) 0x7000222f760 [object Array])) at /root/firefox_fedpkg_sucks/firefox-53.0.3/js/src/builtin/Promise.cpp:2597
#6  0x0000ffffb4345840 in JS::ResolvePromise(JSContext*, JS::Handle<JSObject*>, JS::Handle<JS::Value>) (cx=cx@entry=0xffffffffd3a0, promise=..., 
    promise@entry=(JSObject * const) 0x7000005d850 [object Promise], resolutionValue=..., resolutionValue@entry=$jsval((JSObject *) 0x7000222f760 [object Array]))
    at /root/firefox_fedpkg_sucks/firefox-53.0.3/js/src/jsapi.cpp:4883
#7  0x0000ffffb33a94a8 in mozilla::dom::Promise::MaybeResolve(JSContext*, JS::Handle<JS::Value>) (this=this@entry=0xffff8511ab80, aCx=aCx@entry=0xffffffffd3a0, aValue=aValue@entry=$jsval((JSObject *) 0x7000222f760 [object Array])) at /root/firefox_fedpkg_sucks/firefox-53.0.3/dom/promise/Promise.cpp:275
#8  0x0000ffffb2e44fd0 in mozilla::dom::FetchBody<mozilla::dom::Response>::ContinueConsumeBody(nsresult, unsigned int, unsigned char*) (this=0xffff92508dc8, aStatus=<optimized out>, aResultLength=25392, aResult=0xffff8514b000 "[{\"action\":\"show-heartbeat\",\"approval_request\":{\"approved\":true,\"approver\":{\"email\":\"glind\",\"first_name\":\"\",\"id\":5,\"last_name\":\"\"},\"comment\":\"approving opening to 30 buckets, to check targ"...) at /root/firefox_fedpkg_sucks/firefox-53.0.3/dom/fetch/Fetch.cpp:1305
#9  0x0000ffffb2e45570 in mozilla::dom::(anonymous namespace)::ConsumeBodyDoneObserver<mozilla::dom::Response>::OnStreamComplete(nsIStreamLoader*, nsISupports*, nsresult, uint32_t, uint8_t const*) (this=0xffff85149a80, aLoader=<optimized out>, aCtxt=<optimized out>, aStatus=-10104, aResultLength=25392, aResult=0xffff8514b000 "[{\"action\":\"show-heartbeat\",\"approval_request\":{\"approved\":true,\"approver\":{\"email\":\"glind\",\"first_name\":\"\",\"id\":5,\"last_name\":\"\"},\"comment\":\"approving opening to 30 buckets, to check targ"...)
    at /root/firefox_fedpkg_sucks/firefox-53.0.3/dom/fetch/Fetch.cpp:865
#10 0x0000ffffb1bb7014 in mozilla::net::nsStreamLoader::OnStopRequest(nsIRequest*, nsISupports*, nsresult) (this=0xffff8511d400, request=0xffff92508e50, ctxt=0x0, aStatus=nsresult::NS_OK)
    at /root/firefox_fedpkg_sucks/firefox-53.0.3/netwerk/base/nsStreamLoader.cpp:106
#11 0x0000ffffb1b9a908 in nsInputStreamPump::OnStateStop() (this=0xffff92508e50) at /root/firefox_fedpkg_sucks/firefox-53.0.3/netwerk/base/nsInputStreamPump.cpp:714
#12 0x0000ffffb1b9ac0c in nsInputStreamPump::OnInputStreamReady(nsIAsyncInputStream*) (this=0xffff92508e50, stream=<optimized out>)
    at /root/firefox_fedpkg_sucks/firefox-53.0.3/netwerk/base/nsInputStreamPump.cpp:433
#13 0x0000ffffb1af0d28 in nsInputStreamReadyEvent::Run() (this=0xffff851178c0) at /root/firefox_fedpkg_sucks/firefox-53.0.3/xpcom/io/nsStreamUtils.cpp:95
#14 0x0000ffffb1b14294 in nsThread::ProcessNextEvent(bool, bool*) (this=0xffffb79d3480, aMayWait=<optimized out>, aResult=0xffffffffdd37)
    at /root/firefox_fedpkg_sucks/firefox-53.0.3/xpcom/threads/nsThread.cpp:1240
#15 0x0000ffffb1b38538 in NS_ProcessNextEvent(nsIThread*, bool) (aThread=<optimized out>, aThread@entry=0xffffb79d3480, aMayWait=aMayWait@entry=false)
    at /root/firefox_fedpkg_sucks/firefox-53.0.3/xpcom/glue/nsThreadUtils.cpp:390
#16 0x0000ffffb1e8e804 in mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) (this=0xffffa9d37380, aDelegate=0xffffa9d381c0)
    at /root/firefox_fedpkg_sucks/firefox-53.0.3/ipc/glue/MessagePump.cpp:96
#17 0x0000ffffb1e6f4ec in MessageLoop::RunInternal() (this=<optimized out>) at /root/firefox_fedpkg_sucks/firefox-53.0.3/ipc/chromium/src/base/message_loop.cc:238
#18 0x0000ffffb1e6f4ec in MessageLoop::RunHandler() (this=<optimized out>) at /root/firefox_fedpkg_sucks/firefox-53.0.3/ipc/chromium/src/base/message_loop.cc:231
#19 0x0000ffffb1e6f4ec in MessageLoop::Run() (this=<optimized out>) at /root/firefox_fedpkg_sucks/firefox-53.0.3/ipc/chromium/src/base/message_loop.cc:211
#20 0x0000ffffb3528fec in nsBaseAppShell::Run() (this=0xffffa9dfdee0) at /root/firefox_fedpkg_sucks/firefox-53.0.3/widget/nsBaseAppShell.cpp:156
#21 0x0000ffffb3c52e1c in nsAppStartup::Run() (this=0xffffa055d790) at /root/firefox_fedpkg_sucks/firefox-53.0.3/toolkit/components/startup/nsAppStartup.cpp:283
#22 0x0000ffffb3cd5678 in XREMain::XRE_mainRun() (this=this@entry=0xffffffffe010) at /root/firefox_fedpkg_sucks/firefox-53.0.3/toolkit/xre/nsAppRunner.cpp:4477
#23 0x0000ffffb3cd61a0 in XREMain::XRE_main(int, char**, mozilla::BootstrapConfig const&) (this=this@entry=0xffffffffe010, argc=argc@entry=1, argv=argv@entry=0xfffffffff388, aConfig=...)
    at /root/firefox_fedpkg_sucks/firefox-53.0.3/toolkit/xre/nsAppRunner.cpp:4654
#24 0x0000ffffb3cd6694 in XRE_main(int, char**, mozilla::BootstrapConfig const&) (argc=1, argv=0xfffffffff388, aConfig=...)
    at /root/firefox_fedpkg_sucks/firefox-53.0.3/toolkit/xre/nsAppRunner.cpp:4745
#25 0x0000aaaaaaaaf224 in do_main(int, char**, char**) (argc=1, argv=0xfffffffff388, envp=<optimized out>) at /root/firefox_fedpkg_sucks/firefox-53.0.3/browser/app/nsBrowserApp.cpp:234
#26 0x0000aaaaaaaaea2c in main(int, char**, char**) (argc=1, argv=0xfffffffff388, envp=0xfffffffff398) at /root/firefox_fedpkg_sucks/firefox-53.0.3/browser/app/nsBrowserApp.cpp:305
(gdb) info locals
h1 = <optimized out>
entry = 0x1001a924f0d20
firstRemoved = <optimized out>
(gdb) info registers
x0             0x8f391ac6       2402884294
x1             0x1adab50520     115338446112
x2             0xffffffffd818   281474976700440
x3             0x30     48
x4             0x8f391ac6       2402884294
x5             0xffffb79a0800   281473762068480
x6             0xffffffffd958   281474976700760
x7             0xffffffffd9c0   281474976700864
x8             0xffffffffd960   281474976700768
x9             0xffffffffd3a0   281474976699296
x10            0xffffffffd8a8   281474976700584
x11            0x0      0
x12            0x7000005d8b0    7696581777584
x13            0xffffffffd8a0   281474976700576
x14            0xffffffffd9c0   281474976700864
x15            0x0      0
x16            0xaaaaaaae9d48   187649984732488
x17            0xffffb7f95490   281473768314000
x18            0xffffb4b5adc0   281473713548736
x19            0x1001a924f0d20  281589100514592
x20            0xffffb5908000   281473727889408
x21            0xffffffffd8b0   281474976700592
x22            0xffffffffd8b0   281474976700592
x23            0xffffffffda08   281474976700936
x24            0xffffffffda40   281474976700992
x25            0xffffffffda18   281474976700952
x26            0xffff8514b000   281472914468864
x27            0x6330   25392
x28            0xffffffffd3a0   281474976699296
x29            0xfffffffff250   281474976707152
x30            0xffffb42023a0   281473703748512
sp             0xffffffffd7f0   0xffffffffd7f0
pc             0xffffb4202094   0xffffb4202094 <JSCompartment::wrap(JSContext*, JS::MutableHandle<JS::Value>)+204>
cpsr           0xa0000000       [ EL=0 C N ]
fpsr           0x17     23
fpcr           0x0      0
(gdb) 

   0x0000ffffb4202084 <+188>:   stp     x14, x13, [sp, #40]
   0x0000ffffb4202088 <+192>:   mov     w15, #0x0                       // #0
   0x0000ffffb420208c <+196>:   umull   x1, w0, w3
   0x0000ffffb4202090 <+200>:   add     x19, x5, x1
=> 0x0000ffffb4202094 <+204>:   ldr     w1, [x5, x1]
   0x0000ffffb4202098 <+208>:   cbz     w1, 0xffffb4202130 <JSCompartment::wrap(JSContext*, JS::MutableHandle<JS::Value>)+360>

Comment 3 Jeremy Linton 2017-06-12 16:38:33 UTC
Just an update here, it seems to be affected by compiler flags. Disabling -O2 creates a functional build.

Comment 4 Jeremy Linton 2017-06-14 23:16:46 UTC
Just an update here, I started debugging it with -Os and -O2 and both appear to basically be the same problem. JIT'ed routines are clobbering callee saved registers, which results in random crashes within firefox depending on what gets clobbered. A simple case happens early with an -Os build, where the trace logger is constructed, a JIT'ed regex routine is called and x28 is used to save the stack base, but its never restored on exit. Meaning that the deconstruction of the logger crashes the machine.

Comment 5 Jeremy Linton 2017-06-21 14:43:07 UTC
I'm back looking at this, the exact thing happens with an O2 build where the heartbeat loogic gets json that is parsed, a Jitted routine is called and it moves sp over x28 without saving it. 

For example a call into JIT code via

js::UnboxedPlainObject::createWithProperties (cx=cx@entry=0xffffa7308000, group=group@entry=0x700022ed160, newKind=newKind@entry=js::GenericObject, properties=properties@entry=0xffff92bfebe0) at /root/firefox_fedpkg_sucks/firefox-53.0.3/js/src/vm/UnboxedObject.cpp:678

688                 obj = reinterpret_cast<JSObject*>(CALL_GENERATED_2(function, properties, newKind));


is calling code like:

=> 0x00002ce4f50ca010:  mov     x28, sp
   0x00002ce4f50ca014:  sub     sp, x28, #0x8
   0x00002ce4f50ca018:  str     x30, [x28, #-8]!
   0x00002ce4f50ca01c:  sub     sp, x28, #0x8
   0x00002ce4f50ca020:  str     d31, [x28, #-8]!
   0x00002ce4f50ca024:  cmp     w1, #0x0
   0x00002ce4f50ca028:  b.ne    0x2ce4f50ca094  // b.any
   0x00002ce4f50ca02c:  mov     x17, #0xd178                    // #53624
   0x00002ce4f50ca030:  movk    x17, #0x22e, lsl #16
   0x00002ce4f50ca034:  movk    x17, #0x700, lsl #32
   0x00002ce4f50ca038:  ldr     w16, [x17]
   0x00002ce4f50ca03c:  tst     w16, #0x800000
   0x00002ce4f50ca040:  b.ne    0x2ce4f50ca094  // b.any
   0x00002ce4f50ca044:  mov     x16, #0x89e0                    // #35296
   0x00002ce4f50ca048:  movk    x16, #0xa730, lsl #16
   0x00002ce4f50ca04c:  movk    x16, #0xffff, lsl #32
   0x00002ce4f50ca050:  ldr     x2, [x16]
   0x00002ce4f50ca054:  add     x3, x2, #0x30
   0x00002ce4f50ca058:  mov     x17, #0x89f8                    // #35320
   0x00002ce4f50ca05c:  movk    x17, #0xa730, lsl #16
   0x00002ce4f50ca060:  movk    x17, #0xffff, lsl #32
   0x00002ce4f50ca064:  ldr     x16, [x17]
   0x00002ce4f50ca068:  cmp     x16, x3
   0x00002ce4f50ca06c:  b.cc    0x2ce4f50ca220  // b.lo, b.ul, b.last
   0x00002ce4f50ca070:  mov     x16, #0x89e0                    // #35296
   0x00002ce4f50ca074:  movk    x16, #0xa730, lsl #16
   0x00002ce4f50ca078:  movk    x16, #0xffff, lsl #32
   0x00002ce4f50ca07c:  str     x3, [x16]
   0x00002ce4f50ca080:  ldr     x16, 0x2ce4f50ca270
   0x00002ce4f50ca084:  str     x16, [x2]
   0x00002ce4f50ca088:  mov     x16, #0x0                       // #0
   0x00002ce4f50ca08c:  str     x16, [x2, #8]
   0x00002ce4f50ca090:  b       0x2ce4f50ca184
   0x00002ce4f50ca094:  mov     x16, #0xe0f0                    // #57584
   0x00002ce4f50ca098:  movk    x16, #0x8fa8, lsl #16
   0x00002ce4f50ca09c:  movk    x16, #0xffff, lsl #32
   0x00002ce4f50ca0a0:  ldr     x3, [x16]
   0x00002ce4f50ca0a4:  ldrh    w2, [x3]
   0x00002ce4f50ca0a8:  ldrh    w3, [x3, #2]
   0x00002ce4f50ca0ac:  cmp     w2, w3
   0x00002ce4f50ca0b0:  b.cs    0x2ce4f50ca0d8  // b.hs, b.nlast
   0x00002ce4f50ca0b4:  add     w2, w2, #0x30
   0x00002ce4f50ca0b8:  mov     x16, #0xe0f0                    // #57584
   0x00002ce4f50ca0bc:  movk    x16, #0x8fa8, lsl #16
   0x00002ce4f50ca0c0:  movk    x16, #0xffff, lsl #32
   0x00002ce4f50ca0c4:  ldr     x3, [x16]
   0x00002ce4f50ca0c8:  strh    w2, [x3]
   0x00002ce4f50ca0cc:  sub     w2, w2, #0x30
   0x00002ce4f50ca0d0:  add     x2, x3, x2
   0x00002ce4f50ca0d4:  b       0x2ce4f50ca108
   0x00002ce4f50ca0d8:  cmp     w2, #0x0
   0x00002ce4f50ca0dc:  b.eq    0x2ce4f50ca220  // b.none
   0x00002ce4f50ca0e0:  mov     x16, #0xe0f0                    // #57584
   0x00002ce4f50ca0e4:  movk    x16, #0x8fa8, lsl #16
   0x00002ce4f50ca0e8:  movk    x16, #0xffff, lsl #32
   0x00002ce4f50ca0ec:  ldr     x3, [x16]
   0x00002ce4f50ca0f0:  add     x2, x3, x2
   0x00002ce4f50ca0f4:  sub     sp, x28, #0x8
   0x00002ce4f50ca0f8:  str     x2, [x28, #-8]!
   0x00002ce4f50ca0fc:  ldr     w2, [x2]
   0x00002ce4f50ca100:  str     w2, [x3]
   0x00002ce4f50ca104:  ldr     x2, [x28], #8
   0x00002ce4f50ca108:  ldr     x16, 0x2ce4f50ca278
   0x00002ce4f50ca10c:  str     x16, [x2]
   0x00002ce4f50ca110:  mov     x16, #0x0                       // #0
   0x00002ce4f50ca114:  str     x16, [x2, #8]
   0x00002ce4f50ca118:  b       0x2ce4f50ca184
   0x00002ce4f50ca11c:  sub     sp, x28, #0x10
   0x00002ce4f50ca120:  stp     x0, x2, [x28, #-16]!
   0x00002ce4f50ca124:  mov     x3, #0x8200                     // #33280
   0x00002ce4f50ca128:  movk    x3, #0xa730, lsl #16
   0x00002ce4f50ca12c:  movk    x3, #0xffff, lsl #32
   0x00002ce4f50ca130:  sub     sp, x28, #0x8
   0x00002ce4f50ca134:  str     x30, [x28, #-8]!
   0x00002ce4f50ca138:  mov     x4, x28
   0x00002ce4f50ca13c:  sub     x28, x28, #0x8
  0x00002ce4f50ca140:  and     x28, x28, #0xfffffffffffffff0
   0x00002ce4f50ca144:  mov     sp, x28
   0x00002ce4f50ca148:  str     x4, [x28]
   0x00002ce4f50ca14c:  mov     x1, x2
   0x00002ce4f50ca150:  mov     x0, x3
   0x00002ce4f50ca154:  mov     sp, x28
   0x00002ce4f50ca158:  mov     sp, x28
   0x00002ce4f50ca15c:  mov     x16, #0x8ac8                    // #35528
   0x00002ce4f50ca160:  movk    x16, #0xb42f, lsl #16
   0x00002ce4f50ca164:  movk    x16, #0xffff, lsl #32
   0x00002ce4f50ca168:  blr     x16
   0x00002ce4f50ca16c:  mov     x28, sp
   0x00002ce4f50ca170:  ldr     x28, [x28]
   0x00002ce4f50ca174:  ldr     x30, [x28], #8
   0x00002ce4f50ca178:  mov     sp, x28
   0x00002ce4f50ca17c:  ldp     x0, x2, [x28]
   0x00002ce4f50ca180:  add     x28, x28, #0x10
   0x00002ce4f50ca184:  ldr     x3, [x0]
   0x00002ce4f50ca188:  lsr     x16, x3, #47
   0x00002ce4f50ca18c:  mov     w17, #0xfffb                    // #65531
   0x00002ce4f50ca190:  cmp     w16, w17, lsl #1
   0x00002ce4f50ca194:  b.ne    0x2ce4f50ca208  // b.any
   0x00002ce4f50ca198:  and     x16, x3, #0x7fffffffffff
   0x00002ce4f50ca19c:  str     x16, [x2, #16]
   0x00002ce4f50ca1a0:  ldr     x3, [x0, #16]
   0x00002ce4f50ca1a4:  lsr     x16, x3, #47
   0x00002ce4f50ca1a8:  mov     w17, #0xfffb                    // #65531
   0x00002ce4f50ca1ac:  cmp     w16, w17, lsl #1
   0x00002ce4f50ca1b0:  b.ne    0x2ce4f50ca208  // b.any
   0x00002ce4f50ca1b4:  and     x16, x3, #0x7fffffffffff
   0x00002ce4f50ca1b8:  str     x16, [x2, #24]
   0x00002ce4f50ca1bc:  ldr     x3, [x0, #32]
   0x00002ce4f50ca1c0:  lsr     x16, x3, #47
   0x00002ce4f50ca1c4:  mov     w17, #0xfff8ffff                // #-458753
   0x00002ce4f50ca1c8:  cmp     w16, w17, lsr #15
   0x00002ce4f50ca1cc:  b.ne    0x2ce4f50ca208  // b.any
   0x00002ce4f50ca1d0:  str     w3, [x2, #40]
   0x00002ce4f50ca1d4:  ldr     x3, [x0, #48]
   0x00002ce4f50ca1d8:  lsr     x16, x3, #47
   0x00002ce4f50ca1dc:  mov     w17, #0xfffb                    // #65531
   0x00002ce4f50ca1e0:  cmp     w16, w17, lsl #1
   0x00002ce4f50ca1e4:  b.ne    0x2ce4f50ca208  // b.any
   0x00002ce4f50ca1e8:  and     x16, x3, #0x7fffffffffff
   0x00002ce4f50ca1ec:  str     x16, [x2, #32]
   0x00002ce4f50ca1f0:  mov     x0, x2
   0x00002ce4f50ca1f4:  ldr     xzr, [x28], #8
   0x00002ce4f50ca1f8:  ldr     x30, [x28]
   0x00002ce4f50ca1fc:  add     x28, x28, #0x8


which appears to be generated from:

UnboxedObject.cpp

UnboxedLayout::makeConstructorCode(JSContext* cx, HandleObjectGroup group)

#ifdef JS_CODEGEN_ARM64
    // ARM64 communicates stack address via sp, but uses a pseudo-sp for addressing.
    masm.initStackPtr();
#endif

Comment 6 Jan Horak 2017-06-21 14:48:43 UTC
Looks like you're trying hard. Please add --enable-debug to the .mozconfig (or check the debug_build variable in firefox.spec - be aware that setting it to 1 disables also optimization, which is what you probably don't want to do). There's a chance that some assert will be triggered which moves us to the main cause of this trouble.

Comment 7 Jeremy Linton 2017-06-21 15:33:51 UTC
I've opened an upstream bug.

Comment 8 Jeremy Linton 2017-06-21 15:52:15 UTC
I'm fairly certain i've narrowed down the root cause, the only real question is why its suddenly more visible. Its pretty obvious that small changes in the C++ code generation side, can move the bug around pretty significantly as x28 isn't frequently allocated. Particlarly with lower optimization levels GCC seems to prefer spilling a volatile register onto the stack.

Narrowing down a particular crash is somewhat difficult as they happen at some point after the JIT code is called when x28 is being used. As C++ tends not to use that register it remains untouch for long series of call chains so the actual failures can be far away from the original bug. I've got a pretty creative gdb macro which tracks x28 usage over call sequences, with the goal of finding routines which are failing to restore it properly. But by itself, even with an idea of where the problem is, the machine can literally run for hours before triggering on the fault.

Comment 9 Jeremy Linton 2017-06-23 15:53:17 UTC
Working koji build here:

https://koji.fedoraproject.org/koji/taskinfo?taskID=20102918

Comment 10 Jeremy Linton 2017-08-09 18:45:24 UTC
Created attachment 1311341 [details]
fedora build fix for aarch64 gcc7 register spilling

This is a compiler flags fix to discourage gcc from utilizing x28.

Comment 11 Jeremy Linton 2017-08-09 18:48:03 UTC
The above fix works around the problem by changing the fedora compiler flags used for gcc7. This fixes FF 53 and 54. It seems that FF55 being used by the current fedpkg doesn't need this fix to "run" although that seems to be dumb luck as much as anything as far as I can tell (much like the original problem appeared).

Comment 12 Jan Horak 2017-08-10 07:24:51 UTC
Okay, please let us know if you find out that the patch is required for Firefox 55.

Comment 13 Jeremy Linton 2018-05-01 19:05:17 UTC
This problem appears to be hitting FF59 in F28 as well per bug #1564204. The fix will land in FF61 per the upstream defect.

Comment 14 Fedora End Of Life 2018-05-03 08:12:04 UTC
This message is a reminder that Fedora 26 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 26. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '26'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 26 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Comment 15 Martin Stransky 2018-05-03 11:59:15 UTC
added to rawhide.