Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1358845

Summary: nghttpx crashes on armv7hl while running HTTP/2 curl tests
Product: [Fedora] Fedora Reporter: Kamil Dudka <kdudka>
Component: nghttp2Assignee: Kamil Dudka <kdudka>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: medium    
Version: 25CC: kdudka, pwhalen
Target Milestone: ---   
Target Release: ---   
Hardware: armv7hl   
OS: Linux   
Whiteboard:
Fixed In Version: nghttp2-1.13.0-2.fc25 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-07-29 13:56:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1360319    
Bug Blocks: 245418    
Attachments:
Description Flags
[PATCH] nghttpx: avoid using std::function to fix crash on armv7hl none

Description Kamil Dudka 2016-07-21 15:19:20 UTC
Version-Release number of selected component (if applicable):
nghttp2-1.13.0-1.fc25.armv7hl


Actual results:
[INFO] shrpx_config.cc:2881 Resolving backend address
[INFO] shrpx_config.cc:2891 Host-path pattern: group 0: '/'
[INFO] shrpx_config.cc:2894 group 0 -> 127.0.0.1:8990, proto=http/1.1
[INFO] shrpx_config.cc:2910 Catch-all pattern is group 0
[INFO] shrpx_config.cc:3002 Address resolution for 127.0.0.1 succeeded: 127.0.0.1
[INFO] shrpx_config.cc:2958 Resolved backend address: 127.0.0.1:8990 -> 127.0.0.1:8990
[NOTICE] Listening on 0.0.0.0:9015
[NOTICE] Listening on [::]:9015
[NOTICE] Worker process [29578] spawned
[INFO] shrpx_worker.cc:265 number of http/1.1 backend: 1, number of h2 backend: 0
[INFO] shrpx_worker_process.cc:528 Entering event loop
[INFO] shrpx_connection_handler.cc:365 [LISTEN:0xbefab428] Accepted connection. fd=11
[INFO] shrpx_client_handler.cc:70 [CLIENT_HANDLER:0x80e91750] Time out
[INFO] shrpx_client_handler.cc:449 [CLIENT_HANDLER:0x80e91750] Deleting
[INFO] shrpx_client_handler.cc:472 [CLIENT_HANDLER:0x80e91750] Deleted
[INFO] shrpx_connection_handler.cc:365 [LISTEN:0xbefab428] Accepted connection. fd=11
[NOTICE] Worker process: [29578] exited abnormally with status 8b; exit status 0; signal Segmentation fault(11)
[NOTICE] Shutdown momentarily


Additional info:
http://koji.fedoraproject.org/koji/taskinfo?taskID=14968582
http://koji.fedoraproject.org/koji/taskinfo?taskID=14968582

Comment 1 Kamil Dudka 2016-07-25 15:30:30 UTC
Reproduced on an f24 box:

# rpm -qa libev\* \*nghttp\* | sort -V
libev-4.20-2.fc24.armv7hl
libev-debuginfo-4.20-2.fc24.armv7hl
libev-devel-4.20-2.fc24.armv7hl
libnghttp2-1.13.0-1.fc24.armv7hl
libnghttp2-devel-1.13.0-1.fc24.armv7hl
nghttp2-1.13.0-1.fc24.armv7hl
nghttp2-debuginfo-1.13.0-1.fc24.armv7hl

(gdb) bt
#0  0x7f62ebc4 in std::__invoke_impl<int, int (shrpx::ClientHandler::* const&)(), shrpx::ClientHandler&> (__t=..., __f=<optimized out>) at /usr/include/c++/6.1.1/functional:226
#1  std::__invoke<int (shrpx::ClientHandler::* const&)(), shrpx::ClientHandler&> (__fn=<optimized out>) at /usr/include/c++/6.1.1/functional:260
#2  std::_Mem_fn_base<int (shrpx::ClientHandler::*)(), true>::operator()<shrpx::ClientHandler&> (this=<optimized out>) at /usr/include/c++/6.1.1/functional:613
#3  std::_Function_handler<int (shrpx::ClientHandler&), int (shrpx::ClientHandler::*)()>::_M_invoke(std::_Any_data const&, shrpx::ClientHandler&) (__functor=..., __args#0=...) at /usr/include/c++/6.1.1/functional:1788
#4  0x7f6295b0 in std::function<int (shrpx::ClientHandler&)>::operator()(shrpx::ClientHandler&) const (__args#0=..., this=0x7fadce8c) at /usr/include/c++/6.1.1/functional:2136
#5  shrpx::ClientHandler::do_read (this=0x7fadcc60) at shrpx_client_handler.cc:615
#6  shrpx::(anonymous namespace)::readcb (loop=<optimized out>, w=<optimized out>, revents=<optimized out>) at shrpx_client_handler.cc:94
#7  0xb6e9893c in ev_invoke_pending (loop=0xb6eb0358 <default_loop_struct>) at ev.c:3155
#8  0xb6e9bc18 in ev_run (loop=0xb6eb0358 <default_loop_struct>, loop@entry=0x0, flags=-1226195944, flags@entry=0) at ev.c:3555
#9  0x7f618f30 in shrpx::worker_process_event_loop (wpconf=0x0) at shrpx_worker_process.cc:537
#10 0x7f5bf524 in shrpx::(anonymous namespace)::fork_worker_process (ssv=0xbeaf1c8c) at shrpx.cc:884
#11 shrpx::(anonymous namespace)::event_loop () at shrpx.cc:966
#12 0x7f5b9678 in shrpx::main (argc=6, argv=0xbeaf45b4) at shrpx.cc:3113
#13 0x7f5c1b58 in std::function<int (int, char**)>::operator()(int, char**) const (__args#1=<optimized out>, __args#0=<optimized out>, this=<optimized out>) at /usr/include/c++/6.1.1/functional:2136
#14 nghttp2::run_app(std::function<int (int, char**)>, int, char**) (app=..., argc=<optimized out>, argv=<optimized out>) at template.h:542
#15 0x7f5b7c90 in main (argc=<optimized out>, argv=<optimized out>) at shrpx.cc:3126

If I rebuild/reinstall libev from SRPM locally (the same NVR), nghttpx stops crashing but HTTP/2 curl's tests fail anyway after a while of hanging.

Comment 2 Kamil Dudka 2016-07-25 15:44:04 UTC
> If I rebuild/reinstall libev from SRPM locally (the same NVR), nghttpx stops
> crashing but HTTP/2 curl's tests fail anyway after a while of hanging.

Actually, only the first test hangs whereas the subsequent tests still crash the nghttpx worker.  It is reproducible also with libev-4.22-1.fc25 rebuilt for f24.

Here is another backtrace caught with the updated libev:

(gdb) bt
#0  0x00000000 in ?? ()
#1  std::function<int (shrpx::ClientHandler&)>::operator()(shrpx::ClientHandler&) const (__args#0=..., this=0x81188e8c) at /usr/include/c++/6.1.1/functional:2136
#2  shrpx::ClientHandler::do_read (this=0x81188c60) at shrpx_client_handler.cc:615
#3  shrpx::(anonymous namespace)::readcb (loop=<optimized out>, w=<optimized out>, revents=<optimized out>) at shrpx_client_handler.cc:94
#4  ev_invoke_pending (loop=0xb6f51358 <default_loop_struct>) at ev.c:3288
#5  ev_run (loop=0xb6f51358 <default_loop_struct>, loop@entry=0x0, flags=-1225536504, flags@entry=0) at ev.c:3688
#6  shrpx::worker_process_event_loop (wpconf=0x0) at shrpx_worker_process.cc:537
#7  shrpx::(anonymous namespace)::fork_worker_process (ssv=0xbeb99c8c) at shrpx.cc:884
#8  shrpx::(anonymous namespace)::event_loop () at shrpx.cc:966
#9  shrpx::main (argc=6, argv=0xbeb9c5b4) at shrpx.cc:3113
#10 std::function<int (int, char**)>::operator()(int, char**) const (__args#1=<optimized out>, __args#0=<optimized out>, this=<optimized out>) at /usr/include/c++/6.1.1/functional:2136
#11 nghttp2::run_app(std::function<int (int, char**)>, int, char**) (app=..., argc=<optimized out>, argv=<optimized out>) at template.h:542
#12 main (argc=<optimized out>, argv=<optimized out>) at shrpx.cc:3126

Comment 3 Kamil Dudka 2016-07-25 17:38:25 UTC
Rebuilding nghttp2 with -O1 in CXXFLAGS prevents nghttpx from crashing.

Comment 4 Kamil Dudka 2016-07-25 22:28:32 UTC
-- a/src/Makefile.am
+++ b/src/Makefile.am
@@ -156,8 +156,12 @@ endif # HAVE_MRUBY
 noinst_LIBRARIES = libnghttpx.a
 libnghttpx_a_SOURCES = ${NGHTTPX_SRCS}
 libnghttpx_a_CPPFLAGS = ${AM_CPPFLAGS}

+# needed for shrpx_client_handler.cc and shrpx_http_downstream_connection.cc
+# on armv7hl -- see https://bugzilla.redhat.com/1358845 for details
+libnghttpx_a_CXXFLAGS = -fno-strict-aliasing
+
 nghttpx_SOURCES = shrpx.cc shrpx.h
 nghttpx_CPPFLAGS = ${libnghttpx_a_CPPFLAGS}
 nghttpx_LDADD = libnghttpx.a ${LDADD}

Comment 5 Jan Kurik 2016-07-26 04:30:14 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 25 development cycle.
Changing version to '25'.

Comment 6 Kamil Dudka 2016-07-26 11:11:01 UTC
Created attachment 1184185 [details]
[PATCH] nghttpx: avoid using std::function to fix crash on armv7hl

There seems to be some problem with nghttp's use of std::function.  The attached patch makes the crashes go away.  I am not familiar with std::function myself, so it is not clear to me whether it is a bug of nghttp or a gcc bug.

Comment 7 Kamil Dudka 2016-07-26 12:30:14 UTC
Reported as gcc bug #1360319 with minimal example: attachment #1184246 [details]

Comment 8 Kamil Dudka 2016-07-26 15:07:48 UTC
Reported to nghttp2 upstream: https://github.com/nghttp2/nghttp2/issues/635

Comment 9 Kamil Dudka 2016-07-29 13:56:05 UTC
curl now builds on armv7hl without any workaround in place:

http://koji.fedoraproject.org/koji/buildinfo?buildID=785581