Bug 1482798

Also in dmesg:
[309771.607302] traps: chronyd[20326] trap invalid opcode ip:b73cdaea sp:bfbdd090 error:0 in libfreeblpriv3.so[b73ca000+6d000]

(In reply to Tomasz Torcz from comment #0)
> Description of problem:
> Recently I noticed 'chronyd' aborts during startup with Illegal Instruction.

Does this only happen with NSS 3.32.0, not with 3.31.0?

It happens with nss-3.31.0-6.fc27.i686.rpm, too.

I have the same problem with Fedora 26 on Pentium III:
(dmesg example)
[  113.768573] traps: rpm[1014] trap invalid opcode ip:b6cb1aca sp:bfd6fea0 error:0 in libfreeblpriv3.so[b6cae000+6d000]
[  165.212819] traps: dnf[1050] trap invalid opcode ip:b5207aca sp:bf8778c0 error:0 in libfreeblpriv3.so[b5204000+6d000]
[  184.468669] traps: rpm[1055] trap invalid opcode ip:b6cc9aca sp:bfcb99c0 error:0 in libfreeblpriv3.so[b6cc6000+6d000]
[  287.889120] traps: rpm[1127] trap invalid opcode ip:b6c17aca sp:bfbd8530 error:0 in libfreeblpriv3.so[b6c14000+6d000]
[  633.854490] traps: dnf[1146] trap invalid opcode ip:b5265aca sp:bfd777c0 error:0 in libfreeblpriv3.so[b5262000+6d000]
[  835.699086] traps: curl[1160] trap invalid opcode ip:b5d86aca sp:bfdeaa80 error:0 in libfreeblpriv3.so[b5d83000+6d000]
[  886.525060] traps: rpm[1240] trap invalid opcode ip:b6b84aca sp:bfad5970 error:0 in libfreeblpriv3.so[b6b81000+6d000]
[  886.725567] traps: rpm[1244] trap invalid opcode ip:b6b68aca sp:bf8650f0 error:0 in libfreeblpriv3.so[b6b65000+6d000]

Downgrading nss-softokn-freebl-3.32.0-1.2.fc26.i686 to nss-softokn-freebl-3.30.2-1.0.fc26.i686 solves the issue.

Would it be possible to check which instruction is actually used, with "x/i <address>" in gdb?

Program received signal SIGILL, Illegal instruction.
0xb7afcaea in SHA1_Update (ctx=0x80044c40, dataIn=0x800446a0 "2\263$|m\273\230P\177J\265R\270\367`\016\016L\340\301\001\001\001\001", len=20) at sha_fast.c:101

gives

(gdb) display/i 0xb7afcaea
1: x/i 0xb7afcaea
=> 0xb7afcaea <SHA1_Update+58>:	paddq  %xmm2,%xmm0


According to http://www.felixcloutier.com/x86/PADDQ.html, it is SSE2 instruction. This CPU has

Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse syscall mp mmxext 3dnowext 3dnow cpuid 3dnowprefetch vmmcall

Only SSE1.

I don't think we agreed to require SSE2 on i686.

Davide Repetto's comment was clobbered: restoring it.

--- Comment #1 from Davide Repetto <red> ---
Something similar may have happened to centos in the past:
https://bugs.centos.org/view.php?id=11074

Now that I look at David's comment, It has to do with AES_GCM speedups which looked at the CPU bits to determine if they speedups were doable, but didn't look to see if the OS could save and restore the SSE3 registers. THE SHA1 code is straight C code (except some byte swap macros which don't use SSE2), so it sounds like we need to tweak the compiler options to make sure it's not generating SSE2 instructions.

Same problem here. What makes matters almost catastrophic is that rpm and dnf can't be used any more and one will have to update package by other means, for example by mounting via sshfs, when new nss-softokn build is available.
Attempt to downgrade to the previous version also fails as no lower nss-tools version available.

Hi all, I apologize for my confusion. Has this been resolved? 

FYI, I've also added this to the x86 tracker bug.

No, it hasn't, Jeff.

(In reply to Tomasz Torcz from comment #6)

> (gdb) display/i 0xb7afcaea
> 1: x/i 0xb7afcaea
> => 0xb7afcaea <SHA1_Update+58>:	paddq  %xmm2,%xmm0
[...]
> I don't think we agreed to require SSE2 on i686.

Thank you for investigating this.

(In reply to Bob Relyea from comment #8)
> Now that I look at David's comment, It has to do with AES_GCM speedups which
> looked at the CPU bits to determine if they speedups were doable, but didn't
> look to see if the OS could save and restore the SSE3 registers. THE SHA1
> code is straight C code (except some byte swap macros which don't use SSE2),
> so it sounds like we need to tweak the compiler options to make sure it's
> not generating SSE2 instructions.

I think I managed to disable SSE2 in freebl, at least for the SHA-1 code.  The upstream enabled the -mpclmul compiler option globally, which seems to imply the compiler generates SSE2 code.

The current fix is to limit the effect of -mpclmul to AES and GCM:
http://pkgs.fedoraproject.org/cgit/rpms/nss-softokn.git/tree/nss-softokn-disable-sse2.patch?h=f27

nss-softokn-3.32.0-7.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-d924f2b4ba

nss-softokn-3.32.0-7.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-d924f2b4ba

Will there be an identical package for fc26?

on a F26 system with sse2 CPU I downloaded http://mirror.math.princeton.edu/pub/fedora/linux/updates/testing/27/SRPMS/n/nss-softokn-3.32.0-7.fc27.src.rpm and rebuild using
mock -v --rebuild nss-softokn-3.32.0-7.fc27.src.rpm
then mounted disk on a F26 without sse2 system using sshfs
# sshfs root.x.x:/ /mnt
then copied rebuilt packages and installed them

# rpm -r /mnt -Uvh /mnt/root/{nss-softokn-freebl-3.32.0-7.fc26.i686.rpm,nss-softokn-3.32.0-7.fc26.i686.rpm}

rpm started working but dnf still fails, but in a different function:

Stack trace of thread 4410:
                #0  0x00000000b51afa54 gcmHash_Update (libfreeblpriv3.so)

> The current fix is to limit the effect of -mpclmul to AES and GCM:
> http://pkgs.fedoraproject.org/cgit/rpms/nss-softokn.git/tree/nss-softokn-
> disable-sse2.patch?h=f27

Dump of assembler code for function gcmHash_Update:
   0xb5247a20 <+0>:	sub    $0x3c,%esp
   0xb5247a23 <+3>:	mov    %ebx,0x2c(%esp)
   0xb5247a27 <+7>:	mov    0x48(%esp),%ebx
   0xb5247a2b <+11>:	mov    0x40(%esp),%edx
   0xb5247a2f <+15>:	mov    %esi,0x30(%esp)
   0xb5247a33 <+19>:	mov    %edi,0x34(%esp)
   0xb5247a37 <+23>:	xor    %edi,%edi
   0xb5247a39 <+25>:	lea    0x0(,%ebx,8),%esi
   0xb5247a40 <+32>:	movd   %edi,%xmm0
   0xb5247a44 <+36>:	movd   %esi,%xmm1
   0xb5247a48 <+40>:	mov    0x50(%edx),%esi
   0xb5247a4b <+43>:	punpckldq %xmm0,%xmm1
   0xb5247a4f <+47>:	movq   0x64(%edx),%xmm0
=> 0xb5247a54 <+52>:	paddq  %xmm1,%xmm0


Is there a problem with mock environment or a package?

(In reply to Eugene Kanter from comment #16)

> rpm started working but dnf still fails, but in a different function:
> 
> Stack trace of thread 4410:
>                 #0  0x00000000b51afa54 gcmHash_Update (libfreeblpriv3.so)

It's a separate issue (maybe bug 1485656).  As I indicated in comment 12, I didn't disable optimizations in AES and GCM code because it triggered test failures.

(In reply to Daiki Ueno from comment #18)
> (In reply to Eugene Kanter from comment #16)
> 
> It's a separate issue (maybe bug 1485656).  As I indicated in comment 12, I
> didn't disable optimizations in AES and GCM code because it triggered test
> failures.

The sentence

>The current fix is to limit the effect of -mpclmul to AES and GCM

is hard to interpret and there is no mentions of any filed tests.

I updated bug 1485656 with current assembler dump. Thank you.

(In reply to Eugene Kanter from comment #19)

> I updated bug 1485656 with current assembler dump. Thank you.

Thank you.  In the end I came up with a patch to disable SSE2 even in AES-GCM:
https://github.com/ueno/nss/commit/385ae89cce762030769c7613bd0c81ec7e171bc3

The package is now building in:
https://koji.fedoraproject.org/koji/buildinfo?buildID=971248

Daiki, The "c" compiled versions of AES-GCM shouldn't automatically use SSE2. There are assembler files that use SSE2 instructions if the processor and OS support it (detected at runtime). 

Upstream may have some issues when running Firefox under certain VM's with the unconditional flag.

bob

(In reply to Bob Relyea from comment #21)
> Daiki, The "c" compiled versions of AES-GCM shouldn't automatically use
> SSE2. There are assembler files that use SSE2 instructions if the processor
> and OS support it (detected at runtime). 

Well, it does.  See Eugine's backtrace in comment 17.  Since this upstream commit:
https://hg.mozilla.org/projects/nss/rev/cd068f7ce6ae11120f8e4427aa2e8ac35a69e764
all freebl source files are now compiled with "-mpclmul -maes" and that implies that GCC generates SSE2 instructions to optimize other portions of code.  I tried to disable it with -mno-sse2, but then I got compilation errors in gcm.c and rijndael.c.

I recomplied nss-softokn-3.32.0-8 on F26 and dnf is now working again.

Hi all, thanks for continuing to try to resolve this.

I apologize for my ignorance, but has Daiki's patch resolved this? Are we waiting on a fix for freebl? Are we looking for testers? Thanks for your patience!

(In reply to Jeff Backus from comment #24)
> Hi all, thanks for continuing to try to resolve this.
> 
> I apologize for my ignorance, but has Daiki's patch resolved this? Are we
> waiting on a fix for freebl? Are we looking for testers? Thanks for your
> patience!

Although the fix has been merged in upstream, the final version of the patch touches the C code quite a bit.  So I would like to postpone this to the next rebase of the NSS packages, rather than cherry-picking the patch.

For those who are willing to test, I prepared scratch builds for F27 and F26:
https://koji.fedoraproject.org/koji/taskinfo?taskID=22019235
https://koji.fedoraproject.org/koji/taskinfo?taskID=22019362

Any feedback would be appreciated.

(In reply to Daiki Ueno from comment #25)

> For those who are willing to test, I prepared scratch builds for F27 and F26:
How to test this? Do you have a simple test case?

My real word case to expose the nss_softokn issues so far had been to launch thunderbird on a PIII, but this isn't actually "handy".


> https://koji.fedoraproject.org/koji/taskinfo?taskID=22019235
> https://koji.fedoraproject.org/koji/taskinfo?taskID=22019362
> 
> Any feedback would be appreciated.
I am observing SIGILLs, which I believe are related to nss-softokn with when launching thunderbird.

gdb coredump traceback from thunderbird-52.3.0-1.fc27.i386 w/ nss-softokn-3.33.0-1.0.fc27.i686:
...
#
Core was generated by `/usr/lib/thunderbird/thunderbird'.
Program terminated with signal SIGILL, Illegal instruction.
...
#0  0xb76fbcd9 in __kernel_vsyscall ()
#1  0xb76da7f2 in raise () from /lib/libpthread.so.0
#2  0xb3708488 in nsProfileLock::FatalSignalHandler (signo=4, info=0xbfcd544c, context=0xbfcd54cc)
    at /usr/src/debug/thunderbird-52.3.0-1.fc27.i386/mozilla/toolkit/profile/nsProfileLock.cpp:181
#3  <signal handler called>
#4  0xa6517b4a in SHA1_Update () from /lib/libfreeblpriv3.so
#5  0xa65e8399 in sftkdb_passwordToKey.isra.2 () from /lib/libsoftokn3.so
#6  0xa65e96bf in sftkdb_ChangePassword () from /lib/libsoftokn3.so
#7  0xa65cac7b in NSC_InitPIN () from /lib/libsoftokn3.so
#8  0xb024fc4e in PK11_InitPin () from /lib/libnss3.so
#9  0xb3547b0c in nsPK11Token::InitPassword (this=0x99e47be0, initialPassword=...)
    at /usr/src/debug/thunderbird-52.3.0-1.fc27.i386/mozilla/security/manager/ssl/nsPK11TokenDB.cpp:299
#10 0xb0b83fc4 in NS_InvokeByIndex () from /usr/lib/thunderbird/libxul.so
#11 0xb13d6bb4 in CallMethodHelper::Invoke (this=0xbfcd5d88)
    at /usr/src/debug/thunderbird-52.3.0-1.fc27.i386/mozilla/js/xpconnect/src/XPCWrappedNative.cpp:2058
#12 CallMethodHelper::Call (this=0xbfcd5d88)
    at /usr/src/debug/thunderbird-52.3.0-1.fc27.i386/mozilla/js/xpconnect/src/XPCWrappedNative.cpp:1377
#13 XPCWrappedNative::CallMethod (ccx=..., mode=<optimized out>, mode@entry=XPCWrappedNative::CALL_METHOD)
    at /usr/src/debug/thunderbird-52.3.0-1.fc27.i386/mozilla/js/xpconnect/src/XPCWrappedNative.cpp:1344
#14 0xb13dc24c in XPC_WN_CallMethod (cx=cx@entry=0xaf3ae000, argc=1, vp=0xa9012910)
    at /usr/src/debug/thunderbird-52.3.0-1.fc27.i386/mozilla/js/xpconnect/src/XPCWrappedNativeJSOps.cpp:1000
#15 0xb4177af9 in js::CallJSNative (args=..., native=<optimized out>, cx=0xaf3ae000)
    at /usr/src/debug/thunderbird-52.3.0-1.fc27.i386/mozilla/js/src/jscntxtinlines.h:239
#16 js::InternalCallOrConstruct (cx=0xaf3ae000, args=..., construct=construct@entry=js::NO_CONSTRUCT)
    at /usr/src/debug/thunderbird-52.3.0-1.fc27.i386/mozilla/js/src/vm/Interpreter.cpp:459
#17 0xb4177cf1 in InternalCall (cx=cx@entry=0xaf3ae000, args=...)
    at /usr/src/debug/thunderbird-52.3.0-1.fc27.i386/mozilla/js/src/vm/Interpreter.cpp:504
#18 0xb416ae16 in js::CallFromStack (args=..., cx=0xaf3ae000)
    at /usr/src/debug/thunderbird-52.3.0-1.fc27.i386/mozilla/js/src/vm/Interpreter.cpp:510
#19 Interpret (cx=cx@entry=0xaf3ae000, state=...)
    at /usr/src/debug/thunderbird-52.3.0-1.fc27.i386/mozilla/js/src/vm/Interpreter.cpp:2922
#20 0xb4177540 in js::RunScript (cx=cx@entry=0xaf3ae000, state=...)
    at /usr/src/debug/thunderbird-52.3.0-1.fc27.i386/mozilla/js/src/vm/Interpreter.cpp:405
#21 0xb4177a48 in js::InternalCallOrConstruct (cx=cx@entry=0xaf3ae000, args=..., construct=construct@entry=js::CONSTRUCT)
    at /usr/src/debug/thunderbird-52.3.0-1.fc27.i386/mozilla/js/src/vm/Interpreter.cpp:477
...


# cat /proc/cpuinfo 
...
model name	: Pentium III (Coppermine)
stepping	: 6
...
flags		: fpu vme de pse tsc msr pae mce cx8 sep mtrr pge mca cmov pse36 mmx fxsr sse cpuid

FWIW: chronyd doesn't seem to expose this issue for on the same HW:

# /usr/sbin/chronyd -d
2017-10-07T04:11:18Z chronyd version 3.2 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SECHASH +SIGND +ASYNCDNS +IPV6 +DEBUG)
2017-10-07T04:11:18Z Frequency 147.166 +/- 1.016 ppm read from /var/lib/chrony/drift
2017-10-07T04:11:18Z Using right/UTC timezone to obtain leap second data
2017-10-07T04:11:24Z Selected source 195.186.1.101
2017-10-07T04:11:24Z System clock TAI offset set to 37 seconds

chrony-3.2-1.fc27.i686
nss-softokn-freebl-3.33.0-1.0.fc27.i686
nss-softokn-3.33.0-1.0.fc27.i686

(In reply to Ralf Corsepius from comment #26)

> > https://koji.fedoraproject.org/koji/taskinfo?taskID=22019235
> > https://koji.fedoraproject.org/koji/taskinfo?taskID=22019362
> > 
> > Any feedback would be appreciated.
> I am observing SIGILLs, which I believe are related to nss-softokn with when
> launching thunderbird.
> 
> gdb coredump traceback from thunderbird-52.3.0-1.fc27.i386 w/
> nss-softokn-3.33.0-1.0.fc27.i686:
> ...

While the suggested scratch builds are based on 3.32.1, you are testing 3.33.0, where we reverted the downstream patch to disable SSE2 (see comment 25 for the reason).

The fix is expected to land in Fedora with our rebase to NSS 3.34.

(In reply to Daiki Ueno from comment #28)
> (In reply to Ralf Corsepius from comment #26)
> 
> > > https://koji.fedoraproject.org/koji/taskinfo?taskID=22019235
> > > https://koji.fedoraproject.org/koji/taskinfo?taskID=22019362
> > > 
> > > Any feedback would be appreciated.
> > I am observing SIGILLs, which I believe are related to nss-softokn with when
> > launching thunderbird.
> > 
> > gdb coredump traceback from thunderbird-52.3.0-1.fc27.i386 w/
> > nss-softokn-3.33.0-1.0.fc27.i686:
> > ...
> 
> While the suggested scratch builds are based on 3.32.1, you are testing
> 3.33.0,
0.33.0 was pulled in by dnf. I had not expected you to revert your patches and had not checked details.

However, I happened to have local copies of the binaries from 
https://koji.fedoraproject.org/koji/taskinfo?taskID=22019235
around. 

Result: Unlike with 0.33.0, thunderbird doesn't crash on my PIII and appears to work.

> where we reverted the downstream patch to disable SSE2 (see comment
> 25 for the reason).
Sigh ... This is not helpful for me ;)

> The fix is expected to land in Fedora with our rebase to NSS 3.34.
Will this happen before f27 release?

*** This bug has been marked as a duplicate of bug 1507938 ***

Note that nss-softokn-3.33.0-1.1.fc26 is affected as well, so this bug impacts the Fedora 26 release, too.

*** Bug 1507938 has been marked as a duplicate of this bug. ***

nss-3.34.0-1.0.fc27 nss-softokn-3.34.0-1.0.fc27 nss-util-3.34.0-1.0.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-24169bbb2d

nss-3.34.0-1.0.fc26 nss-softokn-3.34.0-1.0.fc26 nss-util-3.34.0-1.0.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-552febe596

nss-3.34.0-1.0.fc25 nss-softokn-3.34.0-1.0.fc25 nss-util-3.34.0-1.0.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-15b7f781f0

I gave the fc27 and fc26 versions of nss-*3.34.0-1.0 a try on my PIII and they did not trigger the SIGILLs I was facing with *-3.33*.

nss-3.34.0-1.0.fc25, nss-softokn-3.34.0-1.0.fc25, nss-util-3.34.0-1.0.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-15b7f781f0

nss-3.34.0-1.0.fc27, nss-softokn-3.34.0-1.0.fc27, nss-util-3.34.0-1.0.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-24169bbb2d

nss-3.34.0-1.0.fc27 nss-pem-1.0.3-6.fc27 nss-softokn-3.34.0-1.0.fc27 nss-util-3.34.0-1.0.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-24169bbb2d

nss-3.34.0-1.0.fc26, nss-softokn-3.34.0-1.0.fc26, nss-util-3.34.0-1.0.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-552febe596

nss-3.34.0-1.0.fc27, nss-pem-1.0.3-6.fc27, nss-softokn-3.34.0-1.0.fc27, nss-util-3.34.0-1.0.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-24169bbb2d

nss-3.34.0-1.0.fc27, nss-pem-1.0.3-6.fc27, nss-softokn-3.34.0-1.0.fc27, nss-util-3.34.0-1.0.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.

nss-3.34.0-1.0.fc25, nss-softokn-3.34.0-1.0.fc25, nss-util-3.34.0-1.0.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.

*** Bug 1519148 has been marked as a duplicate of this bug. ***

nss-3.34.0-1.0.fc26, nss-softokn-3.34.0-1.0.fc26, nss-util-3.34.0-1.0.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.