Note: This is a public test instance of Red Hat Bugzilla. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback at bugzilla.redhat.com.

Bug 1224945

Summary: mono build failure on ppc64le
Product: [Fedora] Fedora Reporter: Than Ngo <than>
Component: monoAssignee: Xavier Lamien <lxtnow>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 23CC: chkr, claudiorodrigo, dan, itamar, lxtnow, moceap, moez.roy, neale, opensource, paul, pokorra.mailinglists
Target Milestone: ---   
Target Release: ---   
Hardware: ppc64le   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-08-28 11:09:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1199775, 1051573    

Description Than Ngo 2015-05-26 09:31:18 UTC
mono-4.0.1-8.fc23 (with bootstrap 1) does not build on ppc64le in rawhide
it breaks with folling errors.

Error reading namespace XML for global at ./../class/Novell.Directory.Ldap/Documentation/en/ns-global.xml
Warning: couldn't process type file `./../class/Novell.Directory.Ldap/Documentation/en/global/IThreadRunnable.xml' as it doesn't exist
Warning: couldn't process type file `./../class/Novell.Directory.Ldap/Documentation/en/global/Integer32.xml' as it doesn't exist
Warning: couldn't process type file `./../class/Novell.Directory.Ldap/Documentation/en/global/SupportClass.xml' as it doesn't exist
Warning: couldn't process type file `./../class/Novell.Directory.Ldap/Documentation/en/global/SupportClass+SingleThreadModel.xml' as it doesn't exist
Warning: couldn't process type file `./../class/Novell.Directory.Ldap/Documentation/en/global/SupportClass+Tokenizer.xml' as it doesn't exist
Warning: couldn't process type file `./../class/Novell.Directory.Ldap/Documentation/en/global/SupportClass+DateTimeFormatManager.xml' as it doesn't exist
Warning: couldn't process type file `./../class/Novell.Directory.Ldap/Documentation/en/global/SupportClass+DateTimeFormatManager+DateTimeFormatHashTable.xml' as it doesn't exist
Warning: couldn't process type file `./../class/Novell.Directory.Ldap/Documentation/en/global/SupportClass+ArrayListSupport.xml' as it doesn't exist
Warning: couldn't process type file `./../class/Novell.Directory.Ldap/Documentation/en/global/SupportClass+ThreadClass.xml' as it doesn't exist
Warning: couldn't process type file `./../class/Novell.Directory.Ldap/Documentation/en/global/SupportClass+CollectionSupport.xml' as it doesn't exist
Warning: couldn't process type file `./../class/Novell.Directory.Ldap/Documentation/en/global/SupportClass+ListCollectionSupport.xml' as it doesn't exist
Warning: couldn't process type file `./../class/Novell.Directory.Ldap/Documentation/en/global/SupportClass+ArraysSupport.xml' as it doesn't exist
Warning: couldn't process type file `./../class/Novell.Directory.Ldap/Documentation/en/global/SupportClass+SetSupport.xml' as it doesn't exist
Warning: couldn't process type file `./../class/Novell.Directory.Ldap/Documentation/en/global/SupportClass+AbstractSetSupport.xml' as it doesn't exist
Warning: couldn't process type file `./../class/Novell.Directory.Ldap/Documentation/en/global/SupportClass+MessageDigestSupport.xml' as it doesn't exist
Warning: couldn't process type file `./../class/Novell.Directory.Ldap/Documentation/en/global/SupportClass+SecureRandomSupport.xml' as it doesn't exist
Error reading namespace XML for Novell.Directory.Ldap.Asn1 at ./../class/Novell.Directory.Ldap/Documentation/en/ns-Novell.Directory.Ldap.Asn1.xml
mdoc: System.TimeZoneNotFoundException: Exception of type 'System.TimeZoneNotFoundException' was thrown.
  at System.TimeZoneInfo.get_Local () [0x00000] in <filename unknown>:0 
  at System.TimeZoneInfo.GetDateTimeNowUtcOffsetFromUtc (DateTime time, System.Boolean& isAmbiguousLocalDst) [0x00000] in <filename unknown>:0 
  at System.DateTime.get_Now () [0x00000] in <filename unknown>:0 
  at ICSharpCode.SharpZipLib.Zip.ZipEntry..ctor (System.String name, Int32 versionRequiredToExtract, Int32 madeByInfo) [0x00000] in <filename unknown>:0 
  at ICSharpCode.SharpZipLib.Zip.ZipEntry..ctor (System.String name) [0x00000] in <filename unknown>:0 
  at Monodoc.Storage.ZipStorage.SetupEntry (ICSharpCode.SharpZipLib.Zip.ZipOutputStream zipOutput, System.String& id) [0x00000] in <filename unknown>:0 
  at Monodoc.Storage.ZipStorage.Store (System.String id, System.IO.Stream stream) [0x00000] in <filename unknown>:0 
  at Monodoc.Providers.EcmaSpecProvider.PopulateNode (System.Xml.XPath.XPathNodeIterator nodes, Monodoc.Node treeNode) [0x00000] in <filename unknown>:0 
  at Monodoc.Providers.EcmaSpecProvider.PopulateTree (Monodoc.Tree tree) [0x00000] in <filename unknown>:0 
  at Mono.Documentation.MDocAssembler.Run (IEnumerable`1 args) [0x00000] in <filename unknown>:0 
  at Mono.Documentation.MDoc.Run (System.String[] args) [0x00000] in <filename unknown>:0 
  at Mono.Documentation.MDoc.Main (System.String[] args) [0x00000] in <filename unknown>:0 
See `mdoc help' for more information.
mdoc: System.TimeZoneNotFoundException: Exception of type 'System.TimeZoneNotFoundException' was thrown.
  at System.TimeZoneInfo.get_Local () [0x00000] in <filename unknown>:0 
  at System.TimeZoneInfo.GetDateTimeNowUtcOffsetFromUtc (DateTime time, System.Boolean& isAmbiguousLocalDst) [0x00000] in <filename unknown>:0 
  at System.DateTime.get_Now () [0x00000] in <filename unknown>:0 
  at ICSharpCode.SharpZipLib.Zip.ZipEntry..ctor (System.String name, Int32 versionRequiredToExtract, Int32 madeByInfo) [0x00000] in <filename unknown>:0 
  at ICSharpCode.SharpZipLib.Zip.ZipEntry..ctor (System.String name) [0x00000] in <filename unknown>:0 
  at Monodoc.Storage.ZipStorage.SetupEntry (ICSharpCode.SharpZipLib.Zip.ZipOutputStream zipOutput, System.String& id) [0x00000] in <filename unknown>:0 
  at Monodoc.Storage.ZipStorage.Store (System.String id, System.IO.Stream stream) [0x00000] in <filename unknown>:0 
  at Monodoc.Providers.EcmaDoc.PopulateTreeFromIndexFile (System.String indexFilePath, System.String idPrefix, Monodoc.Tree tree, IDocStorage storage, System.Collections.Generic.Dictionary`2 nsSummaries, System.Func`2 indexGenerator, IEcmaProviderFileSource fileSource) [0x00000] in <filename unknown>:0 
  at Monodoc.Providers.EcmaProvider.PopulateTree (Monodoc.Tree tree) [0x00000] in <filename unknown>:0 
  at Mono.Documentation.MDocAssembler.Run (IEnumerable`1 args) [0x00000] in <filename unknown>:0 
  at Mono.Documentation.MDoc.Run (System.String[] args) [0x00000] in <filename unknown>:0 
  at Mono.Documentation.MDoc.Main (System.String[] args) [0x00000] in <filename unknown>:0 
See `mdoc help' for more information.
Makefile:148: recipe for target 'ecma334.tree' failed
make[7]: *** [ecma334.tree] Error 1
make[7]: *** Waiting for unfinished jobs....
Makefile:140: recipe for target 'Novell.tree' failed
make[7]: *** [Novell.tree] Error 1
mdoc: System.TimeZoneNotFoundException: Exception of type 'System.TimeZoneNotFoundException' was thrown.
  at System.TimeZoneInfo.get_Local () [0x00000] in <filename unknown>:0 
  at System.TimeZoneInfo.GetDateTimeNowUtcOffsetFromUtc (DateTime time, System.Boolean& isAmbiguousLocalDst) [0x00000] in <filename unknown>:0 
  at System.DateTime.get_Now () [0x00000] in <filename unknown>:0 
  at ICSharpCode.SharpZipLib.Zip.ZipEntry..ctor (System.String name, Int32 versionRequiredToExtract, Int32 madeByInfo) [0x00000] in <filename unknown>:0 
  at ICSharpCode.SharpZipLib.Zip.ZipEntry..ctor (System.String name) [0x00000] in <filename unknown>:0 
  at Monodoc.Storage.ZipStorage.SetupEntry (ICSharpCode.SharpZipLib.Zip.ZipOutputStream zipOutput, System.String& id) [0x00000] in <filename unknown>:0 
  at Monodoc.Storage.ZipStorage.Store (System.String id, System.IO.Stream stream) [0x00000] in <filename unknown>:0 
  at Monodoc.Providers.EcmaDoc.PopulateTreeFromIndexFile (System.String indexFilePath, System.String idPrefix, Monodoc.Tree tree, IDocStorage storage, System.Collections.Generic.Dictionary`2 nsSummaries, System.Func`2 indexGenerator, IEcmaProviderFileSource fileSource) [0x00000] in <filename unknown>:0 
  at Monodoc.Providers.EcmaProvider.PopulateTree (Monodoc.Tree tree) [0x00000] in <filename unknown>:0 
  at Mono.Documentation.MDocAssembler.Run (IEnumerable`1 args) [0x00000] in <filename unknown>:0 
  at Mono.Documentation.MDoc.Run (System.String[] args) [0x00000] in <filename unknown>:0 
  at Mono.Documentation.MDoc.Main (System.String[] args) [0x00000] in <filename unknown>:0 
See `mdoc help' for more information.
Makefile:136: recipe for target 'Mono.tree' failed
make[7]: *** [Mono.tree] Error 1
mdoc: System.TimeZoneNotFoundException: Exception of type 'System.TimeZoneNotFoundException' was thrown.
  at System.TimeZoneInfo.get_Local () [0x00000] in <filename unknown>:0 
  at System.TimeZoneInfo.GetDateTimeNowUtcOffsetFromUtc (DateTime time, System.Boolean& isAmbiguousLocalDst) [0x00000] in <filename unknown>:0 
  at System.DateTime.get_Now () [0x00000] in <filename unknown>:0 
  at ICSharpCode.SharpZipLib.Zip.ZipEntry..ctor (System.String name, Int32 versionRequiredToExtract, Int32 madeByInfo) [0x00000] in <filename unknown>:0 
  at ICSharpCode.SharpZipLib.Zip.ZipEntry..ctor (System.String name) [0x00000] in <filename unknown>:0 
  at Monodoc.Storage.ZipStorage.SetupEntry (ICSharpCode.SharpZipLib.Zip.ZipOutputStream zipOutput, System.String& id) [0x00000] in <filename unknown>:0 
  at Monodoc.Storage.ZipStorage.Store (System.String id, System.IO.Stream stream) [0x00000] in <filename unknown>:0 
  at Monodoc.Providers.EcmaDoc.PopulateTreeFromIndexFile (System.String indexFilePath, System.String idPrefix, Monodoc.Tree tree, IDocStorage storage, System.Collections.Generic.Dictionary`2 nsSummaries, System.Func`2 indexGenerator, IEcmaProviderFileSource fileSource) [0x00000] in <filename unknown>:0 
  at Monodoc.Providers.EcmaProvider.PopulateTree (Monodoc.Tree tree) [0x00000] in <filename unknown>:0 
  at Mono.Documentation.MDocAssembler.Run (IEnumerable`1 args) [0x00000] in <filename unknown>:0 
  at Mono.Documentation.MDoc.Run (System.String[] args) [0x00000] in <filename unknown>:0 
  at Mono.Documentation.MDoc.Main (System.String[] args) [0x00000] in <filename unknown>:0 
See `mdoc help' for more information.
Makefile:132: recipe for target 'netdocs.tree' failed
make[7]: *** [netdocs.tree] Error 1
mdoc: System.TimeZoneNotFoundException: Exception of type 'System.TimeZoneNotFoundException' was thrown.
  at System.TimeZoneInfo.get_Local () [0x00000] in <filename unknown>:0 
  at System.TimeZoneInfo.GetDateTimeNowUtcOffsetFromUtc (DateTime time, System.Boolean& isAmbiguousLocalDst) [0x00000] in <filename unknown>:0 
  at System.DateTime.get_Now () [0x00000] in <filename unknown>:0 
  at ICSharpCode.SharpZipLib.Zip.ZipEntry..ctor (System.String name, Int32 versionRequiredToExtract, Int32 madeByInfo) [0x00000] in <filename unknown>:0 
  at ICSharpCode.SharpZipLib.Zip.ZipEntry..ctor (System.String name) [0x00000] in <filename unknown>:0 
  at Monodoc.Storage.ZipStorage.SetupEntry (ICSharpCode.SharpZipLib.Zip.ZipOutputStream zipOutput, System.String& id) [0x00000] in <filename unknown>:0 
  at Monodoc.Storage.ZipStorage.Store (System.String id, System.IO.Stream stream) [0x00000] in <filename unknown>:0 
  at Monodoc.Providers.ErrorProvider.CloseTree (Monodoc.HelpSource hs, Monodoc.Tree tree) [0x00000] in <filename unknown>:0 
  at Mono.Documentation.MDocAssembler.Run (IEnumerable`1 args) [0x00000] in <filename unknown>:0 
  at Mono.Documentation.MDoc.Run (System.String[] args) [0x00000] in <filename unknown>:0 
  at Mono.Documentation.MDoc.Main (System.String[] args) [0x00000] in <filename unknown>:0

Comment 1 Timotheus Pokorra 2015-05-27 07:21:52 UTC
You can ignore the errors about Novell.Directory.Ldap, they seem to appear on the build logs for x86_64 as well, but the build should succeed anyway.

The real problem is the System.TimeZoneNotFoundException.
The exception is thrown here: 
https://github.com/mono/mono/blob/mono-4.0.0-branch/mcs/class/corlib/System/TimeZoneInfo.cs#L88

Please have a look at the function TimeZoneInfo CreateLocal () at
https://github.com/mono/mono/blob/mono-4.0.0-branch/mcs/class/corlib/System/TimeZoneInfo.cs#L121

This refers also to https://github.com/mono/mono/blob/mono-4.0.0-branch/mcs/class/corlib/System/TimeZoneInfo.cs#L163
defining timeZoneDirectory enclosed in "#if LIBC".

Will this help you to work on a patch?

Comment 2 Than Ngo 2015-05-28 11:28:49 UTC
thanks for your info, i will give a try and will report the results asap.

Than

Comment 3 Than Ngo 2015-05-28 23:27:53 UTC
Timotheus, i debug the mono codes and found that the error caused by this line

var list = new List<KeyValuePair<DateTime, TimeType>> (count);

in the function.

static List<KeyValuePair<DateTime, TimeType>> ParseTransitions (byte [] buffer, int index, int count, Dictionary<int, TimeType> time_types)

It's really strange, because this line looks correct for me

however i cannot get the backtrace. Do you know how i can get backtrace in mono instead exception?

It will help me to know why it crashes in this line.

Thanks

Comment 4 Dan Horák 2015-06-05 08:02:25 UTC
so it's the hardening (https://fedoraproject.org/wiki/Changes/Harden_All_Packages) that breaks the compiler, with

diff --git a/mono.spec b/mono.spec
index 55d5d8d..33d4ba6 100644
--- a/mono.spec
+++ b/mono.spec
@@ -1,4 +1,5 @@
-%global bootstrap 0
+%undefine _hardened_build
+%global bootstrap 1
 %if 0%{?rhel}%{?el6}%{?el7}
 %if 0%{?el6}
 %define mono_arches %ix86 x86_64 %{arm} sparcv9 alpha s390x ppc ppc64 ppc64le
@@ -13,7 +14,7 @@
 
 Name:           mono
 Version:        4.0.1
-Release:        8%{?dist}
+Release:        8%{?dist}.1
 Summary:        Cross-platform, Open Source, .NET development framework
 
 Group:          Development/Languages
@@ -46,10 +47,10 @@ Obsoletes:      mono-entityframework
 # need to bootstrap mono, comment out this BuildRequires
 # and don't delete the binaries in %%prep.
 
-BuildRequires: mono-core >= 4.0
+#BuildRequires: mono-core >= 4.0
 
 # JIT only available on these:
-ExclusiveArch: %mono_arches
+ExclusiveArch: %mono_arches ppc64le
 
 %define _use_internal_dependency_generator 0
 %define __find_provides env sh -c 'filelist=($(cat)) && { printf "%s\\n" "${filelist[@]}" | /usr/lib/rpm/redhat/find-provides && printf "%s\\n" "${filelist[@]}" | prefix=%{buildroot}%{_prefix} %{buildroot}%{_bindir}/mono-find-provides; } | sort | uniq'

I get http://ppc.koji.fedoraproject.org/koji/taskinfo?taskID=2525410 - all green. It's not only ppc64le that's broken with hardening, on s390x I see segfaults.

Comment 5 Dan Horák 2015-06-05 08:05:41 UTC
For the record, disabled hardening does not help on ppc64. Also the main change between F-22 (where my scratch builds succeeded) and F-23 (where it was failing) is the hardening.

Comment 6 Till Maas 2015-06-05 08:28:24 UTC
Do I understand correctly, that it fails to build on ppc64le because of the hardening but on pcc64 for unknown reasons? And s390x also needs hardening disabled?

Comment 7 Dan Horák 2015-06-05 08:45:08 UTC
(In reply to Till Maas from comment #6)
> Do I understand correctly, that it fails to build on ppc64le because of the
> hardening but on pcc64 for unknown reasons? And s390x also needs hardening
> disabled?

yes, ppc64 is different issue (#1225035), s390x is now tracked in #1228570. IBM should be looking at both the ppc64 failures (with some fixes already in the queue, but not public yet) and I'm in contact with the s390x upstream Mono maintainer.

Comment 8 Timotheus Pokorra 2015-06-18 05:45:41 UTC
I just saw https://github.com/mono/mono/pull/1868 which was merged 6 days ago:
"[ppc] Basic powerpc64 little endian enablement"

would that resolve some issues for the build as well?

Comment 9 Than Ngo 2015-07-06 20:29:34 UTC
no, i have applied the patches and rebuilt moni, the build faulure still happens, but on somewhere.

Comment 10 Jan Kurik 2015-07-15 14:06:42 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 23 development cycle.
Changing version to '23'.

(As we did not run this process for some time, it could affect also pre-Fedora 23 development
cycle bugs. We are very sorry. It will help us with cleanup during Fedora 23 End Of Life. Thank you.)

More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora23

Comment 11 Than Ngo 2015-08-28 11:09:53 UTC
this issue is fixed in rawhide and f23