Bug 1056365

It looks like arm64 support exists in mlton upstream, and there are
debian packages.

I will have hardware soon so I could try to get this working, although
mainly I'll be working on virt stuff.  However you could also try using
this qemu emulation technique:

http://rwmj.wordpress.com/2013/12/22/how-to-run-aarch64-binaries-on-an-x86-64-host-using-qemu-userspace-emulation/

It's relatively easy to set up and works well.

Is there a documented boot strap process? You can also do arm.koji scratch builds if you can put the process into a src.rpm or alternatively if it's documented I can implement

Richard: my success in following your instructions is what motivated me to work on this now. So thanks for that!
Also, I put the arm64 stuff into upstream mlton (but of course forgot one trivial but critical file so it currently won't build).

Peter: there isn't really a documented process. There are some stale instructions on mlton.org and a stale script, but it's really just 3-4 steps for bringing up a new architecture if you have a working C compiler and working mlton on another machine. Then it is a few more steps to actually bootstrap the package into Fedora.

OK well fell free to ping me on anything I can help with

This bug appears to have been reported against 'rawhide' during the Fedora 23 development cycle.
Changing version to '23'.

(As we did not run this process for some time, it could affect also pre-Fedora 23 development
cycle bugs. We are very sorry. It will help us with cleanup during Fedora 23 End Of Life. Thank you.)

More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora23

any updates?

Unfortunately mlton.org and in particular the porting notes
(http://mlton.org/PortingMLton) are both down because of the
ongoing sourceforge outage.  Also because they were using
robots.txt, there is no copy on archive.org.

I also checked the Fedora sources and the porting notes are
there (doc/guide/localhost/PortingMLton) but it's a rather old
copy of the source (from 2013) so probably doesn't contain
aarch64 support.

I guess we'll need to wait to see if sourceforge comes back up.

Upstream added support 10 days ago:
https://github.com/MLton/mlton/commit/c611cbbf9a88eeb86bec51e079ecbfe3c2cb3096

Alas I have not had time to work on this, but all the pieces should be there now.

I can have a go at this if someone can give me some kind of guide
for how to bootstrap it.  What I tried so far:

(1) Reading the PortingMLton notes: Didn't work because only
'-target self' is supported by our compiler.

(2) Looking at the spec file: We can enable the mlton_bootstrap flag,
but that won't work unless we have the %{name}-%{version}.arm64.tar.xz
file (from where?)

It's been a while, let me try to remember...


Take a look at add-cross: https://github.com/MLton/mlton/blob/master/bin/add-cross

Also see the github copy of the documentation:
https://github.com/MLton/mlton/blob/master/doc/guide/src/CrossCompiling.adoc

The basic steps are there. You probably want to do roughly this:
- you have to "make runtime" on the target machine. this should work because it is all C.
- you have to build and run print-constants on the target. you get the C source by running "mlton -build-constants true". save the output into a file called "constants"
- copy the output into build/lib/targets/... (I forget if this needs to be named something special)
- there are probably a few other things that have to go into the targets/.../ directory, look at what's in "self" to see.
- when you have this, you need to build mlton with "-keep g" and the proper -target flag (on the host machine). this will output C source. you can then copy this to the target machine and build mlton

At this point, you have a sort of skeleton mlton. You should do a full bootstrap on mlton on the target machine with this mlton. Once you have this, you can tar up these files and use to bootstrap the RPM.

It would be great to update the script upstream. The "-keep g" method is way better than cross compiling in most cases.

(In reply to Adam Goode from comment #10)

Are you on IRC anywhere?

When I'm writing the 'constants' file, do I need to use the -target parameter?
Apparently not because the mlton I built on the host only understands
'-target self'.

What should be the -target string for Fedora/aarch64?

I got up to this step:

> - when you have this, you need to build mlton with "-keep g" and the proper
> -target flag (on the host machine). this will output C source. you can then
> copy this to the target machine and build mlton

However what do I actually have to type to do the above build?

> At this point, you have a sort of skeleton mlton. You should do a full
> bootstrap on mlton on the target machine with this mlton. Once you have
> this, you can tar up these files and use to bootstrap the RPM.

Can I copy the mlton binary?  Won't it be a host (x86-64) binary and therefore
not run on the target?

$ file ./build/lib/mlton-compile 
./build/lib/mlton-compile: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=fd10b8d49093af830525d66b4891709673bab1c1, not stripped

> It would be great to update the script upstream. The "-keep g" method is way
> better than cross compiling in most cases.

(In reply to Richard W.M. Jones from comment #11)
> (In reply to Adam Goode from comment #10)
> 
> Are you on IRC anywhere?

Yes, I can get on IRC, but my morning (starting now) is full of meetings, so can't really do it right now.


> 
> When I'm writing the 'constants' file, do I need to use the -target
> parameter?
> Apparently not because the mlton I built on the host only understands
> '-target self'.

Any mlton binary should be able to generate the proper constants code. Generate this code on a working mlton installation, then compile and run the C file on your target.


> 
> What should be the -target string for Fedora/aarch64?
> 
> I got up to this step:
> 
> > - when you have this, you need to build mlton with "-keep g" and the proper
> > -target flag (on the host machine). this will output C source. you can then
> > copy this to the target machine and build mlton
> 
> However what do I actually have to type to do the above build?
> 

-keep g is an argument to mlton. You might have to poke into the makefile a bit.


> > At this point, you have a sort of skeleton mlton. You should do a full
> > bootstrap on mlton on the target machine with this mlton. Once you have
> > this, you can tar up these files and use to bootstrap the RPM.
> 
> Can I copy the mlton binary?  Won't it be a host (x86-64) binary and
> therefore
> not run on the target?
> 
> $ file ./build/lib/mlton-compile 
> ./build/lib/mlton-compile: ELF 64-bit LSB executable, x86-64, version 1
> (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for
> GNU/Linux 2.6.32, BuildID[sha1]=fd10b8d49093af830525d66b4891709673bab1c1,
> not stripped
> 

So once you do -keep g, you get a whole bunch of .c files that you have to copy over to the target, compile all together, and you'll get the mlton binary.



I don't have time today to look at this (maybe tonight/tomorrow). I would recommend asking at the mailing list, but the sourceforge outage seems to have brought those down as well.

Looks as if we need a C cross-compiler for this?  In any case I get:

$ cat /tmp/mlton 
#!/bin/bash -
/usr/bin/mlton -keep g "$@"
$ make TARGET=arm64-pc-linux MLTON="/tmp/mlton" -j1
...
cd gdtoa && arm64-pc-linux-gcc -std=gnu99   -fno-common  -pedantic -Wall -W -Wextra -Wformat=2 -Wswitch-default -Wswitch-enum -Wuninitialized -Winit-self -Wstrict-aliasing=2 -Wfloat-equal -Wundef -Wshadow -Wpointer-arith -Wbad-function-cast -Wcast-qual -Wwrite-strings -Waggregate-return -Wstrict-prototypes -Wold-style-definition -Wmissing-prototypes -Wmissing-declarations -Wmissing-field-initializers -Wmissing-noreturn -Wmissing-format-attribute -Wredundant-decls -Wnested-externs -fvisibility=hidden -m64 -I. -Iplatform -Wdisabled-optimization -O2 -fomit-frame-pointer -w -O1 -o arithchk.out arithchk.c
/bin/sh: arm64-pc-linux-gcc: command not found

We do have a cross-compiler (aarch64-linux-gnu-gcc) but it seems
we lack all the machinery to make it work like the cross-compiler
glibc.  (I was however hoping to do this just using the real
hardware rather than involving any cross-compilation).

You can do it without a cross compiler, but you have to manually copy around a lot of things. A new script could mostly automate this. I will file a bug.

Ah, actually add-cross should do this. All you need is ssh access to the remote machine. If it doesn't work, file a bug and/or send patches to https://github.com/MLton/mlton.

I think the script needs to be updated for arm64 though.

Richard: can I assign this to you for now? I'm not working on this right now, but I think there are enough hints here for you to proceed.

Let me know if I can help.

FYI there is this now:
https://fedoraproject.org/wiki/Packaging:Guidelines#Bootstrapping

I'm still lacking clear instructions on what to do.

I suggest getting one of the cheap/crap ARM64 dev boards available
(for around $100) and doing it yourself.

Or even cheaper, run Fedora/aarch64 under virtualization:
https://rwmj.wordpress.com/2015/05/26/fedora-22-aarch64-virt-builder-image/

I have the hardware, I will try to look into doing this. Feel free to take the bug if I don't get to it soon enough.

This bug appears to have been reported against 'rawhide' during the Fedora 24 development cycle.
Changing version to '24'.

More information and reason for this action is here:
https://fedoraproject.org/wiki/Fedora_Program_Management/HouseKeeping/Fedora24#Rawhide_Rebase

Adam: is there a status update for this?

I haven't had time to start this but I do have everything needed.

If anyone is interested in this before I get to it, I am happy to mentor them.

OK, excluded non bootstrapped arches for the moment

Hi,

I have successfully bootstrapped mlton's latest git head on a rawhide aarch64 machine (emulated in QEMU). Thank you for your explanation in this bug report, it was enough to get me started. (I'm interested in this project because I'm interested in improving Fedora support for SML; cf. my recent self-introduction: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/DF35Y2WCXGFTSYMUUEHVZ4CS37CS7OBS/)

I bootstrapped git head because I honestly didn't think this would work, and then once I was far enough along to see it would work, I decided I would see it through. My next plan is to bootstrap the 20130715 release rather than a git snapshot. Hopefully that can then be pushed to Fedora!

Here is a tarball of the install/usr directory in case you want to take a look at the results, unfortunately wordpress mangled the file extension, but I don't have a better hosting solution at present:
https://campuspress.yale.edu/dmoerner/files/2017/01/mlton-bootstrap-git-49e68e391c.tar-1lc9o5x.gz

The process is fairly easy but takes a few days given my limited and slow qemu setup (and my time constraints!). Here's a brief description of what I did:

1. Using add-cross: This basically worked out of the box. I had to apply a minor patch to add-cross to make sure $PATH worked fine. The magic incantation is "./add-cross aarch64-redhat-linux arm64 linux $qemu-machine-ip"

2. Regression testing: This was very slow but I found no regressions, following the instructions here: https://github.com/MLton/mlton/blob/master/doc/guide/src/PortingMLton.adoc

3. Bootstrapping: This worked fine, until I got to mlyacc. mlyacc wouldn't compile. That's for two reasons; first, it tries to use system paths for mllex and mlyacc. Second, and more seriously, it needs mlyacc to produce the src/yacc.grm.{sig,sml} files. I had to use the following work-around:
a. On the original host machine, make -C mlyacc TARGET=aarch64-redhat-linux. This failed to actually build mlyacc for me. But it did get far enough to output the relevant yacc.grm.* files.
b. Copy the yacc.grm.* files to the target machine, cd to the mlyacc directory, and run make MLLEX=../mllex/mllex.
c. Just to be sure that everything is bootstrapped on the target, I deleted mlyacc/src/yacc.grm.*, and then ran again make MLLEX=../mllex/mllex MLYACC=mlyacc.

This produced success!

Daniel:

Awesome!

Do you think you could get your cross compilation fixes upstream? That would help.

The other thing I was looking to try was getting the polyml bootstrap working again (it looks like it is broken in some of the same ways as you mentioned). Even if it is slow, a polyml bootstrap would be nice since polyml bootstraps from C.

Hi Adam,

I submitted the add-cross fix pull request upstream: https://github.com/MLton/mlton/pull/166. I will submit an issue with details about the mlyacc issue after looking into it in more detail.

Bootstrapping with polyml is a great idea. Although, unfortunately, polyml is having its own problems on ppc64 at the moment, https://bugzilla.redhat.com/show_bug.cgi?id=1394556#c6.

The add-cross fixes have been accepted upstream, but they were mostly clarification of the documentation rather than better automation.

Here is the arm64 patch from upstream I applied to the source:

https://dmoerner.fedorapeople.org/mlton-enable-arm64.patch

And here is a tarball of the bootstrapped mlton, it of course includes the generated constants file in usr/lib/mlton/targets/self/constants

https://dmoerner.fedorapeople.org/mlton-20130715.aarch64.tar.xz

And here is a detached signature using my GPG key attached to my FAS account, 6B2F6739, since this is a precompiled binary after all:

https://dmoerner.fedorapeople.org/mlton-20130715.aarch64.tar.xz.sig

I looked a bit into bootstrapping with polyml-mlton. It doesn't look trivial to me; not only do we have to generate mlton/mlton-polyml.use, which looks to be architecture specific, but it also starts generating calls to mlyacc and mllex. The situation is a bit better here, since mlyacc, at least, is used to generate architecture-independent files. So I think, although I haven't tested it, that what you'd have to do is: (a) Use add-cross to generate a cross-compiler mlton target, (b) On the host, use the cross-compiler to generate mlton/mlton-polyml.use, and the relevant *.sml and *.sig files in mlton/front-end, and then (c) copy these files to the target machine and bootstrap with polyml.

Unfortunately, I'm a bit busy right now, so I don't have any more time to devote to this at the moment.

I am working on this now.

https://koji.fedoraproject.org/koji/taskinfo?taskID=26104092