Tagging in Nalin and Miloslav - this looks like a c/image issue.

From the OCI image spec, the architecture field should match GOARCH, which uses aarch64, not arm64

Yes, the image is invalid:
> $ skopeo copy docker://registry.fedoraproject.org/fedora@sha256:6050d1611e8541ee99a4cbe705fed2d29cfacf39985187f6cafe2c9e51639dd2  dir:t
> $ python -mjson.tool t/5490a83c7a4a488e24f32ada02b06a0b5fc9bd3f3ff8745ce949546b256805c2 
> {
>    "architecture": "aarch64",
>    "comment": "Created by Image Factory",
> …

(note that the manifest list already points to the image using the "arm64" architecture name, otherwise it would not be found at all:
> $ skopeo inspect --raw docker://registry.fedoraproject.org/fedora:31 | python -mjson.tool
>            "digest": "sha256:6050d1611e8541ee99a4cbe705fed2d29cfacf39985187f6cafe2c9e51639dd2",
>            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
>            "platform": {
>                "architecture": "arm64",
>                "os": "linux"

For now, the workaround would be to manually inspect the multi-arch manifest list, then refer to the architecture-specific image by digest (as done above), and use --override-arch to ~bypass the check.

Based on the “created by” comment, reassigning to Image Factory.

Yeah just been checking specs https://github.com/opencontainers/image-spec/blob/master/image-index.md#image-index-property-descriptions and it is on the registry/base image pipeline side. Also other workaround is to stay on podman-2:1.6.2-2.fc31.aarch64 for a bit.

This may be resolved by https://github.com/containers/image/issues/799 (we're reverting some of the checks related to multiarch) but I'm not certain. I think that we did establish that "aarch64" is the correct string.

No, arm64 is the correct one, per https://github.com/opencontainers/image-spec/blob/master/config.md#properties .

Assuming "arm64" is the correct value for the architecture on "aarch64"(Go(GOARCH) calls it "arm64"). IMHO the check should be relaxed to accept both(for historical reasons), but we should move to "arm64" in the metadata of images that we produce, so we are producing images that comply with current standards.

I have opened https://github.com/redhat-imaging/imagefactory/pull/434 upstream(it should resolve the issue) could somebody take a look at it please.

I have been trying to test the change in the PR, aplying this on top of the imagefactory and imagefactory-plugins. Running with the koji configs from recent job for the base container image I'm hitting this:
"2020-03-03 15:04:34,911 DEBUG imgfac.FilePersistentImageManager.FilePersistentImageManager thread(c190424e) Message: Saved metadata for image (cb401941-6aed-4f91-ac21-308d8ec8259e): {'type': 'BaseImage', 'data': '/var/lib/imagefactory/storage/cb401941-6aed-4f91-ac21-308d8ec8259e.body', 'template': "<template>\n    <name>Fedora-Container-Minimal-Base-32-20200227.n.0.aarch64</name>\n        <os>\n            <name>Fedora</name>\n            <version>22</version>\n            <arch>aarch64</arch>\n            <install type='url'>\n                <url>https://kojipkgs.fedoraproject.org/compose/branched/Fedora-32-20200227.n.0/compose/Everything/aarch64/os</url>\n            </install>\n        <icicle>\n                <extra_command>rpm -qa --qf '%{NAME},%{VERSION},%{RELEASE},%{ARCH},%{EPOCH},%{SIZE},%{SIGMD5},%{BUILDTIME}\n'</extra_command>\n            </icicle>\n        </os>\n    <description>Fedora-Container-Minimal-Base-32-20200227.n.0.aarch64 OS</description>\n    <disk>\n        <size>5G</size>\n    </disk>\n</template>\n", 'icicle': None, 'percent_complete': 0, 'properties': {}, 'parameters': {'offline_icicle': 'false', 'install_script': '#version=DEVEL\n# Keyboard layouts\nkeyboard \'us\'\n# Root password\nrootpw --iscrypted --lock locked\n# Reboot after installation\nreboot\n# System timezone\ntimezone Etc/UTC --isUtc --nontp\n# Use text mode install\ntext\n# Network information\nnetwork  --bootproto=dhcp --device=link --activate\nrepo --name="koji-override-0" --baseurl=https://kojipkgs.fedoraproject.org/compose/branched/Fedora-32-20200227.n.0/compose/Everything/aarch64/os\n# Use network installation\nurl --url="https://kojipkgs.fedoraproject.org/compose/branched/Fedora-32-20200227.n.0/compose/Everything/aarch64/os"\n\n# System bootloader configuration\nbootloader --disabled\nautopart --type=plain --nohome --noboot --noswap\n# Clear the Master Boot Record\nzerombr\n# Partition clearing information\nclearpart --all\n\n%post --logfile=/root/anaconda-post.log --erroronfail\nset -eux\n\n# Set install langs macro so that new rpms that get installed will\n# only install langs that we limit it to.\nLANG="en_US"\necho "%_install_langs $LANG" > /etc/rpm/macros.image-language-conf\n\n# https://bugzilla.redhat.com/show_bug.cgi?id=1727489\necho \'LANG="C.UTF-8"\' >  /etc/locale.conf\n\n# https://bugzilla.redhat.com/show_bug.cgi?id=1400682\necho "Import RPM GPG key"\nreleasever=$(rpm --eval \'%{fedora}\')\nrpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-primary\n\necho "# fstab intentionally empty for containers" > /etc/fstab\n\n# Remove machine-id on pre generated images\nrm -f /etc/machine-id\ntouch /etc/machine-id\n\n%end\n\n%post --logfile=/root/anaconda-post.log --erroronfail\n# remove some random help txt files\nrm -fv usr/share/gnupg/help*.txt\n\n# Pruning random things\nrm usr/lib/rpm/rpm.daily\nrm -rfv usr/lib64/nss/unsupported-tools/  # unsupported\n\n# Statically linked crap\nrm -fv usr/sbin/{glibc_post_upgrade.x86_64,sln}\nln usr/bin/ln usr/sbin/sln\n\n# Remove some dnf info\nrm -rfv /var/lib/dnf\n\n# don\'t need icons\nrm -rfv /usr/share/icons/*\n\n#some random not-that-useful binaries\nrm -fv /usr/bin/pinky\n\n# we lose presets by removing /usr/lib/systemd but we do not care\nrm -rfv /usr/lib/systemd\n\n# if you want to change the timezone, bind-mount it from the host or reinstall tzdata\nrm -fv /etc/localtime\nmv /usr/share/zoneinfo/UTC /etc/localtime\nrm -rfv  /usr/share/zoneinfo\n\n# Final pruning\nrm -rfv /var/cache/* /var/log/* /tmp/*\n\n%end\n\n%post --nochroot --logfile=/mnt/sysimage/root/anaconda-post-nochroot.log --erroronfail\nset -eux\n\n# https://bugzilla.redhat.com/show_bug.cgi?id=1343138\n# Fix /run/lock breakage since it\'s not tmpfs in docker\n# This unmounts /run (tmpfs) and then recreates the files\n# in the /run directory on the root filesystem of the container\n# NOTE: run this in nochroot because "umount" does not exist in chroot\numount /mnt/sysimage/run\n# The file that specifies the /run/lock tmpfile is\n# /usr/lib/tmpfiles.d/legacy.conf, which is part of the systemd\n# rpm that isn\'t included in this image. We\'ll create the /run/lock\n# file here manually with the settings from legacy.conf\n# NOTE: chroot to run "install" because it is not in anaconda env\nchroot /mnt/sysimage install -d /run/lock -m 0755 -o root -g root\n\n\n# See: https://bugzilla.redhat.com/show_bug.cgi?id=1051816\n# NOTE: run this in nochroot because "find" does not exist in chroot\nKEEPLANG=en_US\nfor dir in locale i18n; do\n    find /mnt/sysimage/usr/share/${dir} -mindepth  1 -maxdepth 1 -type d -not \\( -name "${KEEPLANG}" -o -name POSIX \\) -exec rm -rfv {} +\ndone\n\n%end\n\n%packages --excludedocs --nocore --instLangs=en --excludeWeakdeps\nbash\ncoreutils\nfedora-release-container\nglibc-minimal-langpack\nmicrodnf\nrpm\nopenssh-server\nshadow-utils\nsssd-client\nsystemd\nutil-linux\n-dosfstools\n-e2fsprogs\n-fuse-libs\n-gnupg2-smime\n-grubby\n-kernel\n-libss\n-pinentry\n-shared-mime-info\n-trousers\n-xkeyboard-config\n\n%end\n', 'libvirt_xml': '<domain type="kvm">\n  <name>factory-build-cb401941-6aed-4f91-ac21-308d8ec8259e</name>\n  <memory>2097152</memory>\n  <currentMemory>2097152</currentMemory>\n  <uuid>fbb88ef7-a7c9-40f4-ac69-ea2dd3f4c649</uuid>\n  <clock offset="utc"/>\n  <vcpu>1</vcpu>\n  <features>\n    <acpi/>\n  </features>\n  <cpu mode="custom" match="exact">\n    <model fallback="allow">host</model>\n  </cpu>\n  <os>\n    <type arch="aarch64" machine="virt">hvm</type>\n    <boot dev="hd"/>\n    <loader readonly="yes" type="pflash">/usr/share/AAVMF/AAVMF_CODE.fd</loader>\n    <nvram template="/usr/share/AAVMF/AAVMF_VARS.fd"/>\n  </os>\n  <on_poweroff>destroy</on_poweroff>\n  <on_reboot>destroy</on_reboot>\n  <on_crash>destroy</on_crash>\n  <devices>\n    <graphics port="-1" type="vnc"/>\n    <interface type="bridge">\n      <source bridge="virbr0"/>\n      <mac address="52:54:00:99:45:51"/>\n      <model type="virtio"/>\n    </interface>\n    <input bus="usb" type="tablet"/>\n    <controller type="usb" index="0"/>\n    <input type="keyboard" bus="usb"/>\n    <serial type="pty">\n      <target port="0"/>\n    </serial>\n    <serial type="tcp">\n      <source mode="bind" host="127.0.0.1" service="20720"/>\n      <protocol type="raw"/>\n      <target port="1"/>\n    </serial>\n    <rng model="virtio">\n      <rate bytes="1024" period="1000"/>\n      <backend model="random">/dev/random</backend>\n    </rng>\n    <disk device="disk" type="file">\n      <target dev="vda" bus="virtio"/>\n      <source file="/var/lib/imagefactory/storage/cb401941-6aed-4f91-ac21-308d8ec8259e.body"/>\n      <driver name="qemu" type="raw"/>\n    </disk>\n  </devices>\n</domain>\n'}, 'status': 'FAILED', 'identifier': 'cb401941-6aed-4f91-ac21-308d8ec8259e', 'status_detail': {'activity': 'Base Image build failed with exception.', 'error': 'download: /etc/init.d/sshd: No such file or directory'}}
2020-03-03 15:04:34,912 ERROR imgfac.Builder.Builder thread(c190424e) Message: Exception encountered in _build_image_from_template thread
2020-03-03 15:04:34,912 ERROR imgfac.Builder.Builder thread(c190424e) Message: download: /etc/init.d/sshd: No such file or directory
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/oz/RedHat.py", line 446, in _collect_setup
    self._image_ssh_setup_step_2(g_handle)
  File "/usr/lib/python3.7/site-packages/oz/RedHat.py", line 310, in _image_ssh_setup_step_2
    startuplink = self._get_service_runlevel_link(g_handle, 'sshd')
  File "/usr/lib/python3.7/site-packages/oz/RedHat.py", line 156, in _get_service_runlevel_link
    lines = g_handle.cat('/etc/init.d/' + service).split("\n")
  File "/usr/lib/python3.7/site-packages/oz/GuestFSManager.py", line 183, in cat
    return self.g_handle.cat(fname)
  File "/usr/lib64/python3.7/site-packages/guestfs.py", line 1847, in cat
    r = libguestfsmod.cat(self._o, path)
RuntimeError: download: /etc/init.d/sshd: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/imgfac/Builder.py", line 132, in _build_image_from_template
    self.os_plugin.create_base_image(self, template, parameters)
  File "/usr/lib/python3.7/site-packages/imagefactory_plugins/TinMan/TinMan.py", line 363, in create_base_image
    builder.base_image.icicle = self.guest.customize_and_generate_icicle(libvirt_xml)
  File "/usr/lib/python3.7/site-packages/oz/Linux.py", line 357, in customize_and_generate_icicle
    return self._internal_customize(libvirt_xml, "gen_and_mod")
  File "/usr/lib/python3.7/site-packages/oz/Linux.py", line 307, in _internal_customize
    self._collect_setup(modified_xml)
  File "/usr/lib/python3.7/site-packages/oz/RedHat.py", line 466, in _collect_setup
    self._image_ssh_teardown_step_2(g_handle)
  File "/usr/lib/python3.7/site-packages/oz/RedHat.py", line 192, in _image_ssh_teardown_step_2
    startuplink = self._get_service_runlevel_link(g_handle, 'sshd')
  File "/usr/lib/python3.7/site-packages/oz/RedHat.py", line 156, in _get_service_runlevel_link
    lines = g_handle.cat('/etc/init.d/' + service).split("\n")
  File "/usr/lib/python3.7/site-packages/oz/GuestFSManager.py", line 183, in cat
    return self.g_handle.cat(fname)
  File "/usr/lib64/python3.7/site-packages/guestfs.py", line 1847, in cat
    r = libguestfsmod.cat(self._o, path)
RuntimeError: download: /etc/init.d/sshd: No such file or directory
"
Any hints on what can be going wrong? Can someone take a look on the PR please?

I've built this for Ian, I'm guessing it's going to need to be deployed to infra, which for Beta will need FBR.

For reference the update is https://bodhi.fedoraproject.org/updates/FEDORA-2020-e29a47204e (for some reason bodhi doesn't want to add the RHBZ)

I just want to stress that I have not been able yet to test successfully this change/fix(AFAIK it should fix the issues) locally due to some issue in the OZ(with configs from koji). IMO it shouldn't be directly pushed to prod without testing in stg.

Sorry, have had typo in the patch. Forgot the self... PR updated. And tested locally via imagefactory --debug target_image --template=tdl-aarch64.xml --file-parameter install_script koji-f32-build-41965801-base.ks --parameter offline_icicle true docker. AFAIK should be close approximation how koji is using imagefactory. 

Could you please respin the build with the fixed patch? Thank you.

Updated the patch and pushed builds for F-32+

FEDORA-2020-e858471cbe has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2020-e858471cbe

FEDORA-2020-e858471cbe has been pushed to the Fedora 32 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-e858471cbe`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-e858471cbe

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2020-e858471cbe has been pushed to the Fedora 32 stable repository.
If problem still persists, please make note of it in this bug report.