also being able to create updated isos with livemedia-creator or livecd-creator are coming up with the no kernel found in the dracut stage

I have been having an issue like this. Network installs were working on 22-January-2019 (Tuesday). When I tried on 23-January-2019 (Wednesday) the Grub menu would only contain one item that said "System setup". It would then boot to that and end up in BIOS setup.

As a note I am running this on an aarch64 architecture system.

I am having the same issue (first seen on January 24). 

20-grub.install silently exits if no /etc/machine-id is found, so the kernel does not get copied to /boot from /usr/lib/modules and bootentries are not created. And machine-id is not generated because when post-install script from current systemd rpm runs, there is no ssl installed yet and generation fails silently.

Thanks for the blog ping - I'm backporting the fix for this now, we'll see if it helps. A new openQA test I just wrote should help with it, in fact: it builds a network installer image and tests an install from it. And indeed, it started failing on F29 updates yesterday. I don't know *what* change was sent to F29 which caused this bug to suddenly show up there (I think the change that triggered this on Rawhide was a fedora-release change, but that change doesn't seem to have showed up in F29...), but whatever, adding the dep will probably fix it and should be safe. The openQA test should confirm whether the fix works, also.

systemd-239-9.gite339eae.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-827d700f48

So turns out the openQA test *doesn't* confirm whether the fix works, because of a sort of missing feature in the test: https://pagure.io/fedora-qa/os-autoinst-distri-fedora/issue/89 . So, if someone with the necessary knowledge could grab the updated systemd, create a side repo with it, and run an install with that side repo added to confirm whether the fix works, that'd be great. Thanks!

(In reply to Adam Williamson from comment #6)
> So turns out the openQA test *doesn't* confirm whether the fix works,
> because of a sort of missing feature in the test:
> https://pagure.io/fedora-qa/os-autoinst-distri-fedora/issue/89 . So, if
> someone with the necessary knowledge could grab the updated systemd, create
> a side repo with it, and run an install with that side repo added to confirm
> whether the fix works, that'd be great. Thanks!

TL;DR
I tested it and it fixed the problem. My system booted.

Long version:
I was testing this on our in-house Beaker system and it took me a LONG time to figure out how to get my side-repo added to the ks.cfg via the Beaker job XML. I don't control the Beaker system, so couldn't modify the default ks.cfg. Kind of hacky but I added a %pre section which added a "repo" line to /run/install/ks.cfg. Luckily the ks.cfg gets re-read after the %pre section :)

%pre
echo "repo --name=sysd-test --baseurl=http://example.com/systemd-repo/ > /tmp/my-ks.cfg
cat /run/install/ks.cfg >> /tmp/my-ks.cfg
cp /tmp/my-ks.cfg /run/install/ks.cfg
%end

Hopefully there is some other method that is much easier and more elegant to do this. I would love to know about it! :)

We see this at work starting at the latest on January 23 too (our teams in Dublin and London saw it during their workday).

Right now we're mitigating the issue altogether by disabling the updates repo during netinstall, and applying security updates post-install.

systemd-239-9.gite339eae.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-827d700f48

John: thanks a lot for the confirmation! The update is queued for stable now, so this problem should go away for everyone soon. Sorry again for the inconvenience.

systemd-239-9.gite339eae.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.

*** Bug 1669609 has been marked as a duplicate of this bug. ***

*** Bug 1669055 has been marked as a duplicate of this bug. ***

*** Bug 1670467 has been marked as a duplicate of this bug. ***