Back

Let's first talk about the things that doesn't work:

- Proton games fail to start. I suspect this is a problem of sommelier/Xwayland, but I don't think it'd be wise investing time in this now, as there are just too many moving pieces.

- Mono/XNA games fail with a weird Unicode error. Seems one of those silly issues that are easy to fix.

- Some games, like Return to Monkey Island, require OpenGL 4.x and/or Vulkan, which is not there yet for Asahi.

Now, the good news:

Half-Life 2 works well. The performance is good with some minor punctual stuttering, which I *think* it's CPU-bound as I see the MatQueue0 thread jumping up when that happens. But definitely playable.

Portal 2 has more trouble keeping the framerate than its predecessor, but that won't be too bad if it wasn't for a crash (SIGSEGV) just after the first chamber. I think @lina also hit this issue and fixed it, so perhaps my mesa build is missing some commit.

Terraria says it runs at 50fps, but *feels* slow. I haven't played this game before, so maybe it's just like that?

BONUS TRACK: Let's also give some love to the indie games from itch.io which doesn't force us to run the horrible, horrible Steam launcher. This one in particular, Buck Up And Drive!, has some really nice graphics and works like a charm in this environment.

Okay, so seems like this can be some useful tech to integrate into Fedora Asahi, but how can we achieve that? These are roughly the steps:

1. Clean up the code and create MRs for both mesa and virglrenderer implementing DRM native context for the Asahi driver. Those depend on an stable kernel uAPI, so that's going to take a while to get upstreamed.

2. Finish porting virtio-gpu from crosvm to libkrun, including a partial implementation of rutabaga_gfx.

3. While there, also port virtio-snd and implement a piperwire backend. This one is fairly trivial.

4. Create a ready-to-go OCI image with FEX and a x86_64 root filesystem with the updated mesa drivers that can be consumed by krunvm.

5. Profit?

The end goal is enabling Fedora Asahi users to run Steam on their systems without switching to a 4K kernel, and by running a single command:

krunvm start steamvm

Let's see how far we get.

@slp This sounds really promising for an easy and convenient user perspective! What impact would such an environment have on FEX development/debugging on Asahi?

I don't have much experience with krunvm, but anecdotally containers are always a royal pain to integrate development tooling with. Would a 4K host kernel be the better option here still?

(Asking for a friend )

@neobrain I’m not a FEX expert, but I don’t think debugging it in a VM should by any different than doing it bare metal.

@slp Does this mean you can just use host gdb to start a new FEX session, where FEX is built from a local source tree in the host filesystem?

Or would each source modification require uploading the new FEX build into some container image?

Does krunvm need to be started by gdb as well (how do you attach to FEX specifically then?), or can it run as a "background service" of sorts?

@neobrain It’s actually much simpler. While krunvm uses OCI images as a source for the root fs, it doesn’t follow the container semantics. The filesystem for each microVM is persistent, so you can install gdb, gcc, and whatever other tools and deps you need, and debug a program from inside the VM.

@slp How do I get the actual FEX source into the VM? Are filesystems shared between guest/host, or does one need to map individual directories explicitly? A bit worried about source-level debugging not working for host-built binaries (where the source file paths might not match up the guest filesystem).

Having to build FEX within the VM wouldn't be outrageous, but obviously you'd still want to edit the actual source in an IDE on the host :)

@slp Though I guess even without fully shared filesystems, the VM is slim enough that the FEX source can simply be mapped to the same path that it has on the host.

I've been burnt one too many times by developer UX clearly being an afterthought in Docker, where I constantly ran into issues like this. Sounds like all that can be nicely avoided with this microVM architecture!

@neobrain Indeed! libkrun integrates a virtio-fs server and, by default, the root filesystem is actually serviced from a directory on the host. And additional directories can also be mapped in the guest from arbitrary locations on the host.

@sven @slp it would be still nice if enough HW (mostly PCIe) works on a 4k distro kernel to avoid the need for special install images or 4k and 16k kernels in those images

@sven @slp but Lina’s idea of using 16k folios for DMA might be easier to implement/merge

@janne @sven @slp did you see this? https://lwn.net/SubscriberLink/943758/9abed6231b52be96/
Could this be applicable to Apple silicon hardware?

@lina Ack. I’ll clean up the code and create a draft MR there.

@slp By the way, about game specific issues:

- Stuttering is normal due to FEX JIT jank and/or GPU driver shader variant compilation jank. Both are things that should improve in the future, and shouldn't be related to the VM.
- Terraria was weirdly slow for me too, I think my conclusion was that it was CPU-bound for some reason? But I should look again, it could also be hitting pathological cases in the GPU driver.

Also I'm not sure if you tried forcing the GL version to 3.3 or so (via Mesa environment variables). Anything that doesn't need geometry shaders should work with that hack, which is a nice boost to GL version compat!

Proton *should* work but you need to at the very least force wined3d, since by default it'll try to use dxvk. Also I had to use an older Proton version, not sure if that's fixed (probably a FEX issue?)

Another thing we need to do is figure out how to either forward TSO mode control into the VM, or just run the entire VM in TSO mode. This is super important for FEX performance/correctness!

@lina Thanks for the insight! TBH, after getting the aarch64 stuff working, I didn't invest much time into debugging the issues with x86 software, since I think most of them won't be related to the virtualization layer.

Except, perhaps, some exceptional cases with games getting confused by Sommelier/Xwayland not following the full x11/wm semantics, specially with the ones bundling old libraries (I'm just guessing here). Anyway, those should be easily fixable, once properly debugged.

@lina While I'm kind of tempted to try to get more titles working, I think I'll better spend the time publishing the stuff so others can try it out too. I'd like to also set up a COPR with experimental versions of libkrun/krunvm, so the mesa/virglrenderer changes can be easily tested.

What's the status of TSO management in the kernel? Is there already a way to tag a task to be "TSO-enabled"?

@slp There are patches for TSO in our kernel (using prctl()), though it won't work properly in KVM right now since it has no idea about the feature. I guess we need to add all the stuff to make that work in KVM too...

@lina I see. I’ll take a look at the KVM bits once the other stuff settles, if no one beats me to it.

@slp @lina @marcan has already started discussing with maz in #asahi-dev
see today’s log https://oftc.irclog.whitequark.org/asahi-dev/2023-09-10

@xTeixeira @slp Oh damn, nice!... also for what it's worth the game does run without the Steam Launcher, you just don't get them cheevos and whatnot.