Every time I click on one of these posts, I'm expecting it to be a tiny KVM switch. When did this whole KVM nomenclature catch on for virtual machines?
Is there any way that TinyKVM + KVM Server could ever be made to work with a GUI program? The sandboxing performance seems free and possibly safer than other solutions.
Instead of firejail or bubblewrap would it ever be possible for me to wrap say Firefox (or a much less complicated GUI program) inside of TinyKVM and restrict it to just network access and reading/writing to ~/Downloads? Likely a way more ambitious target than you had ever imagined, but I can dream.
I am wondering if I could default wrap every command on my terminal to run inside a TinyKVM, no network access, and only permissions to the current directory or below.
That really isn't unreasonable at all IMO, it's just that it might be hard to do with userspace syscall emulation, since graphical programs will likely need a lot more of the syscall surface. For X11 and Wayland, you'll need some way of handling UNIX domain sockets. Wayland applications will require shared memory too, though you could get away with something like Waypipe instead to serialize everything. You'd probably want some sort of intermediary between any X11/Wayland communications anyways, just to add additional isolation.
It might be easier to adapt gVisor to handle this sort of workload. Adjacent comment mentions Qubes which does the same thing but uses an entire guest kernel.
(If you are creative enough, you can probably come up with some solutions. Qt apps could be made to work with a custom QPA that can somehow funnel information in and out of the sandbox. You could definitely run something like Waypipe or Xpra in the sandbox too, but again I imagine those would wind up requiring a much greater degree of emulation. It's not like I've actually tried this, though, so I could be off.)
TinyKVM is probably most similar to gVisor in KVM platform mode. TinyKVM implements a smaller number of sys calls and is focussed on making resets as fast as possible.
Running sys calls on the host means there is approximately 1µs overhead per syscall from exiting and entering KVM so I'm not sure how well that would work for GUI applications.
And we currently only have very rudimentary support for threads, enough for a server program with ancillary threads to boot up but the expectation is currently that the call into TinyKVM only runs a single thread and we fork multiple copies of the VM to handle requests in parallel.
> Running sys calls on the host means there is approximately 1µs overhead per syscall from exiting and entering KVM so I'm not sure how well that would work for GUI applications.
That made me rather curious how many syscalls a complex GUI application might issue. I wanted to see how many syscalls were happening across my entire system. Thanks to StackOverflow I have a snippet that seems correct[1]:
> perf stat -e raw_syscalls:sys_enter -a -I 1000 sleep 5
Using this, it seems that most programs (as you would probably guess) don't execute a whole lot of syscalls when they're idle. However, starting a complex GUI program definitely causes a pretty massive flurry of syscalls. Starting winecfg without an already-existing wineserver spews a lot of syscalls, somewhere in the neighborhood of 500,000. If we assume that each syscall takes on average around 2µs including the overhead and that they're all serial, I guess that would add up to about 1 second spent on syscalls. That's probably making way too many assumptions, but it does make me feel like it's not completely infeasible to run GUI applications inside of a sandbox like this, though it may very not be compelling when the overhead is factored in.
And of course, just because it could be done does not mean it should, anyway. Even if this is a good idea, I doubt it makes any sense for TinyKVM to be attempting to do it. What TinyKVM does do is already very interesting and probably a lot more practical anyways. It'd probably be better to fork off or build an entire purpose-built sandbox for GUI software, realistically.
Still, pretty interesting stuff to think about.
> And we currently only have very rudimentary support for threads, enough for a server program with ancillary threads to boot up but the expectation is currently that the call into TinyKVM only runs a single thread and we fork multiple copies of the VM to handle requests in parallel.
BTW, I think this design is really cool. This is something I have wanted to exist for a while, even though I don't practically need it.
Although I didn’t fully grasp half of it, I thoroughly enjoyed reading it. I was hooked from the beginning to the very end. I’m genuinely excited about the potential of TinyKVM. It’s unbelievable how far we’ve come from the early days of VMWare led virtualization, and the fact that we have such powerful machines that anyone can buy! We’ve even got much better tooling to squeeze out more performance without risking safety/security (Rust FTW!).
I want to be more deliberate about securing my tools, but all of the options seem so complex that I do not know where to begin. Then you get various pithy statements like, "chroot is not a security layer", "X cannot be used when you use Y", and it feels hopeless for a novice. Most of the documentation for these tools seem to expect a baseline system administration greater than my own.
I instead lean on heavyweight VMs, but would love something like this which should be a hard security boundary for little cost.
I want to love Qubes, but it is a lot more heavyweight than I want to pursue. I have no crypto fortune or government/industrial secrets worth stealing, so it would be putting on a lot of pain knowing I am not a person of interest. I already run my development work inside a VM, but that has some papercuts. Going full Qubes would probably get even more annoying.
A security/isolation layer like this I could use for free feels like it would get me so close to the Qubes ideal without having to completely change how I interface with my machine.
Given the use of the word "container" that seems to be using Linux namespacing rather than KVM. In case of containers, the isolation is provided solely by the Linux kernel, plus of course any additional defenses you add on top of it. While Guix shell having a built-in way to spawn isolated containers is extremely cool (I use NixOS. As far as I know, Nix does not have an equivalent feature) it seems like from a security standpoint, it would just be similar to using bubblewrap or Firejail directly. Though I like this idea. Seems very useful and convenient.
What I think we're really after though is something like gVisor, where the guest program is completely isolated from the host kernel, and the daemons that allow the guest program to reach the outside world are themselves highly locked down by the host kernel using technologies like seccomp-bpf and namespacing, on top of whatever constraints and validation they apply on their own. While nothing is foolproof, this feels like, if done carefully, it would give you a very good layer of isolation that would be extremely challenging to bypass. I reckon that the sandbox would cease to be the most interesting attack target in a system like gVisor, since in any complicated system, there will probably always be some lower-hanging fruit. (And of course, TinyKVM seems to be basically in the same wheelhouse. None of these solutions are designed to run GUI software, though I reckon it probably could be made to work.)
I admit I havent investigated this thoroughly, but I suspect the low hanging fruit in the tinykvm case is having rw access to /dev/kvm
I think it should be possible to pass /dev/kvm as an open fd to daemons like kvm server and mark it as non-inheritable. As long as the vm is in a subprocess it would be okay I guess.
I'm pretty hopeful that the combination of per-request isolation and the new snapshot functionality we're currently working on will be a big step forward for those running server-side JS at scale.
Having each request start from the exact same program state should make reproducing and fixing production issues easier. In a way it combines the predictability of the CGI programming model with the speed of a warmed modern JIT runtime.
This is amazing! I am also a little bit obsessed with fast-booting kvm for per-request isolation, and have managed to get Linux to pid1 in 3.6ms, I am starting to go a little insane because I don't know how to measure the rest of the CPU time (would love a flamegraph somehow) -- the ftrace data just... confuses me
Assuming your containers are secure to begin with (which can be tricky to set up), when a new container escape kernel bug is inevitably released you're in a race to patch it before someone exploits your system.
Firecracker runs a full Linux guest within KVM while TinyKVM runs just a single process within KVM and handling syscalls on the host by validating permissions then calling the host kernel syscall.
This minimises memory usage and lets us track file descriptors which lets us very quickly reset the guest process (under 100us for deno.)
I spent a while mixing this up with PiKVM and was having trouble understanding how any of it would fit in with that project. Made a lot more sense once I got over that.
I read until "gVisor, system call emulation" and though that this is some kind of IP-KVM project port to RTOS or microcontroller or something other thing which reuses Linux code but does not run Linux.
reply