It would be pretty cool to understand how all of that actually works and generally what the hell is going on when one starts vmconnect.exe. Or WSL for that matter. Okay, let’s do this one last time, yeah? For real this time.

tl;dr - how do I configure Hyper-V and xrdp ?

In 2026 you need the following. You DON’T need to touch anything related to TLS, RDP encryption or whatnot.

• install xrdp v0.14 or later, configure /etc/xrdp/xrdp.ini :

  port=vsock://1:3389
  vmconnect=true
  

• (sound) install pipewire-module-xrdp (debian), for other plateforms: https://github.com/neutrinolabs/xrdp/wiki/How-to-set-up-audio-redirection
• On HyperV: Set-VM -EnhancedSessionTransportType HvSocket <VMNAME>

How video works on Hyper-V

You opened vmconnect.exe, you see picture. What happened behind the scenes? Well… it depends. On a bunch of parameters actually, because Hyper-V has had a pretty long life and so had its video components. The main three things that impact the flow are:

• the VM generation;
• the OS;
• whether “Enhanced Session” or “Basic Session” is used.

The issue with “Basic Sessions” is that they are not super user-friendly. You can’t copy/paste text nor files, can’t resize the screen, can’t use a webcam and so on. That’s why Microsoft provided an upgrade to that with the “Enhanced Session” mode, which works completely differently.

On a Basic Session:

• Generation 1 The Hyper-V host uses an “Emulated S3 controller” (vms3cap.sys). It’s an entirely emulated video card that is detected as actual hardware by your VM. This required no adaptation from an OS to Hyper-V, but it has the downside of requiring a bunch of emulation which impacts not only performance but also increases “the attack surface”, as Microsoft puts it.
• Generation 2 VMs use a “Synthetic display controller”, which is “software based” meaning that it requires a specific driver inside the VM kernel. Hyper-V uses the “VMBUS” (a communication channel between the Host and the VM, see this) to talk to the VM whose kernel is responsible for creating a virtual display adapter.
  • On Windows, this has obviously been integrated day-1 inside the HyperVideo.sys driver (“Microsoft VMBus Video Device Miniport Driver”) driver.
  • On Linux, this started as an “integration pack” before getting contributed by Microsoft to the linux kernel back in 2012 as the “Hyper-V Frame Buffer driver” (or hyperv_fb). This driver was replaced in 2021 by a DRM-compatible variant (hyperv_drm), but essentially works in the same way as it did back in 2012. It’s built on an implementation of the VMBUS bundled in the linux kernel, also contributed by Microsoft, very similarly to Windows.

On an Enhanced Session:

• A RDP server needs to run inside the VM. It sends video but also allows for integration services, copy/pasting, sharing devices, etc.
• On Windows, this is implemented directly by ‘Remote Desktop Services’ that listens directly on the VMBUS.
• On Linux, this has no implementation in the Linux Kernel. This is implemented in userland, typically with a server like xrdp (typical for Hyper-V) or weston (WSL). Because those are in userland, they can’t easily use the VMBUS and therefore bind a special kind of socket (AF_HYPERV / AF_VSOCK) on port 3389, which provides a socket-like mechanism over VMBUS. Note that while AF_VSOCK uses ports, AF_HYPERV uses GUIDs, so 3389 is actually converted into 00000d3d-facb-11e6-bd58-64006a7986d3.

What about the client side? What actually comes out of vmconnect.exe? RDP. It’s only RDP.

The connection flow is as follows:

• The vmconnect.exe connects to the Hyper-V Virtual Machine Management service vmms.exe (yeah, the same that manages the VMs, handles Hyper-V WMI commands, etc.) on port TCP/2179. The fact that it’s a TCP port allows vmconnect.exe to connect remotely.
• The vmconnect.exe sends a special blob called the PCB (“Pre-Connection Blob”) that contains the GUID of the VM to connect to. This blob is specific to Hyper-V RDP, you wouldn’t find it in a connection generated from a mstsc to a remote Windows host. vmms.exe looks at the GUID to find the VM the user wants to connect to, but also handles authentication. When the “Enhanced Session” mode was added, the PCB was extended to have an extra ;EnhancedMode=1 suffix which indicates whether “Enhanced Session” is used or not.
• If EnhancedMode is 0, then the RDP terminates in vmms.exe and Hyper-V feeds it the frames provided by the display controller of the VM matching the GUID in the PCB. If EnhancedMode is 1, it forwards the RDP stream to the VM so that it handles it by itself.
• Modern RDP clients wrap their connection in TLS. What’s a bit weird in this setup is that the TLS needs to happen between the vmconnect.exe client and vmms.exe as it handles the authentication. It wouldn’t make sense to have to re-TLS the RDP stream between vmms.exe and the RDP server. The server therefore needs to be configured in a special mode, where it says that it supports TLS during the RDP negotiation, even though TLS actually never arrives because it’s eaten by vmms.exe.

PR #3514 from xrdp showing the TLS is only between the client and vmms.exe

xrdp (nor most RDP servers on Linux) did not implement this special edge-case, which meant that it crash when receiving unencrypted trafic after having advertised TLS. This is why so many guides disable TLS in the server, leading the client to fallback to “RDP security”, which doesn’t have this edge case. This however means that setting the GPO to enforce TLS security (as you should) would break Enhanced Sessions on Linux. I ended up contributing a PR back in 2025 which ended up in xrdp v0.10.4.0 to add support for this mode, meaning you no longer mean to force the legacy “RDP encryption”. This took the form of adding the following attribute to the configuration.

vmconnect=true

(Bonus) Use alternative RDP clients to connect to Hyper-V VMs

vmconnect.exe

Nothing to be said, it’s what the console uses by default. Did you know you could run it directly? Make sure to launch it with UAC (Windows+R then vmconnect.exe and Shift+Enter).

mstsc.exe (native Windows)

1. Get the VM id through Get-VM -Name <Your VM Name> | fl Id
2. Create a .rdp file with at least:

   full address:s:<Hyper-V machine name. Use FQDN so that kerberos works>
   pcb:s:<Hyper-V VM ID>;EnhancedMode=1
   server port:i:2179
   negotiate security layer:i:0
   sawvmconnect:i:1
   

A few notes:

• pcb is the “PCB”, sent before the connection is actually established as explained before should point towards the ID of the VM. It has an extra field ;EnhancedMode= which, you guessed it, can be 0 or 1 and tells vmms.exe whether you’d like it to try to use the Synthetic controller (Basic Session) or RDP (Enhanced Session) to talk to the VM.
• sawvmconnect is some reverse-engineered magic parameter I found that sets the used SPN to “Microsoft Virtual Console Service/XXXX”. It also disables CredSSP, so it won’t prompt for a password and use Kerberos (hurrah). This should help a lot in hardened environments.

msrdc.exe (Windows App / WSL / Microsoft Remote Desktop)

msrdc is an alternative, “newer” than mstsc. It has the advantage of being able to use some newer, pretty cool features like Multi Media Redirection which allows to stream High FPS video from your VM.

1. Same as for mstsc.exe;
2. sawvmconnect:i:1 isn’t implemented. In a hardened environment, you can work around that by setting the GPO

   Computer Configuration > Administrative templates > System > Credentials Delegation > “Restrict delegation of credentials to remote servers” = Restrict Credential Delegation (at least, RestrictedAdmin also works but is more restrictive)

3. You can add the following to have a dynamic resizing

   dynamic resolution:i:1
   

Setting the GPO will have an impact on you other usages of RDP which might be an issue in corporate environments (you will need to enable RestrictedAdmin/Remote Credential Guard on all other machines via GPO, or your client won’t be able to connect anymore). This is good in terms of security though.