G443 seems broken with Hyper-V

GoGoGadgetPC

Hello,

I have a PowerSpec G443 and pretty much since buying it in October, I've been fighting nearly constant "soon after login" freezes for about two minutes, occasional full hard lockups, and CLOCK_WATCHDOG_TIMEOUT blue screens. I brought the machine in to the local Micro Center, and they ran a bunch of diagnostics and told me everything was fine… so I started reconfiguring the OS, and I think I finally narrowed down the issue. It looks like enabling Hyper-V is the trigger. (I thought it was SR-IOV but even with that disabled, it still acts up.)

This has been through many updates of Windows 11 so I don't think it's a Windows issue. I'm on the official A.MF Sept 27 2023 firmware; shortly after getting the machine I messed around with the MSI firmware but went back to the PowerSpec stock and haven't retried the MSI since then, although I see on their site there's been recent updates. I just don't want to go there if it's not going to specifically address an issue I'm seeing.

I presume most PowerSpec owners are gamers that don't care about Hyper-V, so I might be the only person to even try it out here.

Is there anything known going on here? Any suggestions? I need to be able to run VMs on the machine.

Thanks!

PowerSpec_MikeW

@GoGoGadgetPC

Try disabling Core Isolation/Memory Integrity and see if that solves the issue with Hyper-V. Provide me with a screenshot from bluescreenview: https://www.nirsoft.net/utils/bluescreenview.zip

I'll build one up Monday and see what's going on.

GoGoGadgetPC

Thanks for taking a look! I can't get a usable dump - when the bluescreen happens, the dump process hangs at 0%, never actually writing anything. :(

I'll take a look at the security settings in a bit.

PowerSpec_MikeW

@GoGoGadgetPC

If you have to get at it from outside the OS, registry path is: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\DeviceGuard\Scenarios\HypervisorEnforcedCodeIntegrity

Enabled = 0

GoGoGadgetPC

ok, disabling memory integrity didn't seem to help. I'm still getting the weird temporary lock at start which implies nothing has changed.

PowerSpec_MikeW

@GoGoGadgetPC

Next thing I'd check is the BIOS. Make sure VT-d is enabled. It's a default, but possibly a setting that could have carried over after a flash.

GoGoGadgetPC

I did check and it's set.  IIRC I can't change it post OS install or it breaks Windows, so I am not going to try disabling it 😉

PowerSpec_MikeW

@GoGoGadgetPC

Hyper-V wouldn't be possible without it anyways, just wondering if something is different than what the defaults should be. I've never tested Hyper-V on that unit, I will Monday and I'll let you know if I can replicate it.

PowerSpec_MikeW

@GoGoGadgetPC

Installed our factory image on a G443, AMF BIOS. Enabled Hyper-V. No issues on restart. Built a Ubuntu VM, no issues on restart. Currently running windows updates to see if I can create an issue. A few more questions.

1. Is this the factory install? Have you ever had to reset this system?
2. In Windows Features, is Virtual Machine Platform on? We enable this in our image.
3. Do you have anything else installed that might conflict with Hyper-V? List your appwiz.cpl if you're comfortable.

I have all the virtualization features enabled as per the factory setup. Please compare in msinfo32: Virtualization-based security Available Security Properties Base Virtualization Support, Secure Boot, DMA Protection, UEFI Code Readonly, SMM Security Mitigations 1.0, Mode Based Execution Control, APIC Virtualization

GoGoGadgetPC

Thanks for continuing to look at this! Let me answer what I can:

1. It's been reset several times, including before having the local store look at it.
2. Yes:
   
   
3. Here's the appwiz.cpl list:

It's really not much ultimately, not yet on this load anyway.

With respect to the info in msinfo32:

Virtualization-based security	Running
Virtualization-based security Required Security Properties
Virtualization-based security Available Security Properties	Base Virtualization Support, Secure Boot, DMA Protection, UEFI Code Readonly, SMM Security Mitigations 1.0, Mode Based Execution Control, APIC Virtualization
Virtualization-based security Services Configured
Virtualization-based security Services Running	Credential Guard

This appears to match your list.

The only other difference I can think of right now that would be different than what you would have stock is that the other three M.2 slots have NVMe drives in them - three Crucial CT500P3SSD8 in an Intel VMD RAID-5 setup as a secondary storage drive (the boot drive is a Western Digital 2TB that I believe is the one that shipped with the system as the OS drive).

Another thought I had about what you might have different is that maybe it's related to using all four outputs on the Nvidia card, but I've unplugged all but one and have had the same issue, so that's out.

Luckily I was able to boot it and have this be a "good" boot so I was able to enter this information at least!

PowerSpec_MikeW

@GoGoGadgetPC

Well, VMD is something that's not on by default. Let me test that with a SATA RAID array and see what happens.

PowerSpec_MikeW

@GoGoGadgetPC

Enabled VMD, didn't want to make my system unbootable, so I disabled the global mapping and only mapped SATA. Seems to be operating normally. I assume you're in VMD with global mapping enabled?

GoGoGadgetPC

correct, global mapping. here's what the management tool shows:

you can see the controller for the non-RAID SSD, the boot disk, shows VMD as the controller.

PowerSpec_MikeW

@GoGoGadgetPC

Ah, NVME RAID 5. I'll copy the setup tomorrow and do a clean install so that I can add the RAID driver. Then I'll pull our driver set off Windows Update, enable Hyper-V and see what happens.

PowerSpec_MikeW

@GoGoGadgetPC

Running a 20 cycle reboot test now, but everything looks fine.

GoGoGadgetPC

ugh.  I was a bit afraid of that - if you could repro at least a fix would be possible.  As it stands now, I feel like I'm a bit out of options. 😢

I did try swapping out USB as per the local store and there was no change, even with just a different keyboard and mouse, so I don't think that was it.  

That said, I'm still thinking of what else might be different.  I did add a USB 2.0 four port card to use the two unused headers on the motherboard but when testing around USB, those ports weren't used. (essentially two of these in one: [MC web site]/product/321594/ziotek-dual-usb-port-connector-2-wire )

I know you've spent a bunch of time on this now and I appreciate it.  I'm just not sure on next steps, if any.

PowerSpec_MikeW

@GoGoGadgetPC

Few more questions for you.

1. I assume you've done a clean install rather than resets at this point, but please confirm.
2. Are you updating drivers or just using what we publish on Windows Update? Excluding Nvidia.
3. Break down for me how it crashes. PowerSpec logo with a spinning circle? How consistent is the time?

Edit: Nevermind, it's just a header. Don't worry about the USB unless something is connected. Changed question 3.

GoGoGadgetPC

ok, this is my third time writing this up because the forum keeps eating it, so this time I'm doing it in Notepad first :)

To answer your questions:

1. I am honestly not sure if I did a full "from USB" clean load at this point. I can try that tonight though (with the VMD drivers) to be 100% sure.
2. The drivers depend on which load (I'm not exaggerating when I say this is like the sixth load or something since troubleshooting this) but right now it's just the Windows Update ones I believe for the most part (please see the appwiz.cpl list above)
3. The crash is inconsistent and may be time-based or not, but I don't think it is. I haven't tried with a stopwatch or something yet though. There's really four variations on behavior:
   a. The system gets to the login screen, accepts my PIN / Windows Hello (not configured in this load, but has been in the past with a Hello camera I have), starts to sign me in, then locks up while showing the "logging you in" circle animation. The system will be done at point (keyboard sleep button and case power button not responsive except for "hold down the power button to force a power off")
   b. The system gets me signed in to the desktop, and maybe even lets me start something (e.g. Task Manager), before seeming to lock up, with keyboard input eventually giving the "my input queue is full" beep on keypresses, and done at that point.
   c. scenario a or b except after a time doing the CLOCK_WATCHDOG_TIMEOUT, which stays at 0% forever (I've let it sit for hours).
   d. scenario b, but after a minute or two, the system "recovers" and is fine for the duration, until the next full reboot... system can even be put to sleep and woken up okay

I think there was exactly one time where it did scenario d, before after a few minutes doing a BSOD, but that was only once, and I could have been imagining it at this point. :)

Given your q 1, I think the next step tonight is to do a clean Win11 load from USB (nuke the C: drive partition and let the setup put it back and go from there, with VMD drivers available), then do a minimum setup, just enough to get my login set up, and then doing the Windows Update pass for drivers and updates, and then going one step at a time, validating stability at each step... normally I have a batch file I run that installs Office, some Store apps I use, etc., and enables Hyper-V, and then I reboot and that's the last step. I can not do that this time and instead do one step at a time. This will likely take a few hours to try but maybe I can at least see if a specific point seems to "break" it, then report back, unless you have a better idea.

One thing I haven't done since early after getting the machine is try the MSI BIOS. I did it once just to "be current" (doing the fan fix), but decided that it wasn't worth being different than the "official" PowerSpec version, so I flashed it back and that's what it's on now. I haven't tried going there yet for this process because I'm trying to be as close to what you have as is reasonable, and so far you haven't indicated I should try that. :)

GoGoGadgetPC

(dupe)

PowerSpec_MikeW

@GoGoGadgetPC

Clean install is the way to go, give us something well, clean to work from. Get all the drivers from Windows Update, feel free to update from Nvidia as the driver on windows update is usually a few months behind. Let me know and we'll see where we stand. I assume once the PC is running, you've never had an issue?

GoGoGadgetPC

Correct.  In Scenario 3d, once it is "good", it is good.

I will try to find the time this evening to do the clean installation from USB and will report back.

PowerSpec_MikeW

Appreciate it and we'll do whatever we can to help find the issue. It's very strange. I'd think the VMD array could be a problem, but it's initialized long before that. I can't think of anything Hyper-V that would just randomly crash in that window. Would you mind posting some autoruns screenshots? https://learn.microsoft.com/en-us/sysinternals/downloads/autoruns

GoGoGadgetPC

ok, well, I feel vaguely bad now. I did a clean install, from scratch, from USB, and installed things one at a time, taking notes as I went, and so far, so good. It's getting late for me now so I can't beat on it more at the moment, but so far, it's looking okay. I won't know for sure until I spend some more time with it… if it turns out "all" that was needed was an install from USB, I'll be annoyed that it was necessary, but happy it's fixed.

for now though, it's bedtime :)

GoGoGadgetPC

PowerSpec_MikeW

@GoGoGadgetPC

Let me know if the issue shows up again. Otherwise, it's really hard to say what was in the old image that caused a problem.