Driver Debugging with WinDbg and VMWare

One of the first useful things you will want to do when in the bowels of ring 0 is attack the thing from a debugger point of view. In my case I like using Windows Debugger [windbg] (hey its free, fully functional and does remote debugging really well), and found it neccessary to find a way to have it work with VMWare.

The trick to get it to work in VMWare is to get the host OS to believe it is able to connect to a serial port. Through VMWare, to accomplish this, you need to create a named pipe. The following steps will guide you to configuring VMWare (in my case v3.2):

  1. Open up the Configuration Editor (Settings->Configuration Editor)

  2. Click Add to run the Hardware Wizard

  3. Select Serial Port, and then click the Next button

  4. Select the Use named pipe radio button

  5. Use the default pipe name. It SHOULD be \\.\pipe\com_1. If it is not, change it to that.

  6. Select This end is the Server.

  7. Select The other end is an application.

  8. Click the Advanced Button.

  9. Select the Yield CPU on poll checkbox

    This is an important step as the Kernel in the target virtual machine uses the port in polled mode, not interrupt mode.

  10. Click the Finish button, and then click Ok to close the Configuration Editor.

    Once you have configured your VMWare session, you need to power it on. You will be able to confirm that the new virtual serial port is added by clicking on the Devices->serial0 menu item. You should note it is saying "Connecting". This means its now ready for a remote connection.

    Once the actual virtual machine is configured, you need to configure the target operating system installed in the virtual machine to support remote debugging. This is accomplished by editing the boot.ini found in the root of the C:\ drive. To do this, you need to add a /debugport=some_com_port /baudrate=some_baud_rate to the end of an [operating systems] line. I would not recommend doing it to the one that is there. It would be better to copy the line and paste it with the ammendments, and then use the OS's menu selection to determine which mode you would like to boot into. This is what my boot.ini looks like for XP Pro as the operating system in the virtual machine:

    [boot loader]
    timeout=30
    default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS
    [operating systems]
    multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP Professional" /fastdetect
    multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP Professional - Debug" /fastdetect /debugport=com1 /baudrate=115200

    You will notice that the second option sets the debugport to com1, and sets the baudrate to 115200. I am told you can tweak this out to get even more speed out. But it seems fine for me at this speed, so I haven't mucked with it at all. If you do try this, drop me an email and let me know how it works out.

    If you set up your boot.ini fine, save it and reboot. You should be prompted with something that looks like this:


    At this point you have completed setting up the VMWare side of things. Now you need to set up the host to connect to it. This is actually rather easy. You just need to tell windbg at the command like to connect to the pipe, like this:

    windbg -k com:port=\\.\pipe\com_1,pipe
    
    If you are using WinDbg 6.x or newer, a better alternative is to use:
    windbg -b -k com:pipe,port=\\.\pipe\com_1,resets=0
    

    Thanks needs to go to Randhir Dugal for pointing out the new format for the latest Windbg version.

    I am a rather anal type guy when it comes to security, so I actually made a shortcut on my desktop to WinDbg and added these command line arguments to the Target line. Originally this was so I could run WinDbg with differnet credentials as I do not run with administrator privileges on a day to day basis. I found that with XP's normal security settings for com ports, you can still work in a least privileged environment while doing the development WITHOUT having to use 'runas'. (You are logged into W2K/XP as a least priviledged environment aren't you? If not, you really should read my article on how, and why this is important to do so.)

    At this point fire up the debugger. With any luck you should see something that looks like this:


    If you are using a WinDbg version earlier than 6.x, you will find one issue with this approach. If you cannot seem to connect right away, close Windbg and restart it... it will then work. Seems flaky to me. But it works. And thats a Good Thing™. With the latest versions of WIndbg (6.x and newer) a new resets flag prevents this sort of hanging.

    That is all there is to it. At this point, you can now go nuts with remote debugging. To make sure it works for ya just add some DbgPrint() messages to your ring 0 driver and watch them echo to the screen. Past that, I will leave it to your imagination how to use the debugger :)

    Good luck!