It did cost me two hours to finally install the USB Debugging Driver for my Nexus 4. I got two Laptops with Windows 7 and I didn't have problems installing the driver on one Laptop. But it was a pain installing the driver on the other Laptop and I still don't know why.
Now here comes the workflow that made it work for me.
1. I did change the setup of my Windows 7 and deleted all the drivers that had something to do with Android. For this I had to open the Device Manager and get a list of all hidden / nonpresent devices. Check out this for further information
http://msdn.microsoft.com/en-us/library/windows/hardware/ff553955(v=vs.85).aspx
2. After uninstalling all devices I downloaded the latest USB Debugging version. Doesn't matter whether you use the Eclipse Android SDK Manager or download it directly from Google. Just check out this page.
http://developer.android.com/sdk/win-usb.html
3. I did enable the developer mode on my Nexus 4. Check out this page.
http://www.androidcentral.com/how-enable-developer-settings-android-42
4. I was playing around with plugging in/out and enabling/disabling the DebugMode and the MTP. I think it was the following order.
4.a. Disable DebugMode, Enable MTP, Plugin and select the usb drivers to be installed from <ANDROID-SDK>/extras/usb_drivers
4.b. Enable DebugMode, MTP stays enabled, and select the usb drivers to be installed from <ANDROID-SDK>/extras/usb_drivers
4.c. Once everything is installed correctly an ADB Interface should listed in your DeviceManager and a Popup appears on your phone asking you if the Laptop is authorized to install applications on the Nexus4
I think the most important part was uninstalling all hidden drivers before starting.