Ever since upgrading my home server from Windows Server 2012 to Windows Server 2012 R2, I’ve been unable to RDP to it from my Mac OS X laptop. I use the Microsoft RDP client that comes with Microsoft Office for Mac 2011. There are other alternatives around that may be better – I’ve heard CoRD mentioned a few times, but I’m happy using the Microsoft one. Well, happy enough given I can’t connect to my primary server.

I would always receive the error ‘Remote Desktop Connection cannot verify the identity of the computer that you want to connect to.’ Most frustrating.

Update (22 October): I’ve published a new post here describing the new Microsoft RDP Client for Mac OS X that is perfectly compatible with Windows Server 2012 R2.

I was able to fix the problem so that I could connect. This involved dictating which security layers will be used by the Remote Desktop Session Host on the server itself. There are three options to the security layer configuration, mind you it’s essentially two options with an auto-negotiate setting.

1. Negotiate. As it should be in most situations, Windows’ first thought it to negotiate with the client to select a mutually supported security layer.
2. RDP. This is the original RDP security layer, its supported by 3rd party RDP clients.
3. TLS. TLS is the stronger security layer, but not as widely supported.

This configuration item is applied by Group Policy. Specifically…

Computer Configuration\Administrative Templates\Windows Components\Remote Desktop Services\Remote Desktop Session Host\Security

The setting is called Require use of specific security layer for remote (RDP) connections. Now this can be configured either by a group policy within an Active Directory environment, or by a local policy (Start -> Run -> GPEdit.MSC) if you don’t have a domain, or don’t want to create a GPO specifically for this purpose in your domain.

We’re able to take a leaf out of some government play books here, which is to disable the negotiation option, and then hold fast on a weaker position. The fix here requires us to force the Remote Desktop Session Host to not negotiate the security layer, but rather use a weaker security layer than the default. I certainly don’t mind making that change on my home system, given I’m not overly concerned about the risks associated with lowering the security layer. My home *cough* IPS *cough* will take care of that. With a suitable risk assessment and potential mitigation, this may also be suitable in an enterprise environment.

So, on to the fix. We need to change that setting to the RDP security layer. Why? Is it because the RDP client on the Mac does not support TLS? Actually. No, that’s not the reason. The Mac RDP client does support TLS – however when using TLS to communicate with 2012 R2, it fails and obstinately does not fall back to the RDP security layer. So we’re essentially forcing the RDP client NOT to use TLS. TLS by the way does work when communicating with Windows Server 2008 R2/2012. So, enable the policy setting, force it to RDP, execute a gpupdate to pick up the new policy and then restart the Remote Desktop Session Host service.

Sadly, it doesn’t appear that the RDP client itself can be configured to only use RDP mode. I was hoping the recent update to Office 2011 might have fixed the problem, but as at version 2.1.1 of the Microsoft RDP client, it is still broken.

~ Mike