Why doesn’t the Mac OS X RDP Client trust Windows Server 2012 R2? R2-D2, you know better than to trust a strange computer!

Ever since upgrading my home server from Windows Server 2012 to Windows Server 2012 R2, I’ve been unable to RDP to it from my Mac OS X laptop. I use the Microsoft RDP client that comes with Microsoft Office for Mac 2011. There are other alternatives around that may be better – I’ve heard CoRD mentioned a few times, but I’m happy using the Microsoft one. Well, happy enough given I can’t connect to my primary server.

I would always receive the error ‘Remote Desktop Connection cannot verify the identity of the computer that you want to connect to.’ Most frustrating.

Update (22 October): I’ve published a new post here describing the new Microsoft RDP Client for Mac OS X that is perfectly compatible with Windows Server 2012 R2.

I was able to fix the problem so that I could connect. This involved dictating which security layers will be used by the Remote Desktop Session Host on the server itself. There are three options to the security layer configuration, mind you it’s essentially two options with an auto-negotiate setting.

  1. Negotiate. As it should be in most situations, Windows’ first thought it to negotiate with the client to select a mutually supported security layer.
  2. RDP. This is the original RDP security layer, its supported by 3rd party RDP clients.
  3. TLS. TLS is the stronger security layer, but not as widely supported.

This configuration item is applied by Group Policy. Specifically…

Computer Configuration\Administrative Templates\Windows Components\Remote Desktop Services\Remote Desktop Session Host\Security

The setting is called Require use of specific security layer for remote (RDP) connections. Now this can be configured either by a group policy within an Active Directory environment, or by a local policy (Start -> Run -> GPEdit.MSC) if you don’t have a domain, or don’t want to create a GPO specifically for this purpose in your domain.

We’re able to take a leaf out of some government play books here, which is to disable the negotiation option, and then hold fast on a weaker position. The fix here requires us to force the Remote Desktop Session Host to not negotiate the security layer, but rather use a weaker security layer than the default. I certainly don’t mind making that change on my home system, given I’m not overly concerned about the risks associated with lowering the security layer. My home *cough* IPS *cough* will take care of that. With a suitable risk assessment and potential mitigation, this may also be suitable in an enterprise environment.

So, on to the fix. We need to change that setting to the RDP security layer. Why? Is it because the RDP client on the Mac does not support TLS? Actually. No, that’s not the reason. The Mac RDP client does support TLS – however when using TLS to communicate with 2012 R2, it fails and obstinately does not fall back to the RDP security layer. So we’re essentially forcing the RDP client NOT to use TLS. TLS by the way does work when communicating with Windows Server 2008 R2/2012. So, enable the policy setting, force it to RDP, execute a gpupdate to pick up the new policy and then restart the Remote Desktop Session Host service.

Sadly, it doesn’t appear that the RDP client itself can be configured to only use RDP mode. I was hoping the recent update to Office 2011 might have fixed the problem, but as at version 2.1.1 of the Microsoft RDP client, it is still broken.

~ Mike

13 thoughts on “Why doesn’t the Mac OS X RDP Client trust Windows Server 2012 R2? R2-D2, you know better than to trust a strange computer!

  1. This article explained what happened. However, the MS RDP client I was using is old and not even supported. Microsoft Remote Desktop 8 for OSX was released October 18, 2013 and is available in the App Store.

  2. I can’t even connect to win 8.1 from the new Microsoft App on the App store. It complains about the fqdn, which I have right. If I use the IP it lets me in, but it looks like my mac doesn’t trust the certificate my computer is sending.

    1. Hmm, what error do you get? 8.1 RDP from the Mac works with both the NLA setting and without. You will generally get the certificate error (you should see this on servers as well), as the certificate is not trusted by your Mac. You should be able to click continue when that error occurs and connect anyway.

  3. Hey guys, maybe someone here can shed some light on an issue I’m having with the new MS RDP client. I’m setting up VPN access to let a bunch of designers access a standalone (workgroup) RDS server running 2012 (OG). The RRAS PPTP VPN works great but for some reason no Mac RDP Client I’ve tried, including Microsoft’s, will connect to it. Windows 7 works great. I’ve also tried from multiple Macs. VPN works great every time, but no joy on RDP. The server has all the RDS roles installed and a self-signed certificate but I’m not using RD gateway, just trying to allow Macs to RDP to the server desktop. When the client attempts to connect, it gives a generic error message. I see nothing in the OS X console and nothing on the server event log to give me any leads on where to start troubleshooting. Firewalls are completely off on the server.

    Is there some configuration I’m missing? Should I uninstall RD Gateway? I’m just at a loss here since I’m not a Mac guy. Everything seems in place but not having much luck. Thanks in advance for your assistance.

  4. Everything is very open with a really clear explanation of the
    issues. It was definitely informative. Your site is
    very useful. Thank you for sharing!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s