Why doesn’t the Mac OS X RDP Client trust Windows Server 2012 R2? R2-D2, you know better than to trust a strange computer!

Ever since upgrading my home server from Windows Server 2012 to Windows Server 2012 R2, I’ve been unable to RDP to it from my Mac OS X laptop. I use the Microsoft RDP client that comes with Microsoft Office for Mac 2011. There are other alternatives around that may be better – I’ve heard CoRD mentioned a few times, but I’m happy using the Microsoft one. Well, happy enough given I can’t connect to my primary server.

I would always receive the error ‘Remote Desktop Connection cannot verify the identity of the computer that you want to connect to.’ Most frustrating.

Update (22 October): I’ve published a new post here describing the new Microsoft RDP Client for Mac OS X that is perfectly compatible with Windows Server 2012 R2.

I was able to fix the problem so that I could connect. This involved dictating which security layers will be used by the Remote Desktop Session Host on the server itself. There are three options to the security layer configuration, mind you it’s essentially two options with an auto-negotiate setting.

  1. Negotiate. As it should be in most situations, Windows’ first thought it to negotiate with the client to select a mutually supported security layer.
  2. RDP. This is the original RDP security layer, its supported by 3rd party RDP clients.
  3. TLS. TLS is the stronger security layer, but not as widely supported.

This configuration item is applied by Group Policy. Specifically…

Computer Configuration\Administrative Templates\Windows Components\Remote Desktop Services\Remote Desktop Session Host\Security

The setting is called Require use of specific security layer for remote (RDP) connections. Now this can be configured either by a group policy within an Active Directory environment, or by a local policy (Start -> Run -> GPEdit.MSC) if you don’t have a domain, or don’t want to create a GPO specifically for this purpose in your domain.

We’re able to take a leaf out of some government play books here, which is to disable the negotiation option, and then hold fast on a weaker position. The fix here requires us to force the Remote Desktop Session Host to not negotiate the security layer, but rather use a weaker security layer than the default. I certainly don’t mind making that change on my home system, given I’m not overly concerned about the risks associated with lowering the security layer. My home *cough* IPS *cough* will take care of that. With a suitable risk assessment and potential mitigation, this may also be suitable in an enterprise environment.

So, on to the fix. We need to change that setting to the RDP security layer. Why? Is it because the RDP client on the Mac does not support TLS? Actually. No, that’s not the reason. The Mac RDP client does support TLS – however when using TLS to communicate with 2012 R2, it fails and obstinately does not fall back to the RDP security layer. So we’re essentially forcing the RDP client NOT to use TLS. TLS by the way does work when communicating with Windows Server 2008 R2/2012. So, enable the policy setting, force it to RDP, execute a gpupdate to pick up the new policy and then restart the Remote Desktop Session Host service.

Sadly, it doesn’t appear that the RDP client itself can be configured to only use RDP mode. I was hoping the recent update to Office 2011 might have fixed the problem, but as at version 2.1.1 of the Microsoft RDP client, it is still broken.

~ Mike

33 thoughts on “Why doesn’t the Mac OS X RDP Client trust Windows Server 2012 R2? R2-D2, you know better than to trust a strange computer!

  1. This article explained what happened. However, the MS RDP client I was using is old and not even supported. Microsoft Remote Desktop 8 for OSX was released October 18, 2013 and is available in the App Store.

  2. I can’t even connect to win 8.1 from the new Microsoft App on the App store. It complains about the fqdn, which I have right. If I use the IP it lets me in, but it looks like my mac doesn’t trust the certificate my computer is sending.

    1. Hmm, what error do you get? 8.1 RDP from the Mac works with both the NLA setting and without. You will generally get the certificate error (you should see this on servers as well), as the certificate is not trusted by your Mac. You should be able to click continue when that error occurs and connect anyway.

  3. Hey guys, maybe someone here can shed some light on an issue I’m having with the new MS RDP client. I’m setting up VPN access to let a bunch of designers access a standalone (workgroup) RDS server running 2012 (OG). The RRAS PPTP VPN works great but for some reason no Mac RDP Client I’ve tried, including Microsoft’s, will connect to it. Windows 7 works great. I’ve also tried from multiple Macs. VPN works great every time, but no joy on RDP. The server has all the RDS roles installed and a self-signed certificate but I’m not using RD gateway, just trying to allow Macs to RDP to the server desktop. When the client attempts to connect, it gives a generic error message. I see nothing in the OS X console and nothing on the server event log to give me any leads on where to start troubleshooting. Firewalls are completely off on the server.

    Is there some configuration I’m missing? Should I uninstall RD Gateway? I’m just at a loss here since I’m not a Mac guy. Everything seems in place but not having much luck. Thanks in advance for your assistance.

  4. MDL Marinas in this area include Woolverstone Marina,
    Chatham Maritime Marina, Torquay Marina, Brixham Marina, Queen Anne.
    Enjoy a motor yacht charter or a sailing yacht charter in the Caribbean where there are so
    many luxury mega yachts and luxury sailing yachts available from which to choose for your
    yachting holidays. Launched in 2010, built of steel and finished with a teak
    deck, is 20 meters long and is divided internally into 3 double cabins for guests (a suite in the bow, a VIP
    cabin and a twin bus) and a cockpit for the crew.

  5. Ι’m not sure why but this site is loading incredibly slow for me.
    Is anyone еlse having this issue οr is it a issue on my end?
    I’ll check back later on and see if the problem ѕtill
    exists.

  6. Thanks for the marvelous posting! I really enjoyed reading it, you happen to be a great author.I will make
    sure to bookmark your blog and will come back down the road.

    I want to encourage you to ultimately continue your great work, have a nice morning!

  7. Nice post. I learn something new and challenging on sites I stumbleupon on a daily
    basis. It’s always useful to read through content from
    other authors and use a little something from
    other sites.

  8. You are so cool! I do not suppose I have read through
    a single thing like that before. So good to find someone
    with some genuine thoughts on this subject. Really..
    many thanks for starting this up. This website is something that is needed on the internet, someone
    with a bit of originality!

  9. Greetings I am so excited I found your website, I really found you by error, while I
    was looking on Askjeeve for something else, Nonetheless I am here now and would just like to
    say cheers for a marvelous post and a all round thrilling
    blog (I also love the theme/design), I don’t have time to read through it
    all at the minute but I have bookmarked it and also added in your RSS feeds, so when I have time I
    will be back to read much more, Please do keep up the awesome jo.

  10. Great post. I used to be checking constantly this weblog and I am inspired!
    Extremely useful information specially the last phase :
    ) I take care of such info much. I was seeking this
    certain information for a very long time. Thanks and best of luck.

  11. Woah! I’m really enjoying the template/theme of this website.
    It’s simple, yet effective. A lot of times it’s difficult to get that “perfect balance” between user friendliness and visual
    appearance. I must say that you’ve done a very good job with this.
    In addition, the blog loads very quick for me on Opera.
    Superb Blog!

  12. I see you don’t monetize your website, i think there is one opportunity to
    earn extra cash on your blog, search in google for; idol4jp makes money

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s