HDInsight and WebSSH Security Issue


Background


This post relates to an unpublished 'feature' of Microsoft Azure HDInsight Linux clusters that is misconfigured such that it allows users to obtain root access to clusters without having knowledge of the 'admin' account name or password via a web console.

I originally raised this with Microsoft Support around the end of October / beginning of November 2016. Initially, support informed me that they had discussed it with the product team and that the security issue that I was reporting was not a security issue because:

  • The security boundary of HDInsight is the Virtual Network (VNET) and 
  • The clusters are only intended for single user tenancy (ironically a MSFT Cloud Data Solution Architect recently said to me that HDInsight fully supports multiple users - which I guess is sort of true now with secure clusters being in preview). 
Eventually they agreed that it was indeed an issue and disabled the feature on all new clusters as an interim measure.

This post has been cross-posted to the GI Architect's blog.


What is the issue?


An Azure HDInsight Linux cluster consists of head, worker and zookeeper nodes - these nodes are Azure VMs, although the VMs are not visible nor can the individual VMs be managed in the Azure Portal you can SSH to the cluster nodes.
 When you provision a cluster you are prompted to set to credentials:

  • One that will be used for the Ambari web interface - which you can login to over HTTPS and a <cluster name>.azurehdinsight.net domain.
  • The other for a local account that will be created on ALL nodes in the cluster which you can then use to SSH to the cluster ssh <user>@<cluster name>-ssh.azurehdinsight.net

The SSH account by default has passwordless sudo - that is you can run sudo su and become root without being prompted for your password.

One of the packages that is installed when you provision a HDInsight cluster is hdinsight-webssh running apt-cache show hdinsight-webssh shows us that it is a Microsoft package (there are other Microsoft HDInsight packages they are all prefixed with hdinsight-):


root@hn0-vjt2-h:/home/dhadmin# apt-cache search hdinsight-
 hdinsight-livy - HdInsight livy install
 hdinsight-startupagent - HdInsight Startupagent
 hdinsight-zeppelin - HdInsight zeppelin sample notebooks
 hdinsight-webssh - HDInsight browser terminal emulation
 hdinsight-credentialservice - HdInsight datalake credential service
 

 root@hn0-vjt1-h:~# apt-cache show hdinsight-webssh
 Package: hdinsight-webssh
 Status: install ok installed
 Priority: extra
 Section: universe/admin
 Installed-Size: 22962
 Maintainer: Microsoft
 Architecture: amd64
 Version: 1.0.16990
 Depends: default-jre | java6-runtime
 Description: HDInsight browser terminal emulation
 Description-md5: d6b2cd55c0203fdbed1041a8446ee914

Running netstat you can see that there is a nodejs based web terminal running and listening on port TCPv6 port 3000:

root@hn0-vjt1-h:~# netstat -atunlp | grep node
 tcp6       0      0 :::3000                 :::*                    LISTEN      1361/node
 root@hn0-vjt1-h:~#

 root@hn0-vjt1-h:~# curl -L ':::3000'
<!DOCTYPE html><html><head><meta charset="utf-8"><title>webssh</title><link rel="stylesheet" type="text/css" href="webssh.min.css"><script src="./soc
 ket.io/socket.io.js"></script><script src="./socket.io-stream.js"></script><script src="./terminal.js"></script></head><body><div id="modal-alert" class="modal"><div class="modal-content"><h2>This page is no longer controlling the
 terminal</h2><br><p>It looks like another page has requested the control of this session. For this reason, this page no longer has access to it.</p><br><button id="request_new_session_btn">Request a new session</button></div></
 div><span id="pageId"
 class="identifier">3c6a307a-2bc1-44ef-b620-fe1e6c00e489</span><span id="sessionId" class="identifier">603cae0b-31c0-41a5-9da9-e523889d56b5</span><span id="w" class="charWidth">X</span><pre id="terminaljs" class="terminaljs" data-
 columns="80" data-rows="10"></pre><script src="./webssh.min.js"></script></body></html>


If you run 
ps -fuax | grep webssh

you will see the process (which incidentally also runs as root!).

The configuration for the service/application is here:
/etc/websshd/conf.json

It looks like that a number of python scripts are run when you provision a cluster to start ambari, configure hive etc. one of which is to start this websshd service with /opt/startup_scripts/startup_webssh.py

Impact of the issue

The issue cannot be easily exploited by an external attacker e.g. one that does not already have access to infrastructure in the Azure Virtual Network (VNET) that the HDInsight cluster resides in. Such an external attacker would first need to gain access to (doesn't need to be a privileged account) on any other system hosted in the same VNET and from this point they can easily gain root access on the HDInsight cluster by simply browsing to http://
 <clusternodeipaddress>:3000 which would automatically give them a web based shell as the user that has passwordless sudo without entering any username or password.

However, since the default NSG rules allow connectivity within a VNET (as opposed to a default deny that requires all traffic to be explicitly allowed) this makes it easier for an attacker to extend their reach.

Another possibility is that an external attacker would need to find a vulnerability in the proxy servers and/or the various web interfaces that are accessible via the proxies.

In the case of a malicious user who has authorised access to say an application or web server, they would be able to take advantage of the misconfiguration to obtain root access to the HDInsight cluster as described above.

In either case an external attacker or malicious user can then use the root access to exfiltrate data, plant malicious software etc.


Summary

Microsoft have since disabled the service (although the last time I checked back in December 2016 the package is still installed but the service is not running, nor is there a systemd unit file installed.

Microsoft didn't explain why the package is installed in the first place but I can only assume it was added as a convenience when the product team were developing or testing.

Browser based terminals are problematic when it comes to security but it's worse when the endpoint is

  1. Unencrypted
  2. Performs no authentication
  3. Drops you in as a user that has passwordless sudo
As an added measure you can disable passwordless sudo for the admin account - which probably shouldn't be enabled anyway.

Comments

Post a Comment