Advanced Techniques for Defending AWS ExternalIDs and Cross-Account AssumeRole Access

Last month Kesten Broughton at Praetorian Security released some great research on third party cloud security products using Amazon’s preferred cross-account connection technique – AWS IAM Assume Role Vulnerabilities Found in Many Top Vendors. The opening paragraph is a solid overview of the research:

In this first blog in our series on cross-account-trust we will present the results from 90 vendors showing that 37% had not implemented the ExternalId correctly to protect against confused-deputy attacks. A further 15% of vendors implemented the AWS account integration in the UI correctly, but the ExternalId parameter was not properly validated on the backend, making those sites also vulnerable. We will finish by discussing the new attack surfaces exposed by the AWS cross-account-assume-role trust. We conclude that vendors and clients should critically examine whether role trust is the best trust mechanism for their multi-tenant SaaS solution.

My first response when I read the research was, “why would anybody make a such a bad decision”, but the reality is that the entire concept of a “cloud security expert” is relatively new and while AWS is decent about talking about the confused deputy problem, they don’t cover some of the practical implication issues you don’t really think about until your product contacts the customer. Cross account connections using AssumeRole have a straightforward engineering solution, but without proper threat modeling it is extremely easy to make the mistakes documented in Kesten’s research.

Here at DisruptOps we made some smart decisions early on based on our initial threat modeling that kept us safe. However, we did pick up some tidbits in the research and are adding even more hardening. Plus we have a skunkworks project to wipe out a large portion of the problem anyway and still enable automation at scale without requiring direct write access into customer environments.

I do, however, have one major point of disagreement with the post. I’ll take auto-rotated credentials any day over the recommendation to use static credentials and vaulting. We still have to use them for other cloud providers and are always looking at ways to NOT have static credentials ANYWHERE in our environment.

The Problem

A range of different application types need to connect directly to cloud APIs these days. It could be as simple as accessing an S3 bucket or as complex as a fully automated Cloud Detection and Response platform (I mean, you know, as a random example). These requests require some kind of credentials and those are either static (like a username and password, or an IAM access key and secret key) or dynamic. Dynamic credentials include a token or other ephemeral attribute that is time-limited.

Static application credentials that allow direct access to your cloud management plane are… bad. We can solve this for users with MFA but that doesn’t help us for automated applications like our not-so-theoretical Cloud Detection and Response platform. A while ago Amazon Web Services tackled this problem with the concept of an IAM Role. A role in AWS is essentially a container for permissions which has two policies: what the container can do, and who (or what) can assume the role. Roles are then session based, so when an authorized entity assumes the role they have a set of credentials (access key, secret key, and session token) they can use for the length of the session (from 1 to 24 hours).

Roles in AWS can be assumed through external SAML connection (for users), internal “trusted” connections (from other AWS accounts), or AWS services (like an EC2 instance or Lambda function). Thus you can do cool things like run code in an instance but that instance never stores static credentials. This really removes a lot of headaches.

Allowing connections from an account you don’t control is a bit different; especially if that account is a platform that serves multiple customers. Platforms like that (okay, us) need access to hundreds or thousands of other AWS accounts. Imagine if an attacker could trick the platform into doing something in the wrong account, such as performing a configuration assessment on an account the current user doesn’t own. This is a short version of the confused deputy problem – the deputy is trusted by multiple accounts and the user tricks the deputy into giving them access to an account the current user should never touch.

The most practical exploit for this is if the platform allows a user to enter an account ID for an account they don’t control, add it to their profile, and then abuse the trust given to the platform.

AWS includes a mechanism to defend against this called the AWS ExternalID. This is an arbitrary shared secret the platform provider and the customer exchange out of band. This ID is an attribute passed on during any request to assume the role in the customer account, and the role trust policy in that account has a conditional requirement that checks that the shared secret is correct. When implemented properly this means someone can’t just add an account they don’t control to the deputy since the attacker can’t establish or know the shared secret in the customer account.

Unless…

You see where this is headed. Praetorian identified a large number of providers who use default AWS ExternalIds, or allow customers to set a non-unique ExternalId value for their whole account. They also found products that didn’t validate that the customer even enforces the External ID in their role trust policy. Praetorian found platforms that were actively exploitable, allowing confused deputy attacks due to poor security with cross account roles.

Hardening Cross Account Connections

This was all part of our threat modeling at DisruptOps so we are in good shape, although we did pick up another idea or two to make things better. Let’s walk through each issue identified by Praetorian and look at the best options for security hardening:

• AWS ExternalID uses default settings: We use a random ExternalID.
• AWS ExternalID is shared across multiple customer accounts:We use a random ExternalID on a per-account basis, not a per-customer basis.
• AWS ExternalID is enumerable or guessable: We use fully random/long AWS ExternalIDs. I’m not worried about our PRNG choice thanks to API rate limiting (for you crypto nerds).
• Customers can set their own ExternalID and may repeat or use weak ExternalIDs: We do not support customer’s setting their own ExternalID, although we have definitely had this request. This is where I think some other providers have gotten in trouble. Customers want the capability so they can automate product provisioning themselves; but to do this safely we would need to validate the ExternalID meets length and randomness requirements. With the right validation it can probably be done safely.
• The platform allows the same AWS account to be provisioned multiple times: We don’t allow this.
• The IAM Role is not restricted to a single entity in the provider account, but to any role in the account: We haven’t discussed this in this post, but you can either grant access from any role in the account to the “worker” role in the customer account, or grant access to a single resource or role. We limit our access to the specific roles we use for cross-account access.
• The IAM Role name is static across all customer accounts: The risk here is essentially that the username (role name) is guessable. I don’t consider this a risk at all unless there are other very fundamental mistakes in place. One example is that if an attacker knows the role name and they gain access to the customer account they could potentially add themselves to the role trust policy and then use those privileges (I teach something like this in my incident response training classes). This is a potential risk but one we rate pretty low. If an attacker has that level of access they can likely take over any role in the account.
• The provider does not validate that the ExternalID was set as a condition of access: We provide customers a CloudFormation Template for provisioning that ensures this is set properly. We will be adding support to validate that it stays there, especially as we continue to add other (non-CloudFormation) provisioning options to meet customer requirements.

Kesten seems to prefer the static credential model and use of good vaulting on the provider side for secure the connection. Personally I disagree and I think the AWS model is more secure out of the gates, assuming you follow the basic precautions.

We currently take two additional precautions beyond the recommendations by Praetorian:

• We mask the ExternalID in CloudFormation: We set the ExternalID as a parameter in our CloudFormation templates with the NoEcho option set. This masks it in the console, command line tools, or API. This reduces the risk of exposure to someone with permissions to run CloudFormation that does not have IAM permissions.
• We restrict access to the CloudFormation template to only the target account: This reduces the risk that someone could sweep in and grab the ExternalID during the provisioning process.

Even if the ExternalID is exposed it shouldn’t matter – your platform should ensure that a target account is only registered once, and only with a random ExternalID. Even if an attacker knows the Role Name and the ExternalID they can’t do anything with it since there isn’t anyplace in the platform to enter that information, and AWS themselves enforce that the cross account connection originates from the trusted account. That isn’t something you can spoof.

Hopefully seeing how we handle this gives you some ideas on what to do in your own tooling. I recommend the account registration and random ExternalID requirements even if you use AWS Organizations since adding a condition to restrict access from only your own organization can still open you up to attack from a lower-security account to a higher-security account.

And stay tuned for that new skunkworks project! We should be able to talk about it in a few months and it’s a real game changer for these kinds of problems.