Research

I’ve been teaching cloud incident response with Will Bengtson at Black Hat for a few years now, and one of the cool side effects of running training classes is that we are forced to document our best practices and make them simple enough to explain. (BTW — you should definitely sign up for the 2024 version of our class before the price goes up!) One of the more amusing moments was the first year we taught the class, when I realized I was trying to hand-write all the required CloudTrail log queries in front of the students, because I had only prepared a subset of what we needed. As I wrote in my RECIPE PICKS post, you really only need a handful of queries to find 90% of what you need for most cloud security incidents. Today I want to tie together the RECIPE PICKS mnemonic with the sample queries we use in training. I will break this into two posts — today I’ll load up the queries, and in the next post I’ll show a sample analysis using them. A few caveats: These queries are for AWS Athena running on top of CloudTrail. This is the least common denominator — anyone with an AWS account can run Athena. You will need to adapt them if you use another tool or a SIEM, but those should just be syntactical changes. Obviously you’ll need to do more work for other cloud providers, but this information is available on every major platform. These are only the queries we run on the CloudTrail logs. RECIPE PICKS includes other information these queries don’t cover, or that don’t cleanly match a single query. I’ll write other posts over time, showing more examples of how to gather that data, but none of it takes very long. In class we spend a lot of time adjusting the queries for different needs. For example, when searching for entries on a resource you might need to look in responseElements or requestParameters. I’ll try to knock out a post on how that all works soon, but the TL;DR is: sometimes the ID you query on is used in the API call (request); other times you don’t have it yet, and AWS returns it in the response. RECIPE PICKS is not meant to be done in order. It’s similar to a lot of mnemonics I use in paramedic work. It’s to make sure you don’t miss anything, not an order of operations. With that out of the way, here’s a review of RECIPE PICKS (Canvas FTW): Now let’s go through the queries. Remember, I’ll have follow-on posts with more detail — this is just the main reference post to get started. A few things to help you understand the queries: For each of these queries, you need to replace anything between <> with your proper identifiers (and strip out <>). A “%” is a wildcard in SQL, so just think “*” in your head and you’ll be just fine. You’ll see me pulling different information on different examples (e.g., event name). In real life you might want to pull all the table fields, or different fields. In class we play around with different collection and sorting options for the specific incident, but that is too much for a single blog post. Resource If I have a triggering event associated with a resource, I like to know its current configuration. This is largely to figure out whether I need to stop the bleed and take immediate action (e.g., if something was made public or shared to an unknown account). There is no single query because this data isn’t in CloudTrail. You can review the resource in the console, or run a describe/get/list API call. Events Gather every API call involving a resource. This example is for a snapshot, based on the snapshot ID: SELECTuseridentity.arn,eventname,sourceipaddress,eventtime,resourcesFROM <your table name>WHERE requestparameters like ‘%<snapshot_id%’ OR responseelements like ‘%<snapshot id>%’ORDER BY eventtime Changes Changes is a combination of the before and after state of the resource, and the API call which triggered the change associated with the incident. This is another one you can’t simply query from CloudTrail, and you won’t have a change history without the right security controls in place. This is either: AWS Config A CSPM/CNAPP with historical  inventory A cloud inventory tool (if it has a history) Many CSPM/CNAPP tools include a history of changes. This is the entire reason for the existence of AWS Config (well, based on the pricing there may be additional motivations). My tool (FireMon Cloud Defense) auto-correlates API calls with a change history, but if you don’t have that support in your tool you may need to do a little manual correlation. If you don’t have a change history this becomes much harder. Worst case: you read between the lines. If an API call didn’t error, you can assume the requested change went through and then figure out the state. Identity Who or what made the API call? CloudTrail stores all this in the useridentity element, which is structured as: useridentity STRUCT<type:STRING,principalid:STRING,arn:STRING,accountid:STRING,invokedby:STRING,accesskeyid:STRING,userName:STRING,sessioncontext:STRUCT<attributes:STRUCT< mfaauthenticated:STRING, creationdate:STRING>,sessionissuer:STRUCT< type:STRING, principalId:STRING, arn:STRING, accountId:STRING, userName:STRING>,ec2RoleDelivery:string,webIdFederationData:map<string,string>> The data you’ll see will vary based on the API call and how the entity authenticated. Me? I keep it simple at this point, and just query useridentity.arn as shown in the query above. This provides the Amazon Resource Name we are working with. Permissions What are the permissions of the calling identity? This defines the first part of the IAM blast radius, which is the damage it can do. The API calls are different between user and role, and here’s a quick CLI script that can pull IAM policies. But if you have console access that may be easier: #!/bin/bash# Function to get policies attached to a userget_user_policies() {local user_arn=$1local user_name=$(aws iam get-user –user-name $(echo $user_arn | awk -F/ ‘{print $NF}’) –query ‘User.UserName’ –output text)echo “User Policies for $user_name:”aws iam list-attached-user-policies –user-name $user_name –query ‘AttachedPolicies[*].PolicyArn’ –output text | while read policy_arn; doaws iam get-policy –policy-arn $policy_arn –query ‘Policy.DefaultVersionId’ –output text | while read

A year or so ago I was on an application security program assessment project in one of those very large enterprises. We were working with the security team and they had all the scanners, from SAST/SCA to DAST to vulnerability assessment, but their process was really struggling. It took a long time for bugs to get fixed, things were slow to get approved and deployed, and remediating in-production vulnerabilities could also be slow and inefficient. At one point I asked how vulnerabilities (anything discovered after deployment) were being communicated back to the developers/admins? “Oh, that data is classified as security sensitive so they aren’t allowed access.” Uhh… okay, So you are not letting the people responsible for creating and fixing the problem know about the problem? How’s that going for you? This came up in a conversation today about providing cloud deployment administrators access to the CSPM/CNAPP. In my book this is often an even worse gap, since a large percentage of organizations I work with do not allow the security team change access to cloud deployments, yet issues there are often immediately exploitable over the Internet (or you have a public data exposure… just read the Universal Cloud Threat Model, okay?). Here are my recommendations: Give preference to security tools that have an RBAC model which allows you to provide devs/admins access to only their stuff. If you can’t have that granularity, this doesn’t work well. Communicate discovered security issues in a manner compatible with the team’s tools and workflows. This is often ChatOps (Slack/Teams) or ticketing/tracking systems (JIRA/ServiceNow) they already use. Groom findings before communicating!!! Don’t overload with low-severity or false positives. Focus on critical/high if you are inserting yourself into their workflow, and spend the time to reduce either true false positives (the tools made a mistake) or irrelevant positives (you have a compensating control or it doesn’t matter). On the cloud side this is easier than with code, but it still matters and takes a little time. Allow the admins/leads (at least) direct access to the scanner/assessment tool, but only for things they own. The tools will have more context than an autogenerated alert or ticket. This also allows the team to see the wider range of lower severity issues they still might want to know about. And one final point: Email and spreadsheets are not your friends! When the machines finally come for the humans, the first wave of their attack will probably be email and spreadsheets. One nuance arises when you are dealing with less-trusted devs/admins, which often means “outsourcing”. Look, the ideal is that you trust the people building the things your business runs on, but I know that isn’t how the world always runs. In those cases you will want to invest more in grooming and communications, and probably not give them any direct access to your tooling. I’ve written a lot on appsec and cloudsec over the years, and worked on a ton of projects. This issue has always seemed obvious to me, but I still encounter it a fair bit. I think it’s a holdover from the days when security was focused on controlling all the defenses. But time has proven we can only do so much from the outside, and security really does require everyone to do their part. That’s impossible if you can’t see the bigger security picture. Most of you know this, but if this post helps just one org break through this barrier, then it is worth the 15 minutes it took to write. Share:

Look, we don’t yet know what really happened at Sisense. Thanks to Brian Krebs and CISA, combined with the note sent out by the CISO (bottom of this post), it’s pretty obvious the attackers got a massive trove of secrets. Just look at that list of what you have to rotate. It’s every cred you ever had, every cred you ever thought of, and the creds of your unborn children and/or grandchildren. Brian’s article has basically one sentence that describes the breach: Sisense declined to comment when asked about the veracity of information shared by two trusted sources with close knowledge of the breach investigation. Those sources said the breach appears to have started when the attackers somehow gained access to the company’s code repository at Gitlab, and that in that repository was a token or credential that gave the bad guys access to Sisense’s Amazon S3 buckets in the cloud. And as to where the data ended up? Sure sounds like the dark web to me: On April 10, Sisense Chief Information Security Officer Sangram Dash told customers the company had been made aware of reports that “certain Sisense company information may have been made available on what we have been advised is a restricted access server (not generally available on the internet.)” So if (and that’s a very big if, this early) the first quote is correct, then here’s what probably happened: Someone at Sisense stored IAM user keys in GitLab. Probably in code, but that’s an open question. Could have been in the pipeline config. Sisense also stored a vast trove of user credentials that either were not encrypted before putting them in S3 (I’ll discuss that in a moment), or with the decryption key in the code. Bad guys got into GitLab. Bad guys (or girls, I’m not sexist) tested keys and discovered they can access S3. Maybe this was obvious because the keys were in the code to access S3, or maybe it was non-obvious and they tried good old ListBuckets. Attackers (is that better, people?) downloaded everything from S3 and put it on a private server or somewhere in the depths of Tor. We don’t know the chain of events that lead to the key/credential being in GitLab. There’s also the chance it was more-advanced and the attacker stole session credentials from a runner or something and a static key wasn’t a factor. I am not going to victim blame yet — let’s see what facts emerge. The odds are decent this is more complex than what’s emerged. HOWEVER Some of you will go to work tomorrow, and your CEO will ask, “how can we make sure this never happens to us?” None of this is necessarily easy at scale, but here’s where to start: Scan all your repositories for cloud credentials. This might be as simple as turning on a feature of your platform, or you might have to use a static analysis tool or credential-specific scanner. If you find any, get rid of them. If you can’t immediately get rid of them, hop on over to the associated IAM policy and restrict it to the necessary source IP address. Don’t think S3 encryption will save you. First, it’s always encrypted. The trick is to encrypt with a key that isn’t accessible from the same identity that has access to the data. So… that’s not something most people do. Encryption is almost always incorrectly implemented to stop any threat other than a stolen hard drive. If you manage customer credentials, that sh** needs to be locked down. Secrets managers, dedicated IAM platforms, whatever. It’s possible in this case all the credentials weren’t in an S3 bucket and there were other phases of the attack. But if they were in an S3 bucket, that is… not a good look. Sensitive S3 buckets should have an IP-based resource policy on them. All the IAM identities (users/roles) with access should also have IP restrictions. Read up on the AWS Data Perimeter. Get rid of access keys. Scan your code and repositories for credentials. Lock down where data can be accessed from. And repeat after me, Every cloud security failure is an IAM failure, and every IAM failure is a governance failure. I’m really sorry for any of you using Sisense. That list isn’t going to be easy to get through. Share:

Nearly everyone in the United States (and probably elsewhere) is infected with the Epstein-Barr virus at some point in their life. Most people will never develop symptoms, although a few end up with mono. Even without symptoms you carry this invasive genetic material for life. There’s no cure, and EBV causes some people to develop cancers and possibly Multiple Sclerosis, Chronic Fatigue Syndrome, and other problems. Those later diseases are likely caused by some other precipitating even or infection that “triggers” a reaction with EBV. Look, I have most of a molecular biology degree and I’m a paramedic and I won’t pretend to fully understand it all. The tl;dr is EBV is genetic material floating around your body for life and at some point it activates or interacts with something else and causes badness. (Me write good! Use words!) As I’ve been reading about the XZ Initiative (I’m using initiative deliberately due to the planning and premeditation) the same week that the CISA CSRB released their scathing report on Microsoft, it’s damn clear that our software supply chain issues are as deep as the emptiness of my cat’s soul. (I mean I love him, and I’m excited he’s coming back from the hospital this afternoon, but I couldn’t come up with a more-amusing analogy). If you aren’t up to date on all things XZ I suggest reading Matt Johansen’s rollup in his Vulnerable U newsletter. Here’s how EBV and XZ relate, at least in my twisted mind. XZ was clearly premeditated, well planned, sophisticated, and designed to slowly spread itself under the radar for many years before being triggered. There is absolutely no chance this approach hasn’t already been used by multiple threat actors. As much as I hate FUD and hyperbole, I am 100% confident that there is code in tools and services I use that has been similarly compromised. We didn’t miraculously catch the first ever attempt, because a Microsoft dev is anal-retentive about performance. XZ is the first such exploit which got caught. If I were a cybercriminal or government operative, I would already have several of these long-term attacks underway. You are welcome to believe our record is 1 for 1. I think it’s 1 catch of N attacks, and N scares me. I also do not believe we can eliminate this threat vector. I don’t think the best SAST/SCA tools and a signed SBOM have any chance at making this go away. Ever. That doesn’t mean we give up and lose hope — we just change our perspective and focus more on resilience to these attacks than pure prevention. I don’t have all the answers — not even close — but there are three aspects I think we should explore more. First, let’s make it harder on threat actors. Let’s increase their costs. How? Well, aside from all the improved security scanning over the past few years, I like the idea Daniel Miessler recently mentioned in a conversation and noted in his newsletter: use AI to automatically perform open source intel (OSINT) on OSS contributors. Do they have a history outside that code repo? Any real human interactions? This will be far from perfect, but will likely increase the cost of attack to build a persona which looks sufficiently real. We also have compromises in commercial software (hello Solar Winds). Vendors need to explore better internal code controls, sourcing, and human processes. E.g. require YubiKeys from all devs, side channel notifications and approvals of commits, and I suspect there are some new and innovative scanning approaches we can take as AI evolves (until it evolves past humanity and enslaves us all). E.g. “this may not be a known security defect, but it looks weird compared to this developer’s history, so maybe ping another future energy source human to review it”. I’m also a fan of making critical devs work on dedicated machines separate from the ones they use for email and web browsing, to reduce phishing/malware as a vector. No, I haven’t ever had anyplace I’ve worked approve that, but I *have* heard of some shops which pulled it off. The final part is preparing for the next XZ that slips through and is eventually triggered. Early detection, rapid remediation, and all the other hard expensive things. SBOM/SCA/DevSecOps are key here: you MUST be able to figure out where you are using any particular software package, and be able to implement compensating defenses (e.g., firewalls) and patch quickly. This is NOT SIMPLE AT SCALE, but it’s your best bet as the downstream customer for these things. None of what I suggested is easy. I think this is the next phase of the Assume Breach mindset. You can’t cure EBV. You can’t prevent all possible negative outcomes. But you can reduce some risks, detect others earlier, and react aggressively when those first cancer cells show up. Share: