Automate or Die!How to Survive to an Attack in the Cloud

March 3rd 2017

Toni de la Fuente (@ToniBlyx – blyx.com)Lead Security Operations / Security Architect

Once upon a time…September 2015

Prepare to be hacked!

Foundation

First Things First

*NIST Definition

Ubiquity• Regions• Availability Zones• AWS:• 16 regions (+2)• 42 AZ (+4)

• Azure: • 32 regions

• GCP: • 6 regions (+8)• 18 zones (+16)

*CDN locations not included

AWS Region

Amazon RDS MySQL Master

Internet gateway

Availability Zone 1 Availability Zone 2

Public subnet Public subnet

NAT gateway

EC2

Bastion10.0.128.5

NAT gateway

EC2

Bastion10.0.144.5

Alfresco OneAuto Scaling Group

Elastic Load Balancing

Amazon RDS MySQL Slave

S3 forShared Content

Store

10.0.0.0/1610.0.128.0/20 10.0.144.0/20

10.0.0.0/19 10.0.32.0/19

Alfresco IndexAuto Scaling Group

Private SubnetPrivate Subnet

Alfresco Server Alfresco Server Alfresco ServerAlfresco Server

Index Server Index Server Index Server Index Server

* Immutable infrastructure

Shared Security Model / Responsibility ZonesIaaS• Data• Application• Operating System• Virtualization• Infrastructure• Physical

PaaS• Data• Application• Operating System• Virtualization• Infrastructure• Physical

SaaS• Data• Application• Operating System• Virtualization• Infrastructure• Physical

Shared Security Model / Responsibility Zones

AWS manages the security OF the Cloud

You

AWS foundation services

Compute Storage Database Networking

AWS global infrastructure

Regions

Availability zonesEdge locations

Client-side data encryption Server-side data Encryption Network traffic protection

Platform, applications, identity & access management

Operating system, network & firewall configuration

Customer applications & content

You define your controls IN the Cloud

* Similar on other providers / subject to changes depending on the service or product

*

Shared Security Model / Responsibility Zones

Challenges in Case of an Incident

Disadvantages and ChallengesCloud Forensics and Operations

Ubiquity EnumerationLegal jurisdiction

Elasticity Preservation of evidenceData integrity

Data persistence (replication) Chain of custodyEvidence integrity

Multi-tenancy Data attributionChain of custody

Abstract Determine the best evidencePreservation and visualization of evidence

Quantity of data and Big Data Systems that cannot be investigated or managed in a traditional manner

Knowledge Trained staffContinuous evolution and new features almost daily

Providers Service level agreement / service level objectivesRelationship client-provider / transparency

Traditional vs Cloud ForensicsProcesses Traditional Forensics Cloud Forensics

Identification Identification of an event or incident Multiple tools Few tools

Preservation Securitization and assessment of the scene Yes No

Documentation of the scene Yes NoEvidence collection: origin of the evidence Physical hardware Virtual hardware

Evidence collection: location of the evidence Crime scene Provider’s data center

Marking, packaging and transport Physical Digital through the Internet or physical media

Acquisition /Extraction

Acquisition time Slow FastRAM acquisition Yes DependantHash Slow FastErased data recovery Possible DifficultMetadata acquisition Yes YesTime stamp Precise ComplexInstallation (action) of forensic software Expensive CheapConfiguration and availability of forensic software Expensive CheapTransport Yes No

Analysis Analysis Slow Fast (potentially)

Presentation Documentation of evidence Acquired evidence Data from many sourcesDeclaration Common Difficult to explain to a judge

Storage OptionsType AWS Azure GCPObjects S3 Object Storage

• Buckets• 5TB max per object• Encryption In-flight and at-rest

Azure Storage• Blob storage• 500TB limit per storage account• Encryption In-flight and at-rest

Google Cloud Storage• Buckets• 5TB max per object• Encryption In-flight and at-

rest SAN EBS (Volumes)

• Volume size: 1GB to 16TB (in 1GB increments)

• Magnetic, SSD• Encryption available• Snapshots

Azure Block Storage• Page blobs• Volume size: 1GB to 1TB • Standard (Magnetic), SSD

premium• Snapshots• Encryption available

Google Block Storage• Volume size: 1GB to 10TB • Magnetic, SSD• Snapshots• Encryption by default

NAS Shared Storage (NFS)• EFS

File Storage (CIFS) Single Node File Server + Others

Archive Glacier Azure Backup Google Cloud Storage Nearline

Migration

Import Export / Snowball Import Export Third Party Solution (Iron Mountain, etc.)

CDN AWS CloudFront (CDN) Azure CDN Google Cloud CDN* Ephemeral, DBs, Queues, Caching and Storage GW not included

AWS Specifics

Account and Keys in AWS• Root account: account owner, full access to all resources in the account, very specific tasks (transfer domain,

billing details, support plan)– Email and password + MFA code (if enabled)

• IAM (Identity and Access Management)– User name and password + MFA code (if enabled) to access AWS Management Console, AWS

discussion forums, or AWS support center– SAML– Users, Groups, Roles, Policies. Instance profiles (role)

• Access Keys: AWS SDKs, REST, or Query APIs (AWS CLI)– Access Key i.e: AKIAIOSFODNN7EXAMPLE– Secret Access Key i.e: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

• STS (Security Token Service): – temporary, limited-privilege credentials for IAM users or for users that you authenticate (also for

federated users), and for instances (instance profile)• Key Pairs: Key pairs are used only for:

– Amazon EC2 (SSH) and Amazon CloudFront (sign URLs or sign cookies)

*Become an IAM Ninja: https://youtu.be/Du478i9O_mc

AWS Metadata Server“Instance metadata is data about your instance that you can use to configure or manage the running instance”

“Anyone who can access the instance can view its metadata. Therefore, you should take suitable precautions to protect sensitive data”

# curl http://169.254.169.254/latest/meta-data/ami-idami-launch-indexami-manifest-pathblock-device-mapping/Hostnameiam/instance-actioninstance-idinstance-typelocal-hostnamelocal-ipv4Macmetrics/network/placement/product-codesProfilepublic-keys/reservation-idsecurity-groupsservices/

Attacks

Common incidents• Access Keys compromise, abuses of unconsciously published keys: e.g. a

developer publishing their keys on SourceCode repo (Github, Bitbucket, etc) when commit, keys stolen from workstation, keys hardcoded in application files (bin or config), resources created for criminal purposes, mining, etc.• Phishing attacks: your instance is going to be retired (targeted to admins).

Hard to detect phising bc comes with HTTPS, S3, etc.• Compromised resources: e.g. an unpatched EC2 instance may be infected

with malware and act as a botnet. Poisioned AMI.• Unintentional abuses: e.g. a crawler-type own process being classified as a

DDoS attack by a third party.• Abuses committed by users: e.g. malware or other type of illegal content

being published by the end user of an AWS service on a public S3 bucket.

Common incidents• Application running in a role: can lead to an access to the application

role and stole Access Key or access metadata service• Abuses related to configuration failures: e.g. a web-based proxy service

incorrectly configured being used as an open proxy. SMTP relay, etc. • Infection through 3rd party services: you give them keys to perform

actions (DataDog incident 2016)• Hybrid attacks: attacks partly carried out from a Cloud-based system or

data storing on S3, even when mobile devices or personal computers are used.• Organized crime of all sort• False positives• Did I say CONFIGURATION FAILURES*?

Create a new one:$ aws sts get-session-token --duration-seconds 129600

Persistence• Instance compromised might become Access Keys Compromised– Metadata service (inside an Instance)• curl http://169.254.169.254/latest/meta-data/iam/• curl

http://169.254.169.254/latest/meta-data/iam/security-credentials/<*instanceRole>{

"Code" : "Success", "LastUpdated" : "2017-02-02T03:07:42Z", "Type" : "AWS-HMAC", "AccessKeyId" : "ASIAIWCR2OKMVILEXAMPLE", "SecretAccessKey" : "DVvxzikewoVBrZN30fFBdEQdTczm1WuGOLXC", "Token" : "FQoDYXdzELT//////////wEaDH7/lKtowqytymR0bSK3A0VAup4Atle7I3P6N6aRKCNpPIqt===SHORTENED","Expiration" : "2017-02-02T09:22:37Z”}

*If it has been attached to the instance

Create a new one:$ aws sts get-session-token --duration-seconds 129600

Persistence• Instance compromised might become a Key Compromised– Metadata service (inside an instance)• curl http://169.254.169.254/latest/meta-data/iam/• curl http://169.254.169.254/latest/meta-data/iam/security-

credentials/<instanceRole>{ "Code" : "Success", "LastUpdated" : "2017-02-02T03:07:42Z", "Type" : "AWS-HMAC", "AccessKeyId" : "ASIAIWCR2OKMVILEXAMPLE", "SecretAccessKey" : "DVvxzikewoVBrZN30fFBdEQdTczm1WuGOLXC", "Token" : "FQoDYXdzELT//////////wEaDH7/lKtowqytymR0bSK3A0VAup4Atle7I3P6N6aRKCNpPIqt===SHORTENED","Expiration" : "2017-02-02T09:22:37Z”}

Let’s the Party Begins! Lateral

movement, snapshots, attach

volumes, roles, policies, API GW

+ serverless backdoor,

ransomware… Metasploit is your

friend

Serverless!!!• Who is auditing

serverless?

• Amazon Lambda • CloudWatch

• Azure Cloud Functions• WebJobs

• Google Cloud Functions

MadKing Attack• https://github.com/ThreatResponse/mad-king • Using stolen access keys. Uses Zappa.io. Creates an API Gateway and

Lambda function• Features• Disable CloudTrails• Encrypt CloudTrails• Generate New Developer Access Keys• Stop Instances• Terminate Instances• Burn them all (Destroy all instances) – really Mad King \m/

https://danielgrzelak.com/backdooring-an-aws-account-da007d36f8f9#.ut0x2bjv5

AWS Lambda Infection Toolkit• https://github.com/Miserlou/mackenzie (zappa.io guy)• Persistent Lambda Malware PoC • Features• Encrypt with pubkey• Exfil via POST, S3, Email, SMS, Network Resource Tags• Install Flask backdoor• Infect old package sources• Infect all available functions• Create re-infection handlers

Gone in 60 Millisecons (33c3): https://www.youtube.com/watch?v=YZ058hmLuv0

No code available

Other Attacks Tools• Metasploit AWS module• IAM privilege enumeration module• Lambda module• S3 bucket and access enumeration

• AWS pwn• Reconnaissance, exploitation and exfiltration• https://github.com/dagrz/aws_pwn

• Comulus Cloud Attack Module (not an attack just vulnerable sample code)• Presented at RSA 2017 (Serverless Security)• https://github.com/devsecops/lambhack

Incident Response

Now what?• Control• Impact• Recover• Investigate• Improve

• Notifications from AWS• Access activity (IAM)• Billing activity (Budget alerts)• Logs• Other• Third parties (dedicated tools)• NIDS (Snort, Suricata, etc.)• HIDS (OSSEC, Osquery, rkhunter,

Auditd)• SIEM

Incident Indicators

Sample Task List / WorkflowCompromised

Instance

StartForensic

Workstation

Live or

Dead

Attach the Tools Volume

Apply Isolation

SG

StolenAPI Keys

Check new resources created

Disable Keys

Make API log report if enabled

If foundIsolate them Create a report

Attach the Evidence Collection

Volume

Isolate it?

Log in to the instance

PerformEvidence

Acquisition

Take snapshot to all volumes

Stop itMake Volumes to

Snapshots

Attach Volumes to Forensic

Workstation

Attach the Evidence Collection

Volume

Log in to the Forensic

Workstation

Analyze / Further

InvestigationIncident

Live

Dead

Create Support Case with Provider

Revoke Access

Revoke Sessions

Outside Info Acquisition(instance

profile,endpoints,metadata

, etc)_

Perform Timeline

Pre-built Volatiliy frofile

Pre-built LiME krnl

mod

RAM Acquisition

Yes

NIC Network Scan

Open an Internal Case

Separate Network with Internet

Access to Scan

CAINE / SIFT / DEFT / FCCU /

HELIX3 / FIRE

Windows_Life_ResponseSysinternals

NirsoftFTK Imager

AutopsySleuthkit

Trigger a Network Capture

TAGResources

under investigation

*

Sample Task List / WorkflowCompromised

Instance

StartForensic

Workstation

Live or

Dead

Attach the Tools Volume

Apply Isolation

SG

StolenAPI Keys

Check new resources created

Disable Keys

Make API log report if enabled

If foundIsolate them Create a report

Attach the Evidence Collection

Volume

Isolate it?

Log in to the instance

PerformEvidence

Acquisition

Take snapshot to all volumes

Stop itMake Volumes to

Snapshots

Attach Volumes to Forensic

Workstation

Attach the Evidence Collection

Volume

Log in to the Forensic

Workstation

Analyze / Further

InvestigationIncident

Live

Dead

Create Support Case with Provider

Revoke Access

Revoke Sessions

Outside Info Acquisition(instance

profile,endpoints,metadata

, etc)_

Perform Timeline

Pre-built Volatiliy frofile

Pre-built LiME krnl

mod

RAM Acquisition

Yes

NIC Network Scan

Open an Internal Case

Separate Network with Internet

Access to Scan

CAINE / SIFT / DEFT / FCCU /

HELIX3 / FIRE

Windows_Life_ResponseSysinternals

NirsoftFTK Imager

AutopsySleuthkit

Trigger a Network Capture

TAGResources

under investigation

*

Outside Info Acquisition

PerformEvidence

Acquisition

AWS Infrastructure Logs:CloudTrail and VPCFlowLogs

AWS Service Logs:S3 Logs, RDS Logs, Lambda, etc.

Host Based LogsMessages/System, security, audit, applications, etc.

More Inside: instance profile, endpoints, syslogs, screen, metadata, etc

More Outside: Limits, check resources creation from date (all regions)

IRDF Automation Tools

Tools• March 2016:• https://blyx.com/2016/03/11/forensics-in-aws-an-introduction/

• June 2016:• https://blyx.com/2016/06/16/cloud-forensics-caine7-on-aws/

• August 2016:• Threat Response (Presented in BlackHat 2016)• https://s3-us-west-2.amazonaws.com/threatresponse-static/us-16-Krug-Hard

ening-AWS-Environments-and-Automating-Incident-Response-for-AWS-Compromises-wp.pdf

Pre-Automation POC – AWS CLI (Scripts)

# DISABLE STOLEN KEYS

aws iam update-access-key --access-key-id \ AKIAIOSFODNN7EXAMPLE --status Inactive \ --user-name Bob

aws iam delete-access-key --access-key \AKIDPMS9RO4H3FEXAMPLE --user-name Bob

# LOOK FOR NEW RESOURCES CREATED

aws ec2 describe-instances --region us-east-1 \--query 'Reservations[].Instances[?LaunchTime>=`2017-02-3`][].{id: InstanceId, type: InstanceType, launched: LaunchTime}'

# TAG INSTANCE aws ec2 create-tags --resources i-INSTANCE-ID \--tags “Key=Environment, Value=Quarantine:REFERENCE-ID”

# ISOLATE AN INSTANCE IN A VPC

aws ec2 create-security-group --group-name \ isolation-sg --description “Security group to isolate a EC2-VPC instance” --vpc-id vpc-1a2b3c4d

aws ec2 authorize-security-group-ingress \--group-id sg-BLOCK-ID --protocol tcp --port 22 \--cidr YOUR.IP.ADDRESS.HERE/32

aws ec2 authorize-security-group-egress \--group-id sg-BLOCK-ID --protocol ‘tcp’ \--port 80 --cidr ‘0.0.0.0/0’

aws ec2 modify-instance-attribute --instance-id \i-INSTANCE-ID --groups sg-BLOCK-ID

# CREATE VOLUME SNAPSHOT

aws ec2 create-snapshot –-volume-id vol-xxxx \–-description “IR-ResponderName- Date-REFERENCE-ID”

Threat Response Tool• Incident Response Tool for AWS• http://threatresponse.cloud/

• Compromised AWS API credentials (Access Keys)• Mitigate compromise: Lock

• Compromised EC2 instance• Mitigate compromise

• Isolation

• Collect evidence• Memory acquisition

• Create an Incident Response Workstation in AWS • Start an EC2 instance

• Analysis of collected evidences• WebApp (ThreatResponse Web)

• RAM (volatility)• Disk (Log2time + TimeSketch)

• AWS hardening• Threatprep

• API• Modules

<DEMO>ThreatResponse: aws_ir, margaritashotgun

Assessment and Hardening

Persistence Prevention (AWS)• Instance compromised might become a Key Compromised– UserData in CloudFormation: watch out!

• Prevent it is not very difficult:

• STS tokens can’t be revoked (you only can disable permissions)• CloudTrail may help to detect it (if enabled!)– watch the watcher

• It can shutdown your company! (you won’t be the first one: CodeSpaces)• Use multiple AWS accounts!

iptables -A OUTPUT -m owner ! --uid-owner root -d 169.254.169.254 -j DROP

Instance / Network / Provider• Put all what you need in your well known AMI:

• Hardening applied / Tested (Packer/Vagrant)• CIS Benchmark!

• No config / access needed• Local tools

• Osquery / OSSEC / rkhunter• Update rules / serverless

• local configuration (SELinux/AppArmour)• AuditD

• Collect telemetry host network data (snort/suricata)• Collect everything your provider allows you

• Networking• APIs / Accesses

• Red Team / Third party pentesting*

API calls• Who• When• What call• What resources• Where (from)

Auditing, Assessment and Hardening Tools

• AWS Trusted Advisor• AWS CloudTrail /

Azure Operational Insights • AWS CloudFormation• AWS Config Rules• Alfresco: Prowler / Automate

Hardening CIS Section 3* / OpenSCAP fix (AWS)• Nccgroup: Scout2 (AWS)• Netflix: SecurityMonkey, EDDA,

FIDO (AWS)• Capital One: CloudCustodian (AWS)

• AWS CIS Benchmark Python code and Lambda functions (AWS)• CloudSploit (AWS)• Widdix Hardening Templates (AWS)• Awslimitchecker (AWS)• OMS Security & Compliance (Azure)• Spotify: gcp-audit (GCP)• *Analytics (ELK, Splunk, Nuix etc)• Git Secrets (AWS)

<DEMO>Hardening Automation with templates, Prowler, Security Monkey

TakeawaysSamples, templates, code, links and this presentation is already available at:

https://github.com/toniblyx/rootedcon2017

TL;DR• Automation for everything (deployment multi AZ, hardening,

response, recovery/recreation, centralized logging, log everything!)• Encryption Everywhere (any layer, any content, on-prem, on-

transit)• Account Separation and MFA (prod, test, devel, etc.)• Least Privilege•Go to Immutability / Ephemeral• Expect to be Hacked: Buy Bitcoins…

Questions?

toni@blyx.com - @ToniBlyx

References• Cloud Security Alliance, Mapping the Forensic Standard ISO/IEC 27037 to Cloud Computing, June 2013 • Dr. Keyun Ruan University College Dublin, Designing a Forensic-enabling Cloud Ecosystem, 2013 • International Standard ISO/IEC 27037, Information technology — Security techniques — Guidelines for identification, collection, acquisition, and preservation of digital evidence, October

2012 • Josiah Dykstra, Digital Forensics for IaaS Cloud Computing, June 2012 • Keyun Ruan, Ibrahim Baggili (PhD), Prof Joe Carthy, Prof Tahar Kechadi University College Dublin, Zayed

University, Survey on Cloud forensics and critical criteria for Cloud forensic capability: A preliminary analysis • Keyun Ruan, Joe Carthy, Tahar Kechadi and Mark Crosbie, Cloud Forensics • Keyun Ruan, University College Dublin, Cloud Forensics: challenges & opportunitiess, 2010• NIST Cloud Computing Forensic Science Working Group Information Technology Laboratory, NIST Cloud Computing Forensic Science Challenges, June 2014• Peter Mell Timothy Grance, NIST Special Publication 800-145, The NIST Definition of Cloud Computing, September 2011 • Report From the First Digital Forensic Research Workshop (DFRWS), A Road Map for Digital Forensic Research, August 2001 • Forensics-as-a-Service (FaaS): Computer Forensic Workflow Management and Processing Using Cloud. Yuanfeng Wen, Xiaoxi Man, Khoa Le and Weidong Shi• http://static1.squarespace.com/static/5417f7f9e4b0b77770545590/t/56f3c598906340a7f6e78dbd/1458816415654/AWS_Cloud_and_Security.pdf • https://www.blackhat.com/docs/us-16/materials/us-16-Amiga-Account-Jumping-Post-Infection-Persistency-And-Lateral-Movement-In-AWS-wp.pdf • https://alestic.com/2015/10/aws-iam-readonly-too-permissive/ • Backdooring an AWS account• Exploring an AWS account post-compromise• Disrupting AWS logging• AWS IAM "ReadOnlyAccess" Managed Policy is Too Permissive (For Us)• Access Keys will kill you before you kill the password• Account Jumping Post Infection Persistency and Lateral Movement in AWS• Disrupt CloudTrail and pwning automation tools• RSA 2017 talk: Cloud Security Automate or Die, same tittle as mine but a bit different approach• RSA 2017 talk: Securing Serverless applications in the Cloud• RSA 2017 talk: DevSecOps on the Offense: Automating Amazon Web Services Account Takeover

Thanks!Special Thanks to:

Alfresco DevOps Team Andrew K. @andrewkrug & Joel F., ThreatResponse.cloud Team

Daniel Grzelak @dagrzLorenzo Martinez @lawwait