Hardening cassandra q2_2016

Securing Cassandrafor Compliance (or Paranoia)

Hi, I'm Nate. @zznate https://www.linkedin.com/in/zznate http://www.slideshare.net/zznate/

Co-Founder, CTO The Last Pickle

Cassandra user since 2009 (v0.4) Austin, Texas

https://www.linkedin.com/in/zznate

http://www.slideshare.net/zznate/

Security presentations can be scary. Here's a cat.

First, how did we get here and why is securing Cassandra important?

"Target CEO Gregg Steinhafel Resigns In Data Breach Fallout"

http://www.forbes.com/sites/clareoconnor/2014/05/05/target-ceo-gregg-steinhafel-resigns-in-wake-of-data-breach-fallout/

First, how did we get here and why is securing Cassandra important?

I have

your personal

information

Customers place a lot of trust in technology companies

LOL! Me too!

Sometimes too much.

Ease of scalability comes with a price

HA! A bin-packed message format with no source

verification!*

Ease of scalability comes with a price

* <currently reading o.a.c.net.MessageIn#read>

nmap -Pn -p7000 \-oG logs/cass.gnmap 54.88.0.0/14

I'm publicly discussing your

technical shortcomings

Then you end up in this situation.

Meanwhile, at the FCC...

We have to require two factor, secure socket transport

encryption, something something...

ZZZzzzzzzzZZZzz

We did a regulation!

My staffers still print out my email :)

Why are we doing this again?

Sssshhhh. I'm AES'ing...

...even though the traffic never leaves a backplane.

Some industries will require node to node SSL

1. Encrypting data at rest2. Encrypting data on the wire3. Authentication and authorization4. Management and tooling

Focusing our Discussion: Architecture

1. Encryption at rest

No matter what: understand the failure modes

bit rot, entropy, etc. Horrible things can happen with on disk encryption.

Don't mind me, I'm just your key server.

Haha! Later!

xWhat's on this

disk again?

Shrug.

...but you may not have a choice.

Because we said "at rest"

dmcrypt, eCryptFS

Open source options:

Vormetric, Gazzang

Commercial options:

DSE Encryption

CREATE TABLE users ...WITH compression_parameters:sstable_compression = 'Encryptor' and compression_parameters:cipher_algorithm = 'AES/ECB/PKCS5Padding' and compression_parameters:secret_key_strength = 128;

DSE Encryption

CREATE TABLE users ...WITH compression_parameters:sstable_compression = 'Encryptor' and compression_parameters:cipher_algorithm = 'AES/ECB/PKCS5Padding' and compression_parameters:secret_key_strength = 128;

WARNING:

commitlog not included*

*eCryptFS would work fine for this

EBS Encryption (a.k.a "not my problem")

(Looks like this)

EBS Encryption (a.k.a "not my problem")

http://www.slideshare.net/AmazonWebServices/bdt323-amazon-ebs-cassandra-1-million-writes-per-second

See Crowdstrike's presentation on Cassandra GP2 performance (with encryption):

http://www.slideshare.net/AmazonWebServices/bdt323-amazon-ebs-cassandra-1-million-writes-per-second

Maybe Client Side?

The Java Driver now has custom codecs which would make this easy to implement

https://github.com/datastax/java-driver/tree/3.0/manual/custom_codecs

Maybe Client Side?

The Java Driver now has custom codecs which would make this easy to implement

https://github.com/datastax/java-driver/tree/3.0/manual/custom_codecs

Column-level encryption!

New in Cassandra 3.4 (DSE 5.1?):

Commitlog Encryption: CASSANDRA-6018 Hint File Encryption: CASSANDRA-11040

https://issues.apache.org/jira/browse/CASSANDRA-6018https://issues.apache.org/jira/browse/CASSANDRA-11040

https://issues.apache.org/jira/browse/CASSANDRA-6018


2. Encryption on the wire

Because:It is really easy to attack an un-protected cluster

It takes a single Message to insert an admin account

into the system table

-Dcassandra.write_survey=true

How to steal writes in real time:

The fix is straight forward:

node to node encryption and SSL client certificate authentication to cluster traffic

Awwwwww.



Awwwwww.



Bonus: can be done

with NO downtime!!!

Awwwwww.



Bonus: can be done

with NO downtime!!!

How-to guide: http://thelastpickle.com/blog/2015/09/30/hardening-cassandra-step-by-step-part-1-server-to-

server.html

http://thelastpickle.com/blog/2015/09/30/hardening-cassandra-step-by-step-part-1-server-to-server.html

When you are done it should look like:

Things to note:

Use "dc" or "rack" to limit encryption to connections between racks and data centers

Thanks for that!!

Huzzah!

(But AES on modern hardware will not be a bottleneck)

Things to note:

Keystore and key password must match (artifact of JDK X.509 Impl complexity)

Things to note:

256 bit means export restrictions (requires JCE provider JAR)

http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html

http://docs.oracle.com/javase/8/docs/technotes/guides/security/SunProviders.html#importlimits

http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html

http://docs.oracle.com/javase/8/docs/technotes/guides/security/SunProviders.html#importlimits

Don't forget this part or else...

Things to note:

Hahaha! Now I'm hacking you over SSL.

*Still* vulnerable AND you can't see what the attacker is doing.

Client to Server SSL

Client to Server SSL(see slides 30 to 35)

Client to Server SSL(see slides 30 to 35)

Now with NO downtime!!!

https://issues.apache.org/jira/browse/CASSANDRA-10559Available in: 2.1.12, 2.2.4, 3.0.0


Need to Debug SSL?

-Djavax.net.debug=ssl

http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/ReadDebug.html

http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/ReadDebug.html

Certs are hard :(

Netflix Lemur:x.509 Certificate Orchestration Framework

http://techblog.netflix.com/2015/09/introducing-lemur.htmlhttps://github.com/Netflix/lemur

http://techblog.netflix.com/2015/09/introducing-lemur.html

https://github.com/Netflix/lemur

Certs are hard :(

Hashicorp Vault"secures, stores, and tightly controls access to tokens, passwords, certificates, API keys, and other secrets in modern computing. "

https://www.vaultproject.io/

https://www.vaultproject.io/

2. Encryption on the wire

But wait! There's more!

The internode authentication API:BYO identity verification

Looks like this:

3. Authentication and Authorization

Best practices should not be new to you.

user segmentation schema access limitation etc.

(Everything we did with an RDBMS)





(Everything we did with an RDBMS)

New in 2.2:

Role-based access control!

An Example

An Example

buzzword compliant!

An Example

Turning it all on

authenticator: PasswordAuthenticator

Tip: keep your read-only cqlsh credentials in $HOME/.cassandra/cqlshrc

of the system's admin account

Turning it all on

authorizer: CassandraAuthorizer

Turning it all on

role_manager: CassandraRoleManager

Turning it all on

authorizer: CassandraAuthorizerauthenticator: PasswordAuthenticatorrole_manager: CassandraRoleManager

WARNING:

potential downtime!


Turning it all on

WARNING:

potential downtime!WARNING:

stupid defaults


Turning it all on

WARNING:


stupid defaults

TIP: turn these WAY UP: permissions_validity_in_ms roles_validity_in_ms

Also: use permissions_update_interval_in_ms for async refresh if needed


Turning it all on

WARNING:


stupid defaults

NEW in 3.4:credentials_validity_in_ms*

* https://issues.apache.org/jira/browse/CASSANDRA-7715


Turning it all on

authorizer: TransitionalAuthorizerauthenticator: TransitionalAuthenticator

DSE plugins to avoid downtime

Turning it all on

system.schema_keyspace system.schema_columns system.schema_columnfamilies system.local system.peers

These tables have default read permissions for every authenticated user:

Turning it all on

IMPORTANT cassandra.yaml line note:

"Please increase system_auth keyspace replication factor if you use this..."

Tip: replication factor for the system_auth keyspace should be the same as the number

of nodes in the data center

Turning it all on

IMPORTANT cassandra.yaml line note:

"Please increase system_auth keyspace replication factor if you use this..."

Tip: replication factor for the system_auth keyspace should be the same as the number

of nodes in the data center

WARNING:

stupid defaults*

*https://issues.apache.org/jira/browse/CASSANDRA-11340


4. Management and tooling

Securing JMX

nmap -Pn -p7199 \-oG logs/cass.gnmap 54.88.0.0/14

Always a few suckers that TL,DR'ed

Why do I need to secure JMX?

Works as Advertised!

also good for

some LOLs

Securing JMX

SSL setup is like node to node and client to server

http://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html

Securing JMX

JMX Authentication is straightforward and well documented

$JAVA_HOME/jre/lib/management/jmxremote.access$JAVA_HOME/jre/lib/management/jmxremote.password.template


Securing JMX


Now you can:

nodetool -u admin -pw secret compactionstats



Securing JMX


Now you can:


Tip: -pwf option will read the password from a file



Securing JMX


Now you can:



THIS JUST IN!!!

RBAC for JMX Authentication and Authorization



Thanks!@zznate

Hardening cassandra q2_2016

Technology

Transcript of Hardening cassandra q2_2016