User & Device Identity For Microservices @ Netflix Scale · User & Device Identity for...
Transcript of User & Device Identity For Microservices @ Netflix Scale · User & Device Identity for...
User & Device Identity For Microservices @ Netflix Scale
Satyajit ThadeshwarQCon San Francisco 2019
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Logged out?#$%&!
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Logged out?#$%&!
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Time
Core Streaming Metric
Current Last Week
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Satyajit ThadeshwarProduct Edge Access [email protected]
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Complicated
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
9 teams
57 watchers
Netflix subscribers and the devices that they use
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Where we were
What we did
Wins
Where we were
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Zuul
EDGE
Email: [email protected]: ********ESN: LGTV20165-193456G568
User Login
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Zuul API
EDGE ORIGIN
Email: [email protected]: ********ESN: LGTV20165-193456G568 /login
User Login
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Zuul API
Netflix Microservices
auth service
EDGE ORIGIN MID-TIER SERVICES
Email: [email protected]: ********ESN: LGTV20165-193456G568 /login
success
User Login
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Zuul API
Netflix Microservices
auth service
EDGE ORIGIN MID-TIER SERVICES
Email: [email protected]: ********ESN: LGTV20165-193456G568 /login
success
User Login
customerId: 10192378ESN: LGTV20165-193456G568Expires: In 8 hours
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Zuul API
Netflix Microservices
auth service
EDGE ORIGIN MID-TIER SERVICES
Email: [email protected]: ********ESN: LGTV20165-193456G568 /login
successSet-Cookie
User Login
customerId: 10192378ESN: LGTV20165-193456G568Expires: In 8 hours
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Zuul
EDGE
Authenticate Request
/browse
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Zuul API
EDGE ORIGIN
/browse
Authenticate Request
/browse
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Zuul API
EDGE ORIGIN
/browse
Authenticate Request
success
KEY MANAGEMENTSERVICE
/browse
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Zuul API
Netflix Microservices
EDGE ORIGIN
/browse
Authenticate Request
success
MID-TIER SERVICES
customerId: 10192378ESN: LGTV20165-193456G568
KEY MANAGEMENTSERVICE
/browse
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Zuul API
Netflix Microservices
EDGE ORIGIN
/browse
Authenticate Request
success
MID-TIER SERVICES
customerId: 10192378ESN: LGTV20165-193456G568
KEY MANAGEMENTSERVICE
/browse
More than one service consuming cookies
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Zuul
API
DeviceAuth
Service
Legacy API
Netflix Microservices
SIGNUPFLOW
SERVICE
subscriber
authservice
lolomo / Search
DRM
Other services
EDGE ORIGINS MID-TIER SERVICES
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Zuul
API
DeviceAuth
Service
Legacy API
Netflix Microservices
SIGNUPFLOW
SERVICE
subscriber
authservice
lolomo / Search
DRM
Other services
EDGE ORIGINS MID-TIER SERVICES/ios/android/atv...
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Zuul
API
DeviceAuth
Service
Legacy API
Netflix Microservices
SIGNUPFLOW
SERVICE
subscriber
authservice
lolomo / Search
DRM
Other services
EDGE ORIGINS MID-TIER SERVICES
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Zuul
API
DeviceAuth
Service
Legacy API
Netflix Microservices
SIGNUPFLOW
SERVICE
subscriber
authservice
lolomo / Search
DRM
Other services
EDGE ORIGINS MID-TIER SERVICES
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Zuul
API
DeviceAuth
Service
Legacy API
Netflix Microservices
SIGNUPFLOW
SERVICE
subscriber
authservice
lolomo / Search
DRM
Other services
EDGE ORIGINS MID-TIER SERVICES
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Zuul
API
DeviceAuth
Service
Legacy API
Netflix Microservices
SIGNUPFLOW
SERVICE
subscriber
authservice
lolomo / Search
DRM
Other services
EDGE ORIGINS MID-TIER SERVICES
At massive scale
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Netflix
158M+ subscribers
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Netflix
158M+ subscribers
1B+ devices
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Netflix
158M+ subscribers
1B+ devices
2M peak RPS
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Authenticate Request / Extract Identity
API
ORIGIN
KEY MANAGEMENTSERVICE = 2 million Requests Per Second
More than one token type
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Cookies
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Cookies
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
- Signup
Cookies
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
- Signup
- Login
Cookies
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
- Signup
- Login
- Discovery
MSL Tokens
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
- Device authentication
- Encryption
Message Security Layer (MSL)https://www.infoq.com/news/2014/11/netflix-msl/
MSL Tokens
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
- License
- Playback
CTicket
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
- Legacy devices
Partner Tokens
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
- JWS, JWE
- Non-member experiences
- Signup- Sign-in- Discovery
- License- Playback
- Legacy devices
- Non-member experience
Cookies
MSL Tokens
CTicket
Partner Tokens (JWS, JWE)
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Zuul
API
DeviceAuth
Service
LegacyAPI
Netflix Microservices
SIGNUPFLOW
SERVICE
subscriber
authservice
lolomo / Search
DRM
Other services
EDGE ORIGINS MID-TIER SERVICES
- Multiple services consuming auth tokens
- Multiple types of auth tokens
- Massive scale
- Inefficient, insecure & complicated
Where we were
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Zuul
API
DeviceAuth
Service
LegacyAPI
Netflix Microservices
SIGNUPFLOW
SERVICE
subscriber
authservice
lolomo / Search
DRM
Other services
EDGE ORIGINS MID-TIER SERVICES
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Zuul
API
DeviceAuth
Service
Legacy API
Netflix Microservices
SIGNUPFLOW
SERVICE
subscriber
authservice
EDGE ORIGINS MID-TIER SERVICES
NodeJSServices
Lolomo / Search
DRM
Other services
DiscoveryAPI
PlaybackAPI
What we didUser & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Moved authentication to the edge
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Zuul
API
Device Auth
Service
Legacy API
Netflix Microservices
SIGNUPFLOW
SERVICE
subscriber
authservice
EDGE ORIGINS MID-TIER SERVICES
NodeJSServices
Lolomo / Search
DRM
Other services
DiscoveryAPI
PlaybackAPI
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Zuul
API
Device Auth
Service
Legacy API
Netflix Microservices
SIGNUPFLOW
SERVICE
subscriber
authservice
EDGE ORIGINS MID-TIER SERVICES
NodeJSServices
Lolomo / Search
DRM
Other services
DiscoveryAPI
PlaybackAPICookie
ServiceMSL
ServicePartner Service
EAS
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Zuul
API
Device Auth
Service
Legacy API
Netflix Microservices
SIGNUPFLOW
SERVICE
subscriber
authservice
EDGE ORIGINS MID-TIER SERVICES
NodeJSServices
Lolomo / Search
DRM
Other services
DiscoveryAPI
PlaybackAPICookie
ServiceMSL
ServicePartner Service
EAS
EDGE AUTHENTICATION SERVICES
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Zuul
EDGE
EAS
renewal / device auth / key exchange
Cookie Service
MSL Service
Partner Service
valid and not expired
95%
5%
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Zuul
EDGE
Cookie Service
EAS
valid but expired
renewal call
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Zuul
EDGE
Cookie Service
EAS
valid but expired
renewal call failed
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Zuul
EDGE
Cookie Service
EAS
valid but expired
renewal call rescheduled
resolved identity
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Zuul
EDGE
Cookie Service
EAS
valid but expired
renewal call rescheduled
rescheduled cookie
resolved identity
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Zuul
API
Device Auth
Service
Legacy API
Netflix Microservices
SIGNUPFLOW
SERVICE
subscriber
authservice
EDGE ORIGINS MID-TIER SERVICES
NodeJSServices
Lolomo / Search
DRM
Other services
DiscoveryAPI
PlaybackAPICookie
ServiceMSL
ServicePartner Service
EAS
EDGE AUTHENTICATION SERVICES
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Zuul
API
Device Auth
Service
Legacy API
Netflix Microservices
SIGNUPFLOW
SERVICE
subscriber
authservice
EDGE ORIGINS MID-TIER SERVICES
NodeJSServices
Lolomo / Search
DRM
Other services
DiscoveryAPI
PlaybackAPICookie
ServiceMSL
ServicePartner Service
EAS
EDGE AUTHENTICATION SERVICES
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Passport
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Passport
- Identity structure created at the edge for each request
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Passport
- Identity structure created at the edge for each request
- Contains user & device identity
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Passport
- Identity structure created at the edge for each request
- Contains user & device identity
- Internal to Netflix ecosystem
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Passport
- Identity structure created at the edge for each request
- Contains user & device identity
- Internal to Netflix ecosystem
- Integrity protected by HMAC
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Passport
- Identity structure created at the edge for each request
- Contains user & device identity
- Internal to Netflix ecosystem
- Integrity protected by HMAC
- Protobuf format
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Passportmessage Passport {
Header header = 1; UserInfo user_info = 2; DeviceInfo device_info = 3; Integrity user_integrity = 4; Integrity device_integrity = 5;
}
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Passportmessage Passport {
Header header = 1; UserInfo user_info = 2; DeviceInfo device_info = 3; Integrity user_integrity = 4; Integrity device_integrity = 5;
}
message Header { string originator = 1;}
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Passportmessage Passport {
Header header = 1; UserInfo user_info = 2; DeviceInfo device_info = 3; Integrity user_integrity = 4; Integrity device_integrity = 5;
}
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Passportmessage Passport {
Header header = 1; UserInfo user_info = 2; DeviceInfo device_info = 3; Integrity user_integrity = 4; Integrity device_integrity = 5;
}
message UserInfo { Source source = 1;
AuthenticationLevel auth_level = 2; Int64Wrapper customer_id = 3; Int64Wrapper account_owner_id = 4; repeated UserAction actions = ;
}
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Passportmessage Passport {
Header header = 1; UserInfo user_info = 2; DeviceInfo device_info = 3; Integrity user_integrity = 4; Integrity device_integrity = 5;
}
message DeviceInfo { Source source = 1;
AuthenticationLevel auth_level = 2; StringValue esn = 3; Int32Value device_type = 4; repeated DeviceAction actions = 5;
}
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Passportmessage UserInfo { Source source = 1;
AuthenticationLevel auth_level = 2;}
message DeviceInfo { Source source = 1;
AuthenticationLevel auth_level = 2;}
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Passportmessage UserInfo { Source source = 1;
AuthenticationLevel auth_level = 2;}
message DeviceInfo { Source source = 1;
AuthenticationLevel auth_level = 2;} enum Source {
COOKIE = 1; MSL = 2; PARTNER_TOKEN = 3; CTICKET = 4;
}
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Passportmessage UserInfo { Source source = 1;
AuthenticationLevel auth_level = 2;}
message DeviceInfo { Source source = 1;
AuthenticationLevel auth_level = 2;} enum AuthenticationLevel {
LOW = 1; // untrusted transport HIGH = 2; // secure tokens over TLS HIGHEST = 3; // MSL or user credentials
}
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Passportmessage Passport {
Header header = 1; UserInfo user_info = 2; DeviceInfo device_info = 3; Integrity user_integrity = 4; Integrity device_integrity = 5;
}
message Integrity { string key_name = 1;
bytes hmac = 2;}
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Passport Introspector
- Wrapper over passport binary data
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Passport Introspector
- Wrapper over passport binary data
public interface PassportIntrospector { Long getCustomerId(); Long getAccountOwnerId(); String getEsn(); String getPassportAsString(); ...}
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Passport Introspector
- Wrapper over passport binary data
public interface PassportIntrospector { Long getCustomerId(); Long getAccountOwnerId(); String getEsn(); String getPassportAsString(); ...}
- Consumers create passportIntrospector from binary passport data
factory.createIntrospector(passport);
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Tooling
Self-service tool for teams to decrypt passport
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Passport Actions
message UserInfo { repeated UserAction actions = 6;
...}
message DeviceInfo { repeated DeviceAction actions = 5;
...}
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Passport Actions
message UserInfo { repeated UserAction actions = 6;
...}
message DeviceInfo { repeated DeviceAction actions = 5;
...}
- Explicit signal sent by the downstream services, when an update to user or device identity has been performed
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Passport Actions
message UserInfo { repeated UserAction actions = 6;
...}
message DeviceInfo { repeated DeviceAction actions = 5;
...}
- Explicit signal sent by the downstream services, when an update to user or device identity has been performed
- This "signal" is used by EAS to either create or update the corresponding type of token
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Passport Action
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Passport Action: User Login
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Zuul
EDGE
Email: [email protected]: ********ESN: LGTV20165-193456G568
Passport Action: User Login
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Zuul API
EDGE ORIGIN
Email: [email protected]: ********ESN: LGTV20165-193456G568 /login
Passport Action: User Login
(Device Bound)
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Zuul API
Netflix Microservices
auth service
EDGE ORIGIN MID-TIER SERVICES
Email: [email protected]: ********ESN: LGTV20165-193456G568 /login
success
Passport Action: User Login
(Device Bound)
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Zuul API
Netflix Microservices
auth service
EDGE ORIGIN MID-TIER SERVICES
Email: [email protected]: ********ESN: LGTV20165-193456G568 /login
success
Passport Action: User Login
(Device Bound)
user loginuser login
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Zuul API
Netflix Microservices
auth service
EDGE ORIGIN MID-TIER SERVICES
Email: [email protected]: ********ESN: LGTV20165-193456G568 /login
successSet-Cookie
Passport Action: User Login
Cookie Service
(Device Bound)
user loginuser login
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Passport Action: Profile Switch
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Passport Action: Profile Switch
- Each profile has its own identity
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Passport Action: Profile Switch
- Each profile has its own identity
- Switched profile tokens sent back to the device
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Passport Actions
Separation Of Concerns
Increased Visibility
- Moved authentication to the edge
- Streamlined the identity resolution and mutation path
- Making consumption of user & device identity
- Efficient, secure & simple
What we did
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
WinsUser & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Token Agnostic Identity
Downstream systems don't have to worry about authentication concerns
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Simplified Authorization
Downstream services use authentication level for authorization decisions
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Simplified Authorization
Before:
long customerId = 2123125603L;String ESN = "NFXBOX-235F…";
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Extensible Identity Model
New attributes about user or device can be added
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Local cache for up to date subscriber data message UserInfo {
BytesValue subscriber_account ...}
Placeholder for local cache of subscriber data
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Offloaded &Fine Tuned
Offloaded token processing which resulted into significant gains for
- CPU- Request Latency - GC - Cluster Footprint
We were able to fine-tune EAS systems based on the token processing profile
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Offloaded & Fine Tuned
Offloaded token processing which resulted into significant gains for
- CPU- Request Latency - GC - Cluster Footprint
We were able to fine tune EAS systems based on the token processing profile
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Offloaded &Fine Tuned
- 30% reduction in CPU cost per request
- 40% reduction in load average
CPU to RPS ratio for API instance
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Offloaded &Fine Tuned
- 30% reduction in average latency
- 99th percentile latency dropping by 20%
Response time for API instance
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Offloaded &Fine Tuned
- Significant reduction in GC pressure and GC pause times
Stop the world GC for API cluster
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Increased Visibility
Increased visibility into identities flowing in and out of Netflix ecosystem
...and into the identity mutations happening in a request
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Developer Velocity
Greatly increased developer velocity for authentication related changes
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Team focused onsecurity
Separation of concerns among the teams
User & Device Identity for Microservices @ Netflix ScaleSatyajit Thadeshwar
Key Takeaways
- Token agnostic identity model
- Simplified authorization
- Extensible identity model
- Offloaded all the token processing from many systems
- Fine tuned individual microservices to suit the token processing profile
- Increased visibility into identities flowing and corresponding mutations
- Increased developer velocity for authentication & identity related changes
- Team focused on security
Thank You.
Satyajit [email protected]://www.linkedin.com/in/satyajit-thadeshwar