Multitenant JVMwiki.jvmlangsummit.com/images/8/80/...Multitenancy... · Multitenancy is contrasted...

© 2010 IBM Corporation

Multitenant JVM

Ryan Sciampacone – Senior Software Developer

August 1st 2012

JVM Languages Summit2012

© 2010 IBM Corporation2

Important Disclaimers

§ THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONALPURPOSES ONLY.

§ WHILST EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THEINFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUTWARRANTY OF ANY KIND, EXPRESS OR IMPLIED.

§ ALL PERFORMANCE DATA INCLUDED IN THIS PRESENTATION HAVE BEEN GATHERED IN ACONTROLLED ENVIRONMENT. YOUR OWN TEST RESULTS MAY VARY BASED ON HARDWARE,SOFTWARE OR INFRASTRUCTURE DIFFERENCES.

§ ALL DATA INCLUDED IN THIS PRESENTATION ARE MEANT TO BE USED ONLY AS A GUIDE.

§ IN ADDITION, THE INFORMATION CONTAINED IN THIS PRESENTATION IS BASED ON IBM’SCURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM,WITHOUT NOTICE.

§ IBM AND ITS AFFILIATED COMPANIES SHALL NOT BE RESPONSIBLE FOR ANY DAMAGESARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANYOTHER DOCUMENTATION.

§ NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECTOF:

§ - CREATING ANY WARRANT OR REPRESENTATION FROM IBM, ITS AFFILIATED COMPANIES ORITS OR THEIR SUPPLIERS AND/OR LICENSORS


What’s this talk all about anyways?

Look at the challenges and technical details involved in fitting amultitenancy story onto the Java runtime. This will include goals, basicbuilding blocks, and how existing problems are changed or magnified byhaving multiple tenants.


Dude, what the …?!

This is JVMLS!


Multitenancy == Simplification

Multitenancy refers to a principle in software architecture where a single instance ofthe software runs on a server, serving multiple client organizations (tenants).

Multitenancy is contrasted with a multi-instance architecture where separate softwareinstances (or hardware systems) are set up for different client organizations.

With a multitenant architecture, a software application is designed to virtually partition itsdata and configuration, and each client organization works with a customized virtualapplication instance.

Thanks to


Boiling Multitenancy Down

§ Working Definition– A single instance of a software application that serves multiple customers. Each

customer is a tenant.– Generally tenants can customize some parts of the application (look & feel, security,

business rules) but not the code.– Infrastructure usually opaqueà opportunity for provider

Why? Cost Savings: As compared to single-tenant deployment model


Multitenancy : Trading Isolation for Density

Hardware

OS Images

Middleware

Application

Hardware

OS Images

Middleware

Application

Hardware

OS Images

Middleware

Application

OS Images

Middleware

Application

Hardware

OS Image

Middleware

Application

Middleware

Application

Hardware

OS Image

Middleware (e.g. WAS)

Application Application

Share-nothing(maximum isolation)

Shared hardware Shared OS Shared Process

Hardware

OS Image


Application

Tenant Tenant

Share-everything(maximum sharing)

Isolation

Density

Isolate API

‘Mission critical’apps

‘free’ apps

1+ GB / tenant 100’s MB / tenant1+ GB / tenant 10’s MB / tenant 10’s KB / tenant


Challenge: Lowering the Cost of Entry

§ We need to fix the following– Data Isolation between applications– Control over resource hogs

§ Without forcing people to change their applications! ApplicationChanges


Multitenant JDK: Easy isolation and control

§ Concept: Add a single argument (–Xmt for multi-tenant) to your Javacommand-line to opt into sharing a runtime with others.

§ Result: Your application behaves exactly as it if had a dedicated JVM, but inreality it runs side-by-side with other applications.

§ Benefits: Smaller, faster, and eventually smarter– Less duplication (1 GC, 1 JIT), Heap object sharing– JVM is already running and warm when starting apps


Multitenant JDK: Launch your application

§ Opt-in to multitenancy by adding –Xmt


Multitenant JDK: Register with javad daemon

§ JVM will locate/start daemon automatically

javad

locate


Multitenant JDK: Create a new tenant

§ New tenant created inside the javad daemon

javad

Tenant1


Multitenant JDK: Create a second tenant

§ New tenant created inside the javad daemon

javad

Tenant1

Tenant2

One copy of common codelives in the javad process.

Most runtime structuresare shared.


Middleware == Master Tenant

§ Shared data structures are part of the “global economy”– Growing a shared Hashtable

§ The middleware effectively acts as a tenant– Looser restrictions (?)– Still tries not to interfere with everyone else

§ Consistency in ownership– An object is owned by a Tenant– Private resources may percolate to the global level

Tenant1

Tenant2

Master Tenant


You need an Identifier: TenantContext

§ Basic operations on Tenants available to the middleware– Data Isolation– Resource Management

§ Ability for the middleware to differentiate between Tenants– Which one is causing the problem?

§ Querying the state of Tenants– How much free memory do you have?

Data IsolationData IsolationData Isolation

Create & QueryCreate & QueryCreate & Query

Resource ManagementResource ManagementResource Management


Data Isolation Challenges

§ Static variables are bad (for sharing)

§ Most libraries are full of static variables

Wait! What happens ifeach tenant needs adifferent default locale?


Isolating Statics Through Class Loaders

§ Isolation through the usual class loader tricks– Every time you load a tenant, stick it in a new class loader– Statics are now no longer shared!

ClassLoader

ClassAClass

AClassA

Tenant1

ClassLoader

ClassAClass

AClassA

Tenant2

ClassLoader

ClassAClass

AClassX

Master Tenant




ClassLoader

ClassAClass

AClassA

Tenant1

ClassLoader

ClassAClass

AClassA

Tenant2

ClassLoader

ClassAClass

AClassX

Master Tenant

§ Plenty of duplication!– Class data– JITted code– Duplicated static variables, most of which are likely identical




ClassLoader

ClassAClass

AClassA

Tenant1

ClassLoader

ClassAClass

AClassA

Tenant2

ClassLoader

ClassAClass

AClassX

Master Tenant

§ Per-Tenant “scoping” of master tenant data!


Isolating Statics through Bytecode Re-writing

§ Let’s just focus on the statics– Rewrite the bytecodes!

§ Use an data structure for each static

§ Use property access style to get at statics

§ Still error prone: Class load rules, Reflection, JNI, Other bytecode rewriters

§ Ok, so you need to start marking things as sharable (default) vs. isolated (new)

…

<Tenant1 value>

Tenant1 Contextstatic index


Solving the Data Isolation Challenge

§ What if … the JVM knew about tenants and provided each one with a different view ofstatic variables?

§ Meet the @TenantScope annotation.

§ @TenantScope Semantics: Static variable values are stored per-tenant

§ Each tenant has their own LocaleSettings.defaultLocale

§ Now many tenants can share a single LocaleSettings class

…LocaleSettings.setDefaultLocale(LocaleSettings.UK );

…

…LocaleSettings.setDefaultLocale(LocaleSettings.UK );

…

Tenant1

Tenant2

…LocaleSettings.setDefaultLocale(LocaleSettings.USA );

…

…LocaleSettings.setDefaultLocale(LocaleSettings.USA );

…


Multitenant JDK: Sharing without Changing

§ @TenantScope markup gets added automatically as classes are loaded

§ Tenants see dedicated middleware – but behind the curtains all classes are actuallyshared

merge

Hardware

OS Image


Application Application

Hardware

OS Image


Application

Tenant Tenant

Isolate API

ApplicationChanges


Per-Tenant Heap Consumption

§ IBM JDK’s have new region-based GC technology which maps nicely to tenants(more @ http://ibm.co/JtWfXr)

§ Technique:– Each tenant is initially given enough GC regions to satisfy its minimum reservation– Code running in tenant scope allocates objects in a region it owns– New regions can be requested up to tenant maximum reservation

§ Details:– Finalization needs to run in the proper tenant context– We must be able to map from an objectà tenant easily– GC read/write barriers provide an opportunity to control inter-tenant references

221111

heap (divided into regions)

Tenant1 Tenant2


Hey! We’re done!

Right?

Image source: www.wallpaperslibrary.com


Neighbourhood Watch: Dealing with bad behaviour

images from http://www.rra.memberlodge.org/Neighbourhood-Watch-Reportinghttp://mcsholding.com/DetailsPage.aspx?Page_Id=42

http://bit.ly/ficwkl


Image source: http://bit.ly/ND07gz


Java is Memory: That might be a problem…

§ How do we measure each tenants memory consumption?

§ Shared memory across Tenants– Jar files (R/O class data)

§ Levels of contribution to global data– Shared data structures (add/remove/grow)

§ Implicit vs. Explicit creation of data– String constants– Growth of data structures

§ Querying of System Information– Where does java.lang.management fit in? System.freeMemory() ?

§ Configuration– Quota’s governing maximums, levels of contribution– Simple question: How do you handle a tenant deployment with –Xmx / -Xms options?

§ Associated CPU (and other non memory) costs?– Garbage Collection– Finalization


Dealing with Bad Behavior

§ Scenario: Application needs to be taken down quickly– Bugs – Misbehavior, runaway consumption of resources

§ Operating Systems / Hypervisors great at this– kill -9, ctrl-c, task manager and kill– Shutdown guest O/S

§ JVM not so much– Thread.stop() is a non-starter– Implement TM semantics on all data structures?

§ Worse Case Scenario: 1 Tenant crashes the JVMà All Tenants are torn down

§ Without co-operation, tear down of a single Tenant is a challenge

§ Throttling : Consider slowing down / restricting Tenants with bad behavior

§ The closer your neighbours are the better your controls must be


JSR-284 Resource Consumption Mgmt API

§ Throttling at Java layer for portability

§ Or, leveraging OS WLM directly for efficiency (Linux & AIX)

Network: XXXKB/S

Hardwareresources

CPU: XXX GHZ DISK: XXXKB/SMemory: XXX GB

OS Level Resources Management

Thread

SocketHandler

OS Workload Manager (WLM)

OS resources

JVM Resource Management

Resource Throttle Layer

JSR 284 API

GC(Heap Mgmt)

Tenant Tenant Tenant Tenant Tenant

OS

JVM

Resource native API

Memory CPU Thread File I/O Socket I/O CPU File I/O Socket I/O


JVM vs. Operating System CPU Throttling

Benchmark Setting

§ Accuracy Comparison: Run two CPU-intensive apps each doing the same Fibonaccicalculation, but with different CPU quota: 60% vs 30%, benchmark the accuracy

§ Note: Duration in both OS and JVM controlled environments were roughly the same

Result: JVM control achieves comparable performance, but less accuracy.


Shared State Outside the JVM

§ Native libraries contain state that may not be shareable across tenants

§ Could rewrite the libraries to support tenants– Exactly what we are trying to avoid– May not even be possible (don’t have the source)

§ Remember java –mt –jar one.jar ?

Tenant1

Tenant2

JVM

SharedLibrary

SharedState!


Shared State Outside the JVM

§ Use separate processes to manage different state

§ Each process now holds the context– Cheap relative to JVM cost– Limits on OS capabilities (1000’s of processes)

§ Interesting possibilities– Base class library support– Cross platform support

§ Challenges– Latency

Tenant1

Tenant2

JVM

ProxyLibrary

SharedLibrary

SharedLibrary


Locking Interactions Between Tenants

§ Collisions between tenants on “shared” objects– Java level: Global structures which require synchronized contribution– VM level: String constant table

§ Approaches to locking– Is spinning appropriate anymore?– Tenant Local Locking vs. Global Locking

§ Atomic field access appropriate for 1 tenant but not for many tenants– Code was only expected to be run by 1 or 2 threads– Should be a shared resource?

§ Need to potentially consider different implementations for existing data structures– Starting using concurrent equivalents of known data structures (e.g., Hash tables)

§ Potential for deadlock between different tenants – Debug story is required


Debugging a Tenant

§ Connecting the debugger to a single tenant– Is this something we want to allow?

§ How do you handle stepping through code using shared data?– Master tenant is using a single data structure for all tenant values?– Have to show some opaque value perhaps when stepping?

§ What if you “breakpoint” in the middle of holding a shared monitor?– Need to be prepared for all tenants to “stop”

§ It may well be that if you are supporting debugging in this environment, you must trust things– The debugger connecting is trusted– So don’t allow debugging on shared / super high density deployed systems

§ How do you support other things like Profiling / Monitoring tools?– Profile and you likely need to be on a trusted system– But still need support for safe tooling for “production” systems


Keeping Things Separated: Post Mortem Problem Determination

§ Scenarios– Your VM crashes and you are left with a core file / system dump– You get a heap dump for object liveness / connectivity

§ How do you debug this?– Single tenant view point required, dealing with 1000’s of other tenant data

§ Becomes a runtime / middleware issue– Dump ONLY that data associated with your tenant– What about global shared state?– Core file (etc) rewriters?– Proxy process to help with custom native data?

§ Challenge: Core files are still an issue– “Scrubbing” of details from heap– What about native artifacts? Threads / shared libraries?

§ Tenant space identity within the software stack (finding / separating tenants quickly)– Debugging middleware issues related to tenants– Tenant is causing DB issues – how do you identify the DB transactions to a particular tenant?


Conclusions

§ Think deeper and have the checks and support in the runtime

§ Want to build strong walls– Today lots of applications share common app services– We’re talking about introducing language features for compile time walls– Java EE exploring options

§ May cause us to ask more of the OS and HV– Not necessarily tighter dependencies as much as better APIs for collaboration

§ There are big wins to be found!

§ Barely scratching the surface here

Multitenant JVMwiki.jvmlangsummit.com/images/8/80/...Multitenancy... · Multitenancy is contrasted...

Documents

Transcript of Multitenant JVMwiki.jvmlangsummit.com/images/8/80/...Multitenancy... · Multitenancy is contrasted...