An overview of InfiniteGraph,the distributed graph database.
Darren WoodChief Architect, InfiniteGraph
For today’s discussion…
• InfiniteGraph– Background / History– Architecture– What we are working on
• Consistency Models• Challenges/Requirements of Distibution• Data model – Schema vs Schema-less
Copyright © InfiniteGraph
Building on Objectivity/DB
• Derived from Objectivity/DB core– Distributed Object Database– Native core (broad platform support) with C++,
Java, C# and Python Bindings– Thousands of deployments, many are 24x7x365– Markets : VLDB, Data Fusion / Metadata, Complex
Object Models, Relationship Analytics
Copyright © InfiniteGraph
Relationship Analytics Experience
• Relationship Analytics– Large part of Objy/DB Government business– Significant growth in the commercial space– Many OEM opportunities– Mostly based on graph theory– Hand coded algorithms
Copyright © InfiniteGraph
Needed something more…
• Objectivity Provides– Powerful distributed object storage– Flexible class based persistence– Enterprise DB features
• But….– Lots of custom coding– Complex deployment– No generic algorithm framework
Copyright © InfiniteGraph
Enter InfiniteGraph
• Simple Graph focused API• Automated distribution and deployment• Mostly configuration driven• Java class based persistence• Property model support• Asynchronous navigation• Indexing framework
Copyright © InfiniteGraph
Some code…
Copyright © InfiniteGraph
Vertex alice = myGraph.addVertex(new Person(“Alice”)); Vertex bob = myGraph.addVertex(new Person(“Bob”)); Vertex carlos = myGraph.addVertex(new Person(“Carlos”)); Vertex charlie = myGraph.addVertex(new Person(“Charlie”));
alice.addEdge(new Meeting(“Denver”, “5-27-10”), bob);bob.addEdge(new Call(timestamp), carlos);carlos.addEdge(new Payment(100000.00), charlie);bob.addEdge(new Call(timestamp), charlie);
Alice Carlos CharlieBobMeets Calls Pays
Calls
Basic Architecture
Copyright © InfiniteGraph
IG Core/API
ConfigurationNavigation Execution
Management Extensions
BlueprintsUser Apps
Objectivity/DB Distributed Database
Session / TX ManagementPlacement
Copyright © InfiniteGraph
Targeting Large Graphs
• Graphs grow quickly– Billions of phone calls / day in US– Emails, social media events, IP Traffic– Financial transactions
• Some analytics require navigation of large sections of the graph
• Must distribute data and go parallel
Copyright © InfiniteGraph
Consistency Models
• Trading off full consistency for performance• Relax locking, allow stale reads• Gain 100x edge ingest rates• Great for “social graph” applications• These tend to have relaxed consistency
requirements• API allows choice per operation
Copyright © InfiniteGraph
Pipelining
Copyright © InfiniteGraph
IG Core/API
ConfigurationNavigation Execution
Management Extensions
Session / TX Management
Placement(MDP)
Placement(Pipelining)
V1V1
V2V2
V3V3
E12E12
E23E23
Pipeline
Manager
Sta
ging
Con
tain
ers P
ipeline Containers
E(1->2)
E(3->1)
E(2->3)
E(2->1)
E(2->3)E(3->1)
E(1->2)
E(3->2)
E(1->2)
E(2->3)
E(3->1)
E(2->1)
E(2->3)
E(3->1)
E(3->2)
E(1->2)
Distributing Navigation
• Graph algorithms naturally branch• Breaking up the process is relatively simple• Orchestrating it is more challenging
Copyright © InfiniteGraph
Alice
Carlos CharlieBobMeets Calls Pays
Dave EveChuckCallsLives With
Meets
Distributed API
Application(s)
Partition 1 Partition 3Partition 2 Partition ...n
Processor Processor Processor Processor
Partitioned Graphs are Ugly
Copyright © InfiniteGraph
Making it scale…
• A distributed data tier supports parallel IO• A distributed cache/processor tier for
processing “navlets”• Ability to deal with remote data reads (fast)• Navlets hit processor/cache instances based
on consistent hashing of their start vertex
Copyright © InfiniteGraph
Flexible Data Models
• Schema isn’t for everyone– Great for relatively static models– Support for inheritance helps– So does auto-evolution
• Looking to support document style elements– Support for ad-hoc attributes– Ability to mix both approaches
Copyright © InfiniteGraph
Other Projects
• Blueprints / Gremlin support• Native Visualizer and Management Console• Built in Qualifiers• More Indexing Enhancements• Algorithms• Large scale samples
Copyright © InfiniteGraph
Top Related