Using Spring with NoSQL databases (SpringOne China 2012)
-
Upload
chris-richardson -
Category
Technology
-
view
3.903 -
download
7
description
Transcript of Using Spring with NoSQL databases (SpringOne China 2012)
Using Spring with NoSQL databasesChris Richardson, Author of POJOs in Action, Founder of the original CloudFoundry.com @crichardson [email protected] http://plainoldobjects.com/
Presentation goal
NoSQL databases: what, why and how
How Spring Data simplifies the development of NoSQL applications
About Chris
(About Chris)
About Chris()
About Chris
About Chris
http://www.theregister.co.uk/2009/08/19/springsource_cloud_foundry/
vmc push About-Chris
Developer Advocate
Signup at http://cloudfoundry.com
Agenda
• Why NoSQL?
• Overview of NoSQL databases
• Introduction to Spring Data
• Using Spring Data for Redis
• Using Spring Data for Mongo
• Deploying on Cloud Foundry
Relational databases are great...
• SQL
• High-level
• Sorting
• Aggregation
• ACID semantics
• Well supported
• JDBC
• Hibernate/JPA
• Spring
• Well understood
• Developers
• Operators
... but they have limitations
• Object/relational impedance mismatch
• Complicated to map rich domain model to relational schema
• Difficult to handle semi-structured data, e.g. varying attributes
• Schema changes
• Extremely difficult/impossible to scale
• Poor performance for some use cases
Solution: Spend Money
http://upload.wikimedia.org/wikipedia/commons/e/e5/Rising_Sun_Yacht.JPG
OR
http://www.trekbikes.com/us/en/bikes/road/race_performance/madone_5_series/madone_5_2/#
• Hire more DevOps• Use application-level sharding• Build your own middleware• …
Solution: Use NoSQL
Benefits• Higher performance• Higher scalability• Richer data-model• Schema-less
Drawbacks• Limited transactions• Relaxed consistency• Unconstrained data
Growing in popularity…
But don’t get too excited
Solution: Use NewSQL• Relational databases with SQL and ACID transactions
AND• New and improved architecture - designed for modern hardware• Radically better scalability and performance
• NewSQL vendors: Gemfire/SQLFire, VoltDB, ...
Future = multi-paradigm data storage for enterprise applications
IEEE Software Sept/October 2010 - Debasish Ghosh / Twitter @debasishg
Agenda
• Why NoSQL?
• Overview of NoSQL databases
• Introduction to Spring Data
• Using Spring Data for Redis
• Using Spring Data for Mongo
• Deploying on Cloud Foundry
• Advanced key-value store
• Written in C
• Very fast, e.g. 100K reqs/sec
• Optional persistence
• Transactions with optimistic locking
• Master-slave replication
• Sharding using client-side consistent hashing
Redis
K1 V1
K2 V2
... ...
Using Redis (via CLI)redis 127.0.0.1:6379> set foo 1OKredis 127.0.0.1:6379> get foo"1"
Datatypes:•Strings•Hashes•Maps•Lists•Sets•Sorted sets
redis 127.0.0.1:6379> sadd myset a(integer) 1redis 127.0.0.1:6379> sadd myset b(integer) 1redis 127.0.0.1:6379> smembers myset 1) "a"2) "b"redis 127.0.0.1:6379> srem myset a(integer) 1redis 127.0.0.1:6379> smembers myset1) "b"
Redis use cases• Replacement for Memcached
• Session state
• Cache of data retrieved from system of record (SOR)
• Replica of SOR for queries needing high-performance
• Handling tasks that overload an RDBMS
• Hit counts - INCR
• Most recent N items - LPUSH and LTRIM
• Randomly selecting an item – SRANDMEMBER
• Queuing – Lists with LPOP, RPUSH, ….
• High score tables – Sorted sets and ZINCRBY
• …
MongoDB
• Document-oriented database
• JSON-style documents: objects, lists, primitives
• Schema-less
• Transaction = update of a single document
• Rich query language for dynamic queries
• Geospatial queries
• Grid FS provides file storage
• Very fast, asynchronous writes
• Highly scalable and available
Server
Database: Food To Go
Collection: Restaurants
Data model = Binary JSON documents
{ "name" : "TGI Fridays", "type" : ”American", "serviceArea" : [ "94619", "94618" ], "openingHours" : [ { "dayOfWeek" : "Wednesday", "open" : 1730, "close" : 2230 } ], "_id" : ObjectId("4bddc2f49d1505567c6220a0")}
Sequence of bytes on disk è fast i/o
MongoDB CLI> r = {name: 'Ajanta'}> db.restaurants.save(r)> r{ "_id" : ObjectId("4e555dd9646e338dca11710c"), "name" : "Ajanta" }> r = db.restaurants.findOne({name:"Ajanta"}){ "_id" : ObjectId("4e555dd9646e338dca11710c"), "name" : "Ajanta" }> r.type= "Indian”> db.restaurants.save(r)> db.restaurants.update({name:"Ajanta"}, {$set: {name:"Ajanta Restaurant"}, $push: { menuItems: {name: "Chicken Vindaloo"}}})> db.restaurants.find(){ "_id" : ObjectId("4e555dd9646e338dca11710c"), "menuItems" : [ { "name" :
"Chicken Vindaloo" } ], "name" : "Ajanta Restaurant", "type" : "Indian" }> db.restaurants.remove(r.id)
MongoDB query by example{ serviceArea:"94619", openingHours: { $elemMatch : { "dayOfWeek" : "Monday", "open": {$lte: 1800}, "close": {$gte: 1800} } }}
DBCursor cursor = collection.find(qbeObject); while (cursor.hasNext()) { DBObject o = cursor.next(); … }
Find a restaurant that serves the 94619 zip code and is open at 6pm on a Monday
Mongos
Scaling MongoDBShard 2
Mongod(replica)
Mongod(master)
Mongod(replica)
Shard 1Mongod(replica)
Mongod(master)
Mongod(replica)
Mongos
Client
Config Server
mongod
mongod
mongod
MongoDB use cases
• Use cases
• High volume writes
• Complex data
• Semi-structured data
• Who is using it?
• Shutterfly, Foursquare
• Bit.ly Intuit
• SourceForge, NY Times
• GILT Groupe, Evite,
• SugarCRM
Other NoSQL databases
http://nosql-database.org/ lists 122+ NoSQL databases
Type Examples
Extensible columns/Column-oriented HbaseSimpleDB, DynamoDBCassandra
Graph Neo4j
Key-value Voldemort, Riak
Document CouchDbSorry if
I left
out you
r favori
te
Agenda
• Why NoSQL?
• Overview of NoSQL databases
• Introduction to Spring Data
• Using Spring Data for Redis
• Using Spring Data for Mongo
• Deploying on Cloud Foundry
Spring Data is here to help
http://www.springsource.org/spring-data
NoSQL databases
For
Spring Data sub-projects
• Relational
• JPA
• JDBC Extensions
• NoSQL
• Redis
• Mongo
• HBase
• Neo4j
• Gemfire
• Lucene
• QueryDSL
• Big Data
• Hadoop
• HDFS and M/R
• Hive
• Pig
• Cascading
• Splunk
• Access
• REST
What you get
• Template classes that hide the boilerplate code
• Auto-generated (generic) repositories for some NOSQL databases
• Java ⇔ NoSQL mapping
• Cross Store Persistence
• Support in Roo
Get the book!
Agenda
• Why NoSQL?
• Overview of NoSQL databases
• Introduction to Spring Data
• Using Spring Data for Redis
• Using Spring Data for Mongo
• Deploying on Cloud Foundry
Redis challenges
• Connection management: need to get and reliably close connections
• Data mapping: application objects ⇔ Redis binary/strings
• Multiple client libraries with gratuitously different APIs
Spring Data for Redis
• Low-level - RedisConnection(Factory)
• Supports Jedis, Jredis, Rjc and Srp
• Insulates client code from underlying library
• High-level - RedisTemplate
• Builds on RedisConnection(Factory)
• Connection management
• Pluggable Java ⇔ binary conversion
• Support classes:
• Collections-backed by RedisTemplate
• Atomic Counters
• Support for Redis pub/sub
Low-level API = RedisConnection(Factory)
Using RedisConnectionFactorypublic class LowLevelRedisTest {
@Autowired private RedisConnectionFactory redisConnectionFactory;
@Test public void testLowLevel() { RedisConnection con = null; try { con = redisConnectionFactory.getConnection();
byte[] key = "foo".getBytes(); byte[] value = "bar".getBytes(); con.set(key, value);
byte[] retrievedValue = con.get(key);
Assert.assertArrayEquals(value, retrievedValue);
} finally { if (con != null) { con.close(); } } }
Ugly byte arrays L
Library independent code J
Need to clean up L
Configuring RedisConnectionFactory
@Configurationpublic class RedisConfiguration {
@Value("${databaseHostName}") protected String databaseHostName;
@Bean public RedisConnectionFactory jedisConnectionFactory() { JedisConnectionFactory factory = new JedisConnectionFactory(); factory.setHostName(databaseHostName); factory.setPort(6379); factory.setUsePool(true); return factory; }
}
High-level API = RedisTemplate
• Builds on RedisConnection(Factory)
• Analogous to JdbcTemplate
• Parameterized type
• K - Key type
• V – Value type
• Handles Java Key/Value ⇔ Redis byte[]
• Maps Redis exceptions ⇒
DataAccessException
• StringRedisTemplate
• Extends RedisTemplate<String, String>
• Keys and values are Strings
Using StringRedisTemplate
public class RedisTemplateTest {
@Autowired private StringRedisTemplate stringRedisTemplate;
@Test public void testGetAndSet() { stringRedisTemplate.opsForValue().set("foo", "bar"); assertEquals("bar", stringRedisTemplate.opsForValue().get("foo")); } @Test public void testHashOps() { stringRedisTemplate.opsForHash().put("myHash", "myKey", "value");
assertEquals("value", stringRedisTemplate.opsForHash().get("myHash", "myKey"));
assertEquals(Collections.singleton("myKey"), stringRedisTemplate.opsForHash().keys("myHash"));
assertEquals(Collections.singletonMap("myKey", "value"), stringRedisTemplate.opsForHash().entries("myHash")); }
Converts between Strings and byte[]
Returns KV type specific interface
Configuring StringRedisTemplate
@Configurationpublic class RedisConfiguration {
@Bean public RedisConnectionFactory jedisConnectionFactory() { … }
@Bean public StringRedisTemplate stringRedisTemplate(RedisConnectionFactory factory) { StringRedisTemplate template = new StringRedisTemplate(); template.setConnectionFactory(factory); return template; }}
RedisTemplate: Java objects ⇔ binary data
• DefaultSerializer - defaults to JdkSerializationRedisSerializer
• KeySerializer
• ValueSerializer
• HashKeySerializer
• HashValueSerializer
StringRedisTemplate uses StringRedisSerializer
• Stores keys and values as Strings
Register serializers to override the default behavior
Converted to JSON by RedisTemplate
Redis-backed Collections
@Test public void testRedisSet() {
Set<String> mySet = new DefaultRedisSet<String>("mySet", stringRedisTemplate);
Assert.assertTrue(mySet.isEmpty());
mySet.add("a");
assertEquals(Collections.singleton("a"), stringRedisTemplate.opsForSet().members("mySet"));
}
The key
Redis Atomic Counters
Redis Pub/Sub - Consumer
@Configurationpublic class RedisConfiguration {
@Bean public MyRedisSubscriber myListener() { return new MyRedisSubscriber(); } @Bean public RedisMessageListenerContainer redisMessageListenerContainer( RedisConnectionFactory redisConnectionFactory, MyRedisSubscriber myListener) { RedisMessageListenerContainer c = new RedisMessageListenerContainer();
c.setConnectionFactory(redisConnectionFactory); c.addMessageListener(new MessageListenerAdapter(myListener), new ChannelTopic("myChannel")); return c; }
public class MyRedisSubscriber { String message;
void handleMessage(String text) { this.message = text; } }
Redis Pub/Sub - Producer
public class RedisPubSubTest {
@Autowired private StringRedisTemplate stringRedisTemplate;
@Autowired private MyRedisSubscriber myListener; @Test public void testPubSub() throws InterruptedException { … stringRedisTemplate.convertAndSend("myChannel", "hello"); TimeUnit.SECONDS.sleep(4); Assert.assertEquals("hello", myListener.message); }}
Redis caching support
Template needs to (de)serialize K and VKVs = <prefix + K, V>
Sorted set of all keys for clear()
Agenda
• Why NoSQL?
• Overview of NoSQL databases
• Introduction to Spring Data
• Using Spring Data for Redis
• Using Spring Data for Mongo
• Deploying on Cloud Foundry
MongoDB API usage patterns
• Create and store Mongo singleton
• Externalized server host, port etc.
• Inserts/Updates
• Map application POJO ⇒ DBObject
• mongo.getDatabase(…).getCollection(…)
• Partial document updates
• Configure asynchronous vs. synchronous writes
• Queries
• Construct query object
• mongo.getDatabase(…).getCollection(…)
• Iterate through Cursor
• Map DBObject ⇒ application POJO
⇒ Higher-level than JDBC but still
repetitive, …
Spring Data - MongoDB
• MongoTemplate
• Query, Criteria, and Update DSLs
• Generic repositories
• Querydsl integration
• Cross-store persistence
• GeoSpatial integration
• Map-Reduce integration
• GridFS support
MongoTemplatedatabaseNameuserIdPassworddefaultCollectionName
writeConcernwriteResultChecking
save()insert()remove()updateFirst()findOne()find()…
MongoTemplate
Mongo(Java Driver class)
<<interface>>MongoConvertor
write(Object, DBObject)read(Class, DBObject)
uses
POJO ó DBObjectmappingSimplifies data access
Translates exceptions
MongoMappingConverter
Example entitypublic class Restaurant { private String id; private String name; private List<MenuItem> menuItems;
public Restaurant(String name) { this.name = name; … }
...}
public class MenuItem { private String name; private double price;
public MenuItem(String name, double price) { this.name = name; this.price = price; }
.....
Spring Data uses fields and non-default constructors
Example data access code
@Repositorypublic class RestaurantRepository {
@Autowired private MongoTemplate mongoTemplate; public void add(Restaurant restaurant) { mongoTemplate.save(restaurant); }
public List<Restaurant> findRestaurantsByName(String restaurantName) { return mongoTemplate.find( query(where("name").is(restaurantName)), Restaurant.class); }
A document in the restaurant collection
{ "_id" : ObjectId("4d977f55d3fe3119c904e026"), "_class" : "net.chrisrichardson.mongodb.example.mongotemplate.Restaurant", "name" : "Ajanta" "menuItems" : [ { "name" : "Tandoori Portobello Mushrooms", "price" : 5.5 }, { "name" : "Duck Curry Kerala", "price" : 15 } ]}
Spring MongoDB Example - Config 1@Configurationpublic class MongoExampleConfig extends AbstractMongoConfiguration {
private @Value("#{mongoDbProperties.databaseName}") String mongoDbDatabase;
private @Value("#{mongoDbProperties.host}") String mongoDbHost;
public Mongo mongo() throws Exception { return new Mongo(mongoDbHost); }
@Override protected String getDatabaseName() { return mongoDbDatabase; }...}
databaseName=demo1host=192.168.253.150
mongodb.properties:
<beans> <context:annotation-config/> <context:component-scan base-package="net.chrisrichardson.mongodb.example"/>
<util:properties id="mongoDbProperties" location="mongodb.properties"/>
</beans>
External Config
Defines a MongoTemplate
Spring MongoDB Example - Config 2<bean id="mongoTemplate" class="org.springframework.data.mongodb.core.MongoTemplate"> <constructor-arg ref="mongoFactory"/></bean>
<mongo:db-factory id="mongoFactory" host= "#{mongoDbProperties.host}"dbname="#{mongoDbProperties.databaseName}" />
<util:properties
id="mongoDbProperties" location="mongodb.properties"/>
Update example@Repositorypublic class RestaurantRepository { public void addMenuItem(String restaurantId, MenuItem newMenuItem) { mongoTemplate.updateFirst( query(where("_id").is(new ObjectId(restaurantId))), new Update().push("menuItems", newMenuItem), Restaurant.class); }
Atomic, in-place update of document
Geospatial example 1
@Componentclass MongoFriendService extends FriendService {
@Autowired var mongoTemplate: MongoTemplate = _
@PostConstruct def createGeoIndex { val dbo = new BasicDBObject dbo.put("location", "2d") mongoTemplate.getCollection("friendRecord").ensureIndex(dbo) }
Create geospatial 2d index
Collection name
case class FriendRecord(id : String, name : String, location : Point)
Geospatial example 2 - finding nearby
@Componentclass MongoFriendService extends FriendService {
override def findNearbyFriends(request: NearbyFriendsRequest) = { val location = new Point(request.longitude, request.latitude) val distance = new Distance(3, Metrics.MILES) val query = NearQuery.near(location).maxDistance(distance) val result = mongoTemplate.geoNear(query, classOf[FriendRecord])
val nearby = result.getContent.map(_.getContent) FindNearbyFriendsResponse(nearby.map(f => FriendInfo(f.name, f.id))) }
Callbacks – access driver API with exception translation
@Test public void testDbCallback() { Restaurant ajanta = makeAjantaRestaurant(); restaurantRepository.add(ajanta); assertCollectionExists("restaurants2"); }
private Void assertCollectionExists(final String collectionName) { return mongoTemplate.execute(new DbCallback<Void>(){ @Override public Void doInDB(DB db) { Set<String> collectionNames = db.getCollectionNames(); Assert.assertTrue("Missing from " + collectionNames, collectionNames.contains(collectionName)); return null; }}); }
Exceptions are translated
Defining a Mongo Generic Repository
interface PersonRepository extends MongoRepository<Person, ObjectId> { List<Person> findByLastname(String lastName);}
public class Person { private ObjectId id; private String firstname; private String lastname;… getters and setters}
Person p = new Person("John", "Doe");personRepository.save(p);
Person p2 = personRepository.findOne(p.getId());
List<Person> johnDoes = personRepository.findByLastname("Doe"); assertEquals(1, johnDoes.size());
Mongo Repository configuration<bean>
<mongo:repositories base-package="net.chrisrichardson.mongodb.example.mongorepository" mongo-template-ref="mongoTemplate" />
</beans>
Scans classpath looking for subtypes of MongoRepository in the base package
Richer mapping@Document(collection=”people”)public class Person {
@Id private ObjectId id; private String firstname; @Indexed private String lastname;
@PersistenceConstructor public Person(String firstname, String lastname) { this.firstname = firstname; this.lastname = lastname; }….}
Annotations define mapping: @Document, @Id, @Indexed, @PersistanceConstructor, @CompoundIndex, @DBRef, @GeoSpatialIndexed, @Value
Map fields instead of properties ⇒ no getters or setters
required
Non-default constructorIndex generation
Richer mapping configuration@Configurationpublic class MongoExampleConfig extends AbstractMongoConfiguration { private @Value("#{mongoDbProperties.databaseName}") String mongoDbDatabase;
private @Value("#{mongoDbProperties.host}") String mongoDbHost;
@Override public Mongo mongo() throws Exception { return new Mongo(mongoDbHost); } @Override public String getDatabaseName() { return mongoDbDatabase; }
@Override public String getMappingBasePackage() { return Person.class.getPackage().getName(); }}
Defines MongoTemplate bean
Configures classpath scanning
Support for the QueryDSL project
QPerson person = QPerson.person;
Predicate predicate = person.homeAddress.street1.eq("1 High Street") .and(person.firstname.eq("John"))
List<Person> people = personRepository.findAll(predicate);
assertEquals(1, people.size());assertPersonEquals(p, people.get(0));
Generated from domain model class
Type-safe composable queries
Cross-store/polyglot persistence
@Entitypublic class Person { // In Database @Id private Long id; private String firstname; private String lastname; // In MongoDB @RelatedDocument private Address address;
{ "_id" : ObjectId(”….."), "_entity_id" : NumberLong(1), "_entity_class" : "net.. Person", "_entity_field_name" : "address", "zip" : "94611", "street1" : "1 High Street", …}
Person person = new Person(…);entityManager.persist(person);
Person p2 = entityManager.find(…)
Agenda
• Why NoSQL?
• Overview of NoSQL databases
• Introduction to Spring Data
• Using Spring Data for Redis
• Using Spring Data for Mongo
• Deploying on Cloud Foundry
Using Mongo and Redis with Cloud Foundry
• Create a Mongo or Redis service
• Bind the service to your application
• Access the service
• Via auto-reconfiguration
• Using <cloud:*/> namespace
Creating a Redis Server
Deploying a Redis application
Redis bean definitions
USING THE APPLICATION
About <cloud:redis-connection-factory/>
<cloud:redis-connection-factory id="redisConnectionFactory" service-name="redis1" />
Use when multiple services are bound
Deploying a Mongo application
MongoDB bean definitions
Using the Mongo Application
About <cloud:mongo-db-factory/><cloud:mongo-db-factory id="mongoFactory" service-name="mongo1" > <cloud:mongo-options connections-per-host="..." max-wait-time="..." /></cloud:mongo-db-factory>
Use when multiple services are bound
]
NoSQL and Caldecott
• Caldecott let’s you tunnel to a NoSQL service
• Use Redis CLI
• Explore database, adhoc operations
• ...
• Use Mongo CLI etc
• Explore database, adhoc operations
• Mongo dump/restore
• ...
Summary
• NoSQL databases sometimes offer a combination of:
• Higher scalability and performance
• Schema less, richer data models
• Spring Data simplifies the development of NoSQL applications
• Cloud Foundry supports Mongo and Redis
84
Cloud Foundry 启动营在www.cloudfoundry.com注册账号并成功上传应用程序,即可于12月8日中午后凭账号ID和应用URL到签到处换取Cloud Foundry主题卫衣一件。
85
iPhone5 等你拿第二天大会结束前,请不要提前离开,将填写完整的意见反馈表投到签到处的抽奖箱内,即可参与“iPhone5”抽奖活动。
86
Birds of a Feather 专家面对面所有讲师都会在课程结束后,到紫兰厅与来宾讨论课程上的问题