Pg amqp
13
/ PostgreSQL: meet your Queue Presentation / Theo Schlossnagle 1
-
Upload
command-prompt-inc -
Category
Documents
-
view
2.747 -
download
3
description
Theo SchlossnagleOften times, when we make changes in a database we also want to "do something useful." Simply sending and email or triggering an ETL into a non-postgres system like Lucene is also a bit convoluted and painful. By using asynchronous messaging, we can make this simple. There have been a few attempts in the past to add into postgres the ability to queue external tasks, but with AMQP you get a complete package.pg_amqp marries the AMQP standard into PostgreSQL providing transaction-aware message queueing. We'll talk about how easy it can be to get an AMQP enabled postgresql up and running with pg_amqp. We'll also discuss the caveats that exist in pg_amqp (currently implementing AMQP 0.8) and how AMQP 1.0 will eventually solve them.
Transcript of Pg amqp
/
PostgreSQL:meet your Queue
Presentation / Theo Schlossnagle
1
PostgreSQL is Awesome
• Fast.
• Extensible.
• Tablespaces.
• Robust data types.
• Partitioning (albeit fake).
• Partial and functional indexes.
• Extremely supportive community.
• Extremely compliant with database standards.
2
PostgreSQL is not the “world”
• Inevitably, we must interact with the rest of the world.
• “non-SQL” components:
• nosql systems
• caching systems
• search systems (solr/lucene)
• management processes
3
Appropiare Typicalis
• Enforce in the application:
• the application code that updates the price or description of a product in the products table;
• the application submits the updates to the search index system.
• The flaw:
• psql# UPDATE products SET description = REPLACE(description, ‘behaviour’, ‘behavior’);
• Administrative fixes like that require out-of-band dependency handling.
4
A Solution
• Ideally, the database would notify all of these systems.
• The most common case I see: memcached.
• app: pull from memcached user::[email protected] if not found: select * from users where email=‘[email protected]‘ put row in memcached at user::[email protected]
• app: update users set mood=‘happy‘ where email=‘[email protected]‘ (a) purge memcached record (b) get full row and replace in memcached
• hence: pgmemcache
• Problem:
• need a Postgres module for each remote component
5
Enter Queueing
• Queueing?
• A generic message bus that allows PostgreSQL to communicate with other components in the architecture.
• Enter AMQP: “Advanced Message Queueing Protocol”
• Why not STOMP?
• Why not Starling?
• AMQP has been around the block, and the specification is quite complete.
• Almost every “real” message broker implementation supports AMQP
6
Setups: Installing
• svn export \ https://labs.omniti.com/pgtreats/trunk/contrib/pg_amqp
• cd pg_amqp
• make USE_PGXS=1
• make install
• add to postgresql.conf:shared_preload_libraries = 'pg_amqp.so'
• (re)start postgres
• load the pg_amqp.sql file into your database.
7
Setup: configuring your broker
INSERT INTO amqp.broker (host,port,vhost,username,password) VALUES (‘localhost’,5672,NULL,‘guest’,‘guest’) RETURNING broker_id
8
Setup: declaring an exchange
• This can often be done outside of the AMQP client
• using an AMQP management process(that is just, in fact, an AMQP client)
• Often, another component has already created the exchange.
• If you really need to do it within PostgreSQL:
SELECT amqp.declare_exchange(broker_id, exchange_name, exchange_type, passive, durable, auto_delete)
9
Usage: sending messages
• How do I connect?
• It’s implicit... you don’t need to.
• How do I disconnect?
• Broker connections are cached and live across transactions.
• If you want to explicitly disconnect:
SELECT amqp.disconnect(broker_id);
10
Usage: sending messages for real
• Sending messages as a part of my transaction:
SELECT amqp.publish(broker_id, exchange, routing_key, message);
• This will publish the “message” over the specified “exchange” using the specified “routing_key,” but only on transaction commit within Postgres.
• Sending messages NOW:
SELECT amqp.autonomous_publish(broker_id, exchange, routing_key, message);
• Publish the same message, but do it immediately.
11
A dark side: unsafe? WTF?
• Currently, pg_amqp uses the AMQP 0.8 specification.
• The AMQP 1.0 specification introduces formal 2PC.
• It is possible in the current implementation to have AMQP transaction fail when we commit it (in postgres’s commit txn hook) in a fashion that we cannot act on. Ouch.
• This only happens when the AMQP broker crashes between the last in-transaction publish call and the COMMIT on the database side.
• Once RabbitMQ supports AMQP 1.0, I’ll update the driver to use 2PC.
12
/
Thank you for listening.https://labs.omniti.com/pgtreats/trunk/contrib/pg_amqp/
Presentation
13
