Post on 16-Jun-2015
description
Tricking out your
Memcached Setup
Wednesday, July 28, 2010
Chaos FactorWednesday, July 28, 2010
ASCII Protocol[brian@gaz libmemcached]$ memcp foo --servers=localhost[brian@gaz libmemcached]$ telnet localhost 11211Trying 127.0.0.1...Connected to localhost.Escape character is '^]'.get fooVALUE foo 0 4bar
ENDversionVERSION 1.4.5
Wednesday, July 28, 2010
Binary ProtocolMAGIC
(1 byte)Key Length
(2 bytes)Opcode
(1 byte)
Extra Length(1 byte)
Data Type(1 byte)
Reserved(2 bytes)
Total Body Length(4 bytes)
Opaque(4 bytes)
CAS (Compare and Swap)(8 bytes)
Wednesday, July 28, 2010
Pipelining!
GET FOO GET DOG GET HELP
PACKET
Wednesday, July 28, 2010
Consistent Hashing
New Record
Item goes into nearest node
Wednesday, July 28, 2010
libhashkit
• One_at_a_time
• MD5
• HSIES (patents anyone?)
• CRC
• murmur (pretty simple)
Wednesday, July 28, 2010
Replica
Node: B
Node: C
Store:Shopping_key
Node: D
Node: A
Wednesday, July 28, 2010
Replica
Node: B
Node: C
GET:Shopping_key
Node: D
Node: A
Wednesday, July 28, 2010
STAT!
• memstat --servers=localhost “args”
• slab
• items
• sizes
Wednesday, July 28, 2010
slabSTAT 1:chunk_size 104STAT 1:chunks_per_page 10082STAT 1:total_pages 1STAT 1:total_chunks 10082STAT 1:used_chunks 10081STAT 1:free_chunks 1STAT 1:free_chunks_end 10079STAT 9:chunk_size 696STAT 9:chunks_per_page 1506STAT 9:total_pages 63STAT 9:total_chunks 94878STAT 9:used_chunks 94878STAT 9:free_chunks 0STAT 9:free_chunks_end 0STAT active_slabs 2STAT total_malloced 67083616END
Wednesday, July 28, 2010
itemsSTAT items:2:number 1STAT items:2:age 452STAT items:2:evicted 0STAT items:2:evicted_nonzero 0STAT items:2:evicted_time 2STAT items:2:outofmemory 0STAT items:2:tailrepairs 0...STAT items:27:number 1STAT items:27:age 452STAT items:27:evicted 0STAT items:27:evicted_nonzero 0STAT items:27:evicted_time 2STAT items:27:outofmemory 0STAT items:27:tailrepairs 0
Wednesday, July 28, 2010
sizes96 35128 38160 807192 804224 410256 222288 83320 39352 53384 33416 64448 51480 30512 54544 39576 10065
Wednesday, July 28, 2010
Dump your data?
• memdump
• memcached_dump(memcached_st *ptr, memcached_dump_fn *function, void *context, uint32_t number_of_callbacks);
Wednesday, July 28, 2010
Drain
for (items= memdump()){store(items);memcached_delete();
}
Wednesday, July 28, 2010
libmemcachedutil
• libmemcached_util_ping();
• libmemcached_util_version_check();
• memcached_pool_st
Wednesday, July 28, 2010
memcached_pool_st* pool= memcached_pool_create(memc, 5, 10);memcached_st* mmc[10];
for (size_t x= 0; x < 10; ++x) { mmc[x]= memcached_pool_pop(pool, false, &rc);
…; } memcached_pool_push(pool, mmc[x])
Wednesday, July 28, 2010
...and now for databases
Wednesday, July 28, 2010
Wednesday, July 28, 2010
Setting via SELECT
select id, url, memc_set(concat('feeds', md5(url)), url) from feeds;
select memc_get(concat('feeds', md5(url))) from feeds;+-------------------------------------------------------+| memc_get(concat('feeds', md5(url))) |+-------------------------------------------------------+| http://feeds.feedburner.com/littlegreenfootballs/Ilds | | http://del.icio.us/rss/jacomien | | http://del.icio.us/rss/tags/rmj20 | | http://www.jihadwatch.org/index.rdf | | http://www.connotea.org/rss/user/rmj20 | | http://devblog.grazr.com/?feed=rss2x | | http://feeds.feedburner.com/littlegreenfootballs/kyeH | +-------------------------------------------------------+
Wednesday, July 28, 2010
Trigger
insert into feeds (url) values ('http://grazr.com/feedlist.xml');
select memc_get(concat('feeds:', md5('http://grazr.com/feedlist.xml')));+------------------------------------------------------------------+| memc_get(concat('feeds:', md5('http://grazr.com/feedlist.xml'))) |+------------------------------------------------------------------+| http://grazr.com/feedlist.xml | +------------------------------------------------------------------+
DROP TRIGGER IF EXISTS feed_insert;CREATE TRIGGER feed_insert BEFORE INSERT ON feeds FOR EACH ROW BEGIN SET @mm= memc_set(concat('feeds:',md5(NEW.url)), NEW.url);END |
Wednesday, July 28, 2010
Memcached Replication via MySQL
INSERT INTO table_a VALUES (“key”, “value”, “values”); INSERT INTO blackhole_table VALUES (memc_delete(“key”));
Wednesday, July 28, 2010
drizzle> select table_name -> from information_schema.tables -> where table_name like '%MEMCACHED%';+--------------------+| table_name |+--------------------+| MEMCACHED_STATS || MEMCACHED_ANALYSIS |+--------------------+2 rows in set (0 sec)
Wednesday, July 28, 2010
drizzle> desc information_schema.memcached_stats;+-----------------------+-------------+------+-----+---------+-------+| Field | Type | Null | Key | Default | Extra |+-----------------------+-------------+------+-----+---------+-------+| NAME | varchar(32) | NO | | | || PORT_NUMBER | bigint | NO | | 0 | || PROCESS_ID | bigint | NO | | 0 | || UPTIME | bigint | NO | | 0 | || TIME | bigint | NO | | 0 | || VERSION | varchar(8) | NO | | | || POINTER_SIZE | bigint | NO | | 0 | || RUSAGE_USER | bigint | NO | | 0 | || RUSAGE_SYSTEM | bigint | NO | | 0 | || CURRENT_ITEMS | bigint | NO | | 0 | |
Wednesday, July 28, 2010
drizzle> desc information_schema.memcached_analysis;+--------------------------------+-------------+------+-----+---------+-------+| Field | Type | Null | Key | Default | Extra |+--------------------------------+-------------+------+-----+---------+-------+| SERVERS_ANALYZED | bigint | NO | | 0 | || AVERAGE_ITEM_SIZE | bigint | NO | | 0 | || NODE_WITH_MOST_MEM_CONSUMPTION | varchar(32) | NO | | | || USED_BYTES | bigint | NO | | 0 | || NODE_WITH_LEAST_FREE_SPACE | varchar(32) | NO | | | || FREE_BYTES | bigint | NO | | 0 | || NODE_WITH_LONGEST_UPTIME | varchar(32) | NO | | | || LONGEST_UPTIME | bigint | NO | | 0 | || POOL_WIDE_HIT_RATIO | bigint | NO | | 0 | |+--------------------------------+-------------+------+-----+---------+-------+
Wednesday, July 28, 2010
libmemcachedprotocol
• Ascii
• Binary
Wednesday, July 28, 2010
memcached_binary_protocol_callback_st interface_v1_impl= { .interface_version= MEMCACHED_PROTOCOL_HANDLER_V1, .interface.v1= { .add= add_handler, .append= append_handler, .decrement= decrement_handler, .delete= delete_handler, .flush= flush_handler, .get= get_handler, .increment= increment_handler, .noop= noop_handler, .prepend= prepend_handler, .quit= quit_handler, .replace= replace_handler, .set= set_handler, .stat= stat_handler, .version= version_handler }};
Wednesday, July 28, 2010
./examples/memcached_light(everyone needs their own persistent Memcached server)
Wednesday, July 28, 2010
Examples!
• tests/mem_functions.c
• man “libmemcached_examples”
Wednesday, July 28, 2010
Brian Aker (brian.aker@datadifferential.com)
twitter:brianaker
url: Krow.net
http://DataDifferential.com
Wednesday, July 28, 2010