Page 1 Queue Storage Jeff Chu | 2010-6-24. Page 2 Agenda Queue Storage Overview Programming Queue...
-
Upload
charlene-reynolds -
Category
Documents
-
view
218 -
download
1
Transcript of Page 1 Queue Storage Jeff Chu | 2010-6-24. Page 2 Agenda Queue Storage Overview Programming Queue...
Page 1
Queue Storage
Jeff Chu | 2010-6-24
Page 2
Agenda• Queue Storage Overview
• Programming Queue Storage
• Queue Storage tips
• Lab Time
Page 3
Queue Storage Overview
MessageQueueAccount
order processing
customer ID order ID http://…
customer ID order ID http://…
cohowinery
Page 4
Queue Storage Overview• Data Structure…
– FIFO (First-in, First-out)– FCFS (First-come, First-serve)
• Unlimited message stores– Message size limits: 8KB– Base on byte count, not text length.
• Inter-role communication– Web to Worker communication– Worker to Web communication
Page 5
Queue Storage Overview
Azure Queue
Input Queue (Work Items)
Page 6
QUEUE’S RELIABLE DELIVERYGuarantee delivery/processing of messages (two-step consumption)
1. Worker Dequeues message and it is marked as Invisible for a specified “Invisibility Time”
2. Worker Deletes message when finished processing it
If Worker role crashes, message becomes visible after the invisibility time for another Worker to process
Azure QueueInput Queue (Work Items)
Page 7
21
21
11
11
C1C1
C2C2
Removing Poison Messages
11
11
21
21
3340
40
Producers Consumers
P2P2
P1P1
30
30
2. GetMessage(Q, 30 s) msg 2
1. GetMessage(Q, 30 s) msg 1
11
11
21
21
7
10
10
20
20
Page 8
C1C1
C2C2
Removing Poison Messages
340
40
Producers Consumers
P2P2
P1P1
11
11
21
21
2. GetMessage(Q, 30 s) msg 23. C2 consumed msg 24. DeleteMessage(Q, msg 2)7. GetMessage(Q, 30 s) msg 1
1. GetMessage(Q, 30 s) msg 15. C1 crashed
11
11
21
21
6. msg1 visible 30 s after Dequeue30
30
8
12
1211
1112
12
Page 9
C1C1
C2C2
Removing Poison Messages
340
40
Producers Consumers
P2P2
P1P1
12
12
2. Dequeue(Q, 30 sec) msg 23. C2 consumed msg 24. Delete(Q, msg 2)7. Dequeue(Q, 30 sec) msg 18. C2 crashed
1. Dequeue(Q, 30 sec) msg 15. C1 crashed10. C1 restarted11. Dequeue(Q, 30 sec) msg 112. DequeueCount > 213. Delete (Q, msg1)
12
12
6. msg1 visible 30s after Dequeue9. msg1 visible 30s after Dequeue
30
30
9
13
1312
1213
13
Page 10
Programming Queue Storage• Get Client
– CloudQueue.CreateIfNotExist()
• Create Message– CloudQueueMessage class
– CloudQueue.AddMessage()
• Get Message– Get CloudQueueMessage, and change message’s property to invisible.
– GetMessages() accept max 32 messages.
• Peek Message– Get CloudQueueMessage, but DON’T change invisible property.
• Delete Message– CloudQueue.DeleteMessage()
– CloudQueue.Clear ()
Page 11
Programming Queue Storage• Get queue count
– RetrieveApproximentMessageCount() : must call this to update active message count.
– ApproximentMessageCount Property
• CloudQueueMessage class– Message can archive any string or data, but length cannot over 8KB.
– DequeueCount
• Times– ExpirationTime: Time of message will be deleted by Queue Storage.
– InsertionTime: Time of message inserted into queue.
– NextVisibleTime: Time of next visible time.
• Metadata– Queue Container support metadata (like BLOB storage).
Page 12
Programming Queue StorageCloudStorageAccount account = CloudStorageAccount.FromConfigurationSetting("CloudStorageAccount");CloudQueueClient queueClient = new CloudQueueClient( account.QueueEndpoint, account.Credentials);
//Create QueueCloudQueue queue = queueClient.GetQueueReference(queueName);queue.CreateIfNotExist();
//Add MessageCloudQueueMessage message = new CloudQueueMessage(“some content");queue.AddMessage(message);
//Get Messagemessage = queue.GetMessage(TimeSpan.FromMinutes(3) /*Invisibility timeout*/);
// Process Message within the Invisibility Timeout
//Delete Messagequeue.DeleteMessage(message);
Page 13
Queue Storage tips• Messages can be up to 8KB
– Use blob to store large messages, and store blob ref in message
• A message may be processed more than once– Make message processing idempotent
• Work should be repeatable and can be done multiple times
• Assume messages put into queue can be processed in any order
• For higher throughput– Batch multiple work items into a single message or into a blob– Use multiple queues
• Use DequeueCount to remove poison messages– Enforce threshold on message’s dequeue count
• Monitor message count to dynamically increase/reduce workers
Page 14
Queue Storage tips• If your message must over 8KB…
– Store your message into BLOB storage and pass the reference to queue.
• Producer and Consumer Problem– Delete after data consumed.– Pass reference to consumer to process
finalized data.
Page 15
Show time for you!
Page 16
References• Windows Azure SDK – Table Storage
– http://msdn.microsoft.com/en-us/library/dd179423.aspx• Tech.Ed 2010 North America Session
– COS310: Windows Azure Storage Deep Dive– http://www.msteched.com/2010/NorthAmerica/COS310
• 邊做邊學 Windows Azure: Table, BLOB and Queue – http://msdn.microsoft.com/zh-tw/azure/ff477598.aspx
• 認識 Table Storage– http://www.dotblogs.com.tw/regionbbs/archive/
2010/03/01/vs2010.and.azure.platform.understanding.table.storage.service.aspx