Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Practical concurrent systems made simple using Kamaelia
Michael Sparks (BBC Research)http://edit.kamaelia.org/GetKamaelia
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaeliaMichael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
What is the problem?
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaeliaMichael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Concurrencyis considered
HARD
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaeliaMichael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
WHY ?(is that really the problem?)
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaeliaMichael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
What is Kamaelia's
Goal?
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaeliaMichael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
to make
Concurrency
EASYto use
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaeliaMichael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
WHY ?(is that really the goal?)
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaeliaMichael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
What would Knuth do?
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaeliaMichael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
I can’t think of even five of those programs that would
have been enhanced noticeably by parallelism
or multithreading.
http://flickr.com/photos/scottlaird/
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaeliaMichael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Hmm...
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaeliaMichael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
They think a magic bullet will come along to make multicores speed up my
kind of work; I think it’s a pipe dream.
http://flickr.com/photos/scottlaird/
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaeliaMichael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Not helping...
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaeliaMichael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
(No—that’s the wrong metaphor! "Pipelines"
actually work for me, but threads don’t.)
http://flickr.com/photos/scottlaird/
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaeliaMichael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Any formal support?
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaeliaMichael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Why don't (more) people know
about it?
http://www.inf.u-szeged.hu/stf/eloado.php
It was written by Tony Hoare in 198523 years ago
Communicating Sequential Processes
essentially supports this in a formal way
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaeliaMichael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaeliahttp://www.inf.u-szeged.hu/stf/eloado.php
Scary?
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Why is
concurrency(considered)
hard?
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
~/Images/Rest/Artwork> ls -l |grep -v "^d" |sed -e "s/ */ /g"|cut -d' ' -f5,8|sort -nr|head -7 |tail -3652109 Circus.jpg648385 background6.jpg596044 Monster.jpg
8 concurrent things...Doesn't look
so hard really...
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Why is
concurrency(considered)
hard?
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
BankCashiers£5
£10Alice
BobAccount starts at £2.50
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Valid case 1Alice enters Bank cashier sharedCashier A, gets balance £ 2.50 £ 2.50A Takes £10 from Alice, add to balance £12.50 £ 2.50A Stores balance £12.50 £12.50
Bob enters Bank cashier sharedCashier B, gets balance £12.50 £12.50B Gives Bob £5, removes from balance £ 7.50 £12.50B Stores balance £ 7.50 £ 7.50
Final Balance: £7.50
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Valid case 2Bob enters Bank cashier sharedCashier B, gets balance £ 2.50 £ 2.50B Gives Bob £5, removes from balance £ -2.50 £ 2.50B Stores balance £ -2.50 £ -2.50
Alice enters Bank cashier sharedCashier A, gets balance £ -2.50 £ -2.50A Takes £10 from Alice, add to balance £ 7.50 £ -2.50A Stores balance £ 7.50 £ 7.50
Final Balance: £7.50
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
But concurrency allows any order...
Final Balance: £.......
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
But concurrency allows any order...Bob enters Bank cashier sharedAlice enters Bank cashier sharedCashier B, gets balance £ 2.50 £ 2.50Cashier A, gets balance £ 2.50 £ 2.50B Gives Bob £5, removes from balance £ -2.50 £ 2.50A Takes £10 from Alice, add to balance £12.50 £ 2.50B Stores balance £ -2.50 £ -2.50A Stores balance £12.50 £12.50
Final (wrong) Balance: £12.50
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
But concurrency allows any order...Bob enters Bank cashier sharedAlice enters Bank cashier sharedCashier B, gets balance £ 2.50 £ 2.50Cashier A, gets balance £ 2.50 £ 2.50B Gives Bob £5, removes from balance £ -2.50 £ 2.50A Takes £10 from Alice, add to balance £12.50 £ 2.50B Stores balance £ -2.50 £ -2.50A Stores balance £12.50 £12.50
£12.50Lost update
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaeliaMichael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
What is the problem?
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaeliaMichael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Concurrencyis considered
HARD?
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Concurrency is hard due tounconstrained shared data
Threads provide precisely thatenvironment
Concurrency is hard because weuse hard to use tools to useconcurrency
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
But what about...
● Mutexes, Semaphores,Locks?
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
But what about...
● Mutexes, Semaphores,Locks?
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
But what about...
● Mutexes, Semaphores,Locks?
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
But what about...
● Mutexes, Semaphores,Locks?
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
But what about...
● Mutexes, Semaphores,Locks?
“I accidentally discovered I had a second thread updatinga dictionary value without locking the dictionary first”
Pycon attendee yesterday
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
But what about...
● Mutexes, Semaphores,Locks?
They work, but many people find them hard to get right
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
However, threads, mutexes etcare not the problem
Remember:
Concurrency is hard due tounconstrained shared data
Can we do better?
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Kamaelia's Underlying Goals
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
To assist with building and maintaining practical systems using communicating components to simplify them in a usable way such that it allows even novice developers to build safe and efficient highly concurrent systems. (rather than a niche of developers)
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
To assist with building and maintaining practical systems using communicating components to simplify them in a usable way such that it allows even novice developers to build safe and efficient highly concurrent systems. (rather than a niche of developers)
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
To assist with building and maintaining practical systems using communicating components to simplify them in a usable way such that it allows even novice developers to build safe and efficient highly concurrent systems. (rather than a niche of developers)
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
“Even Novice Developers?”
Trainee engineers
Google Summer of Code students
+ other small number ofrandom novices :)
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
14+ developers with limited levelsof experience
All produced interesting (safe) systems with high levels of
concurrency
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Applications?(aka What sort of systems?)
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Brisk walk through of a few...
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Email Greylisting(spam resistant mail server)
Recognises that in event of an error...
● Most spammers don't resend● Most real mail does resend
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Content Transcoderheadless toolchain for transcoding & resizing images/videos (eg for web)
The sort of thing youtube, flickr, slideshare, etc need as a background job
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Content Transcoder● Watches directories● Sorts content by type● Transcodes user video to flash● Converts images to variety of sizes● Moves to moderation queue
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Content Transcoder● Watches directories● Sorts content by type● Transcodes user video to flash● Converts images to variety of sizes● Moves to moderation queue
(Part of a wider project)
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Speak 'n WriteGraphical desktop tool to allow a child to learn to read and write...
Speaks and displays:“write the word <some word>”
The child copies the wordReads back what the child wroteTells them if they succeeded or failed
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Speak 'n WriteGraphical desktop tool to allow a child to learn to read and write...
● Uses espeak for speech● Pygame used for display● Uses gesture recognition for
determining what was written.
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
ER ModellerGraphical desktop tool that renders a conceptual data model
● Textual description -> visual layout.● Specifically designed to work at conceptual rather than visual level.
● Aids data model design
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
IRC LoggerYep, rather hackneyed...
● But really about IRC components allowing remote control of systems
● Also have AIM components, and there's a community library for XMPP components (etc)
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Others... (far from exhaustive)
Digital TV tools (eg timeshifting everything through to agent based recording), whiteboarding, simple games, personal webserving, music generation, bit torrent distribution, AIM/IRC/XMPP based systems, Pop3 proxying, Pop3 spam filtering, PyQT/tkinter/pygame tools, video shot change detection, etc
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
OK, that's the problems, background, and some
usecases. I want to see what this looks like.
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
from Kamaelia.Chassis.Pipeline import Pipeline
from Kamaelia.Apps.Games4Kids.BasicSprite import BasicSpritefrom Kamaelia.Apps.Games4Kids.SpriteScheduler import SpriteSchedulerfrom Kamaelia.Apps.Games4Kids.MyGamesEventsComponent import MyGamesEventsComponent
Pipeline( MyGamesEventsComponent(up="p", down="l", left="a", right="s"), BasicSprite("cat.png", name = "cat", border=40),).activate()
Pipeline( MyGamesEventsComponent(up="up", down="down", left="left", right="right"), BasicSprite("mouse.png", name = "player", border=40),).activate()
SpriteScheduler(BasicSprite.allSprites()).run()
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Bunch of imports
Pipeline( MyGamesEventsComponent(...), BasicSprite(...),).activate()
Pipeline( MyGamesEventsComponent(...), BasicSprite(...),).activate()
SpriteScheduler(BasicSprite.allSprites()).run()
Pipeline
MyGamesEvents
Component
BasicSprite
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Bunch of imports
Pipeline( MyGamesEventsComponent(...), BasicSprite(...),).activate()
Pipeline( MyGamesEventsComponent(...), BasicSprite(...),).activate()
SpriteScheduler(BasicSprite.allSprites()).run()
Pipeline
MyGamesEvents
Component
BasicSprite
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Note: this focusses on what we want to do, at a high level.Pipeline( MyGamesEventsComponent(up="p", down="l", left="a", right="s"), BasicSprite("cat.png", name = "cat", border=40),).activate()
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
What if we want to update and act on a global state?(eg based on distance between players?)
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
GlobalState
{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Create a place to publish locations to.from Kamaelia.Util.Backplane import *Backplane("PLAYERS").activate()
Backplane(“Players”)
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Make our Cat publish to it.Pipeline( MyGamesEventsComponent(up="p", down="l", ...), BasicSprite("cat.png", name = "cat", border=40), PureTransformer(lambda x: ("Cat ", x)), PublishTo("PLAYERS"),).activate()
Backplane(“Players”)
MyGamesEventsComp...
BasicSprite
PureTransformer
PublishTo
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Make our Mouse publish to it.Pipeline( MyGamesEventsComponent(up="p", down="l", ...), BasicSprite("cat.png", name = "cat", border=40), PureTransformer(lambda x: ("Mouse ", x)), PublishTo("PLAYERS"),).activate()
Backplane(“Players”)
MyGamesEventsComp...
BasicSprite
PureTransformer
PublishTo
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Make our analyser & subscribe itPipeline( SubscribeTo("PLAYERS"), PlayerAnalyser(), Distancer(), ConsoleEchoer(),).activate()
Backplane(“Players”)
SubscribeTo
PlayerAnalyser Distancer
ConsoleEchoer
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Full System View
Backplane(“Players”)
PublishTo
PureTransformer
BasicSprite
MyGamesEvents...
PublishTo
PureTransformer
BasicSprite
MyGamesEvents...
SubscribeTo
PlayerAnalyser
Distancer
ConsoleEchoer
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Full System View
Backplane(“Players”)
PublishTo
PureTransformer
BasicSprite
MyGamesEvents...
PublishTo
PureTransformer
BasicSprite
MyGamesEvents...
SubscribeTo
PlayerAnalyser
Distancer
ConsoleEchoer
The ERModeller code
also uses this
sort of code
structure
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Full System View
Backplane(“Players”)
PublishTo
PureTransformer
BasicSprite
MyGamesEvents...
PublishTo
PureTransformer
BasicSprite
MyGamesEvents...
SubscribeTo
PlayerAnalyser
Distancer
ConsoleEchoer
(No—that’s the wrong metaphor!
"Pipelines" actually work for me, but threads don’t.)
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Full System View
Backplane(“Players”)
Cat SpritePipeline
Mouse SpritePipeline
Player AnalyserPipeline
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
GlobalState
{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
GlobalState
{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}
Cat SpritePipeline
Mouse SpritePipeline
Player AnalyserPipeline
Backplane(“Players”)
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
GlobalState
{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}
Cat SpritePipeline
Mouse SpritePipeline
Player AnalyserPipeline
Backplane(“Players”)
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
GlobalState
{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}
Cat SpritePipeline
Mouse SpritePipeline
Player AnalyserPipeline
Backplane(“Players”)
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
GlobalState
{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}
Cat SpritePipeline
Mouse SpritePipeline
Player AnalyserPipelineBackplane
(“Players”)
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
GlobalState
{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}{('Mouse', 'Cat '): 244, ('Cat ', 'Mouse'): 244}
Cat SpritePipeline
Mouse SpritePipeline
Player AnalyserPipeline
Backplane(“Players”)
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Observations● Code structure follows problem structure
● 1 to 1 correspondence between diagram and code
● We're focussing on the problem, not code, not concurrency
● BUT we're looking for how things talk and transform.
Just like web mashups, or hardwareor unix pipelines, or ...
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Observations● Significantly easier to think about if you
anthropomorphise the problem
● Or “if there was a small army of little robots inside your computer, what would they do, who would they need to talk to”.
Each component is one of these “robots”
● Yes, I know its sad, and I do rather think that way far too often, but it works - so shoot me :-)
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Internal Component Metaphor
inbox
control
...
outbox
signal
...
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
System Metaphor
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Building a Network Server
MainServer Core
Remote User
Protocol Handler Factory
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Building a Network Server
MainServer Core
Remote User
Protocol Handler Factory
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Building a Network Server
MainServer Core
Remote User
Protocol Handler Factory
Socket handler
Protocol handler
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Building a Network Server
MainServer Core
Remote User
Protocol Handler Factory
Socket handler
Protocol handler
datato
user
datafromuser
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Building a Network Server
Protocol Handler FactoryProtocol handler
This is the part you focus on:
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Simplest Possible protocol handlerimport Axon
class Echo(Axon.Component.component): def main(self): while 1: for data in self.Inbox(“inbox”): self.send(data, “outbox”) yield 1
datato
userdatafromuser
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Anatomy of a more complex http-ish protocol handler
One approach is a bit like how an animal is “constructed”:
● Skeleton● Muscle● Tendons● Skin
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Building a More Complex Protocolclass ProtocolSkeleton(Axon.Component.component): def main(self): while not self.dataReady(“control”): for data in self.Inbox(“inbox”): response = self.handleMessage(data) self.send(response, “outbox”) if not self.anyReady(): self.pause() yield 1
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Building a More Complex Protocol
class HTTP_ish_Muscle(ProtocolSkeleton): def handleMessage(self, data): args = self.parse(data) handler_app = self.findHandler(*args) for line in handler_app(*args): self.send(line, “outbox”)
def parse(self, data): args = data.split() return args[:1], args[1:-1] args[-1:]
def findHandler(meth, args, version): def null(*args): yield 1 return null
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Building a More Complex Protocol
class SimpleFileProtocol(HTTP_ish_Muscle):
def findHandler(meth, args, version):
def fileReader(*args): filename = “ “.join(args).replace(“..“,““) for i in open(filename): yield i
return fileReader
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Using our Protocol Handlerfrom Kamaelia.Chassis.ConnectedServer import ServerCore
ServerCore(protocol = SimpleFileProtocol, port=1180).run()
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
or... you can also do this
from Kamaelia.Chassis.ConnectedServer import \ServerCore
class SimpleFileServer(ServerCore): protocol = SimpleFileProtocol port=1180
SimpleFileServer().run()
The greylistin
g code
essentially us
es this
sort of code
structure
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Graphlines
Not every system makes sense as a pipeline, nor make sense as backplanes.
Example structures:
... ... ... ...
...
...
... ...
...
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Graphline eg: Simple Slideshow Graphline( NEXT = Button(caption="Next", msg="NEXT", position=(72,8)), PREVIOUS = Button(caption="Previous", msg="PREV",position=(8,8)), FIRST = Button(caption="First", msg="FIRST",position=(256,8)), LAST = Button(caption="Last", msg="LAST",position=(320,8)), CHOOSER = Chooser(items = files), IMAGE = Image(size=(800,600), position=(8,48)), linkages = { ("NEXT","outbox") : ("CHOOSER","inbox"), ("PREVIOUS","outbox") : ("CHOOSER","inbox"), ("FIRST","outbox") : ("CHOOSER","inbox"), ("LAST","outbox") : ("CHOOSER","inbox"), ("CHOOSER","outbox") : ("IMAGE","inbox"), }).run()
...
...
...
... ...
...
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Simple Slideshow (args/links trimmed) Graphline( NEXT = Button(...), PREVIOUS = Button(...), FIRST = Button(...), LAST = Button(...), CHOOSER = Chooser(...), IMAGE = Image(...), ...).run()
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Simple Slideshow (args/links trimmed) Graphline( NEXT = Button(...), PREVIOUS = Button(...), FIRST = Button(...), LAST = Button(...), CHOOSER = Chooser(...), IMAGE = Image(...), ...).run()
FIRST(button)
Chooser
LAST(button)
PREVIOUS(button)
NEXT(button)
Image
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Recap...● What's the problem with concurrency?● Approaches that work:● “shared nothing” - aka message passing● Friendlier metaphor (inboxes/outboxes)● Overview of systems built this way
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Recap...
Nitty gritties:● Composing basic systems● Evolving basic systems
● Similar basic structure to the ERModeller● Building basic network servers● Evolving more complex network servers
● Similar basic structure to the greylister
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
More information?
http://edit.kamaelia.org/GetKamaelia● Links to code● Links to tutorials● Links to examples● Links to reference documentation
IRC: freenode.net , channel #kamaeliaLogs: http://edit.kamaelia.org/logs/
Michael Sparks (BBC Research)
http://edit.kamaelia.org/GetKamaelia
Thank you for listening
Questions?
more info:● http://edit.kamaelia.org/GetKamaelia● freenode.net , channel #kamaelia
Top Related