Post on 10-May-2015
description
budapest.py
Google Appengine as a perfectmashup platform
Janos Hardijanos@liligo.com
http://github.com/yaanno
What is a mashup?
Some original idea implemented on the top of various 3rd party web
services by mixing them.
What do you need for a good mashup?
A smashing idea
Technical background
3rd party services
An example
A good idea (me thinks):
A travel search bot based on a 3rd party engine plus Twitter (+ Jabber) messaging.
Search types: flight, hotel, inspiration, etc.
Sideslide
Twitter bots: http://twitter.pbworks.com/Bots
Twitter mashups: http://twitter.pbworks.com/Mashups
Google AppEngine
Technical background:
Database (datastore)
Cache (memcache)
3rd party apis via HTTP (urlfetch)
... and many many more ...
3rd party web services
Good starting points:
programmableweb.com (apis)
mashable.com (news)
Mashup ingredients
Twitter API
Jabber API
Yahoo Trip Planner API
Travel Search APIs
Travel Search APIk
Kayak
Liligo
Cleartrip
Kayak Travel Search API
- Kayak infrastructure
- Slightly complicated data exchange
- XML based
- Poor documentation
- Discontinued! Now you have “RSS APIs” ...
Liligo Travel Search API
- Liligo infrastructure
- Complicated setup procedure
- XML based
- Good documentation
Cleartrip Travel Search API
- Cleartrip infrastructure
- Dozens of services
- Simple data exchange
- XML based
- Excellent documentation
- From India with love
Yahoo! API
- Stone simple
- XML based
- Good documentation
Twitter API
- Simple as stone
- XML/JSON based
- Good documentation
Google (Data) APIs
Loads of stuff, amazing.
http://code.google.com/p/gdata-python-client/
Now what?
What we have is ether nothing or almost useless.
Web based bot?
Cron service calls /cron resource
/cron sets up default tasks
Task manager processes these tasks (queue api)
Default tasks
- Do we have new messages?
- Do we have new searches stored?
- Do we have unreplied messages?
... and a lot more ...
How?
Apparently we don't have parallel / async events, processes and so on but we have a nice Task Queue api.
Task Queue
class MessageHandler(webapp.RequestHandler): def get(self): task_type = self.request.get('type') queue = Queue(name='messaging') if task_type == 'collect': task = Task( url='/worker/collect_messages', method='GET', ) queue.add(task)
Task Queueclass SearchHandler(webapp.RequestHandler): def get(self): task = Task( url = '/worker/search', params = { 'engine' : 'yahoo', 'searchtype' : 'inspire', 'query' : query['query'], 'maxresults' : '30', 'request_id' : query['request_id'], }, method = 'GET', ) queue.add(task)
Message Workerimport loggingfrom worker import Workerfrom .modules.messaging import Messaging
class MessageWorker(Worker): def __init__(self): logging.debug('new messaging worker spawned') def get(self): messaging = Messaging() messaging.collect() messaging.process() messaging.save()
Search Worker
import loggingfrom worker import Workerfrom .modules.search import Search
class SearchWorker(Worker): def __init__(self): logging.debug('new search worker spawned') def get(self): request = self.request.query_string query = self.normalize(request) search = Search(query) raw_result = search.search()
Processing #1
def collect(self): self.messages = t.status_replies()
def process(self): self.messages = self._serialize(self.messages)
def _serialize(self,serializable): return json.loads(serializable)
Processing #2
import xml.etree.ElementTree as ET
xml = ET.fromstring(self.result)
trips = xml.findall("trips/trip")
for trip in trips:id = trip.get("id")priceelem = trip.find("price")price = priceelem.texturl = priceelem.get("url")
Page render & Cache
class UserPage(Page):
def get(self,*args):
userpageid = str('userpage_' + args[0]) page = memcache.get(userpageid)
if page is None: userid = long(args[0]) author = Author.all().filter('author_id', userid).get() messages = Message.all().filter('author_id',
author.key().id()).fetch(20) results = { 'author': author, 'results':[] } page = self.prerender('user_page', {'results':results}) memcache.put(page) self.response.out.write(page)
Natively http://googleappengine.blogspot.com/2009/02/roadmap-update.html
External service http://morethanseven.net/2009/02/21/example-using-xmpp-app-engine-imified/
PyGtalkRobot basedhttp://code.google.com/p/aiassistant/
Jabber on Appengine
Thanks for your time!