Chapter 9: Web Services, APIs and Maswolber/bookChapters/App_Inventor_Ch_9.pdf · Chapter 9: Web...
14
Transcript of Chapter 9: Web Services, APIs and Maswolber/bookChapters/App_Inventor_Ch_9.pdf · Chapter 9: Web...
Chapter 9: Web Services, APIsand Mashing DataThis chapter introduces web services, APIs, RSS, and mashups, then stepsthrough the development of an Android app that communicates with theTwitter API.
To computer users, the web is a bunch of sites that can be visited with abrowser. Programmers see things differently-- to them the web is also ahuge database of information which can be collected and processed(mashed) to create new and interesting apps.
Consider, for instance, the custom bookstore sites that sell books usingAmazon's database of information along with their facilities for on-linetransactions.
Such custom bookstores are possible because Amazon, like many sites,provides two levels of access to information: a human interface and anapplication programmer interface (API):
Most of us have used the human interface by typing in amazon.com into abrowser. But let's say you were a basketball enthusiast and you wanted tosetup a site for selling books about basketball. You could write a web app-- acomputer program-- that made calls to Amazon to perform searches and get
information about particular books. Your program could organize the data,combine it with other data (e.g., pictures of NBA stars), and display thatinformation to visitors of your "basketballBooks.com" site.
Your program would communicate with Amazon, but instead of usingamazon.com, as a person would do, your program would communicate withtheir API interface. API stands for application programmer interface and anapplication programmer is one who writes apps-- the basketball storecreator, in this case. An API provides for computer-computercommunication, one program talking to another, as opposed to a web app,which provides for human-computer interaction.
Technically, an API is the specification that describes how your programshould talk to the web service at some server's site-- the type of requests itexpects, and the format of the data that will be returned to your program.Often, the data returned is some form of extensible markup language (XML),though there are also JSON APIs. The API tells the client programmer how tocommunicate with the web service.
Whereas HTML is the language used when a server sends information to abrowser for human consumption, XML is generally used when a server sendsinformation to a program on another server. Both HTML and XML includedata within them (e.g., book information), but HTML also has presentationinformation-- specifications for how the information should be displayed--while XML does not.
Here's an example of a search at amazon.com for Books about Baseball. Thesearch returns a page that looks like this:
The HTML behind this page is extremely complex and includes both the data
you see and code for styling that data (e.g., code specifying that the bookimages should be aligned). Here's an abridged version of the HTML code:
<html><head>
<style type="text/css"><!--BODY { font-family: verdana,arial,helvetica,sans-serif; font-size: small; background-color: #FFFFFF; color: #000000; margin-top: 0px; }...
<div class="productImage"><a href="http://www.amazon.com/Baseball-Prospectus-2010/dp/0470558407/ref=sr_1_1?ie=UTF8&s=books&qid=1264011867&sr=1-1"> <imgsrc="http://ecx.images-amazon.com/images/I/51kRbJCrDoL._SL160_AA115_.jpg"class="" border="0" alt="Product Details" width="115" height="115"/> </a></div><div class="productData"><div class="productTitle"><a href="http://www.amazon.com/Baseball-Prospectus-2010/dp/0470558407/ref=sr_1_1?ie=UTF8&s=books&qid=1264011867&sr=1-1"> Baseball Prospectus2010</a> <span class="ptBrand">by Baseball Prospectus</span><spanclass="binding"> (<span class="format">Paperback</span> - Feb 22,2010)</span></div>
The important thing to understand is that browsers understand HTML codeand use all the added styling information to render a page. However, it isclear to see that extracting just the data, e.g., the title of the books, is notso easy. A computer program reading this data in order to just grab thetitles would have to know within what HTML tags that data resides. The codeto parse this HTML-- to extract the data- would be non-trivial. Furthermore,if the HTML styling changed, the program code to parse it would no longerwork!
This is the reason many web sites provide a non-HTML API web service thatreturns data in some well-documented XML language. The client code cancall the service with a URL, as in the following example:
http://ecs.amazonaws.com/onca/xml?AWSAccessKeyId=1x1x1x1x1x1x1x11 &Keywords=baseball&Operation=ItemSearch&SearchIndex=Books&Service=AWSECommerceService&Timestamp=2010-01-20T18%3A10%3A14.000Z&Version=2009-03-31&Signature=LAu8sU4nKvh0R6AbEQuFfAne27mCFDqVYiqUudGpkcc%3D
This URL is in the same format of the URLs that humans use to navigate thehuman web. But instead of returning HTML, it returns XML in Amazon's(fairly) well documented API. You can see the response I got from thisrequest here:http://www.cs.usfca.edu/~wolber/amazonResponse.xml. Here is anabridged version:
1. The 1x1x1x1x1x1x must be replaced with an Amazon access key for this URL to work.Anyone may sign up for such a key.
<ItemSearchResponse><Items>
<Item><ASIN>0470558407</ASIN><ItemAttributes>
<Author>Baseball Prospectus</Author><Manufacturer>Wiley</Manufacturer><ProductGroup>Book</ProductGroup><Title>Baseball Prospectus 2010</Title>
</ItemAttributes></Item><Item>
<ASIN>1600783554</ASIN><ItemAttributes>
<Author>Ron Shandler</Author><Manufacturer>Triumph Books</Manufacturer><ProductGroup>Book</ProductGroup>−<Title>2010 Baseball Forecaster (Ron Shandler's Baseball Forecaster)</Title>
</ItemAttributes><Items>
<ItemSearchResponse>
The response has a list of items, and each item has a set of taggedattributes including title, ISBN, etc. The XML is much easier to parsecompared to HTML and the expectation is that it won't change without thesource site giving ample warning (and they don't need to change it forstylistic reasons, as with the HTML interface).
So a mashup program can call the amazon API, and perhaps call other APIssuch as the Google Books API or a library's API. After the program parsesthe XML, the various data is all stored in the program's memory. It can thenprocess it, organize it, and eventually display it in any manner it would like.If its goal is to sell books, Amazon provides methods for allowing custombook sites to earn money on click-throughs.
Here's how product advertising works, from Amazon web site:
1Your application uses the Product Advertising API to supply item descriptions and images,and customer and seller reviews, which you present to your customers.
2Customers shop on your web site.
3
When the customer is ready to purchase the items in their e-commerce shopping cart,your application sends an HTML form to Product Advertising API and Amazon completesthe purchase by getting purchase information, such as payment method and shippingaddress, and then Amazon fulfills the order by shipping the items.
http://docs.amazonwebservices.com/AWSECommerceService/latest/DG/
Some APIs are simple than Amazon's. For instance, Yahoo provides financialinformation with a URL that returns simple comma separated data. Forinstance, the URL:
http://download.finance.yahoo.com/d/quotes.csv?f=sl1d1t1c1ohgv&e=.csv&s=IBM
returns the text:
"IBM",75.97,"11/19/2008","4:01pm",-4.11,79.93,81.00,75.73,12606807
The API specification is simply this: the commas separate these nine piecesof information:
• Ticker symbol• Last price (after a 20-minute delay)• Date of that price• Time of that price• Change since the day's opening• Opening price• Day's high price• Day's low price• Trade volume
Mashups
The term mashup perhaps originated in the music field, with Eminem'salbum The Slim Shady LP combining samples from various other songs andartists.Today, numerous sites like ccmixter help artists mashup samples and createnew art. Combining previously created art is controversial of course, due tocopyright issues. For a great introduction to the general concept of mashupsand the societal implications, see Lawrence Lessig's book Remix (you canfind it for free on-line at http://www.bloomsburyacademic.com/remix.htm)
Just as music can be remixed, so can web programs and information. To aprogrammer, a mashup is a program that takes data from multiple sources(APIs) and combines it in interesting ways. Custom bookshops that combinedata from multiple APIs would be considered mashups, but there are many,many more that use APIs from all walks of life. The sitehttp://www.programmableweb.com/ provides a dynamic list of interestingAPIs and mashups. Perhaps the most popular example, and one of the first,is housingmaps.com which combines data from craigslist.com and Google'smaps API. Here's a snapshot of it showing craigslist listings plotted onto a
map of San Francisco:
Tweeting Big Stock Swings To Twitter
Another interesting example of a mashup was provided by Bob DuCharme athttp://www.devx.com/webdev/Article/40511/0/page/1. Bob combinedYahoo's stock information with the Twitter API to create a mashup thattwitters him when a particular stock has a swing of 50 or more points. Thisdiagram illustrates how it works:
In this mashup, DuCharme's server program requests data from YahooFinance, which returns it in comma-separated value format. His program thenprocesses it by checking it against its previous readings. If there is more than a 50 pointdifference, it uses the Twitter API to change the status on his twitter page. Bob and otherhumans can then check his twitter page for updates.
RSS
RSS stands for Really Simple Syndication. RSS is a specific XML languagethat is used by most blogs and in general any service that just needs to sendout a simple list of items in an XML format.
Because RSS is so common, there are many clients and code libraries forprocessing RSS data, so you can help people access your data by putting itin this format. Note, however, that RSS is a fixed protocol-- if you havecustom data you either have to fit it into RSS fields or define your own XMLschema. For instance, the XML returned by Amazon, shown above, is not inRSS because Amazon provides pricing and other information that doesn't fitinto the RSS protocol.
Creating Mashups
Mashups are generally created in high-level programming languages such asPython, Java, or Perl. Most information sources generally provide clientlibrary code in these popular languages that makes data access easy. Ingeneral, however, creating mashups requires sophisticated programmingknowledge.
There have been some advancement in tools for "end-user" creation ofmashups. The term end-user in this context means someone who doesn't
know how to program in a traditional textual language such as Python orJava (which includes most of the world).
Yahoo Pipes is one such end-user tool. With it, you can link various webservices and RSS data sources together to create mashups. Here's anexample of a pipe that mashes NY City apartment information with userinput:
This pipe allows the user to enter a NYC neighborhood, a keyword, and aminimum distance. In the sample shown, the user is looking for apartmentsnear a park in Manhattan.
The pipe was created without any programming, at least of the traditionaltextual kind. Here is what development looks like:
The developer just drags in functional nodes for getting user input,extracting data from sources, fitering, and displaying data. In the sample,for instance, the top node prompts the user for the neighborhood (the"where"), then funnels that information to a URL builder that creates a callto the API http://newyork.backpage.com/online/exports/Rss.xml. The resultsare then filtered by location and another input by the user, "what" (bydefault, the "what" is "park", but the user can enter any place for whichthey'd like to find an apartment that is near).
Sergey Brin Quote, stats about more computer->computer communicationthan human-communication
Twitter API
Twitter has an API that return data in RSS format. For instance, one of theAPI commands returns the public timeline (the newest twits from the world).if you enter:
http://twitter.com/statuses/public_timeline.rss
in a browser, the browser will render something like:
This is just because most browsers will render RSS in a format that's nice forviewing content. Really, the RSS sent to the browser looks like:
<?xml version="1.0" encoding="UTF-8"?><rss xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"xmlns:georss="http://www.georss.org/georss"><channel><title>Twitter public timeline</title><link>http://twitter.com/public_timeline</link><atom:link type="application/rss+xml" href="http://twitter.com/statuses/public_timeline.rss" rel="self"/><description>Twitter updates from everyone!</description><language>en-us</language><ttl>40</ttl><item><title>Juliana_Madeira: "É triste pensar que a natureza fala e que o
gênero humano não a ouve." Victor Hugo</title><description>Juliana_Madeira: "É triste pensar que a natureza fala e queo gênero humano não a ouve." Victor Hugo</description><pubDate>Wed, 20 Jan 2010 20:10:23 +0000</pubDate><guid>http://twitter.com/Juliana_Madeira/statuses/7998009238</guid><link>http://twitter.com/Juliana_Madeira/statuses/7998009238</link></item>
and the browser has processed it to render it.
This URL returns a particular person's friend timeline:http://twitter.com/statuses/friends_timeline.xml
If you type this command in a browser, it will prompt for a valid twitterusername and password. You can also enter a URL of the form:
http://username:[email protected]/statuses/friends_timeline.xml
replacing "username" and "password" with your own authentication.
Putting REST URLs directly in a browser gives you an idea of how an APIworks, but generally these requests will come not from a person, but from aprogram. Your program will make an HTTP request, generally using high-level "client" library code for the particular language you are using, andhiding the URL details. The library code also takes care of parsing the XMLreturned from the service, providing your client with an easy-to-process listof data.
An App Inventor Twitter Client for Android
An Android programmer, using the Java SDK, could communicate with anyAPI. App Inventor, however, does not yet provide blocks for calling anarbitrary API. It does, however, provide blocks for communicating with theTwitter API.
The blocks consist of request blocks and request-received blocks. Requestblocks are used to send requests to the Twitter API: a request to login, arequest to get the friends_timeline, etc. Typically these requests will bemade from within some event-handler (e.g., when the user clicks a buttonlabeled "view friends timeline".
Requests are not processed immediately-- when you request a timeline, forinstance, the timeline is not provided as the return value of the block.Instead, request and reply are asynchronous. You send the request with oneblock, then specify an event-handler that will be triggered when the appreceives the request from Twitter.
This separation of request and reply is mandated by the Android system, asAndroid wants apps to always be ready to respond to external events,instead of waiting for some external request to complete.
Here's some program blocks that make use of two request-->handle eventsequences:
In these blocks, we first make a Login request to Twitter. While thisinteraction is occurring, the program could handle other user or externalevents. When Twitter logs the user in, it sends notification back to the phoneapp, and the IsLoggedIn event occurs, triggering the corresponding event-handler. The programer has set things up so that, in the handler for"IsLoggedIn", it issues another request with RequestFriendTimeline block.Once again the app is free to handle other events while Twitter is doing itsjob. When Twitter returns the timeline, the FriendTimelineReceived eventoccurs, and the program processes it by displaying the information.
Problems
0. Create a mashup using Yahoo Pipes
1. If you don't have a Twitter account, register and get one. If you have onealready, you may want to register with another account for the tutorials we'llbe doing.
2. Add at least one status message on the account, and follow at least oneperson and get at least one person to follow you.
3. On your phone, download an Android Twitter client and post a statusmessage from it. What features does the Twitter Client you chose provide?Provide a screenshot of your Twitter Client and a description of its features
4. Now create a Twitter Client of your own using App Inventor. You can doso using the tutorial at: http://sites.google.com/site/appinventorhelp/tutorials/twitterdemo
5. Write an Android application that combines texting and twitting. Forinstance, the app might text one of your friends when you've added a newstatus message on Twitter. You don't have to do exactly this-- be creative!
