#Pharo Days 2016 Data Formats and Protocols
-
Upload
philippe-back -
Category
Software
-
view
414 -
download
0
Transcript of #Pharo Days 2016 Data Formats and Protocols
Pharo Object World
• Object Memory + Virtual Machine
• Graphics + Interaction, OS + Libraries
• Tools, IDE
Pharo
• One simple language used everywhere
• True open source for all parts
• Understandable from high to low level
Outside World
• Infinitely larger
• Our world is part of it
• We need to interact with it & affect it
OS (Sub) Process
• Execute arbitrary programs as sub processes
• Communicate via standard input/output/error
• Huge availability, high quality, reuse, NIH
OS (Sub) Process
• Large overhead
• Opaque, not Object Oriented
• Awkward interface, Not always good fit
Network Services
• Fully implemented in Pharo, Open standards
• Client/server, Distributed, IOT, DB
• Marshalling & communication overhead
Character Streams
• One of the simplest data formats
• ASCII, Latin1, UTF-8, UTF-16, UTF-32
• Fully covered & implemented in Pharo
| array readStream |
array := ByteArray streamContents: [ :out | (ZnCharacterWriteStream on: out encoding: #iso88591) nextPutAll: 'Les élèves français' ].
readStream := array readStream.
(ZnCharacterReadStream on: readStream encoding: #iso8859) upToEnd.
-> 'Les élèves français'
Referencehttp://files.pharo.org/books/enterprise-pharo/book/Zinc-
Encoding-Meta/Zinc-Encoding-Meta.html
Internal Formats
• FUEL (binary)
• STON (textual)
• Very cool, very useful
• Limited to the Pharo World
PVTRecord
• timestamp <DateAndTime>
• id <String>
• position <longitude@latitude> <Float@Float>
• speed <Integer>
CSV
• textual
• 1 record per line (any EOL)
• fields separated by delimiter
• fixed number of fields
id,time,long,lat,speedv1,2016-03-29T15:21:00+00:00,5.3403835,50.918625,37v1,2016-03-29T15:22:00+00:00,5.3397698,50.915192,27v1,2016-03-29T15:23:00+00:00,5.3415751,50.911633,47
(FileLocator desktop / 'pvtrecords.csv')readStreamDo: [ :in |
(NeoCSVReader on: in) skipHeader;addField;addFieldConverter: [ :string |
DateAndTime fromString: string ];addFloatField;addFloatField;addIntegerField;upToEnd ].
(FileLocator desktop / 'pvtrecords.csv')readStreamDo: [ :in |
(NeoCSVReader on: in) skipHeader;recordClass: PVTRecord;addField: #identification:;addField: #timestamp: converter: [ :string |
DateAndTime fromString: string ];addFloatField: #longitude:;addFloatField: #latitude:;addIntegerField: #speed:;upToEnd ].
(FileLocator desktop / 'pvtrecords2.csv')writeStreamDo: [ :out |
(NeoCSVWriter on: out)writeHeader: #(id time long lat speed);nextPutAll: ( {
PVTRecord one. PVTRecord two. PVTRecord three } collect: [ :each | { each identification. each timestamp. each longitude. each latitude. each speed } ] ) ].
"id","time","long","lat","speed""v1","2016-03-29T15:21:00+00:00","5.3403835","50.918625","37""v1","2016-03-29T15:22:00+00:00","5.3397698","50.915192","27""v1","2016-03-29T15:23:00+00:00","5.3415751","50.911633","47"
(FileLocator desktop / 'pvtrecords2.csv')writeStreamDo: [ :out |
(NeoCSVWriter on: out)writeHeader: #(id time long lat speed);addFields: #(
identification timestamp longitude latitude speed);
nextPutAll: { PVTRecord one. PVTRecord two. PVTRecord three } ].
Referencehttp://files.pharo.org/books/enterprise-pharo/book/
NeoCSV/NeoCSV.html
JSON
• Simple, universal, partially self describing
• Maps well to Pharo (Dictionary & Array)
• No class info, no complex graphs
[{ "id":"v1", "time":"2016-03-29T15:21:00+00:00", "long":5.3403835, "lat":50.918625, "speed":37},{ "id":"v1", "time":"2016-03-29T15:22:00+00:00", "long":5.3397698, "lat":50.915192, "speed":27},{ "id":"v1", "time":"2016-03-29T15:23:00+00:00", "long":5.3415751, "lat":50.911633, "speed":47}]
(FileLocator desktop / 'pvtrecords.json')readStreamDo: [ :in |
(NeoJSONReader on: in)for: DateAndTime customDo: [ :mapping |
mapping decoder: [ :string | DateAndTime fromString: string ] ];
for: PVTRecord do: [ :mapping | mapping
mapAccessor: #speed;mapAccessor: #identification to: #id;mapAccessor: #longitude to: #long;mapAccessor: #latitude to: #lat.
(mapping mapAccessor: #timestamp to: #time)valueSchema: DateAndTime ];
nextListAs: PVTRecord ].
(FileLocator desktop / 'pvtrecords2.json')writeStreamDo: [ :out |
(NeoJSONWriter on: out)nextPut: {
{#id->#v1. #time->'2016-03-29T15:21:00+00:00'. #long->5.3403835. #lat->50.918625. #speed->27 } asDictionary
} ].
(FileLocator desktop / 'pvtrecords2.json')writeStreamDo: [ :out |
(NeoJSONWriter on: out)prettyPrint: true;newLine: String lf;for: DateAndTime customDo: [ :mapping |
mapping encoder: [ :dateAndTime | dateAndTime asString ] ];
for: PVTRecord do: [ :mapping | mapping
mapAccessor: #speed;mapAccessor: #identification to: #id;mapAccessor: #longitude to: #long;mapAccessor: #latitude to: #lat.
(mapping mapAccessor: #timestamp to: #time) ];nextPut: {
PVTRecord one. PVTRecord two. PVTRecord three };newline ].
[{
"speed" : 37,"long" : 5.3403835,"lat" : 50.918625,"time" : "2016-03-29T15:21:00+00:00","id" : "v1"
},{
"speed" : 27,"long" : 5.3397698,"lat" : 50.915192,"time" : "2016-03-29T15:22:00+00:00","id" : "v1"
},{
"speed" : 47,"long" : 5.3415751,"lat" : 50.911633,"time" : "2016-03-29T15:23:00+00:00","id" : "v1"
}]
Referencehttp://files.pharo.org/books/enterprise-pharo/book/
NeoJSON/NeoJSON.html
<records><pvt> <id>v1</id> <time>2016-03-29T15:21:00+00:00</time> <long>5.3403835</long> <lat>50.918625</lat> <speed>37</speed></pvt><pvt> <id>v1</id> <time>2016-03-29T15:22:00+00:00</time> <long>5.3397698</long> <lat>50.915192</lat> <speed>27</speed></pvt><pvt> <id>v1</id> <time>2016-03-29T15:23:00+00:00</time> <long>5.3415751</long> <lat>50.911633</lat> <speed>47</speed></pvt></records>
((XMLDOMParser parseFileNamed: (FileLocator desktop / 'pvtrecords.xml') fullName)
allElementsNamed: 'pvt')collect: [ :each |
PVTRecord new timestamp: (DateAndTime fromString:
(each contentStringAt: 'time'));identification: (each contentStringAt: 'id');longitude: (each contentStringAt: 'long') asNumber;latitude: (each contentStringAt: 'lat') asNumber;speed: (each contentStringAt: 'speed') asInteger;yourself ]
as: Array.
(FileLocator desktop / 'pvtrecords2.xml')writeStreamDo: [ :out |
| writer |(writer := XMLWriter on: out)
enablePrettyPrinting;lineBreak: String lf;xml.
writer tag: #records with: [{ PVTRecord one. PVTRecord two. PVTRecord three }
do: [ :each | writer tag: #pvt with: [
writer tag: #id with: each identification;tag: #time with: each timestamp asString;tag: #long with: each longitude asString;tag: #lat with: each latitude asString;tag: #speed with: each speed asString ] ] ] ].
<?xml version="1.0"?><records> <pvt> <id>v1</id> <time>2016-03-29T15:21:00+00:00</time> <long>5.3403835</long> <lat>50.918625</lat> <speed>37</speed> </pvt> <pvt> <id>v1</id> <time>2016-03-29T15:22:00+00:00</time> <long>5.3397698</long> <lat>50.915192</lat> <speed>27</speed> </pvt> <pvt> <id>v1</id> <time>2016-03-29T15:23:00+00:00</time> <long>5.3415751</long> <lat>50.911633</lat> <speed>47</speed> </pvt></records>
ZnClient newurl: 'http://easy.t3-platform.net/rest/geo-ip';queryAt: 'address' put: '81.83.7.35';get.
=> '{"latitude" : 50.8333,"address" : "81.83.7.35","country" : "BE","longitude" : 4.0
}'
ZnClient newsystemPolicy;url: 'http://easy.t3-platform.net/rest/geo-ip';queryAt: 'address' put: '81.83.7.35';accept: ZnMimeType applicationJson;contentReader: [ :entity |
STONJSON fromString: entity contents ];get.
Server Side
• Representational State Transfer (REST)
• Plain Zinc (BYO)
• Zinc-REST, Seaside-REST
• Teapot
| client |client := MDBasicClient new.[ client at: 'foo-key' ifAbsentPut: [ 'my-long-query-result' asByteArray ].
] ensure: [ client close ]
MOM
• infrastructure supporting sending & receiving messages between distributed systems
• heterogeneous & decoupled
• asynchronous
| server |server := self stampClient.[ server open. server subscribeTo: 'factorial'. server runWith: [ :message | | number | message body = 'quit' ifTrue: [ ConnectionClosed signal ]. number := message body asInteger. server sendText: number factorial asString to: message replyTo ] ] fork.
| client request |client := self stampClient.client open.request := client newSendFrameTo: 'factorial'.request text: 42 asString.request replyTo: '/temp-queue/factorial'.client write: request.response := client readMessage.self assert: response body equals: 42 factorial asString.client sendText: 'quit' to: 'factorial'.client close.
Finding Formats & Protocols
• The Pharo Catalog
• Spotter
• http://catalog.pharo.org
• Ask on the mailing lists