DEVNET-2002Coding 201: Coding Skills 201: Going Further with REST and Python: JSON Parsing

Coding 201 Parsing JSON in Python

Mike Maas, Technical Evangelist – IoE, DevNet, @mike_maas

© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public Presentation ID

•  Introductions •  Overview

•  Requirements

•  Tools

•  Requesting Content

•  Parsing Content

•  Resources

Agenda

3


Introductions •  Who Am I?

•  Mike Maas, Technical Evangelist – Internet of Everything, DevNet •  Based out of Seattle, Washington

•  Why this session? Why Python? •  It is a great way to learn programming skills

•  http://cacm.acm.org/blogs/blog-cacm/176450-python-is-now-the-most-popular-introductory-teaching-language-at-top-us-universities/fulltext

•  It is a very flexible and quick "glue" language •  Lots of bindings and existing packaged functionality •  Ported to lots and lots of architectures/platforms

4


Overview

5


Overview •  Goals

•  Build upon Coding 101 •  Deeper understanding of using Python to collect information from the network •  Further inspection of JSON and the tools available to evaluate JSON content •  Understand the methods and implementation of parsing JSON

•  What we won't be covering •  Authentication

•  Oauth, Digest/Form/etc.


Overview •  What we are going to work on

•  Using Postman •  Learning a little bit about CMX •  Understanding how to use Python to parse data •  Understanding some of the challenges of parsing data •  Seeing some of the errors you might encounter


Requirements

8


Requirements: Tools •  A web browser (Chrome)

•  http://getchrome.com

•  Python 3.4+ •  https://www.python.org/downloads/release/python-342/

•  Text Editor (Or you can use IDLE installed by Python) •  Git (Optional)

•  http://git-scm.com/

•  Postman (Optional) •  Get from the Chrome Web Store or the standalone version at

http://www.getpostman.com/


Requirements: Expertise •  Basic computer skills / Accessing web sites

•  Using the command-line for your Operating System

•  Some skills in scripting or computer automation


Requirements

•  Install Python 3.4 if you don't have it already installed •  https://www.python.org/downloads/release/python-342/

•  Open a console on your Operating System •  Windows

•  Start > Run… > cmd •  OS X

•  Spotlight > Terminal •  Type python –-version

•  We should get "Python 3.4.2" •  If we don't – that is OK

•  On OS X : Install Python 3.4.2 and just run command on OS X as python3 •  On Windows: Install Python 3.4.2 and modify your Path •  On Linux: I will assume you know what you are doing and modify your aliases/exports as needed

Pre-Flight Check: Python


Requirements

•  Install Postman if you don't already have it installed •  This isn't required but it is a very useful tool •  Available at http://www.getpostman.com/ •  If you are used to a different tool (SoapUI, Chrome DevTools, etc.) – Feel free

•  Browser-based version or Standalone packaged application is fine

Pre-Flight Check: Postman


Requirements: Code

13

•  GitHub •  https://github.com/CiscoDevNet/coding-skills-sample-code

•  coding202-parsing-json

•  To Clone locally run the following command: •  git clone https://github.com/CiscoDevNet/coding-skills-sample-code.git

All the samples used are available online


Optional: Tools

14

•  JSON Test •  http://www.jsontest.com/

•  JSON Lint •  http://jsonlint.com/

•  JSON Placeholder •  http://jsonplaceholder.typicode.com/

Other online resources for our foray into JSON parsing


Requesting Content

15


Request and Response

•  Request •  What you want •  Parameters

•  Response results can often be controlled via Request elements •  The URI itself or URL query params •  Form-based transactions •  HTTP Headers •  Pseudo Extensions (https://stream.twitter.com/1.1/statuses/sample.json)

•  Response •  What you get •  Content

•  Can often return the same data in different formats per request •  XML, JSON, ATOM, …

What you want; what you get


HTTP

•  What does HTTP stand for? (HT) & (TP) •  What is the most relevant part of the definition for us today?

•  Headers •  Authorization •  Accept

•  Charset, Encoding, Language •  Actions

•  GET – Actually get content •  POST – Create new content •  DELETE – Delete content •  PUT – Update content

•  Content Description •  MIME Types •  For HTTP the content is negotiated

A brief word about HTTP


Requesting Content via Postman

18

•  From Postman •  In the Request field enter our test URL

•  http://jsonplaceholder.typicode.com/users •  Click Send

•  From a Browser •  Just put http://jsonplaceholder.typicode.com/users in as the URL

•  We should get some JSON returned

Let’s get something to look at

http://jsonplaceholder.typicode.com/users http://jsonplaceholder.typicode.com/todos


Understanding the structure of the Response

[

{

"id" : 1,

"name": "Leanne Graham"

}, { "id": 2, "name": "Ervin Howell" }

]

Let's look at this in JSON Lint and then JSON.org.

Let’s look at what we got (Condensed)



[ (Array)

{ (Object)

"id" : 1, (Value)

"name": "Leanne Graham" (Value)

}, { (Object) "id": 2, (Value) "name": "Ervin Howell" (Value) }

]

Let’s look at what we got (Condensed)



•  In Postman •  In the Request field enter our test URL

•  https://msesandbox.cisco.com/api/contextaware/v1/maps •  Click Send – we should get an error

•  Need to add Authentication and Accept headers •  Again, we should get some data returned

Let’s look at something else… Cisco CMX!

https://msesandbox.cisco.com/api/contextaware/v1/maps



{

"Maps": {

"Campus": [

{

"name": "DevNetCampus",

"Building": [

{

"name": "DevNetBuilding",

"Floor": [

{

"name": "DevNetZone"

}

]

}

]

}

]

}

}

Let’s look at what we got from CMX (Condensed)


Activity: Requesting Content


Requesting Content via Python

•  Get Code Via Git •  Git clone the following command:

•  git clone https://github.com/CiscoDevNet/coding-skills-sample-code.git

Let’s get something to look at in our console

•  Get Code Via Download •  Download the following file:

•  https://github.com/CiscoDevNet/coding-skills-sample-code/archive/master.zip

•  When you get a coding-skills-sample-code directory/folder •  Launch a terminal and navigate to coding-skills-sample-code •  Locate the coding202-parsing-json directory/folder •  Pass the get-cmx-json.py file to the Python interpreter •  Execute python get-cmx-json.py

•  You may need to run python3 get-cmx-json.py on OS X



{ (Object)

"Maps": { (Object)

"Campus": [ (Array)

{ (Object)

"name": "DevNetCampus", (Value)

"Building": [ (Array)

{ (Object)

"name": "DevNetBuilding", (Value)

"Floor": [(Array)

{ (Object)

"name": "DevNetZone" (Value)

}

]

}

]

}

]

}

}

Let’s look at what we got from Python (Condensed) - python get-cmx-json.py


Parsing Content

26


•  Why Parse? •  Methods of Parsing

•  Don't Parse •  Output is truly human-readable

•  Do-It-Yourself (DIY) String Parsing •  Useful for really, really simple string extraction

•  Build or Use a Framework •  A lot of ready-made libraries and tools are already available

Parsing Content

27


•  Available Libraries •  json (Python Standard Library) •  simplejson •  ultrajson •  …

•  JSON.org has lists of utilities at http://json.org/

Methods of Parsing Content

28


Parsing and Extraction

•  Understand •  The Content

•  What it represents – A user? A book? A log entry? A data sample? An access point? •  Schema – What is the order, form, and requirements of the data? •  Size – How much content will you get back? And will that be a problem?

•  What you need •  Names and Email Addresses, X and Y coordinates, Patient ID numbers

•  How much you need •  Do you need a count of items, a representative sample, or everything?

•  Be aware that there is sometimes more than one path to the content… •  Different web libraries (urllib, urllib2, Requests, etc.) •  Different JSON parsers (JSON, SimpleJSON, UltraJSON, etc.)

Syntactic analysis and processing of the result


Introduction to the code

•  Python syntax •  Indentation

•  Python code blocks do not have explicit statement scope delimiters •  Indentation amount is relative

•  Variables •  Assignment •  Arrays •  Statements

•  Import statements •  Print statements •  Conditional statements •  Looping constructs

Visit https://docs.python.org/3/reference/index.html for the complete docs

Lets orient ourselves to Python and how it is used in the example code


import json from urllib.request import Request, urlopen

string_variable = "A String"

number_variable = 7

if life > death:

print('Still Alive!')

def foo():

print('bar')

jsonContent = json.loads(contents)

for apple in basket:

wash(apple)

Import Statements

Variable Assignment

Method Invocation

Looping

Conditional Statements

Method Definition


opened_url = urllib2.urlopen(request)

content = opened_url.read()

- is functionally equivalent to the following -

content = urllib2.urlopen(request).read()

an_array = {'first element', 'second element', 'third element'}

element = an_array[1]

some_value = some_matrix[0][2]

buildings_array['Building']['Floor']

for entry in jsonContent['Device']['entries']:

if entry == "Device007":

print(entry['macAddress'])

Method Chaining

Arrays

Multidimensional Arrays

Compound Statements

Multidimensional Dictionaries


Activity: Parse and Extract

33



•  Open activity-parse-1.py

•  Review the Code •  This is where it is important to know what the content looks like •  Access an Array / Output via Print •  Let's get the location of "Chelsey Dietrich"

•  Run the code with python activity-parse-1.py (or use python3)

Extract a single piece of information




•  Review the Code •  More content, More complications •  Looping through Collections •  Access an Array / Output via Print


Extract multiple pieces of information




•  Review the Code •  Keep what we want and show it how we want to •  Looping through Collections •  Storing Variables •  Formatted output via Print


Extract and combine multiple pieces of information




•  Review the Code •  Storing relevant information •  Python arrays •  Querying a source (same or different)


Extract and combine multiple pieces of information forming a new Request


Demo: IOx

38


•  IOx •  Edge Computing

•  Python on the Edge

Demonstration

39


Resources


Resources

•  On-Site •  Learning Labs

•  https://learninglabs.cisco.com •  DevNetizens

Information to help you on your way


Resources

•  On-Line •  Learning Labs

•  https://learninglabs.cisco.com •  Python

•  http://python.org •  JSON : JSON.org

•  JSON Test : http://www.jsontest.com/ •  JSON Lint : http://jsonlint.com/ •  JSON Placeholder : http://jsonplaceholder.typicode.com/

Information to help you on your way


Debugging


Debugging

•  What is Debugging?

•  Python is an interpreted language •  Although it can be compiled •  Weakly Typed

•  Approaches to Debugging •  Print •  PDB

•  Breakpoints •  Try something else

•  It is difficult to trust your own code sometimes – try a different tool

High-level concepts for a Low-level topic


Debugging

•  Detecting Problems •  It blows up •  It doesn't do what you expect it to •  Nothing seems to happen at all

•  This can be a bit more difficult to isolate as it could be a performance issue (network, disk, etc.)

•  Traceback •  A call stack that shows you the path of execution

•  Python Lint •  PIP, pychecker, pylint, pep8, etc.

How do we know something went wrong?


Issues Requesting Data via Python

mikemaas$ python getcontent.py File "getcontent.py", line 1 bad_import json ^ SyntaxError: invalid syntax

An error in syntax



mikemaas$ python3.4 getcontent.py File "getcontent.py", line 11 print entry['macAddress']

^

SyntaxError: Missing parentheses in call to 'print'

Version-specific Errors



mikemaas$ python getcontent.py Traceback (most recent call last): File "getcontent.py", line 6, in <module> contents = urllib2.urlopen(request).read() File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 127, in urlopen return _opener.open(url, data, timeout) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 404, in open response = self._open(req, data) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 422, in _open '_open', req) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 382, in _call_chain result = func(*args) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1222, in https_open return self.do_open(httplib.HTTPSConnection, req) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1184, in do_open raise URLError(err) urllib2.URLError: <urlopen error [Errno 51] Network is unreachable>

No Network


Activity: Debug


Activity: Debug

•  Open activity-debug-1.py

•  Review the Code •  Taking control of execution •  Looking at things as they execute

•  Run the code •  python -m pdb activity-debug-1.py

Step through a script


Activity: Debug

•  b(reak) Set a breakpoint •  c(ontinue) Continue debugging until you hit a breakpoint •  d(own) Navigate down a stack frame •  l(ist) List the source code for the current file •  n(ext) Go to the next line of code •  p(print) Print an expression value •  r(eturn) Continue execution from function return •  s(tep) Step through the code •  u(p) Navigate up a stack frame

Debugger Commands - https://docs.python.org/2/library/pdb.html

DEVNET-2002Coding 201: Coding Skills 201: Going Further with REST and Python: JSON Parsing

Technology

Transcript of DEVNET-2002Coding 201: Coding Skills 201: Going Further with REST and Python: JSON Parsing