RAIN RFID - A world of business & consumer connectivity - Real … · 2020-01-04 · RFID Reader...
Transcript of RAIN RFID - A world of business & consumer connectivity - Real … · 2020-01-04 · RFID Reader...
Real-time RAIN Data in the CloudBy: Colin BookmanJune 9th, 2016
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Realtime Inventory
Proprietary + Confidential
70%Of the items seen in inventory
systems are actually where they
claim to be. We've found this to
hold true for retail and
distribution centers.
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Where are my items What do I need to do How do I get this data
Performing quick lookups of what
physical locations have a particular
item, and what items are in a particular
location.
What items need to be moved from
one location to another. How many of
each item needs to be moved, and to
where.
My Current inventory system onlysupports a count, not EPC level data.My inventory system cannot handlemany minor adjustments of inventorydata per second.
Problems Faced
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Key GCP Technology
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Pubsub
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Storage
NoSQLBlob SQL
CloudStorage
CloudBigtable
CloudDatastore Cloud SQL
Good for:Structured and unstructuredbinary or object data
Such as:Images, large media files,backups
Good for:Getting started,App Engine,serve use cases
Such as:User profiles,product catalog
Good for:Web frameworks,existing applications
Such as:User credentials, customerorders
Good for:Heavy read +write, events, andanalytical data
Such as:AdTech, Financialand IoT data
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Bigtable
Read/Write Latency at 99% (ms)
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
BigTable
Proprietary + Confidential
Dataflow
Proprietary + Confidential
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Dataflow
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Dataflow
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Dataflow
Proprietary + Confidential
BigQuery
Proprietary + Confidential
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Kubernetes
kubelet
UI
kubeletCLI
API
users master nodes
etcd
kubelet
scheduler
controllers
apiserver
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Overwhelmed,So am I
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Realtime Inventory
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Low Latency Simple Stupid onsiteequipment
Real Time InventoryIngestion & Aggregation
Ideally takes server less than 100ms
to prepare a response to an api
request
Equipment will be in remote hard to
get places. Keeping it simple stupid
makes support easier.
Inventory data should be ingested bythe system to as close of real-time aspossible. Ideally no cron jobs tocalculate inventory counts.
Technical Requirements
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
http://www.dailyhaha.com/_pics/failure-dog.htm
Failures
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Attempt #1, Firebase
RFIDReader
● Fixed readers all over the environment
● Readers performing a Session 1 Target AB read
Ingestion ● RPI ingests RFID data, and pushes it to Firebase
Application ● Webapp queries Firebase for all the EPC tags and calculates item counts
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Low Latency Simple Stupid onsiteequipment
Real Time InventoryIngestion & Aggregation
Attempt #1, Firebase
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Attempt #2, MySQL
RFIDReader
● Fixed readers all over the environment
● Readers performing a synchronized Session 1 Target AB Session 3 Target A read
Ingestion ● RPI ingests RFID data, and pushes it to Firebase AppEngine
● AppEngine writes each tag into MySQL
Application ● Webapp queries Firebase AppEngine for EPC tags Gtin14 Counts
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Attempt #2, MySQL
EPC Location Gtin14 LastSeen
3018E511C735D2C0000000EA MTV.CYD1.Hanger1 12345678901231 2016-06-09T01:00:00+00:00
3018E511C735D2C0000000EB MTV.CYD1.Hanger1 12345678901231 2016-06-09T01:00:00+00:00
3018E511C735D2C0000000EC MTV.CYD1.Hanger3 12345678901231 2016-06-09T01:00:00+00:00
3018E511C735D2C0000000ED MTV.CYD1.Hanger2 12345678901231 2016-06-09T01:00:00+00:00
MySQL Schema
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Low Latency Simple Stupid onsiteequipment
Real Time InventoryIngestion & Aggregation
Attempt #2, MySQL
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Attempt #3, Datastore (NoSQL)
RFIDReader
● Fixed readers / Phased Arrays all over the environment
● Readers performing a Session 3 target A read
Ingestion ● RPI ingests RFID data, and pushes it to AppEngine Pubsub
● AppEngine pulls pubsub messages and writes each tag into MySQL Datastore
Application ● Webapp queries AppEngine for Gtin14 Counts
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Attempt #3, Datastore (NoSQL)
Key Tags LastUpdated
<ReaderId> EPCSGTIN96[] <UTC Time>
Google.MTV.Foo [3018E511C735D2C0000000EB,
3018E511C735D2C0000000EC,
3018E511C735D2C0000000E0]
2016-06-
09T01:00:00+00:00
Raw RFID Tag Documents
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Attempt #3, Datastore (NoSQL)
Key Count LastUpdated
<Location>.<Gtin14> <Integer> <UTC Time>
Google.MTV.Foo.12345678901231 10 2016-06-
09T01:00:00+00:00
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Attempt #3, Datastore (NoSQL)
Low Latency Simple Stupid onsiteequipment
Real Time InventoryIngestion & Aggregation
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
http://www.dailyhaha.com/_pics/failure-dog.htm
What worked
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Attempt #4, Dataflow + Datastore + BigQuery
RFIDReader
● Fixed readers / Phased Arrays all over the environment
● Readers performing a Session 3 target A, followed by a shorter Session 3 Target B read
Ingestion ● RFID Reader ingests RFID data, and pushes it to Pubsub
● A sliding window dataflow job processes tags and outputs to a pubsub topic `inventory-
aggregation` aggregation counts, which app engine ingests and pushes to datastore.
● Streaming inserts to bigquery also process the raw rfid tag stream
Application ● Webapp queries AppEngine for Gtin14 Counts
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Dataflow + Datastore + BigQuery
RFID ReaderCapture RFID Data
Physical Location
Cloud Pub/SubStreaming Changes
Messaging
Cloud DataflowAggregation
Processing
Cloud DataStoreInventory Store
Events
Cloud BigQueryData Warehouse
Analytics
App EngineInventory APIs
Services
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Attempt #4, Dataflow + Datastore + BigQuery
Low Latency Simple Stupid onsiteequipment
Real Time InventoryIngestion & Aggregation
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
http://www.dailyhaha.com/_pics/failure-dog.htm
Future Plans
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
RFIDReader
● Fixed readers / Phased Arrays / Mobile Readers all over the environment
● Fixed Readers performing a Session 3 target A, followed by a shorter Session 3 Target B read
Ingestion ● RFID Reader ingests RFID data, and pushes it to Pubsub
● A dataflow job pushes the raw RFID data to bigtable and bigquery. It also analyzes the rfid data
on a sliding window and writes the results to bigtable and bigquery.
Application ● Webapp queries AppEngine Managed VMs for Gtin14 Counts
Dataflow + BigTable + BigQuery
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Dataflow + BigTable + BigQuery
RFID ReaderCapture RFID Data
Physical Location
Cloud Pub/SubStreaming Changes
Messaging
Cloud DataflowInventory Streams
Processing
Cloud BigTableInventory Store
Events
Cloud BigQueryData Warehouse
Analytics
App EngineInventory APIs
Services
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Dataflow + BigTable + BigQuery
Row Key EPC: POSCounts
<Location>|<Grouping>.<Gtin14> <EPCSgtin96> 3019C56A40A1288000000002 Square
LastSeen
Latitude
Longitude
ReaderType
lastseen: 2016-06-09T01:00:00+00:00
latitude: 37.422428
longitude: -122.084329
readerType: fixed
25
BigTable Schema
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Dataflow + BigTable + BigQuery
Low Latency Simple Stupid onsiteequipment
Real Time InventoryIngestion & Aggregation
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
http://cdn-d4d.kxcdn.com/wp-content/uploads/2015/02/4.jpg
Security
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Security
PKI Each RFID Reader has its own Public and Private key pair
TLS All of our cloud endpoints use http over TLS
IAM Cloud services use Identity Access Management tolimit access to minimal permission set
Logs Store logs for post mortem analysis
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
http://treasure.diylol.com/uploads/post/image/222172/resized_the-most-interesting-man-in-the-world-meme-generator-i-have-a-question-for-you-what-do-you-want-to-ask-me-46a76c.jpg
Questions