Intelligent Machines with MongoDB

48
BRÜCKNER MASCHINENBAU STRETCHING THE LIMITS http://www.brueckner.com/ http://tiny.cc/brueckner

description

Smart Factories with M2M communications is the main goal of the fourth industrial revolution. The “Industrial Internet” as it is called by GE or the “Industry 4.0” project by the German government revolve around Big Data and Analytics. The machine building industry faces challenges in emerging markets such as high energy costs and a shortage of skilled staff, which drive the need for highly-optimized and self-optimizing machines. We will use the example of a world leader in this sector to describe how we built a system for real-time analytics of machine sensor data, using MongoDB as the heart of an easily scalable and decentralized storage system for operational data. Customers of Brückner Maschinenbau GmbH use the system to gather data from their production lines, monitor the quality of their products and optimize their production-processes with prediction analyses and deep production insights through machine learning algorithms.

Transcript of Intelligent Machines with MongoDB

Page 1: Intelligent Machines with MongoDB

BRÜCKNER MASCHINENBAU

STRETCHING THE LIMITShttp://www.brueckner.com/http://tiny.cc/brueckner

Page 2: Intelligent Machines with MongoDB

Team Leader Software Development at Brückner Maschinenbau

@mkappeller

AboutMatthias Kappeller

Page 3: Intelligent Machines with MongoDB

Chef de Cuisine at MongoSoup

@loomit

AboutJohannes Brandstetter

Page 4: Intelligent Machines with MongoDB

We Build the Largest Lines in the IndustryBOPP 10.4m (35ft) – winder

© Brückner Maschinenbau 4

Page 5: Intelligent Machines with MongoDB

We Build the Fastest Lines in the IndustryTDO of a 8.7 m BOPP line – 525 m/min production speed

© Brückner Maschinenbau 5

Page 6: Intelligent Machines with MongoDB

Are You in Packaging Films?

© Brückner Maschinenbau 6

Page 7: Intelligent Machines with MongoDB

Are You in Technical Films?

© Brückner Maschinenbau 7

Page 8: Intelligent Machines with MongoDB

How we deploy our products

© Brückner Maschinenbau 8

Page 9: Intelligent Machines with MongoDB

010010100111 010101101101 10110101 1011

Sensor Data

Page 10: Intelligent Machines with MongoDB

Temperature

Speed

Pressure

Thickness

Density

Alarms

Sensor-Status

~100'000 Datapoints per Line1-8 lines per customer

Page 11: Intelligent Machines with MongoDB

~100000 data points in total ~4000 variables are high frequency

Sensor Data

Page 12: Intelligent Machines with MongoDB

Sensor DataFrequency of incoming data at 5-10Hz (100-200ms)

http://www.fantom-xp.com/en_19__Intel_Core_i7_high_speed.html

Page 13: Intelligent Machines with MongoDB

Sensor DataTime Series Graph Track Single Parameter

© Brückner Maschinenbau 13

Page 14: Intelligent Machines with MongoDB

Document for every product of a campaign Quality Profile Thickness Scan Scan speed: 300-500 m/min Analyzed via Heatmap

Scanner Data

Page 15: Intelligent Machines with MongoDB

Difficult and time-consuming configuration and setup Typical ETL (information loss) Many Stored-Procedures Low performance Low availability Outdated UI-Technology (Delphi) Proprietary PLC-Driver (to read sensor data)

Our current approach and its problems

© Brückner Maschinenbau 15

Page 16: Intelligent Machines with MongoDB

High performance

• Write> 3'000 updates / sec

• Read> 2 queries / sec

Scalability

Low System-Complexity

Near Future Goals

Page 17: Intelligent Machines with MongoDB

Intelligent Line

Management Cockpits

Smart Recipes

Far Future Goals

Page 18: Intelligent Machines with MongoDB

OEM Customer has no IT-Administration or low IT-KnowHow Low-Cost Server Infrastructure Highly scalable Server infrastructure Bad network connection Many power shutdowns Production 24/7

• High availability• Complex system update

Challenges

© Brückner Maschinenbau 18

Page 19: Intelligent Machines with MongoDB

PoC

http://farm8.static.flickr.com/7396/8718123610_09e70f6d90.jpg

Page 20: Intelligent Machines with MongoDB

MongoDB

Page 21: Intelligent Machines with MongoDB

MongoDB Hosting from Germany

Bring your own data center

Customer-focused solutions

Page 22: Intelligent Machines with MongoDB

Web based management toolsReal-time visualizationReal-time loggingSSL connector

Page 23: Intelligent Machines with MongoDB
Page 24: Intelligent Machines with MongoDB

Schema free Ease and speed of development Ease of operations Realtime analysis of streaming data Possibility to add Hadoop for heavier Analytics later

Reasons for choosing MongoDB

Page 25: Intelligent Machines with MongoDB

Architecture Overview (simplified)

OPC

REST

Data Input

Streaming

MongoDB

MetaData

Service

• Scan / Lab Data

• Machine Configuration

• Recipe

Streaming Data

Document Data

Streaming Data

Hadoop

Analytics

Page 26: Intelligent Machines with MongoDB

Schema Design

Time Series Schema Bucket Schema “Simple Schema” : one document per event

Page 27: Intelligent Machines with MongoDB

Schema Design

http://blog.mongodb.org/post/65517193370/schema-design-for-time-series-data-in-mongodb

Page 28: Intelligent Machines with MongoDB

{ _id: {timestamp_hour: ISODate("2013-10-10T23:00:00.000Z"), type: “velocity_m1”},

values: { 0: { 0: 93, 1: 91, …, 59: 95 }, 1: { 0: 95, 1: 89, …, 59: 87 }, …, 59: { 0: 91, 1: 90, …, 59: 92 } }

}db.metrics.update(

{ _id.timestamp_hour: ISODate("2013-10-10T23:00:00.000Z"), _id.type: “velocity_m1” }, {$set: {“values.59.59”: 92 } })

Schema Design “time series”

Page 29: Intelligent Machines with MongoDB

Preallocate space with empty documents for upcoming time periods -> in-place updates

Primary Index: timestamp Good fit for streaming data But lots of sparse documents in our scenario

Schema Design “time series” schema: Performance

Page 30: Intelligent Machines with MongoDB

Schema Design “time series”

http://www.culatools.com/wp-content/uploads/2011/09/sparse_matrix.png

Page 31: Intelligent Machines with MongoDB

Schema design for time series data well understood Sensor Data is not time series data Sensors have own thresholds

Schema Design

Page 32: Intelligent Machines with MongoDB

Schema Design “bucket” schema

http://flickr.com/photos/99255685@N00/2063575447

Page 33: Intelligent Machines with MongoDB

“A bucket is most commonly a type of data buffer or a type of document in which data is divided into regions.“http://en.wikipedia.org/wiki/Bucket_(computing)

Schema Design “bucket” schema

Page 34: Intelligent Machines with MongoDB

Schema Design “bucket” schema

Have one bucket per sensor type

Fill up with values till it‘s full Use next bucket

Page 35: Intelligent Machines with MongoDB

{ _id: {timestamp_start:ISODate("2013-10-10T23:00:00.000Z"), type:“velocity_m1”

}, state: “OPEN”,

values: [ { t: 456, v: 93 }, { t: 572, v: 89 }, …, { t: 2344, v: 92 }

{ t: -1, v: 0 }, ...] }db.metrics.update( {_id.timestamp_start: ISODate("2013-10-10T23:00:00.000Z"), _id.type: “velocity_m1” }, {$set: {“values[761]”: {t: 2500, v: 90}})

Schema Design “bucket” schema

Page 36: Intelligent Machines with MongoDB

Preallocate space with empty documents for upcoming data count -> in-place updates

Different bucket sizes for different sensors Still good for range based queries Have to keep counter in app to determine bucket state Good fit for sparse data

Schema Design “bucket” schema: Performance

Page 37: Intelligent Machines with MongoDB

Bucket Schema vs. Simple Schema

Fragmentation•disk and memory will get fragmented over time•even more so the smaller the documents are

Page 38: Intelligent Machines with MongoDB

Update driven vs. insert driven• individual writes are smaller• performance and concurrency benefits

Performance

Page 39: Intelligent Machines with MongoDB

Index Size• Bucket decreases index size by factor of number of documents it

holds• Index should fit in RAM + working set• Index updates cause locks and I/O

Bucket Model vs. Simple Schema

Page 40: Intelligent Machines with MongoDB

Read Performance• Simple Schema: Document per event: 3600 reads• Bucket Schema: Document per minute: 60 reads

Read performance is greatly improved Optimal with tuned block sizes and read ahead

• Fewer disk seeks• Fewer random I/O

Bucket Model vs. Simple Schema

Page 41: Intelligent Machines with MongoDB

Hardware Challenges

Growth of data Data retention HW belongs to customer, can’t upgrade Support for legacy MongoDB versions

Page 42: Intelligent Machines with MongoDB

Hardware Overview

Page 43: Intelligent Machines with MongoDB

SSD vs. HDD

SSD

HDD

Page 44: Intelligent Machines with MongoDB

Schema design matters Increase performance:

• In-Memory caching• Concurrency• Queue• Core-Sharding

Flexible and scalable system• We can build a system with low complexity for simple use cases• We can provide a system for „bigger“ use cases by increasing

complexity

Lessons learned

Page 45: Intelligent Machines with MongoDB

Conclusion

Development of appliance like systems can be challenging Ease of use Resilience

Page 46: Intelligent Machines with MongoDB

Where we are now

PoC and its results approved by management Workout / Design the UI/UX Develop the software system

Ship prototypes to some sites Explore and develop analytic-algorithms Fieldtest for our software system with MongoDB First machine with this solution to be deployed in 2016

Page 47: Intelligent Machines with MongoDB

Questions?

Don’t forget to grab a free MongoSoup T-Shirt

Page 48: Intelligent Machines with MongoDB

@mongosoup

www.mongosoup.de