Af11psug Cs05 Bama-companies
-
Upload
04422236558 -
Category
Documents
-
view
216 -
download
0
Transcript of Af11psug Cs05 Bama-companies
-
8/13/2019 Af11psug Cs05 Bama-companies
1/22
Copyright 2011 Rockwell Automation, Inc. All rights reserved.
Insert Photo Here
Rockwell AutomationProcess Solutions User Group (PSUG)
November 14-15, 2011
Chicago, IL McCormick Place West
A PlantPAx Virtual
Batch Upgrade
Sean Nichols
Controls Engineering Lead
The Bama Companies
-
8/13/2019 Af11psug Cs05 Bama-companies
2/22
The Bama Companies
History: Founded in the early 20th century
Privately held
All US production based in Tulsa, Oklahoma
Inclusive environment (all stakeholders at every level
involved)
Products made include:
Biscuits
Pizza dough
Hand held pastries
Pie crusts
Breadsticks
Copyright 2011 Rockwell Automation, Inc. All rights reserved.
-
8/13/2019 Af11psug Cs05 Bama-companies
3/22
Sean Nichols Bio
Educational
Nuclear power (Navy)
Electrical Engineering from OSU
Extensive Rockwell training
Occupational
5 years with the Naval NPTU
7 years with Bama Companies
5 years maintenance
2 years as Corporate Controls Engineer
Current responsibilities include plant floor controls
and visualization for all Tulsa based plant
locations Personal
Married, 3 kids, one on the way; live in Tulsa,
Oklahoma
Interest include: Family activities, computers
-
8/13/2019 Af11psug Cs05 Bama-companies
4/22
Process Description
Freezer
Packaging
Ingredients
Mixing
Sheeting Line
-
8/13/2019 Af11psug Cs05 Bama-companies
5/22
Goal: Address Challenges with Frozen Dough
Pre Improvement State: System could get out of sequence
Had difficulty recovering from power outages
Legacy technology (PLC-5 and RSView32 based platform) Maintenance and troubleshooting had to occur at a Ladder Logic level
Changed programing on average once a day (Bama-tized)
Constantly tweaking to try and make process run better or make recipe changes
Desired Improved State: Environmental and UPS considerations to make system more reliable Improvements for recipe security and operator tracking
S88 based programming (modular equipment/phase coding and flexible recipes)
Complete documentation sets around functional operations
VMware and thin client architecture ( Jointly driven by IT and Engineering ) Electronic hand adds and other SOPs (goal to eliminate batch sheets)
Electronic lot tracking to meet regulatory compliance
Whatever chosen would be planned at other facilities
-
8/13/2019 Af11psug Cs05 Bama-companies
6/22
Solutions Evaluated
All technology was to remain Allen-Bradley based
Installed base and familiarity with products
Three independent implementations reviewed:1) PC based batch manager integrated with ControlLogix; delivered by a System
Integrator
- Recipe configuration, Campaign management, Batch reporting
2) ConrolLogix and RSView32 system custom created for Bama; delivered by SystemIntegrator
- Lowest HW/SW investment
- Already familiar with the process
- Concerns around bugs; (Bama only knowledge)
3) PlantPAx (FTBatch, FTView, ControlLogix) system delivered by Rockwell Automation
- Recognized Industry leader (Price Premium)
- Dedicated Food and Beverage group
- Off the shelf application
Copyright 2011 Rockwell Automation, Inc. All rights reserved.
-
8/13/2019 Af11psug Cs05 Bama-companies
7/22
Vendor Selected
Copyright 2011 Rockwell Automation, Inc. All rights reserved.
PlantPAx (FTBatch, FTView, ControlLogix) system delivered by Rockwell
Automation
Only one to formalize/organize a User Requirements Spec
(Was a billable activity but of value)
System interruption was designed to go into a Hold state
Power outages and brownouts are common at our facilities
Other systems would end up scrapping batches
Flexible recipe management
True S88 compliant offering
-
8/13/2019 Af11psug Cs05 Bama-companies
8/22
Delivery Team
Bama Foods
Sean Nichols:
Project Manager & EngineeringLead
Nick Huddleston
Process Improvement Mngr.
Operations Lead
John Blaylock
Networking and SW Installs
Additional Bama Staff SQL integration team
4-5 Automation Techs
4-5 Operator Blackhats
Executive Sponsorship (Budget rework)
Rockwell Automation Solutions
Lynn Hecathorn:
Project Manager Dwayne McAlexander:
Lead Engineer
David Keen:
HMI Implementation
Jed Hepfner:
Field Support, Training, Testing,
Startup support
Copyright 2011 Rockwell Automation, Inc. All rights reserved.
Industrial Controls of Ok.
Troy Bridenstine:
Electrical Foreman
-
8/13/2019 Af11psug Cs05 Bama-companies
9/22
Project Execution: Foundation Work
User Requirements Specification and Proposal
Interview and captured voice/concerns from IT, Operations (Plant and Production),
Maintenance, and Quality
Allowed for project to be on fast track Knowledge of third party expert (and implementer) was trusted in design
Justified a network audit prior to startup
Contract Negotiations (Good Faith Start)
Mutual investment in creating a sustainable relationship
Required the same engineering lead to design and implement (to the best extent
possible)
Functional Specification Mapped out entire process (Blueprint)
Completed and approved before design/coding started
9 revisions by Bama lead operators, engineering, maintenance (inclusivity)
Contractually binds the deliverables and acceptance criteriaCopyright 2011 Rockwell Automation, Inc. All rights reserved.
-
8/13/2019 Af11psug Cs05 Bama-companies
10/22
Project Execution: System Topology
Copyright 2011 Rockwell Automation, Inc. All rights reserved.
19 Logical Racks 500+ Control Modules
9 Weigh Scales
22 Batch Units 105 Phases
25 Active Recipes
1 ESX Server FTView SE VM
FTBatch VM
SQL VM
Citrix Published Apps
10 Thin Clients
Redundant HardDrives
Failover Scheme
-
8/13/2019 Af11psug Cs05 Bama-companies
11/22
Project Execution: Hardware Design
ControlLogix L74 Controller Supports existing Control Modules plus room
for spare w/ redundancy
Primarily Ladder Logic (standardprogramming language for Bama)
Limited Function Block used for certaincapabilities
Reuse as much RIO as possible
(2)1756 Remote I/O Scanners
1771 Racks
Weigh scales
Local PLC (rack 0) replaced with ASB
-
8/13/2019 Af11psug Cs05 Bama-companies
12/22
Project Execution: Hardware Design
Added new FLEX I/O to Ethernet
Multiverter (hardware that was not previously
automated) Added Operator Interfaces for Hand Adds
Designed to be PanelView;
Switched to thin client to becompatible with Windows
Domain Servers (required
for tracking scheme)
-
8/13/2019 Af11psug Cs05 Bama-companies
13/22
Project Execution: Software Design
Copyright 2011 Rockwell Automation, Inc. All rights reserved.
FTView SE 250 Screen License
RA standard GEMS Library version C2 (~6 module classes)
Event logs from each client and alarm logs are dumped to a SQL database through anODBC connection
FTBatch 22 Units
Incorporated all Hand Adds, Equipment Checks, and SOPs through operator prompting Batch journal is dumped to a SQL database through an ODBC connection
All recipes created by Bama
Found to be much more flexible then previous system
Plans to talk with Product group about ideas for future enhancements
Ultimate recipe and procedural edits will be owned by Quality and Process Improvement
SQL Database Existing System prior to upgrade
Uses Business objects (our ERP)
Data table work required but ultimately working with our existing reports
-
8/13/2019 Af11psug Cs05 Bama-companies
14/22
Project Execution: Virtual Configuration &
Deployment
Hardware:
Dual ESX servers; Hard drive SAN controllers connected by dual fibers (14 HDspartitioned as 2 arrays)
One HD in each array is a hot spare (Standard RAID technology) Each ESX server has a 1GB switch with dual connections
Redundant fiber to core switch
Off site backup process for all images
Software
System originally built for existing IT/Business infrastructure
Process system added with minimal changes
New VLAN and a few switches standard growth
Not completely new concept (running RSView32 Virtually for 3 years)
Ten (10) Thin clients being managed by single terminal server image Install once, use multiple, replace easily
Only ini file needs to be updated with MAC ID of new hardware
Three (3) virtual servers dedicated to the PlantPAx system
Fourth virtual server (SQL) offsite Three (3) Citrix Applications
-
8/13/2019 Af11psug Cs05 Bama-companies
15/22
Project Execution: Control Code
Implementation
RA standard GEMS Control Modules version C2
Phase Manager using a sequencer that can be controlled internally or
externally (operators can walk through an entire batch manually if need be)
Supports strutt (scheduled checks/test of equipment)
Needed additional units to insure appropriate arbitration
Comments can be added at any point to the batch record
Broke recipes into a wet mix and mixer class (3 mixers)
Batch ID transfers automatically
Does a material check to make sure product compatibility
Can run multiple products in unison
Copyright 2011 Rockwell Automation, Inc. All rights reserved.
-
8/13/2019 Af11psug Cs05 Bama-companies
16/22
Control Module Checkout
Copyright 2011 Rockwell Automation, Inc. All rights reserved.
Purpose to verify that all GEMS control modules used were compatible with
the 1771 I/O and that the ControlLogix controller can communicate with all
the racks
This process requires downtime as it temporarily puts the new hardware
topography in place and test the system to manage the risk of a swap over
During our specific checkout:
4 racks visible to the PLC5 but invisible to the ControlLogix until we fixed one dipswitch setting and repaired a damaged wire
Found the weigh scales were wired for DH+ instead of RIO
This was a lifesaver for startup because it eliminated hurdles that would
have existed otherwise
-
8/13/2019 Af11psug Cs05 Bama-companies
17/22
-
8/13/2019 Af11psug Cs05 Bama-companies
18/22
Startup and Commissioning
Copyright 2011 Rockwell Automation, Inc. All rights reserved.
Application Training: Valuable but could have been better
No one was previously at formal batch training (informal application training)
Plan was to train first and then do FAT
Was a lecture/watch format with electronic content provided by Rockwell instead of
do format (pros and cons)
7 day shutdown Standard Bama SOPs included Rockwell (and all
other contractors, over 40 total) in planningmeetings to create master schedule and resolveconflicts up front
Used a field resource (Jed) to accompany Dwayneof the 24/7 coverage. Jed had worked on theproject and come onsite with Dwayne for a fewdays beforehand to insure a smooth supportstructure.
Monday startup gave plenty of run time todebug/familiarize and allowed for 24/7 coverage togo to remote support by the weekend
-
8/13/2019 Af11psug Cs05 Bama-companies
19/22
Lessons Learned
Copyright 2011 Rockwell Automation, Inc. All rights reserved.
Not all Units are Created Equal: Typical Units = mixers, hoppers, slurry tanks, etc We found others such as a Salt Chute.
Remote ESX Images: Installations and operations from VPNs and firewalls can be a challenge. Using vSphere as a Citrix App
seems to remedy the situation.
Local RA Field Support: Do your best to have Rockwell Automation use a local resource with the appropriate skill set so that when
the delivery team is complete you have expertise around the corner. (We are also attempting to do thisthrough ICO)
Dont use Microsoft auto updates: IT and Eng have an agreement to work on required patches together and determine how to deploy
FAT: Was short as we were trying to train at the same time we
were checking out; should not have tried to address issues
all at once instead of working on a fix at the time.
Virtual Servers: Use Snapshots to keep fresh rollback points.
-
8/13/2019 Af11psug Cs05 Bama-companies
20/22
Benefits Seen
Still in the evaluation processes but
Looking to see about improvements around power outages
Flexible recipes trials (no salt) completed without interrupting production
Found a lot of hardware and material deficiencies as part of the process
i.e. RPM sensor preventing packing flour sifter at least once since install
Double signoff now needed which improves/enforces QA involvement on signoff
Eliminated paperwork for hand add tracking
Improved manual equipment control capabilities that did not exist before
Improved alarming and troubleshooting capabilities from the plant floor
Inclusivity of system selection and deployment improved plant adoption
-
8/13/2019 Af11psug Cs05 Bama-companies
21/22
Plans Moving Forward
Copyright 2011 Rockwell Automation, Inc. All rights reserved.
Offsite data replication to further improve disaster recovery
More in-depth training on batch and phase manager
Pursuing continuing the rollout of the new standards across theorganization
Review new data being collected to see how they may be incorporated into
reporting/process improvements
-
8/13/2019 Af11psug Cs05 Bama-companies
22/22
Copyright 2011 Rockwell Automation, Inc. All rights reserved.
Insert Photo Here
Rockwell AutomationProcess Solutions User Group (PSUG)
November 14-15, 2011
Chicago, IL McCormick Place West