AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager [email protected]...
-
Upload
allyson-russell -
Category
Documents
-
view
215 -
download
0
Transcript of AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager [email protected]...
![Page 1: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/1.jpg)
AGRESEARCH SHOW AND TELL
Greg Peyroux, Information Solutions [email protected]
Alan McCulloch, Software [email protected]
Information Services Software Development And Specialists Team
Development, Web and Data Services
![Page 2: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/2.jpg)
OVERVIEW
Demonstrations of applications, services and data of interest, discussion, expressions of interest
• Group Structure• Team
• Who and What We Do• New Direction and Business Focus
• Development and Management Tools• Open source and cross platform tools and
frameworks• Examples
![Page 3: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/3.jpg)
INFORMATION SERVICES GROUP
Information Technology and Knowledge Services
• Knowledge• Library • Records management a• Knowledge advisory
• IT• Technical• User Support• Development, Web, Data and Bioinformatics
![Page 4: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/4.jpg)
Recent History of AgResearch:•Series of Reviews
•‘Fit for Purpose’ Initiatives
•Significant Team Turnover
Opportunity to innovate and improve
THE TEAM – REBUILT AND REPOSITIONED
![Page 5: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/5.jpg)
How do you know that you add value?
•Restructured team: Aligned to Outcomes
•Emphasis on “All AgResearch” + the Bigger Picture
•Focus on strategic goals
•Greater Management and Business Awareness
•Transparency: Work request portal on intranet
THE TEAM – REBUILD AND REPOSITION
![Page 6: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/6.jpg)
Development Services•Priority on solutions for science and farms
•Request Analysis
– Determine optimum solution
»Alternative approach e.g. open source
»Reuse/repurpose existing system
»Purchase shrink wrapped S/W or consultancy
»Develop
•Close-off, Feedback and Executive Reporting
A NEW DIRECTION FOR DEVELOPMENT
![Page 7: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/7.jpg)
THE TEAM – REBUILDING
Is B “Best”?
Desperately Seeking:•Fresh Ideas and Concepts
• Interns•Effective Communication Skills• Increased Science Capability:
• PHD, MScs, BScs• Statistical, Research + Technician Experience
•Critical Mass
![Page 8: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/8.jpg)
THE EXTENDED TEAMSoftware Development And Specialists Team (Information Solutions)
Development, Web and Data Services
• Developers– Web, SharePoint– Windows, Mobile Devices– Lab (LIMS) and Farm Systems
• Bioinformatics
• Specialists– GIS– ERP analyst– Reporting Specialist– Database administrator– Records
![Page 9: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/9.jpg)
TEAM IMPACT• Broad Span
– Internal – All AgResearch
– Science
– Support
– Farms
– External
– Farms
– Meat processors
– Commercial Labs
– Accountants and consultancy firms
Group and Team Structures Create Consistent , Aligned Service
![Page 10: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/10.jpg)
Is B “Best”?
Microsoft Stack
• NET framework• Migrating to TFS 2010• SQL Server• BizTalk• SharePoint 2010 (CMS, team, documents, My Sites etc)
Keen to collaborate on:• Public Records Act• Research Data Management• Mapping Systems
Open Source and Cross Plaform Tools (Alan)
DEVELOPMENT TOOLS
![Page 11: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/11.jpg)
Is B “Best”?
Organisation-Wide Data Integration
• Support
• Farm-Systems
• Science data
• Location
• Field Data recording
KEY PROJECTS
![Page 12: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/12.jpg)
INTEGRATION
• Resolution Farm Management and Planning
Integration of Information Systems
![Page 13: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/13.jpg)
INTEGRATION - LOCATIONZ
• Resolution Farm Management and Planning
![Page 14: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/14.jpg)
INTEGRATION - LOCATION
• Resolution Farm Management and Planning
Global Location Integration and Management
• Spatial Location• Silver light application – draw, annotate, search and view• Import
• GPS points• SHP• KML etc
• Known• Desktop Application
• Primarily for sample tracking
• Global Location ID• Web service to guarantee retrieval
![Page 15: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/15.jpg)
OVERVIEW OF FARMING/RESEARCH DATA
Farm and Related Research Data Components
EnvironmentalEnvironmental AnimalAnimal
Animal locations and movements
Animal locations and movements
Milk yield and vat
Herd
Tests and results etc
Primary data setsPrimary data sets
Location/GIS-relatedPaddock/Transect
Points, Lines Polygons
Integrate into data warehouse, mine for traitse.g. Best calvers in the wet
![Page 16: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/16.jpg)
SMART DATA RECORDING AND STORAGE
Dynamic task creation and recording
• Mobile device– User creates a custom task on their PC
– Records task with a range of checks and rich validation
– User uploads task data into a database
Excel Import and Metadata annotating
![Page 17: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/17.jpg)
OVERVIEW OF FARMING/RESEARCH DATATask Recorder – Integrated Task Creation Framework
![Page 18: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/18.jpg)
TOOLS FOR MANAGING PEOPLE, PRODUCTS, PROJECTS AND CHANGE
Ugh! Look at his new recording system, that won’t last!
![Page 19: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/19.jpg)
• Porter’s 5 Forces• PEST• SWOT• Fatal Flaw• Stakeholder mapping• Tipping Point
• Use of levers• Incubate advocates• Plan for resistors
Value of Tools
MANAGEMENT TOOLS AND FRAMEWORKS
![Page 20: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/20.jpg)
MANAGEMENT TOOLS
Kotter, J. (2007). Leading change: Why transformational efforts fail? Harvard Business Review , 85(1), 96-103.
Managing Change: Kotter’s 8 Steps to Transformation
![Page 21: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/21.jpg)
OPEN SOURCE AND CROSS-PLATFORM ENVIRONMENTS AND FRAMEWORKS
Allan McCulloch
![Page 22: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/22.jpg)
WINDOWS AND UNIX HPC RESOURCES
CROSS-PLATFORM SOFTWARE DEVELOPMENT - STATUS
EXTENDED DATA MODELLING
DESIGN PATTERNS FOR DATA MODELLING
COLLABORATION SUPPORT - WIKIS AND DRUPAL
![Page 23: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/23.jpg)
WINDOWS AND UNIX HPC RESOURCES
We use the Condor scheduler - creates and manages both Windows and Unix compute pools. Mature, scalable, cross-platform
• Windows = desktop machines , jobs run in background. Users approached and volunteer machines to the pool (+ “captive” machines in training rooms and IS and departmental staff !) as and when required . Max size so far, simulation run on about 40 machines. Saved $$$ on leased high performance windows boxes.
• Unix = Centos Redhat / Centos based , HP blades (mix of 32 and 64 bit , 2 and 4 core, 2Gb RAM ). B-word data pre-processing mainly
![Page 24: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/24.jpg)
“LARGE MEMORY” SMP UNIX MACHINES
• 1 x Redhat Linux database server (postgres , mysql, Oracle)
• Data warehousing • B-word• GIS data (postgres)
• 1 x Redhat Linux compute server
• Data Mining • Stats (R , ASREML)• B-word miscellany• Virtual machine provisioning (early days)
![Page 25: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/25.jpg)
CROSS-PLATFORM (LINUX-WINDOWS) IN-HOUSE SOFTWARE DEVELOPMENT
• Java based middleware supporting various B-word web interfaces (not being further developed)
• Small scale python, perl and R scripting for specific problems or pipelines (/ bucket brigades). Unusual for us to develop anything significant from scratch – huge communities, many packages and solutions already out there.
• Python based Object-Relational system supporting mainly B-word databases
• Mathematical modellers developing C++ based models
![Page 26: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/26.jpg)
EXTENDED DATA MODELLING
What is it ?
• ER : Entity-Relationship model : Chen (1976), formalized in Ng (1981). Standard Terminology.
• Extended Entity Relationship (EER) : little standard terminology , many contributors
• EER only one of a number of semantic models• Developed to meet complex requirements beyond data-
processing in business and industry – CAD/CAM db, image and graphics db, cartographic and geological db.
• Ref ElMasri, R.A. , Navathe , S.B. : Fundamentals of Database Systems
![Page 27: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/27.jpg)
BRDF FRAMEWORK – SOME EER ASPECTS
• Inheritance used to provide a core data provenance and record identification and classification layer for all tables• Automatic date stamps and user ids for updates on any table• Keywords field for search engines• Record status for versioning• Class typeid for object-relational mapping support• Universal unique record identifier across all tables – really useful !
• Heterogenous lists can include member records from many tables – e.g. search engine results; projects manifests (subjects, samples, lab resources etc)
• Normalised free text comments attached to any record• Provenance of all data imports - records are linked to the data source
they came from .• Entity types and relationship types treated uniformly
![Page 28: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/28.jpg)
BRDF FRAMEWORK
• Postgres based (….but portable to dbs like Oracle, SQLServer)• Easy to install - single setup script• Around 8 instances mostly b-word related but also hosting
genetics and epidemiological data• Python based object-relational mapping and web interface • About to begin work on a C# /.net based OR mapping and new
front-end• (Custom object relational mappings are cost-effective and work
well for data processing , but objects do not usually plug into standard GUI componentry , and so therefore are not cost effective for UI development )
• (and vice-versa !!! )
![Page 29: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/29.jpg)
DESIGN PATTERNS FOR DATA MODELLING
• “Design Patterns” (Gamma, Helm etc al.) – reusable pieces of design in Object Oriented software engineering (e.g. observer, visitor, factory method, façade, flyweight,….)
• We need Design Patterns for databases to control complexity and cost
• Design patterns easier to define for EER based systems
![Page 30: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/30.jpg)
IMPORT (IN) AND DISPLAY (OUT) DESIGN PATTERNS
![Page 31: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/31.jpg)
STUDY-OBSERVATION DESIGN PATTERN IN BRDF FRAMEWORK
![Page 32: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/32.jpg)
STUDY-OBSERVATION DESIGN PATTERN SUGGESTED DISCRETIZING CONTINUOUS MS OBSERVATIONS INTO
BINS
![Page 33: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/33.jpg)
DRUPAL BASED COLLABORATION SUPPORT : MULTI-TENANTED DESIGN
![Page 34: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/34.jpg)
APPLICATIONS
Some examples of applications
![Page 35: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/35.jpg)
FARM APPLICATIONS
• Resolution Farm Management and Planning
• ETS Tools - Forestry
• Resolution Farm Management and Planning
![Page 36: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/36.jpg)
FARM APPLICATIONS - ETS
• MAF lookup tables and rules
• Google Map interface
• Extended GPS support
![Page 37: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/37.jpg)
![Page 38: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/38.jpg)
SUMMARY
• Open source and cross platform• Windows and Unix HPC resources• Cross-platform Software development – status• Extended Data Modelling • Design patterns for data modelling• Collaboration support - wikis and Drupal
• Development and Management Tools• Data integration• Change management
• About Us• Team • Who and What We Do• Direction and Business Focus
![Page 39: AGRESEARCH SHOW AND TELL Greg Peyroux, Information Solutions Manager greg.peyroux@agresearch.co.nz Alan McCulloch, Software Engineer alan.mcculloch@agresearch.co.nz.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649e005503460f94ae9382/html5/thumbnails/39.jpg)
AGRESEARCH SHOW AND TELL
Greg Peyroux, Information Solutions [email protected]
Alan McCulloch, Software [email protected]
Thank you for your attention
Questions
Please feel free to contact us