Capabilities and Process Application Integration.
-
Upload
cuthbert-cross -
Category
Documents
-
view
220 -
download
0
Transcript of Capabilities and Process Application Integration.
![Page 1: Capabilities and Process Application Integration.](https://reader034.fdocuments.us/reader034/viewer/2022051401/5697bfa51a28abf838c97aa9/html5/thumbnails/1.jpg)
Capabilities and Process
Application Integration
![Page 2: Capabilities and Process Application Integration.](https://reader034.fdocuments.us/reader034/viewer/2022051401/5697bfa51a28abf838c97aa9/html5/thumbnails/2.jpg)
Goals
• Explain the importance of a single software stack for all BIRN servers
• Describe software integration, deployment and upgrade processes (in random order)
• Discuss possible extensions and improvements
![Page 3: Capabilities and Process Application Integration.](https://reader034.fdocuments.us/reader034/viewer/2022051401/5697bfa51a28abf838c97aa9/html5/thumbnails/3.jpg)
BCC Software Integration Responsibilities
• System Architecture
• Development and Integration– System software– Application software
• Deployment– Reliability– Scalability– Consistency– Flexibility
![Page 4: Capabilities and Process Application Integration.](https://reader034.fdocuments.us/reader034/viewer/2022051401/5697bfa51a28abf838c97aa9/html5/thumbnails/4.jpg)
BIRN-CC Responsibilities (cont’d)
• Maintenance and Support for a Continually Growing Community– 24/7 production quality grid– Monitoring– Troubleshooting– Help Desk– Training
![Page 5: Capabilities and Process Application Integration.](https://reader034.fdocuments.us/reader034/viewer/2022051401/5697bfa51a28abf838c97aa9/html5/thumbnails/5.jpg)
Applications/Servers
• BIRN 1.0– GPOP– GComp– NAS– SRB– Oracle
• BIRN 2.0 added– Nagios– Condor– Scientific Applications
• BIRN 3.0 added– More Sciapps– Oracle– Postgres– Tomcat– Gridsphere– HID– Mediator– XNAT (in progress)– Xwiki (in progress)
![Page 6: Capabilities and Process Application Integration.](https://reader034.fdocuments.us/reader034/viewer/2022051401/5697bfa51a28abf838c97aa9/html5/thumbnails/6.jpg)
Sources for BIRN Software
• Source code, RPMs and tarballs from:– Participant CVS/SVN repositories– Participant websites– Opensource community
• Configuration instructions– Make scripts– Hand edit config files– Website instructions– BIRN knowledge-base
![Page 7: Capabilities and Process Application Integration.](https://reader034.fdocuments.us/reader034/viewer/2022051401/5697bfa51a28abf838c97aa9/html5/thumbnails/7.jpg)
Currently integrated on GComp
• AFNI• AIR• Brains2• Caret• Freesurfer
• FSL• LDDMM• LONI• Mipav• Slicer
![Page 8: Capabilities and Process Application Integration.](https://reader034.fdocuments.us/reader034/viewer/2022051401/5697bfa51a28abf838c97aa9/html5/thumbnails/8.jpg)
If Done by Hand at 25+ BIRN Sites
CHAOS!
![Page 9: Capabilities and Process Application Integration.](https://reader034.fdocuments.us/reader034/viewer/2022051401/5697bfa51a28abf838c97aa9/html5/thumbnails/9.jpg)
The BIRN Software Stack to the Rescue
• Every application/server gets– Same software version– Same software configuration– Same security
• Fewer manhours spent– Debugging– Propagating fixes
• Synergy
![Page 10: Capabilities and Process Application Integration.](https://reader034.fdocuments.us/reader034/viewer/2022051401/5697bfa51a28abf838c97aa9/html5/thumbnails/10.jpg)
RPMs - a Cornerstone in the Path to Sanity
• Create users
• Unpack tarballs
• Modify configuration files
• Adjust firewalls
• Set up databases
• Start/stop services
• Make changes based on “zone”, “site”, etc.
![Page 11: Capabilities and Process Application Integration.](https://reader034.fdocuments.us/reader034/viewer/2022051401/5697bfa51a28abf838c97aa9/html5/thumbnails/11.jpg)
The RPM Database
• RPM database on each server knows…– What has been installed– What functionality is available from where
• Examples[root@ncmir-gcomp ~]# rpm -qa | grep slicerslicer-2.5.1-0
[root@ncmir-gcomp ~]# rpm -q --requires slicer/usr/local/bin/tclsh libGL.so.1
[root@ncmir-gcomp ~]# rpm -q --whatprovides libGL.so.1 xorg-x11-Mesa-libGL-6.8.2-1.EL.13.25.1
![Page 12: Capabilities and Process Application Integration.](https://reader034.fdocuments.us/reader034/viewer/2022051401/5697bfa51a28abf838c97aa9/html5/thumbnails/12.jpg)
Specific Examples for BIRN Sciapps
• 3DSlicer– RPM from tarball
• Brains2– Installs RPM from site
– Creates symbolic link
• Caret– RPM from .zip file
– Permissions corrected
• LDDMM– RPM created from tarball
– Environment variables set
• MIPAV– RPM from spec file and tarball
provided by site
– Verification script
[root@ncmir-gcomp ~]# for i in afni air brains2 caret freesurfer fsl lddmm loni mipav slicer ; do rpm -qa | grep -i $i ; done
AFNI-2006_03_21_1314-0 LONI_Pipeline_Client-1-0caret-5.50-1 LONI_Pipeline_Server-1-0freesurfer-3.0.2-0 mipav-1.59-1fsl-3.3.5-0 slicer-2.5.1-0lddmm-1.0.1-3 brains2-RedHatEnterpriseWS4-20060327-032445
![Page 13: Capabilities and Process Application Integration.](https://reader034.fdocuments.us/reader034/viewer/2022051401/5697bfa51a28abf838c97aa9/html5/thumbnails/13.jpg)
• Alternative to up2date from RedHat
• Used for automated software updates
• Relies on RPM database
YUM (Yellowdog Updater, Modified)
[root@ncmir-gcomp ~]# for i in afni air brains2 caret freesurfer fsl lddmm loni mipav slicer ; do yum list | grep -i $i ; done
AFNI.i386 2006_03_21_1314-0 installed brains2-RedHatEnterpriseWS4.i386 20060327-032445 installedcaret.i386 5.50-1 installed freesurfer.i386 3.0.2-0 installed fsl.i386 3.3.5-0 installed lddmm.i386 1.0.1-4 installed LONI_Pipeline_Client.i386 1-0 installed LONI_Pipeline_Server.i386 1-0 installed mipav.i386 1.59-1 installed slicer.i386 2.5.1-0 installedslicer.i386 2.6-0 birncentral-dev-
![Page 14: Capabilities and Process Application Integration.](https://reader034.fdocuments.us/reader034/viewer/2022051401/5697bfa51a28abf838c97aa9/html5/thumbnails/14.jpg)
YUM-my features
• Specific packages can be excluded temporarily
• Packages can be “installonly”, never updated
• Repository is same as that used for installation of new servers
![Page 15: Capabilities and Process Application Integration.](https://reader034.fdocuments.us/reader034/viewer/2022051401/5697bfa51a28abf838c97aa9/html5/thumbnails/15.jpg)
• Development & Integration– CVS & SRB for version control facilitates
collaborative development– Automated build and deployment mechanisms for
repeatability and scalability– Flexible development environment to meet
changing requirements– Continually improving infrastructure in terms of
repeatability, tracking, monitoring, automation, and scalability
Software Standards
![Page 16: Capabilities and Process Application Integration.](https://reader034.fdocuments.us/reader034/viewer/2022051401/5697bfa51a28abf838c97aa9/html5/thumbnails/16.jpg)
Features of Software Build Process
• “Normal” source code from CVS
• Large source, especially binary, from SRB
• Automatic tagging during build
• Tags match RPM version info
• Updates automatically added to YUM
![Page 17: Capabilities and Process Application Integration.](https://reader034.fdocuments.us/reader034/viewer/2022051401/5697bfa51a28abf838c97aa9/html5/thumbnails/17.jpg)
![Page 18: Capabilities and Process Application Integration.](https://reader034.fdocuments.us/reader034/viewer/2022051401/5697bfa51a28abf838c97aa9/html5/thumbnails/18.jpg)
Integrating/Updating via RPM
• Commit RPM to BIRN CVS repository• Where matters
– Which roll (sciapps, freesurfer, …)– Which architecture(s) (i386, noarch, ia64, …)
• Basename can’t change• If integrating, add nodes and graph files• Example for brains2
– BIRN/rocks/src/roll/sciapps/RPMS/i386/brains2-RedHatEnterpriseWS4-20060327-032445.i386.rpm
– BIRN/rocks/src/roll/sciapps/nodes/brains2.xml– BIRN/rocks/src/roll/sciapps/graphs/default/sciapps.xml
![Page 19: Capabilities and Process Application Integration.](https://reader034.fdocuments.us/reader034/viewer/2022051401/5697bfa51a28abf838c97aa9/html5/thumbnails/19.jpg)
Updating/Integrating from CVS/SVN/SRB
• Import/Update/Upload source– CVS/SVN source repositories– Large tarballs, especially binaries, into SRB
• Update BIRN CVS files– Makefile– Version.mk– *.spec.in
• Examples– Slicer for tarball in SRB– Gridsphere for CVS/SVN
![Page 20: Capabilities and Process Application Integration.](https://reader034.fdocuments.us/reader034/viewer/2022051401/5697bfa51a28abf838c97aa9/html5/thumbnails/20.jpg)
Updating BIRN CVS files
• Create/Update Makefile– Checks out/Downloads source– Prepares for build– Has build rules for spec.in file, if not in spec.in
file
• Create/Update version.mk– Sets up variables used in building and tagging– Has version # to track source changes– Has revision # to track spec.in changes– Important for YUM updating
![Page 21: Capabilities and Process Application Integration.](https://reader034.fdocuments.us/reader034/viewer/2022051401/5697bfa51a28abf838c97aa9/html5/thumbnails/21.jpg)
Updating BIRN CVS files (cont’d)
• Create/Update *.spec.in– Standard boilerplate/template to start from– Pre-, install, Post- sections– Sets up requirements for building and running– http://fedora.redhat.com/docs/drafts/rpm-guide-en/
![Page 22: Capabilities and Process Application Integration.](https://reader034.fdocuments.us/reader034/viewer/2022051401/5697bfa51a28abf838c97aa9/html5/thumbnails/22.jpg)
Software Deployment Paradigm
• Deployment of software at different levels of maturity to independent, isolated areas– Development
• Supports short-term specialized testing (network issues, database upgrades, etc.)
• Integrate testbed software prior to beta testing
– Staging• Provides area for complete, end-to-end testing
• Allows demonstration of latest software development efforts– Prior to production deployment– Without disruption to production systems
– Production• Allows reliable, repeatable, consistent infrastructure for collaboration of
neuroscience researchers
![Page 23: Capabilities and Process Application Integration.](https://reader034.fdocuments.us/reader034/viewer/2022051401/5697bfa51a28abf838c97aa9/html5/thumbnails/23.jpg)
![Page 24: Capabilities and Process Application Integration.](https://reader034.fdocuments.us/reader034/viewer/2022051401/5697bfa51a28abf838c97aa9/html5/thumbnails/24.jpg)
BIRN Uses Rocks
• Efficient, Reliable, Repeatable, Scalable Deployment • Systems can be reinstalled, reconfigured or
duplicated quickly and easily• Consistent software installation and configuration
without sacrificing flexibility• Standardized software delivery process• Server functionality is organized into rolls
– Reusable– Generic or specific
• Feedback incorporated in new Rocks releases by the SDSC Rocks team
![Page 25: Capabilities and Process Application Integration.](https://reader034.fdocuments.us/reader034/viewer/2022051401/5697bfa51a28abf838c97aa9/html5/thumbnails/25.jpg)
Kicking Off Server Installation
• Choose rolls
• Partition & format the disk (/export is preserved)
• Enter network information & hostname
• Select operating environment (dev/stage/prod)
• Select timezone
• Enter root password
• Walk away
![Page 26: Capabilities and Process Application Integration.](https://reader034.fdocuments.us/reader034/viewer/2022051401/5697bfa51a28abf838c97aa9/html5/thumbnails/26.jpg)
What the Blazes is a Roll?!
• A roll is a complete, reusable “set” of software (RPMs) needed to fulfill the functionality definition of the roll
• Examples
CentOS Provide RedHat O/S
ganglia or nagios Monitoring tools
condor or srb Generic condor / SRB
birncondor or birnsrb BIRN specific modifications
sciapps BIRN specific neuroscience tools
![Page 27: Capabilities and Process Application Integration.](https://reader034.fdocuments.us/reader034/viewer/2022051401/5697bfa51a28abf838c97aa9/html5/thumbnails/27.jpg)
Server Functionality Defined by Roll Selection
• BIRN Gridsphere Portal roll list includes:– CentOS, birn, area51, nagios, etc.– java (to compile gridsphere code)– tomcat (portlet container)– gridsphere (basic)
• gridsphere and gridportlets RPMs
– birnportal (adds BIRN specific functions)• birnportal, birnportlets, gama RPMs
– srbportlets (adds SRB specific functions)– postgres (database)
![Page 28: Capabilities and Process Application Integration.](https://reader034.fdocuments.us/reader034/viewer/2022051401/5697bfa51a28abf838c97aa9/html5/thumbnails/28.jpg)
• Contacts– [email protected]– [email protected]– Vicky Rowley, [email protected]– Kennon Kwok, [email protected]
• THE END