BEST AND WORST PRACTICES DEPLOYING IBM CONNECTIONS
Click here to load reader
-
Upload
nico-meisenzahl -
Category
Technology
-
view
940 -
download
0
Transcript of BEST AND WORST PRACTICES DEPLOYING IBM CONNECTIONS
Toronto, June 6-7 2016
Best and Worst Practices Deploying
IBM Connections
Christoph StoettnerNico Meisenzahl
PLATINUM&SPOTLIGHTSPONSORS
GOLDSPONSORS
SILVERSPONSORS
BRONZESPONSORS
Christoph Stoettner
• Senior Consultant – panagenda• IBMNotes/Dominosince1999• IBMConnectionssinceversion2.5/2009
• Many years of experience in:• Migrations• Administrationundinstalls• Performanceanalysis
• Joined panagenda in 2015 focusing in:• IBMConnectionsdeploymentund
optimization• IBMConnectionsmonitoring
• Husband of one & father of two, Bavarian
3
@stoeps linkedin.com/in/christophstoettnerwww.stoeps.de christophstoettner [email protected]
Nico Meisenzahl
• Consultant at panagenda• IBM Notes / Domino since 2008• IBM Connections since version 3.0 / 2010• Many years of experience in:
• Consulting• Migrations&Administration
• Joined panagenda in 2016 focusing in:• IBMConnectionsConsulting• ICSdeployment&optimization
4
@nmeisenzahl linkedin.com/in/nicomeisenzahlmeisenzahl.org nico.meisenzahl [email protected]
IBM Connections Request Flow
5
Databases
WebSphereApplication Server
IBMHTTPServerWebSpherePlugins
Application DBsPEOPLEDB
LDAPServer
TDI
Forward toApplicationServerandPort
(LoadbalancingandFailover)
Redirectunknown
URL
UploadandDownloadofFiles,Attachments
Common:AccessCustomization,Webressources
ReadandWrite
Authentication
Users/Groups
ReadsUserdataforProfiles
Create,Update,DeleteandInactivateProfiles Shared
Directory
Linkto
Attachments
ProfilechangessynchronizetoMembertablesthroughJMSQueue
Optional:
DirectAccesstoAttachments
Toronto, June 6-7 2016
Installation & Requirements
System Requirements
• Regularly check requirement documents
• All versions• http://short.stoeps.de/vwzrv
• IBM Connections 5• http://short.stoeps.de/mspdi
• IBM Connections 5.5• http://short.stoeps.de/cnx55sysreq
• Check all notes, Download PDF
Connections5.0CR3
Connections5.5
Connections5.5CR1
7
System Requirements
8
• Be careful with installation documents• Sometimeswrongdependenciesmentioned• Supportedstatementdoesnotmeanit’slicensed
• http://www-01.ibm.com/support/docview.wss?uid=swg21683118• Supported: DB210.5FP4->butConnections5.0LicenseAgreement:
• YouwillnotfindaDB210.5licenseinyourPAAccount
Sizing
• Be prepared for future growth• Do not overact
• Afewhundredusersdoesn’tmeanyouneedalargedeployment
• Not fans of multi-instance database machines• IfIrunindatabaseperformanceissuesIsplitthedatabasesto
differentmachines• Performancetuningguide
• Multi-instanceisbestpractice,ifyouhaveenough resources
PerformanceTuningGuide5CR1– page10
9
Sizing
• A word on requirements• 4|8GBmemoryminimumisoftentooless,
bettertostartwith10or12GB• Memoryswappingkillsalltuningefforts
• CPU cores• 2coresminimumonlyonsmalldeployments• Thumbrule:calculateonecoreforeachjvm
(expensivewithPVUlicense)
• Disk• Usingnetworkstorageorvirtualizedservers• Easiertoextend
Connections5.0Connections5.5
10
Prepare for your Installation
• Download all software packages• CheckSystemRequirements!
• Paths shouldn't contain spaces• Nospacesinsourceanddestinationfolders
• Use a dedicated administration user • EspeciallyonWindowsavoiduserswithappliedgrouppolicies• IfpossibledisableUserAccountControl(UAC)• RunallInstallerandScriptswithoption“RunAsAdministrator”
11
Security Settings
• During installation you should disable all "Security" Software• SELinux• AppArmor• Antivirus• Firewalls• Selfdevelopedscriptsandextensions
• It's not fun, when a script deletes databases, because you forgot to add the directory to the script exclusions
12
Operating System specific settings -Linux
13
• Different operating systems need special settings• Always use the operating system where you have the best
skills• Linux
• /etc/security/limits.conf• Increasenofileandnproc(seetuningguides)
• Example from tuning guide
• Defaultnproc(maxnumberofprocesses)foruserroot2047• Youcanextendthenprocwithulimit–pupto16384(e.g.within.bashrc
or.profile)• Orsetsoftandhardlimittoequalsizes,avoidsadditionalchangeswith
profile
root soft nproc 2047 root hard nproc 16384
Operating System specific settings -Windows
14
• Always use UNC path as Shared Directory• EasiertoaddadditionalWebSphereNodes
forfailoverorloadbalancing
• WebSphere services • Technicaluseraccount
• Enable“Passwordneverexpires”• Disable“Mustchangepasswordonnextlogin”
• Default:LocalSystemhasnonetworkaccess
• Check access rights on Shared Directory
Network
• Name lookup / DNS• Allserversmustberesolvable(hostsisnotasuitableworkaround)• Knowingtheprotocol
• AvoidRoundRobin• NoAuthentication failoverinWebSpherewithRoundRobin!
• Network storage (file locking is important)• NFSv4/SMB|CIFS• NoDFS
• Reverse Proxies / Proxies• Alwaystestyourdeploymentwithoutproxies• Activateaftersuccessfultesting
15
Register WAS as a service
• Register WAS as a service • Services for Deployment Manager and
NodeAgent(s)• wasservice.bat|sh
• Map service to a technical user• anyActiveDirectoryUserispossible• allowedtoread/writenetworksharewithShared
Content
• Service can parse commands to nodeagent• -stopArgs "<NAcommands>"
• Configure monitoring policy (if required)
16
WasService.bat|sh – Register service
17
cd D:\IBMCNX\WebSphere\AppServer\bin
WASService.exe -add CnxNode01 -serverName nodeagent -profilePath d:\ibmcnx\websphere\appserver\profiles\CNXNode01 -stopArgs "-username wasadmin -password password -stopservers" -userid cnxtec -password password -encodeParams -restart true -startType automatic
parsed to nodeAgentstops AppServer
Monitoring Policy
• Each Application Server• ChangeNoderestartstateto
"RUNNING"• Large deployment on Windows
• Defaulttimeoutforserviceshutdown=20seconds
• IncreaseValueat:HKEY_Local_Machine:SYSTEM\CurrentControlSet\Control\WaitToKillServiceTimeout
• Must set this to STOPPED before performing updates• OrusesyncNode.bat|shafterapplyingtheupdate
18
Directories & Synching
• Prepare your LDAP• BetterdatawithinLDAP→betterProfiles
• Switching Authentication directories is possible• Needsomeplanning
• Dependencies• QualityofLDAPdata• PlanstoactivateSPNEGO• DominoMailIntegration
19
Federated Repositories Best Practice
• Leave the file based wasadmin with WebSphere Application Server• FallbackifLDAPBindCredentialschanged• Solvingproblemswith
FederatedRepositories
• Default does not allow this (you have to disable security to change configuration)
Checkthisbox
20
Logs – adjust language WebSphere
• Change log language to English (IBM will love you for this)• WebSphere
Add "-Duser.language=en –Duser.region=US" to Generic JVM arguments of• Eachapplicationserver(Processdefinition – JavaVirtualMachine)• dmgr (SystemAdministration – DeploymentManager– ProcessDefinition ...)• nodeagents (SystemAdministration– Nodeagents– nodeagent– ProcessDef
...)
21
Logs – adjust language TDI
22
• TDI• editibmdisrv.bat|sh• add-Duser.language=en–Duser.region=UStoLOG_4Jvariable• Linux:
• Windows:
Rotate Logs
• WebSphere Logs too small for Troubleshooting• Default:5Logs1MBeach(SystemOutandSystemErr)• Better5-10Logs20MBeach
• SettingforeachApplicationServer• rememberNodeagentsandDmgr
• Change this as soon as your servers have been created
23
Rotate Logs
• IBM Connections 5.5 – SET BY DEFAULT!!• Install.log
• Result:
• Soyourlogsarestored30days,independentofsize
24
Rotate IBM HTTP Server Logs
• Default: no max size for access_log and error_log• Often some GB of Log files
• OpenwithanEditor?• Disksize
• Search for this lines in httpd.conf:
• Comment out:
CustomLog log/access_log common
ErrorLog logs/error_log
# CustomLog log/access_log common
# ErrorLog logs/error_log
25
Rotate IBM HTTP Server Logs
• Add:
• Delete Log Files older than x days• Linux
• Windows(BatchthroughTaskSchedulerorPowershell)
Linux:CustomLog "|/opt /IBM/HTTPServer/bin/rotatelogs /opt/IBM/HTTPServer/logs/access_log.%Y%m%d 86400" commonErrorLog "|/opt/IBM/HTTPServer/bin/rotatelogs /opt/IBM/HTTPServer/logs/error_log.%Y%m%d 86400“
Windows:CustomLog "|D:/IBM/HTTPServer/bin/rotatelogs.exe D:/IBM/HTTPServer/logs/access_log.%Y%m%d 86400" commonErrorLog "|D:/IBM/HTTPServer/bin/rotatelogs.exe D:/IBM/HTTPServer/logs/error_log.%Y%m%d 86400"
crontab -e# Delete logfiles older than 3 days in logs10 0 * * * find /opt/IBM/HTTPServer/logs/*_log.* -mtime +3 -exec rm -rf {} \;
forfiles -p "D:\IBM\HTTPServer\logs" -s -m *_log.* -d -3 -c "cmd /c echo @file"
26
Rotate Logs DB2
• db2diag.log• Default: no maximum size
• Default:%PROGRAMDATA%\IBM\DB2\instancename\DB2• FullC-PartitioninWindowsstillhardtosolve
[db2inst1@cnx-db2 ~]$ db2 get dbm cfg |grep -i diagsizeSize of rotating db2diag & notify logs (MB) (DIAGSIZE) = 0
[db2inst1@cnx-db2 ~]$ db2 update dbm cfg using DIAGSIZE 1024DB20000I The UPDATE DATABASE MANAGER CONFIGURATION command completed successfully.
[db2inst1@cnx-db2 ~]$ db2 get dbm cfg |grep -i diagsizeSize of rotating db2diag & notify logs (MB) (DIAGSIZE) = 1024
27
HTTP Server Keystore
28
• Several Reviews showed• Keystore ofWebSpherePluginusedforIHSSSLKey• Whyisthisworse?• WhatwouldyoudowhenyougetSSLErrorswithinConnections?
• Thisoverwritesplugin-key.kdb onyourWebserver• SSLKeydeleted• Backup?
HTTP Server Key store
29
• When you want to reuse Plugin Key store• ImportSSLKeyintoCMSKeyStore• Butneverseenthisinthewild
HTTP Server Keystore
• Best Practice - Create a separate key store for IHS• Ikeymanwillhelpyou• Possibletouseawildcard
• Thenyoucanjustcopyittouseondev/testmachines
• Backupthekeystore beforechanges• Don’tactivate“Expirationtime”• “Stashpasswordtoafile”
• Createsacnx-key.sth file• UsedbyIHStoopenkeystore
30
Toronto, June 6-7 2016
Security
J2EE Roles
• Some Applications are public readable after installation• Profiles• Communities• Blogs• Wikis
• Check after Updates• Google:“Site:myconnections-host”• Shouldonlyshowaloginpage
• Use the Community Scripts to do this or change in the ISC
32
Harden HTTP
• DisableSSLv2/v3• Automaticallydisabledwith8.5.5.4• SSLProtocolDisable SSlv2SSLv3
• Checkwithhydra,nmap orssllabs.com/ssltest/
• Defaulthttpd.conf uses:TLS_RSA_WITH_3DES_EDE_CBC_SHA
# Ciphers TLS1.0, 1.1SSLCipherSpec TLS_RSA_WITH_AES_256_CBC_SHASSLCipherSpec TLS_RSA_WITH_AES_128_CBC_SHA# Additional Ciphers TLS1.2SSLCipherSpec TLS_RSA_WITH_AES_128_GCM_SHA256 SSLCipherSpec TLS_RSA_WITH_AES_256_GCM_SHA384 SSLCipherSpec TLS_RSA_WITH_AES_128_CBC_SHA256 SSLCipherSpec TLS_RSA_WITH_AES_256_CBC_SHA256
33
Harden HTTP
• IfyouuseSSLKeyslongerthan2048bit,youmustreplaceJavapolicy• DownloadandreplaceJava(unrestricted)policyfiles• https://www-01.ibm.com/marketing/iwm/iwm/web/reg/pick.do?source=jcesdk
• AlsoneededifDomino(MailIntegration)orSametimeProxyuselongerkeys
• RemoveServerInformation(HTTPHeader,Errorpages)• ServerSignatureOff• ServerTokens Prod(DEFAULT)• AddServerHeaderOff
Default
34
Remove Index
• RemoveallFilesexceptindex.htmlfrom<IHS_ROOT>/htdocs• Renameindex.html(e.g.0815.html)
• echo1>0815.html• Fortestingyoucanaccess thefile
• Addrobots.txt
35
Toronto, June 6-7 2016
Tuning
Performance Tuning Guides
• 4.0• http://www-
10.lotus.com/ldd/lcwiki.nsf/dx/IBM_Connections_4.0_Performance_Tuning_Guide
• 4.5 Addendum• http://www-
10.lotus.com/ldd/lcwiki.nsf/dx/IBM_Connections_4.5_Performance_Tuning_Guide_Addendum
• 5.0 CR1• http://www-
10.lotus.com/ldd/lcwiki.nsf/dx/IBM_Connection_V5_CR1_Tuning_guide• Read everything carefully• check and understand dependencies
37
Worst Practise Example - Tuning
• Customer showed me a system with following infrastructure
• WebSphere• Largedeployment• 16GBRAM• 4Cores
• DB2• 12instances• 8GBRAM• 4Cores
• Connections restart 22 minutes
Web Serverihs.example.local
WebSpherewas1.example.local Db2 / TDI
db2.example.localFreigabe
LDAPdomino1.example.local
User SynchronisationAuthentication
38
Solving the problem
• Large deployment means about 15 JVM on the machine• Restartshows15min100%CPUusage• Adding4coresandrestarttimegetdownto7minutes• Otheroptionwouldbemidsizedeployment,butthenyouhavetoreinstall
Connections• Java Heap Sizes set to default (256 MB and 768 MB) ->
increase to 1.5 – 2.5 GB• Perf Guide mentions that multiple instances on DB2 only
increase performance with enough resources• Butthatwasnottherealproblem
• DataSource connectionPool Sizes are set to Default 1/10 • Increasethisvaluestotheproposalsintheguideand...• Restarttimecomesdownunder3minutes
• Key point: read the complete guide
39
Java Heap
• Default Java Heap Sizes on Midsize Deployment: 2506 MB / application server
• Large Deployment depends on application: 0.5 to 2.5 GB• Main point in memory tuning
• Neverexceedthesystemmemory• Swappingkillsallyour tuningefforts
• Counting the JVM Heap sizes is not enough• Maximumheapisnotthemaximumamountofmemorythejvm uses!• Libraries,jarsandsooncountadditional tomemoryusage• JVMmemoryusagemaybe3*JVMmaximumHeap
• Initial and maximum Heap Size should be equalized• mentioned inIBMConnections5.0tuningguide
40
IBM HTTP Server
• Enable compression• Important!!!!!• SeeSlidesfromBP307- IBMConnect2014• Saveupto70%networktraffic• MinimalincreaseofCPUload
• Enable file download through IHS• Dependonyourdeployment• OftensecurityforbidsstorageaccessfromDMZ
• If you have no access to file share from IHS -> Files should be installed in a separate Cluster
41
Midsize Deployment Files
• Often IHS positioned in the red zone (DMZ)• Mostly No Access to SHARED DIRECTORY
• CreateaClusterforFiles• NoProblemwithLargeDeployments• WithMidsizeyoucanaddanadditionalClusterduringSetup
(LooksdifferentonConnections5.5!)
http://www-01.ibm.com/support/docview.wss?uid=swg21317658
42
Activate Synchronous File transfer
• Servers -> Application Servers -> serverName -> Web Container Settings -> Web Container -> Custom Properties• com.ibm.ws.webcontainer.channelwritetype=sync
43
Toronto, June 6-7 2016
Enhance User experienceHappy admins with happy users
Single Sign On - LtpaToken
45
• Single Sign On within IBM portfolio• Domino only supports one domain per Web SSO Document
• Youcancopy&pasteWebSSODocumentsandchangeDomainnames(seee.g.PaulMooney- AdminBlast2012– Tip#4
• DNSDomainismultivalue(worksuntilDomino8.5.x,butnotwithDomino9.x)
• ServerswithmixedInternetSiteandNon-InternetSiteusage:copy&pastetoo!
• Often internal servers use local domains, when Connections is external accessible SSO needs workaround• addingadditionalhostnamestodomino• YoucanuseIHS(IBMHTTPServer)asareverseproxytoaccessiNotes
Single Sign On - LtpaToken
46
• It’s possible but not recommended to change the cookie name
• Default one are• LtpaToken• LtpaToken2
• Name dependencies• Web-SSOdocumentnameforDomino• WebSSOwithinWebSphere• sametime.ini
• ST_TOKEN_TYPE=CustomLtpaName• Connectionsscheduler
Single Sign On - SPNEGO
47
• Requirements• Windows2003/2008/2012ActiveDirectory
• Configure use documentation and http://de.slideshare.net/david_hay/dave-hay-desktop-single-signon-in-an-active-directory-world?related=1
• Real additional value for users• Easy to deploy, when you have the rights and clue what to do• Do not test Browser Single Sign On with Chrome, because process
does not end when you close the last window• Tip: SPNEGO is not working with a SPN which is a CNAME DNS
alias
Mail integration
48
• Use IBM HTTP Server as reverse proxy to access iNotes
LoadModule rewrite_module modules/mod_rewrite.so <IfModule mod_ibm_ssl.c>
Listen 0.0.0.0:1443<VirtualHost *:1443>
ServerName connections.example.com SSLEnable RewriteEngine onProxyRequests OffProxyPass / http://inotes.example.local/ProxyPassReverse / http://inotes.example.local/
</VirtualHost> </IfModule>
iNotes Web Mail Redirect
49
https://connections.example.com:1443
Socialmail-config.xml
50
• When you use reverse proxy to access iNotes• Mailintegrationworksonlywhenyouusehttporhttps• AddUseConfiguredProtocol toyourconfiguration
• Problem when you need to access multiple iNotesServers
<ServerConfig name="domino-redirect"><ConfigType>REDIRECT</ConfigType><RedirectURL>https://connections.example.com:1443/iwaredir.nsf</RedirectURL><MailPattern type="example.com" />
</ServerConfig><GadgetConfig>
<GadgetPreference id="UseConfiguredProtocol">true</GadgetPreference></GadgetConfig>
Mail integration and SPNEGO
51
• LtpaToken contains AD $DN• Lookup in Domino Directory with this DN -> user is not
allowed to open mail• Solution
• AddAD$DNtoACL• OraddAD$DNtoDominoFullname (ADDNcontains,asdelimiter
betweenou)
• Or: • http://tdiblog.anderls.com/2015/02/adding-user-active-directory.html• ThanksAndreasArtner
Toronto, June 6-7 2016
Backup
What to Backup
• UsinganexampleConnections installationguiderarelyexplainsbackups• Theseguidesnormallydonotmentionbackup,orwhattobackup• Diskcrashmeansdataloss
• Databasebackupsthrough filebackuparenotsupported andmostlynotrestorable
• Important!!!• DatabaseBackupthroughOnlineBackupscanbetakenwhenConnectionsisup• Offlinebackupsarealsopossible
• Ensurethefilesystem&DBbackuparerunatthesametimeofday• DBandFilesystemdatawill stayinsync– ifyoutakeyourDBbackupatmidnightandthefile
systematmiddaytheywillbeoutofsync
53
Backup
• Most important (minimum daily)• Databases(offlineoronline)• Sharedcontent
• Important• Configuration
• WebSphereApplicationServer• Connections• IBMHTTPServer• TDISolution
• Test if restore is possible!!!!• SeveralissueswithWebSphererestores,wherebinariesweren'ton
thetape
54
Toronto, June 6-7 2016
Checklists
Checklist
Do• Documentyourinstallationsteps
• Theofficialdocumentationissometimesconfusing,becauseallOSwithinonedocument
• UseaLDAPuserforconnectionsAdmin• Bepreparedforscaling
• ShareddirectoryonUNCpath• Nosmalldeploymentinstallations
• Tuneyourenvironment• READTHEDOCUMENTATION!!!!
Don’t• Usemultiple instancesDB2withsmall
resources• Installonasinglemachine (unlessthe
environmentisverysmallorfortest)• Copycustomizationstonewerversions
• jsp,ftl copywillbreaksomething• Useunstablefile shares• TestdeploymentwithserverIE• Testwithonlyonelanguage
56
Install Checklist• WebSphereApplicationServer
• ConfigureFederatedRepository• LtpaToken,enablesecurity
• WebSphereApplicationServerSupplements(IHS,Plugins)• DB2(orotherDBM)• TDI• AddWebserver toDmgr (useconfigurewebserver.bat)• EnableSSLonIHS• ImportIHSRootKeywithinWebSphere cell trustkeystore (retrieve fromport)• ConfigureCCM• InstalloptionalAddons likeFormsExperience Builder,IBMDocs,Cognos
57
Documentation
DocumentEVERYTHING!!!becauseyoucanremembereverythingyoudid….
58
Documentation
• Everyone hates writing documentation• BUT – make notes as you go, it doesn’t need to be a full
step by step guide with screenshots• Document all customizataions• Any additional changes made• Anything of note that deviates from the guides• Lessons learnt or how you solved issues• Use the scripts to output some of it
59
Toronto, June 6-7 2016
Resources
Useful Tools
• Editor with syntax highlighting• vim,geany• notepad++,pspad,UltraEdit
• Tail• baretail• multitail• mtail
• Proxy• Fiddler(oftenaskedforbyIBM
Support)• Burpsuite (interceptproxy)
• Browser• Firefox(portable)/Firefox
ESR• Chrome• IE(downloadvmwith
differentversions)• https://www.modern.ie
• Network analyzer• Wireshark• tcpdump
• Unzip / Unarchiver• 7-zip• WinRar
61
Links and References
• IBMConnectionsSystemRequirements• http://www-01.ibm.com/support/docview.wss?uid=swg27012786
• IBMConnectionsFamilyDocumentation• http://www.ibm.com/support/knowledgecenter/SSYGQH/welcome
• IBMConnections4PerformanceTuningGuide• https://www-10.lotus.com/ldd/lcwiki.nsf/dx/IBM_Connections_4.0_Performance_Tuning_Guide
• IBMConnections4.5PerformanceTuningGuideAddendum• https://www-10.lotus.com/ldd/lcwiki.nsf/dx/IBM_Connections_4.5_Performance_Tuning_Guide_Addendum
• IBMConnections5CR1PerformanceTuningGuide• https://www-10.lotus.com/ldd/lcwiki.nsf/dx/IBM_Connection_V5_CR1_Tuning_guide
62
Useful Blogs• http://ibmconnections.com• http://turtleblog.info• http://portal2portal.blogspot.de• https://www.urspringer.de• http://socialconnections.info• http://blog.robertfarstad.com• http://www.curiousmitch.com• http://www.ramsit.com/category/blog• http://techblog.gis-ag.info• https://milanmatejic.wordpress.com• http://ibmdocs.com• http://domino.elfworld.org• https://dontforgetthe0.com
• http://dilf.me.uk/socialshazza• http://www.stoeps.de• http://scripting101.org• http://meisenzahl.org• http://martin.leyrer.priv.at• http://kbild.ch• http://www.notesgoddess.net• http://www.dominodiva.com• http://notesbusters.com• https://rob59blog.wordpress.com• http://connections101.info• http://brandlrainer.blogspot.de• https://collaborationben.com
63
Thank you very much for your attention!
panagenda GmbH – MakeYourDataWorkforYou
Lahnstr. 17● 64646 Heppenheim (Germany)
Skype:christophstoettner ● Cell:+49173 8588719
E-Mail:[email protected]
ChristophStoettnerSeniorConsultant
panagenda GmbH – MakeYourDataWorkforYou
Lahnstr. 17● 64646 Heppenheim (Germany)
Skype:nico.meisenzahl ● Cell:+49170 7355081
E-Mail:nico.meisenzahl@panagenda .com
NicoMeisenzahlConsultant
64
PLATINUM&SPOTLIGHTSPONSORS
GOLDSPONSORS
SILVERSPONSORS
BRONZESPONSORS