EMC Documentum System6Upgrade · EMC®Documentum® System6Upgrade Version6 CaseStudy P/N300006414...

22
EMC ® Documentum ® System 6 Upgrade Version 6 Case Study P/N 300006414 EMC Corporation Corporate Headquarters: Hopkinton, MA 01748‑9103 1‑508‑435‑1000 www.EMC.com

Transcript of EMC Documentum System6Upgrade · EMC®Documentum® System6Upgrade Version6 CaseStudy P/N300006414...

Page 1: EMC Documentum System6Upgrade · EMC®Documentum® System6Upgrade Version6 CaseStudy P/N300006414 EMCCorporation CorporateHeadquarters: Hopkinton,MA01748‑9103 1‑508‑435‑1000

EMC® Documentum®

System 6 Upgrade

Version 6

Case StudyP/N 300­006­414

EMC CorporationCorporate Headquarters:

Hopkinton, MA 01748‑91031‑508‑435‑1000www.EMC.com

Page 2: EMC Documentum System6Upgrade · EMC®Documentum® System6Upgrade Version6 CaseStudy P/N300006414 EMCCorporation CorporateHeadquarters: Hopkinton,MA01748‑9103 1‑508‑435‑1000

Copyright © 2008 EMC Corporation. All rights reserved.Published January 2008EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.THE INFORMATION IN THIS PUBLICATION IS PROVIDED AS IS. EMC CORPORATIONMAKES NO REPRESENTATIONS ORWARRANTIESOF ANY KINDWITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIESOF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.For the most up‑to‑date listing of EMC product names, see EMC Corporation Trademarks on EMC.com.All other trademarks used herein are the property of their respective owners.

Page 3: EMC Documentum System6Upgrade · EMC®Documentum® System6Upgrade Version6 CaseStudy P/N300006414 EMCCorporation CorporateHeadquarters: Hopkinton,MA01748‑9103 1‑508‑435‑1000

EMC Documentum Technical Document

Table of Contents

Introduction ....................................................................................................... 4Approach................................................................................................... 4

System Planning and Topologies ...................................................................... 6Pre‑upgrade topology ................................................................................. 6Concerns and dependencies ........................................................................ 8Post upgrade topology ................................................................................ 9Resource requirements............................................................................ 12High availability, backup, and recovery.................................................... 14

Upgrade Tasks ................................................................................................... 15Creating a new full‑text index on the test machine ........................................ 15Main upgrade procedure ............................................................................ 15Webtop links .......................................................................................... 17Migrating the full‑text index from test to production................................. 18Shutting down BEA ............................................................................ 19BEA permissions ............................................................................ 19

Other issues ............................................................................................... 19Webtop permissions ............................................................................... 19Content Server and application server clocks synchronization ................... 20eRoom issue ........................................................................................... 20

Go­live Activities ............................................................................................... 21

3 EMC Documentum System 6 Upgrade Case Study

Page 4: EMC Documentum System6Upgrade · EMC®Documentum® System6Upgrade Version6 CaseStudy P/N300006414 EMCCorporation CorporateHeadquarters: Hopkinton,MA01748‑9103 1‑508‑435‑1000

EMC Documentum Technical Document

IntroductionThis document presents a case study for upgrading an internally deployed Documentumproduction system to the latest release of Documentum products (Documentum 6). Historically,Documentum has made it a practice for internal groups to use our own products to helpimprove product quality and refine feature sets. For Documentum 6, the goal was to upgradeDocumentum’s production system immediately upon general availability and to better serveour customer base by sharing the same experiences our customers would encounter. The mainbenefits to be derived from this approach include the following:• Find or fix product issues early by leveraging internal user community.• Gain internal knowledge about system deployment issues and pass this knowledge on to

customers.• Upgrade hardware, for increased capacity and performance, to meet the internal needs or a

growing organization.• Enable internal customers to take advantage of new features in the D6 platform.• Enable internal customers to take advantage of additional product capabilities provided in the

Documentum production system.A summary of the upgrade is as follows:• Upgrade our main corporate repository (dm_notes) from 5.3 SP1 to D6.• Upgrade our main client (WDK/Webtop) from 5.3 SP3 to D6.• Deploy DCE to provide collaboration capability.The following additional products are planned for future phased deployments:• Branch Office Caching Services (BOCS) and Document Messaging Service (DMS) – to improve

performance over a globally distributed network

Business Process Services (BPS) and Business Process Integrator (BPI) – for designingworkflows and extending them outside the firewall

• XML Transformation Services (XTS) – for rendering XML source documentsThe case study captures information related to the following areas:• System Planning and Topologies, page 6• Upgrade Tasks, page 15• Go‑live Activities, page 21

ApproachThe basic approach for conducting the upgrade involves duplicating the production environmentonto another system, upgrading the duplicate system, and switching over to the duplicate systemas the production system. Prior to performing the final cut‑over to make the duplicate system theproduction system, we perform a final synchronization of all the data files from the productionsystem to the duplicate system.

The upgrade project was conducted in three phases (development, test, and production). Themain objectives of each phase were as follows:

4 EMC Documentum System 6 Upgrade Case Study

Page 5: EMC Documentum System6Upgrade · EMC®Documentum® System6Upgrade Version6 CaseStudy P/N300006414 EMCCorporation CorporateHeadquarters: Hopkinton,MA01748‑9103 1‑508‑435‑1000

EMC Documentum Technical Document

• Development – Customizations are typically tested during the development phase, but sincethere are no customizations in the Documentum production system, we used this phase towork with some of the early system builds. This enabled us to acquire early familiarity withthe upgrade process as well as to identify product issues prior to release. We also focused onmaking sure we had the right hardware resources in place to handle product requirementsand anticipated requirements of our growing internal customer base.

• Testing – During the testing phase, we focused on making sure the various productsand machines comprising the system worked together as expected. We conductedsome basic testing of new features in the Documentum 6 products. We also engagedwith the main internal stakeholders of the Documentum production system (TechPubs,Program Management, Finance, Human Resources, Legal). These tests were conducted byrepresentatives from each of the groups, and involved using the system as they normallywould use the system in performance of their day‑to‑day activities.

• Production – During the production phase, we tweaked the hardware environment basedon what we learned during the testing phase. The production upgrade occurred over aweekend, during which time we brought down the production system and performed a finalsynchronization of the data files from the previous production system to the new productionsystem. Following the switch over, we repeated the same set of system level tests and engagedthe internal stakeholders to repeat their tests to ensure everything was working as expected.

EMC Documentum System 6 Upgrade Case Study 5

Page 6: EMC Documentum System6Upgrade · EMC®Documentum® System6Upgrade Version6 CaseStudy P/N300006414 EMCCorporation CorporateHeadquarters: Hopkinton,MA01748‑9103 1‑508‑435‑1000

EMC Documentum Technical Document

System Planning and TopologiesPrior to the upgrade, the Documentum production system comprised a variety of differentproducts and product versions. Only a portion of this entire system was to be upgraded; therest of the system needed to be kept as it was.

Pre­upgrade topologyFigure 1, page 6 shows the entire system prior to the upgrade. Other repositories, not relatedto the dm_notes repository or involved in the upgrade, are shown for reference because theywere all served by a common Webtop instance.

Figure 1. Pre­upgrade topology

6 EMC Documentum System 6 Upgrade Case Study

Page 7: EMC Documentum System6Upgrade · EMC®Documentum® System6Upgrade Version6 CaseStudy P/N300006414 EMCCorporation CorporateHeadquarters: Hopkinton,MA01748‑9103 1‑508‑435‑1000

EMC Documentum Technical Document

Table 1, page 7 provides additional technical information for each of the pre‑upgrade productsserving the dm_notes repository.

Table 1. Pre­upgrade machine specifications

Nickname NetworkDesignation

Machine Specs(CPU, RAM,HD)

InstalledSoftwareModules

Notes

AutoRenderServer

wcw‑ferret.corp.emc.com

Machine:VMWare

CPU: (1) 2.66GHz

RAM: 1 GB

HD: 20 GBvirtual disk

OS: Windows2000SP4

AutoRender Pro4.2.2

eRoom Server CambISeroom01.documentum.com

Machine: HPNetServer

CPU: 4x 550MHz

RAM: 1.5 GB

HD: 204 GB

OS: Windows2000 Server SP4

eRoomEnterprise 7.3

Full‑textIndexing Server

Copperhead.lss.emc.com

Machine: FujitsuPRIMEPOWER450

CPU: (4) 1.3 GHz

RAM: 8 GB

HD: 984 (111used)

OS: Solaris 9

Full‑text indexserver 5.3 SP1

Part of ContentServer 5.3 SP1

Content Server Python.documentum.com

Machine: FujitsuPRIMEPOWER450

CPU: (4) 1.3 GHz

RAM: 8 GB

HD: 1228 (1062used)

OS: Solaris 9

RDBMS: Oracle10g

Content Server5.3 SP1

EMC Documentum System 6 Upgrade Case Study 7

Page 8: EMC Documentum System6Upgrade · EMC®Documentum® System6Upgrade Version6 CaseStudy P/N300006414 EMCCorporation CorporateHeadquarters: Hopkinton,MA01748‑9103 1‑508‑435‑1000

EMC Documentum Technical Document

Nickname NetworkDesignation

Machine Specs(CPU, RAM,HD)

InstalledSoftwareModules

Notes

ApplicationServer

Beautyprod.lss.emc.combeauty.documentum.com

Machine:VMWare

CPU: (2) 3 GHz

RAM: 3.5 GB

HD: 96 (17 used)

OS: Red HatLinux ES release4 (update 5)

ApplicationServer: BEAWebLogic 8.1SP5

Webtop 5.3 SP3

DocumentumAdministrator5.3 SP1

Enterprise SearchServer

EcisProd.corp.emc.com

Machine:VMWare

CPU: (2) 2400MHz

RAM: 3.6 GB

HD: 65 GB (SAN)

OS: Windows2003 SP2

ECIS 5.3 SP3

Concerns and dependenciesThe main concern we had early on was how the Documentum 6 platform would interact with thewide variety of product versions currently constituting the system. For example, the followingrepositories, and their associated Content Server versions, were being served by a single Webtopinstance (beauty 5.3 SP3):• dm_notes (5.3 SP1)• Us_global (4.2.6)• ge_internal (5.1)• uk_internal (5.1)• wcmprod (5.3)• deploy_support (5.3 SP1)• dm_notes (5.3 SP1)• support (5.3 SP2)

In addition, we had the following concerns regarding how existing version products in theDocumentum system would interact with those products in the system being upgraded toDocumentum 6 (Table 2, page 9 ).

8 EMC Documentum System 6 Upgrade Case Study

Page 9: EMC Documentum System6Upgrade · EMC®Documentum® System6Upgrade Version6 CaseStudy P/N300006414 EMCCorporation CorporateHeadquarters: Hopkinton,MA01748‑9103 1‑508‑435‑1000

EMC Documentum Technical Document

• eRoom integration. We were unsure how the current eRoom integration would work. In fact,we discovered (post upgrade) that the eRoom server needed to be patched with an upgradedversion of DFC.

• Desktop client compatibility (5.3x, 5.2.5x, 4.2.6x, 3.1.7). We were unsure whether theseversions of Documentum Desktop would work against a D6 repository.

• ECIS integration. We did not know what we would have to do to make D6 ECIS work withD6 Webtop.

• Full‑text index server. Prior to the upgrade, one instance of the full‑text index serversupported both the dm_notes and support repositories. We were unsure whether we couldcontinue to have one full‑text index server serving both repositories (since they would nowbe at widely divergent versions) or whether we would need to create another instance ofthe full‑text index server.

• Routers vs. Workflow on dm_notes. Routers are WorkSpace related, which is not supportedin D6. All open routers needed to be completed prior to the upgrade.

• AutoRender ‑ 4.2. Since AutoRender 4.2 is not supported with D6 repositories, we had toinstall a new rendering application for D6, Content Transformation Services, for which wehad no domain expertise.

Table 2. Upgraded products

Upgraded Product Ending Version

Content Server 5.3 SP1 Content Server D6

Web Development Kit (WDK) 5.3 SP3 WDK D6

Webtop 5.3 SP3 Webtop D6

Documentum Administrator 5.3 SP1 Documentum Administrator D6

eRoom Enterprise 7.3 eRoom Enterprise 7.3

Enterprise Content Integration Services (ECIS)5.3 SP3

ECIS D6

AutoRender Pro 4.2.2 Replaced by Advanced DocumentTransformation Services (ADTS) D6

Post upgrade topologyDue to incompatibilities between the Documentum 6 version products and pre‑5.3 versionproducts, the pre‑upgrade system (Figure 1, page 6 ) was split into two different subsystems, onethat supports 5.3 and later Content Servers (Figure 2, page 10) and one that supports pre‑5.3Content Servers (Figure 3, page 11).

EMC Documentum System 6 Upgrade Case Study 9

Page 10: EMC Documentum System6Upgrade · EMC®Documentum® System6Upgrade Version6 CaseStudy P/N300006414 EMCCorporation CorporateHeadquarters: Hopkinton,MA01748‑9103 1‑508‑435‑1000

EMC Documentum Technical Document

Figure 2. D6­based system for 5.3 and later components

10 EMC Documentum System 6 Upgrade Case Study

Page 11: EMC Documentum System6Upgrade · EMC®Documentum® System6Upgrade Version6 CaseStudy P/N300006414 EMCCorporation CorporateHeadquarters: Hopkinton,MA01748‑9103 1‑508‑435‑1000

EMC Documentum Technical Document

Figure 3. Non­upgraded system for pre 5.3 components

WDK/Webtop 6 is not backward compatible with Content Server versions prior to 5.3 (us_global,uk_internal, ge_internal). Since upgrading these repositories to D6 was out of scope, we had toimplement two different WDK/Webtop‑based systems, one for 5.3 and later Content Servers andthe other for pre‑5.3 version Content Servers. The existing WDK/Webtop instance was retained tosupport the pre‑5.3 Content Servers and a new D6 WDK/Webtop instance was created to supportthe 5.3 and later Content Servers.

In the pre‑upgrade system, one full‑text index server supported both the support and dm_notesrepositories. In order to continue to support the dm_notes (version 6) and support (version 5.3SP2) repositories, the full‑text index server would have to have been upgraded to 5.3 SP5. Sincethe full‑text index server is part of the Content Server, and no Content Servers were planned tobe upgraded to 5.3 SP5, this option did not make sense. Also, since one of the purposes of theupgrade was to assist in testing the D6 product, it made sense to have two full‑text index servers;one for the upgraded dm_notes repository and one for the 5.3 SP2 support repository.

In the pre‑upgrade system, AutoRender Pro provided PDF and HTML rendering for boththe us_global and dm_notes repositories. Document Transformation Services (DTS) providescomparable functionality for D6 repositories. Since neither AutoRender Pro nor DTS supportsthe full range of Content Servers in use (4.2.2 thru 6), AutoRender Pro was retained for thenon‑upgraded system and DTS was used for the D6‑based system.

EMC Documentum System 6 Upgrade Case Study 11

Page 12: EMC Documentum System6Upgrade · EMC®Documentum® System6Upgrade Version6 CaseStudy P/N300006414 EMCCorporation CorporateHeadquarters: Hopkinton,MA01748‑9103 1‑508‑435‑1000

EMC Documentum Technical Document

Resource requirements

The creation of an additional D6‑based system required additional machines and OS softwarelicenses for those machines. The pre‑upgrade system involved 12 hosts. The two post‑upgradesystems together involved 17 hosts, or 5 additional machines. While we were able to procure fouradditional machines from other completed projects (the distributed beta program and from theinfrastructure team), we did have to purchase one additional Linux license for those machines. Wealso had to purchase one new machine for the ADTS.

These systems were used because of their availability, not because they met any independentlyassessed sizing guidelines. We would have preferred to use our own generally available sizingguidelines to independently assess machine requirements and purchase or reuse machines thatfit those requirements. However, we were budget and time contrained, so we made use of thosemachines that were available.

For the full‑text index server, we generally recommend an allocation of three times more storageto the full‑text index than to the content filestores. However, during the development and testphases of the project, we found that the full‑text storage area was only 40% the size of the contentfilestores. Since our SAN storage architecture supports relatively easy expansion of storage,should the need arise, we went with a full‑text storage allocation in line with the 40% ratio.

No benchmark testing was performed to validate the new machine topology. Our enterpriseengineering group validated the basic topology. Various internal stakeholders (TechPubs,Program Management, Finance, Human Resources, Legal) conducted their normal activities to seeif there were any issues. There were cases where internal users needed to update their version ofJava and/or Documentum Desktop.

Table 3, page 13 provides additional technical information for each of the post upgradecomponents serving the dm_notes repository.

12 EMC Documentum System 6 Upgrade Case Study

Page 13: EMC Documentum System6Upgrade · EMC®Documentum® System6Upgrade Version6 CaseStudy P/N300006414 EMCCorporation CorporateHeadquarters: Hopkinton,MA01748‑9103 1‑508‑435‑1000

EMC Documentum Technical Document

Table 3. Post upgrade machine specifications

Nickname NetworkDesignation

Machine Specs(CPU, RAM,HD)

InstalledSoftwareModules

Notes

Content Server PYTHON Computer:Fujitsu 450

CPU: 4 X 1.3 Ghz

RAM: 8 GB

Storage: 1118(999 used)

OS: Solaris 9

RDBMS: Oracle10g R2

Content Server 6(2 instances, 1 forglobal registry)

New machine forD6 system.

Changedmachine fromSun Enterprise420R in devand Sun Fire480R in testdue to sluggishperformance.

Switched fromSolaris 10 in devto Solaris 9 intest.

ApplicationServer

beautyprod(VMware Image)

Machine:VMWare

CPU: (2) 3 GHz

RAM: 3.5 GB

Storage: 97 GB(5.9 GB used)

OS: RedHatEnterprise LinuxES release 4(Update 5)

ApplicationServer: BEAWebLogic Server9.2

Java: Sun J2SEJDK 5.0 Update10

New machine forD6 system.

Added DCE andDA during test.

eRoom Server cambiseroom01 Same machinefrom pre‑upgrade (Table 1,page 7 ).

Full‑textIndexing Server

copperhead.lss.emc.com

Same machinefrom pre‑upgrade (Table 1,page 7 ).

OS: Solaris 9

Full‑text indexserver 5.3 SP1

Fast index server6

UNIX allows twoinstallations ofthe FAST indexserver on thesame machine.

EMC Documentum System 6 Upgrade Case Study 13

Page 14: EMC Documentum System6Upgrade · EMC®Documentum® System6Upgrade Version6 CaseStudy P/N300006414 EMCCorporation CorporateHeadquarters: Hopkinton,MA01748‑9103 1‑508‑435‑1000

EMC Documentum Technical Document

Nickname NetworkDesignation

Machine Specs(CPU, RAM,HD)

InstalledSoftwareModules

Notes

ECIS Server ecisprod.corp.emc.com

Same machinefrom pre‑upgrade (Table 1,page 7 ).

OS: Windows2003 SP2

ECIS 5.3 SP3

DTS Server wcw‑rendserv.corp.emc.com

Machine: DellPoweredge 1750

CPU: (1) 3.2 GHz

RAM: 4 GB

HD: 73 GB

OS: WindowsServer 2003 SP2

New machineduring test.

Changedmachinefrom test toproduction.

High availability, backup, and recovery

Our internal deployment of dm_notes does not implement highly available or failoverarchitectures. Full backups of dm_notes are made to tape on a weekly basis and incrementalbackups are made on a daily basis. Database archive logs are backed up every 12 hours.

For the upgrade, we used a BCV (Business Continuous Volume) mirroring device to create a fullcold backup of dm_notes. The BCV backup only takes 10 minutes to create in comparison to the 12to 14 hours it takes to create a cold backup to tape. If we needed to recover from a cold backup, wewould mount the BCV backup, which remains on disk.

If we needed to recover from tape, we would restore content files and metadata, and the DBA teamwould synchronize restored information with the file system. While the restore/synchronizationprocess may result in a loss of some data, typically it is no more than one day’s worth.

14 EMC Documentum System 6 Upgrade Case Study

Page 15: EMC Documentum System6Upgrade · EMC®Documentum® System6Upgrade Version6 CaseStudy P/N300006414 EMCCorporation CorporateHeadquarters: Hopkinton,MA01748‑9103 1‑508‑435‑1000

EMC Documentum Technical Document

Upgrade TasksThis chapter highlights the main tasks that we performed to upgrade the production system.

Creating a new full­text index on the testmachineSince the existing 5.3 SP1 index could not be migrated to Documentum 6, a new D6 full‑text indexwas generated for the test environment. Dm_notes is a medium‑sized repository with 793 GB ofcontent. While we are not sure how much of that content is indexable, 2,174,579 documents wereindexed. The fixml directory, which is the index server’s internal representation of all the contentfiles and properties to be indexed, was 14 GB and took about a week to generate. The indexgenerated from the fixml is about 31 GB and took about 22 hours to generate. Though we raninto bugs and problems while creating the index, it probably would have taken roughly sameamount of time had everything gone smoothly.

Main upgrade procedureThe following indicates the main tasks we conducted during the upgrade. Go‑live Activities, page21 indicates the timing of these tasks and the actor who performed them.

1. Installed ADTS on a new machine (Wcw‑rendserv).

2. Deployed the new splash page to redirect 5.3 Webtop users to the D6 Webtop (Webtop links,page 17).

3. Installed a new Documentum 6 full‑text indexing server (FAST) instance on Copperhead.

Note: There is no upgrade for the 5.3 SP1 full‑text indexing server (or full‑text index) toDocumentum 6, so a new Documentum 6 instance of the full‑text indexing server was installedon the same host (Copperhead) as the 5.3 SP1 instance.

4. Sent out a notification to the users.

5. Performed an incremental backup for the following servers:• Beautyprod (Webtop/DA)• Ecisprod (ECIS)• Copperhead (FAST)

Note: We performed incremental backups as a precaution, in case we needed to recoverspecific files. Otherwise, if a problem occurred, we would reinstall the software, which isfaster than restoring from a backup.

6. Performed a full cold backup of dm_notes.

a. Shut down Content Server (including the Java method server) on Python.

b. Shut down Oracle.

EMC Documentum System 6 Upgrade Case Study 15

Page 16: EMC Documentum System6Upgrade · EMC®Documentum® System6Upgrade Version6 CaseStudy P/N300006414 EMCCorporation CorporateHeadquarters: Hopkinton,MA01748‑9103 1‑508‑435‑1000

EMC Documentum Technical Document

c. Performed a full cold backup of Python.

d. Started up Oracle.

Note: Content Server is on SAN storage, which enabled us to use BCV software to mirrorour data files. After we shut down the applications on Python, we completed a finalsynchronization of data and the mirror was created. If needed, we could have mounted themirror and been back to a pre‑upgrade state in minutes, as opposed to hours/days required fora tape restore.

7. Changed DNS entries for beauty (beauty.documentum.com), beauty1 (Webtop 5.3), andbeauty2 (Webtop 6).

a. Moved the main Webtop URL (beauty.documentum.com) to a different host with adifferent IP address.

b. Updated the IP address for beauty, beauty1, and beauty2 on the EMC internal DNS servers.

c. Reduced the TTL (time to live) on DNS servers so that IP address changes were effectiveimmediately.

8. Pointed the 5.3 Webtop instance to Viper (port 1489).

Note: Though Viper already hosted the US_Global Docbase/connection broker, we could haveused a different server with an appropriate connection broker. We needed to repoint the5.3 SP3 Webtop instance so that dm_notes would not be displayed as a possible repositoryto 5.3 SP3 Webtop users.

9. Installed new application server (Beautyprod).

a. Shut down Beautyprod.

b. Installed BEA Weblogic 9.2 MP1 on Beautyprod.

10. Installed D6 global registry repository on Python.

11. Installed D6 Webtop and DA on Beautyprod.

Note: D6 Webtop and DA were installed under a different user on the same machine as the5.3 SP3 Webtop and 5.3 SP1 DA.

12. Uninstalled and re‑installed ECIS on Ecisprod.

a. Uninstalled ECIS 5.3 SP3 from Ecisprod.

b. Installed ECIS 6 on Ecisprod.

c. Updated Webtop’s dfc.properties file to point to the D6 ECIS instance.

13. Migrated the full‑text index from test to production (Migrating the full‑text index from test toproduction, page 18).

14. Ran the D6 Content Server upgrade on dm_notes (Python).

15. Ran the report.sh script.

Note: The report.sh monitors repository, connection broker, and Oracle availability. If any oneof these applications crashes, the script restarts the application and notifies us. If we don’tcomment out the script, the applications are automatically restarted after we shut them down.

16. Installed the D6 DCE docapp.

17. Synchronized processes and configured the indexagent.

16 EMC Documentum System 6 Upgrade Case Study

Page 17: EMC Documentum System6Upgrade · EMC®Documentum® System6Upgrade Version6 CaseStudy P/N300006414 EMCCorporation CorporateHeadquarters: Hopkinton,MA01748‑9103 1‑508‑435‑1000

EMC Documentum Technical Document

Note: We mounted the storage drives from the production repository host to the productionindex server host so that content to be indexed can be accessed as if it resides on the indexserver host instead of having to transport the content over the network. This mounting methodenhances performance because the index server can use the get_path method to obtain thecontent to be indexed instead of having to use the slower get_file method. To set this up, we setthe <all_filestore_local> element to true within the indexagent.xml file on the index server host.

18. Pointed D6 webtop to 1490 and global registry to 1499.

Note: To control which repositories were available to each Webtop instance, we neededto configure two more connection brokers. The dm_notes and support repositories wereprojected to the Python connection broker on port 1490 for use with the D6 Webtop instance.Since dm_notes (and only dm_notes) also needs to communicate with the global registry, weprojected the global registry and dm_notes repository to the Python connection broker onport 1499. The global registry also projects to port 1490, but we will likely change this whenwe upgrade to SP1.

We left all the repositories projecting to the Python connection broker on port 1489 so thatDocumentum Desktop users can see all the repositories without modifying their dmcl.ini files.

19. Tested basic functionality (jobs, checkin/checkout, user administration, etc.).

20. Verified eRoom servers connectivity and eRoom link functionality.

21. Tested TechPubs XML application functionality.

22. Tested Account Admin/firewall access.

Webtop links

Due to incompatibility of Webtop 6 with pre‑5.3 repositories, the existing 5.3 SP1 Webtop instanceneeded to be retained to support those repositories. The two Webtop instances required uniqueURLs, which meant that the URL (http://beauty.documentum.com/webtop), DRLs, and virtuallinks for some of the Webtop users would be broken. To solve this problem, we added a “landingpage” to redirect users to the correct Webtop instance and added some code to fix the URLs andvirtual links so they pointed to the correct Webtop instance.

During the system upgrade, the existing WebTop 5.3 SP1 instance (beauty1) was left in place and anew D6 Webtop instance (beauty2) was installed on the same host, but with a different IP address.The production configuration resulted in the following:• beauty (beauty.documentum.com) – splash page that redirects users to the appropriate

Webtop instance (5.3 or 6) depending on the selected repository. Resolves old DRLs andbookmarked URLs to the appropriate Webtop instance (beauty1 or beauty2).

• beauty1 (Webtop 5.3 SP1) – serves all old 5.2.5 and below repositories (deploy support,ge_internal, uk_internal, us_global, wcmprod)

• beauty2 (Webtop 6) – serves all 5.3 and above (including Documentum 6) repositories(dm_notes, support)

EMC Documentum System 6 Upgrade Case Study 17

Page 18: EMC Documentum System6Upgrade · EMC®Documentum® System6Upgrade Version6 CaseStudy P/N300006414 EMCCorporation CorporateHeadquarters: Hopkinton,MA01748‑9103 1‑508‑435‑1000

EMC Documentum Technical Document

Migrating the full­text index from test to production

Since the entire upgrade happened over a weekend, and involved bringing down the system forall the users, we did not want to spend the time to generate a new full‑text index from scratch onour production environment. Instead, we copied both the fixml and index directories from the testenvironment to the production environment. On the production environment, we then indexedthe difference in the number of documents created between the time the test environment wasrefreshed with production content and the time of the actual indexing of the production content(between 8/13 and 9/7 or approximately 40 thousand documents). That saved us approximately20 hours of indexing.

The full‑text index was created for D6 dm_notes as follows:

1. Generated a new full‑text index on our test version of dm_notes.

Note: The test machine was refreshed with production content on 8/13.

2. Stopped all processes writing fxml on the test system.

a. On the test environment, stopped the index agent and index server. This was done fromthe Index Agent/Server Administration page.

b. Stopped the BEA Weblogic processes (Shutting down BEA, page 19).

3. Copied the full‑text index from the test machine to the production machine.

a. Copied the entire fixml directory (/export1/6.0/data/fulltext/fixml) from the test machine(diamondback) to the production machine (copperhead).

Note: We used the cpio copy utility. At the start of the copy, an index of the files wastaken. If source files were deleted during the copy, cpio would have reported an errorwhen the copy was completed. We checked the sizes after the copy to make sure thesource and destination were roughly the same size. The du ‑sk directoryname or du ‑sk *commands were used to check the sizes.

The two fixml directories should have been the same size, but in our case they were notbecause BEA processes were continuing to write fixml on the test machine (Shuttingdown BEA, page 19).

b. Copied the entire index directory (/export/6.0/data/fulltext/index ) from the test machine(diamondback) to the production machine (copperhead).

4. Updated the full‑text index on the production machine.

a. Assessed the difference between the fixml documents on the test machine and the fixml onthe production machine. The difference represented newly created fixml documents thatwere created on the production environment since the test environment was refreshedwith production content on 8/13, or about 40 thousand documents.

b. Indexed the 40 thousand newly created documents (created between 8/13 and 9/7). Thisonly took about two hours, compared to the 22 hours it took to create the entire indexfrom all the fixml.

18 EMC Documentum System 6 Upgrade Case Study

Page 19: EMC Documentum System6Upgrade · EMC®Documentum® System6Upgrade Version6 CaseStudy P/N300006414 EMCCorporation CorporateHeadquarters: Hopkinton,MA01748‑9103 1‑508‑435‑1000

EMC Documentum Technical Document

Shutting down BEA

When we compared the fixml file sizes between the test environment and the productionenvironment, we discovered that the file sizes did not match. This was determined to be due toBEA processes that continued to generate fixml on the test (or production) environment, despitethe index agent and index server being stopped.

The solution was to shut down all BEA processes to discontinue writing to the fixml directory.This was done by entering the following commands.cd $DOCUMENTUM/fulltext/IndexServer/bin/source setupenv.csh$DOCUMENTUM/fulltext/IndexServer/bin/shutdown.sh$DOCUMENTUM/bea9.2/domains/DctmDomain/stopIndexAgent2.sh$DOCUMENTUM/bea9.2/domains/DctmDomain/bin/stopWebLogic.sh

There was also an issue associated with shutting down BEA (BEA permissions, page 19).

BEA permissions

BEA created a directory, /var/tmp/wlstTemp, which was owned by gradmin. When dm20 triedto shutdown the BEA instance it also referred to this folder. Since dm20 did not own it or haveaccess, we got an error message. Deleting the directory made it so dm20 could shut down BEAwithout error, but then gradmin would have the same issue. Since we were not running BEA withthe global registry repository, this was not be an issue for us, however other sites may have anissue. For these other sites, the following fix should resolves the issue.

1. Back up the <DctmDomain> dir/bin/stopWebLogic.sh file.

2. Open the <DctmDomain> dir/bin/stopWebLogic.sh file.

3. Change the following:${JAVA_HOME}/bin/java ${JAVA_OPTIONS} weblogic.WLST shutdown.py 2>&1to${JAVA_HOME}/bin/java ‑Djava.io.tmpdir=<your temp dir> ${JAVA_OPTIONS}weblogic.WLST shutdown.py 2>&1where: <your temp dir> is what you want to use (for example, the user’s home directory).Explicitly setting the directory resolved the issue.

Other issues

Webtop permissions

Many users in 5.3 SP1 dm_notes have admin rights. In Documentum 6, those users with adminrights are able to create or change a preset, which has the potential for creating user accessproblems. Post upgrade, we plan to change many users from having admin rights to the rightsof a coordinator.

EMC Documentum System 6 Upgrade Case Study 19

Page 20: EMC Documentum System6Upgrade · EMC®Documentum® System6Upgrade Version6 CaseStudy P/N300006414 EMCCorporation CorporateHeadquarters: Hopkinton,MA01748‑9103 1‑508‑435‑1000

EMC Documentum Technical Document

Content Server and application server clockssynchronization

If the Content Server and application server clocks are out of synch by more than six hours (thedefault), URLs will expire and content transfer for both uploading and downloading of contentwill fail. The clocks for Content Server and the application server should be synched up with atime server. The time interval which, when exceeded, causes the URL to expire, is set in theacs.properties file on the Accelerated Content Services (ACS) host.

eRoom issue

When we upgraded the dm_notes repository to version D6, a new attribute was added(r_aspect_name). When an object is updated via Webtop (DCE), the r_aspect_name attribute ispopulated. DFC versions prior to DFC 6 do not handle this attribute. The problem was solved bypatching eRoom with the D6 version of DFC.

20 EMC Documentum System 6 Upgrade Case Study

Page 21: EMC Documentum System6Upgrade · EMC®Documentum® System6Upgrade Version6 CaseStudy P/N300006414 EMCCorporation CorporateHeadquarters: Hopkinton,MA01748‑9103 1‑508‑435‑1000

EMC Documentum Technical Document

Go­live ActivitiesThis section outlines the sequence and timing of upgrade activities during the switch over to theproduction system. These activities took place over a weekend, to cause the minimal amount ofdisruption to the end users, because the system must be taken down during the upgrade.

Day Task Time Period Actor

1 Checkpoint #1 2:00 p.m. team

1Installed ADTS onwcw‑rendserv 2:00 ‑ 3:00 p.m. IT person 1

1Deployed the newsplash page 5:00 ‑ 5:30 p.m.

IT person 2, IT person3

1

Installed/upgradedD6 full‑text indexserver on Copperhead 5:00 ‑ 6:00 p.m. IT person 4

2Sent out notificationto the users 9‑9:15 a.m.

IT person 5/IT person6

2 Checkpoint #2 10:00 a.m. team

2

Performedincremental backupon Beautyprod,Ecisprod andCopperhead servers 10:00 a.m. ‑12:00 p.m. IT person 8

2 Checkpoint #3 1:00 p.m. team

2

Shut down ContentServer (and JavaMethod Server) onPython 5:00 ‑ 5:10 p.m. IT person 1

2 Shut down Oracle 5:10 ‑ 5:15 p.m.DBA person 1 (DBAgroup)

2Performed BCV Split(backup Python) 5:15 ‑ 5:45 p.m. IT person 9

2 Started up Oracle 5:45 ‑ 6:00 p.m.DBA person 1 (DBAgroup)

2

Made DNS entrychanges forWebtop/TTL (beauty,beauty1 and beauty2) 5:00 ‑ 5:30 p.m. IT person 10

2Pointed 5.3 Webtop toViper (1489) 5:00 ‑ 5:30 p.m. IT person 3

2Shut downbeautyprod 5:00 ‑ 5:30 p.m. IT person 3

EMC Documentum System 6 Upgrade Case Study 21

Page 22: EMC Documentum System6Upgrade · EMC®Documentum® System6Upgrade Version6 CaseStudy P/N300006414 EMCCorporation CorporateHeadquarters: Hopkinton,MA01748‑9103 1‑508‑435‑1000

EMC Documentum Technical Document

Day Task Time Period Actor

2

Installed BEAWeblogic 9.2 MP1on beautyprod 5:30 ‑ 6:30 p.m. IT person 3

2Installed globalregistry D6 on Python 6:00 ‑ 7:30 p.m. IT person 1

2Installed D6Webtop/DA 6:30 ‑ 7:30 p.m. IT person 3

2Tested ECISfunctionality 7:30 ‑ 8:00 p.m.

ECM team (IT person1, IT person 3, ITperson 4, IT person 11)

2Upgraded Ecisprodserver 5:00 ‑ 6:00 p.m. IT person 4

2Moved fixml from testto production 6:00 ‑ 7:00 p.m. IT person 10

2Upgraded dm_notesto D6 7:30 ‑ 11:30 p.m. IT person 1

2 Enabled Report.sh 11:30 ‑ 11:45 p.m. IT person 1

3Installed DCE D6docapp 11:30 p.m. ‑ 12:30 a.m. IT person 1

3

Synched up processand configuredindexagent 12:30 ‑ 6:30 a.m. IT person 4

3

Pointed D6 Webtopto 1490 and globalregistry to 1499 6:30 ‑ 7:00 a.m. IT person 3

3

Tested basicfunctionality (jobs,checkin/checkout,user administration,etc.) 12:30 a.m. ‑ 1:30 a.m.

ECM team (IT person1, IT person 3, ITperson 4, IT person 11)

3

Verified eRoomservers connectivityand eRoom linkfunctionality 12:30 a.m. ‑ 1:30 a.m. IT person 1

3Tested TechPubsapplication 12:30 a.m. ‑ 1:30 a.m. Tech Pubs person

4Tested AccountAdmin/firewall access 8:00 ‑ 10:00 p.m.

HCL India(repository accountadministrators)

22 EMC Documentum System 6 Upgrade Case Study