Post on 08-Nov-2018
Quick and EasyOpenEdge AI File Management
PAUL KOUFALISPRESIDENT
PROGRESSWIZ CONSULTING
Progresswiz Consulting
Based in Montréal Québec CanadaBased in Montréal, Québec, CanadaProviding technical consulting in Progress®, UNIX, Windows, MFG/PRO and moreUNIX, Windows, MFG/PRO and moreSpecialized in performance tuning, system availability and business continuity planningy y p g…and security of Progress-based systems
© 2008 Progresswiz Consulting2 Quick and Easy OpenEdge AI File Management
Agenda
IntroductionIntroductionAfter-Imaging (AI) BasicsPre-OpenEdge Implementation ChallengesPre-OpenEdge Implementation ChallengesThe New AI File Management DaemonStep by Step ImplementationStep-by-Step ImplementationUnderstanding the ResultsA F Fi l StA Few Final StepsQuestions
© 2008 Progresswiz Consulting3 Quick and Easy OpenEdge AI File Management
Introduction
What is After-Imaging?What is After Imaging?
• Quick Definition:Quick Definition:
A logging system that stores all informationA logging system that stores all information required to reapply all changes made to a database between two points in time
© 2008 Progresswiz Consulting4 Quick and Easy OpenEdge AI File Management
Introduction
OpenEdge AI Gremlins Hard at Work!
© 2008 Progresswiz Consulting5 Quick and Easy OpenEdge AI File Management
Introduction
Recipe for RecoveryRecipe for Recovery• One backup• All AI files generated since that backupAll AI files generated since that backup
ResultResult• One fully recovered database with near-zero
data loss• “Near-zero” because open transactions will be
rolled back
© 2008 Progresswiz Consulting6 Quick and Easy OpenEdge AI File Management
Introduction
Why bother?Why bother?• Let’s see, because you like your job?
• Wait…wait…I know…data in a database is like, important, right?g
© 2008 Progresswiz Consulting7 Quick and Easy OpenEdge AI File Management
Introduction
But seriously folksBut seriously folks…• Most business’ only backup once per day• Without AI any changes made since the lastWithout AI, any changes made since the last
backup are lost
Best of all, it’s 100% free• Both Workgroup and Enterprise g p p• And Personal DB too!
© 2008 Progresswiz Consulting8 Quick and Easy OpenEdge AI File Management
After-Imaging (AI) Basics
Basic transaction process:Basic transaction process:1. Begin Transaction
a) Make some changeb) Before and after value written to BIc) Before and after value written to AId) Repeatd) Repeat…
2. End Transaction
Note that writes to the data files are not in the list• It will happen eventually
© 2008 Progresswiz Consulting9 Quick and Easy OpenEdge AI File Management
After-Imaging (AI) Basics
What happens to all these BI/AI notes?What happens to all these BI/AI notes?
• OpenEdge automatically manages the reuse• OpenEdge automatically manages the reuse of BI file space
• AI files must be archived before allowing OpenEdge to reuse them
© 2008 Progresswiz Consulting10 Quick and Easy OpenEdge AI File Management
After-Imaging (AI) Basics
Data Files BI Fil AI Fil 1 F llData Files(.d*)
BI File AI File 1 Full
D t
BI Notes Written
AI Notes Written
AI File 2
..
Full
Data Changes
AI File n
.
U P
Full
User Processes
Now What?
© 2008 Progresswiz Consulting11 Quick and Easy OpenEdge AI File Management
After-Imaging (AI) Basics
AI File 1EFull
AI File 1Empty
FullAI File 2
...
Empty Copy/Archive Safe Storage
AI File n
.
EmptyFull
© 2008 Progresswiz Consulting12 Quick and Easy OpenEdge AI File Management
After-Imaging (AI) Basics
Sounds simple enough:Sounds simple enough:• When full, copy/archive AI files to a safe
location• Mark them as empty so OpenEdge can reuse
the space in the files
What’s the problem?
© 2008 Progresswiz Consulting13 Quick and Easy OpenEdge AI File Management
Pre-OpenEdge Implementation Challenges
Problem #1: ScriptingProblem #1: Scripting
Commands are simple enough:Commands are simple enough:
• rfutil sports –C aimage extent full• rfutil sports -C aimage query extnum by name sports.a4p g q y y p• cp sports.a4 /archive/sports.${extnum}• rfutil <dbname> -C aimage empty
But adding the logic around them requires work
© 2008 Progresswiz Consulting14 Quick and Easy OpenEdge AI File Management
Pre-OpenEdge Implementation Challenges
Scripts to automate AI file management canScripts to automate AI file management can be as simple or complex as you like• Depends how “smart” you want them to bep y
Reality: Not that difficultReality: Not that difficult• Plenty of example scripts available online• Search the peg (www.peg.com)Search the peg (www.peg.com)
© 2008 Progresswiz Consulting15 Quick and Easy OpenEdge AI File Management
Pre-OpenEdge Implementation Challenges
Problem #2: Performance ImpactProblem #2: Performance Impact
Ye Olde Days of Yore yesYe Olde Days of Yore…yesToday: Please…• The effect is minimal unless you have really• The effect is minimal unless you have really
old junky hardware• Of course, there are exceptionsOf course, there are exceptions
– But if you’re that busy you really need AI!
© 2008 Progresswiz Consulting16 Quick and Easy OpenEdge AI File Management
Pre-OpenEdge Implementation Challenges
Resist the “If it ain’t broke ” mentalityResist the If it ain t broke… mentality• If you are not running AI your system IS
broken
With modern hardware the performance issue t ode a d a e t e pe o a ce ssueis mootAnd now with the AI File Management gDaemon, so is the scripting issue
© 2008 Progresswiz Consulting17 Quick and Easy OpenEdge AI File Management
The New AI File Management Daemon
Available as of 10 1AAvailable as of 10.1AAutomates:• Rotation and reuse of AI files• Rotation and reuse of AI files• Archiving of full extents• The <ahem> intelligent <ahem> naming ofThe <ahem> intelligent <ahem> naming of
archived AI extents– More on this later
It’s about time!!!
© 2008 Progresswiz Consulting18 Quick and Easy OpenEdge AI File Management
The New AI File Management Daemon
Can be activated in one of two modes:Can be activated in one of two modes:• Time interval: AI file rotation occurs every x
seconds– Min: 2 minutes– Max: 24 hours
• Full AI File: Rotation occurs when the currently active AI file is filled
T i ll d ith fi d l th AI t t– Typically used with fixed length AI extents
© 2008 Progresswiz Consulting19 Quick and Easy OpenEdge AI File Management
The New AI File Management Daemon
Two configurable parameters:Two configurable parameters:• -aiarcdir: Directory where full AI extents will
be copiedp– Can specify multiple directories in case writes
to the first fail• -aiarcinterval: Time between AI file rotations
– If not specified, rotation occurs when extent fills
© 2008 Progresswiz Consulting20 Quick and Easy OpenEdge AI File Management
How it Works1. Daemon wakes up and checks for full AI files
or elapsed aiarcinterval
2. Full AI files are copied to the archive di t d d i t l
AI Daemondirectory and renamed appropriately
3. Once successfully copied, AI files are marked as empty and available for reuse
DB
AI Files ArchivedAI Files
BI File
© 2008 Progresswiz Consulting21 Quick and Easy OpenEdge AI File Management
Step-by-Step Implementation
Four ways to activate AI and AI File Mgmt:Four ways to activate AI and AI File Mgmt:
After-Imaging AI File ManagementOFFLINE OFFLINEONLINE OFFLINEOFFLINE ONLINE
N t O li ti ti f 10 1B
OFFLINE ONLINEONLINE ONLINE
Note: Online activation as of 10.1B
© 2008 Progresswiz Consulting22 Quick and Easy OpenEdge AI File Management
Add AI Extents
Can be done offline or online as of 10 1ACan be done offline or online as of 10.1ACreate add.st file# Three variable length AI files in /db/proda /db/proda /db/proda /db/prod
Apply to database$ prostrct add sports add.st$ prostrct addonline sports add.st
© 2008 Progresswiz Consulting23 Quick and Easy OpenEdge AI File Management
Enable AI & AI Mgmt Daemon Offline
Take a full backup of your databaseTake a full backup of your database$ probkup sports /backup/sports.probkp
Enable after-imaging$ rfutil sports –C aimage begin
Enable AI Management Daemon$ rfutil sports C aiarchiver enable$ rfutil sports –C aiarchiver enable
© 2008 Progresswiz Consulting24 Quick and Easy OpenEdge AI File Management
Enable AI & AI Mgmt Daemon Offline
Start your databaseStart your database• Be sure to include the –aiarcdir and, if
required, -aiarcinteval startup parametersq , p p
$ proserve sports –pf sports.pf –aiarcdir /aiarch /aiarch2aiarcdir /aiarch,/aiarch2-aiarcinterval 900
© 2008 Progresswiz Consulting25 Quick and Easy OpenEdge AI File Management
Enable AI & AI Mgmt Daemon Online
Enabling AI & the AI File ManagementEnabling AI & the AI File Management Daemon offline required three distinct steps:• Backupp• Enable AI• Enable AI File Managementg
As of 10.1B, do it all in one commandAs of 10.1B, do it all in one command
© 2008 Progresswiz Consulting26 Quick and Easy OpenEdge AI File Management
Enable AI & AI Mgmt Daemon Online
$ probkup online sports/backup/sports.probkpenableaienableaiarchi erenableaiarchiver-aiarcdir /aiarch,/aiarch2-aiarcinterval 900
© 2008 Progresswiz Consulting27 Quick and Easy OpenEdge AI File Management
Mixing Online and Offline
Parameters enableai and enableaiarchiverParameters enableai and enableaiarchiverare optionalCan use one or other in probkup online cmdCan use one or other in probkup online cmd• Of course, AI must be enabled in order to
enable AI Archiver
© 2008 Progresswiz Consulting28 Quick and Easy OpenEdge AI File Management
Mixing Online and Offline
Example:Example:• Backup offline
$ probkup sports /backup/sports.probkp
E bl AI ffli• Enable AI offline$ rfutil sports –C aimage begin
• Start DB$ proserve sports –pf sports.pf
• Enable AI Archiver online$ probkup online sports/backup/sports.probkp$ probkup online sports/backup/sports.probkp
enableaiarchiver-aiarcdir /aiarch,/aiarch2-aiarcinterval 900
© 2008 Progresswiz Consulting29 Quick and Easy OpenEdge AI File Management
Disabling the AI Archiver Daemon
To permanently disable the AI ArchiverTo permanently disable the AI Archiver$ rfutil sports –C aiarchiver disable
To stop the daemon temporarily without disabling the functionality• I.e. for maintenance$ rfutil sports –C aiarchiver end
© 2008 Progresswiz Consulting30 Quick and Easy OpenEdge AI File Management
Disabling the AI Archiver Daemon
No documented way to restart AI ArchiverNo documented way to restart AI Archiver• I.e. there is no “aiarchiver begin” • Will start automatically at next proserveWill start automatically at next proserve
• Or run the command:Or run the command:$ _mprshut sports -C aimgt
© 2008 Progresswiz Consulting31 Quick and Easy OpenEdge AI File Management
AI Archiver Status
Activated?Activated?• Use “proutil <dbname> -C describe
Database Features
ID Feature Active Details---- --------------------------------- ------ -------
8 After Image Mangement/Archiver Yes9 64 Bit DBKEYS Yes10 Large Keys Yes11 64 Bit Sequences Yes
© 2008 Progresswiz Consulting32 Quick and Easy OpenEdge AI File Management
AI Archiver Status
Running?Running?• Look in promon – R&D – 1 – 4 – 1 10/22/08 Stat s All Processes10/22/08 Status: All Processes7:17:24
Usr Name Type Wait Trans id Login timeyp g
0 Paul Kou BROK -- 0 0 10/22/08 17:095 AIMD -- 0 0 10/22/08 17:096 Paul Kou MON -- 0 0 10/22/08 17:16
© 2008 Progresswiz Consulting33 Quick and Easy OpenEdge AI File Management
Changing Parameters Online
Both parameters (-aiarcdir and –aiarcinterval)Both parameters ( aiarcdir and aiarcinterval) can be changed online
$ f til t C i hi tdi / i h$ rfutil sports –C aiarchiver setdir /aiarchnew
$ rfutil sports –C aiarchiver setinterval 1800
The second option is particularly useful• Decrease interval during high activityDecrease interval during high activity• Increase interval during quiet times
© 2008 Progresswiz Consulting34 Quick and Easy OpenEdge AI File Management
Understanding the Results
Enabling AI Archiver affects three outputEnabling AI Archiver affects three output streams:• DB Log Fileg• AI Archive output directory (-aiarcdir)• AI Archiver logg
– New
© 2008 Progresswiz Consulting35 Quick and Easy OpenEdge AI File Management
Database Log File
AI Archiver entries show up as “AIMGT”AI Archiver entries show up as AIMGTExample:[2008/10/22@17:27:16.078-0400] P-5316 T-912 I AIMGT 5: (13213) A new archive interval 120 has been set.
Please don’t go blind!AIMGT 5: (13213) A new archive interval 120 has been set.AIMGT 5: (13213) A new archive interval 120 has been set.
© 2008 Progresswiz Consulting36 Quick and Easy OpenEdge AI File Management
Archived AI Files
The good:The good:• All the information you need is stored in the
file nameThe bad:• All the information you don’t need is stored in y
the file nameThe ugly:C!~TEMP~oewrk~db~pk.20081022.164924.00002.00000017.pk.a2
• Now that’s a file name!
© 2008 Progresswiz Consulting37 Quick and Easy OpenEdge AI File Management
Archived AI Files
C!~TEMP~oewrk~db~pk.20081022.164924.00002.00000017.pk.a2
a b c d e f a b c d ea) The full file path of the source DBb) The date of the last backup) pc) The time of the last backupd) The sequence number of the last backup) q pe) The AI sequence numberf) The original AI file name
© 2008 Progresswiz Consulting38 Quick and Easy OpenEdge AI File Management
AI Archiver Log
AI Archiver creates it’s own log:AI Archiver creates it s own log:• <DBNAME>.archival.log
Whoa!! • A little bit challenging to read• A little bit challenging to read
© 2008 Progresswiz Consulting39 Quick and Easy OpenEdge AI File Management
AI Archiver Log
0001,C:\TEMP\oewrk\db\pk,20081022,181456,20081022,164924,2,36,C:\TEMP\oewrk\db\pk.a3,C:\TEMP\oewrk\db,C:\TEMP\oewrk\db\C!~TEMP~oewrk~db~pk.20081022.164924.00002.00000036.pk.a3
0032,C:\TEMP\oewrk\db\pk,20081022,181652,1,3,20081022,181652,38,1,pk.bkp.3,BACKUP SET 20081022 181652p.3,BACKUP_SET_20081022_181652
0001,C:\TEMP\oewrk\db\pk,20081022,181656,20081022,164924,3,37,C:\TEMP\oewrk\db\pk.a1,C:\TEMP\oewrk\db,C:\TEMP\oewrk\db\C!~TEMP~oewrk~db~pk.20081022.164924.00003.00000037.pk.a1
Black text is AI file rotation log entryBlue text is a probkup• Note the backup timestamp and sequence number
changed in AI file rotation entry
© 2008 Progresswiz Consulting40 Quick and Easy OpenEdge AI File Management
Careful!
AI Archive files and log show AI file #37AI Archive files and log show AI file #37 associated with backup #3:C:\TEMP\oewrk\db\C!~TEMP~oewrk~db~pk.20081022.164924.00003.00000037.pk.a1
Intuitively believe AI 37 should be applied to backup 3backup 3
– Incorrect– AI 37 is last AI file before backup 3AI 37 is last AI file before backup 3
© 2008 Progresswiz Consulting41 Quick and Easy OpenEdge AI File Management
Careful…continued…
From DB Log File:From DB Log File:Full backup started.Switched to ai extent C:\TEMP\oewrk\db\pk.a2. This is after-image file number 38 since the last AIMAGE BEGIN
Therefore AI file #38 is the first AI file to apply after restoring a backupafter restoring a backup• Not a problem• Rfutil utility will complainRfutil utility will complain
– “…expected file 38…”
© 2008 Progresswiz Consulting42 Quick and Easy OpenEdge AI File Management
Careful…continued…
UNC pathnames in -aiarcdirUNC pathnames in aiarcdir• I.e “-aiarcdir \\bkpserver\aidir”• There was a bug on WindowsThere was a bug on Windows• Not sure if fixed in 10.1C+
© 2008 Progresswiz Consulting43 Quick and Easy OpenEdge AI File Management
A Few Final Steps
AI Archive DirectoryAI Archive Directory• Ideally, directory mounted from another server
– NOT in the same building as productionNOT in the same building as production• Cleanup AI files older than x days
– Delete files older than x days after saving to y gtape
– 5 is good starting numberY ill h t i t thi lf– You will have to script this yourself
© 2008 Progresswiz Consulting44 Quick and Easy OpenEdge AI File Management
A Few Final Steps
MonitoringMonitoring• One more thing to monitor• mprshut <dbname> -C aimgt_mprshut <dbname> C aimgt
Log File Maintenance• <Dbname> archival log<Dbname>.archival.log
– Archive and truncate periodically
© 2008 Progresswiz Consulting45 Quick and Easy OpenEdge AI File Management
That’s It!
You’re done! Congratulations!You re done! Congratulations!
Restores and roll-forwards are easyRestores and roll-forwards are easy • AI Files are semi-intelligently named• AI Archiver log file shows which files go with• AI Archiver log file shows which files go with
which backupJust remember to cleanup AI ArchiveJust remember to cleanup AI Archive directory
© 2008 Progresswiz Consulting46 Quick and Easy OpenEdge AI File Management
Questions
Questions?Questions?
© 2008 Progresswiz Consulting47 Quick and Easy OpenEdge AI File Management
Progresswiz Consulting
Questions or comments? Send me an email:Questions or comments? Send me an email:pk@progresswiz.com
© 2008 Progresswiz Consulting48 Quick and Easy OpenEdge AI File Management