The Why's and How's of Migrating to Foundation

The Why's and The Why's and How's How's

of Migrating to of Migrating to FoundationFoundation

Carlton DoeSenior Systems Engineer

NewCo Software and Solutions, Inc.

2

user.conferenceInformix

ISBN 0-13-605296-7 ISBN 0-13-080533-5

?

Who I Am• 10 + years of experience as DBA and engine

admin mainly in retail environments• Co-founded, presided over, and currently sit on

the Board of Directors of the International Informix Users Group (IIUG)

• Written two Informix Press books:

3


What I’m Going to Cover• What is IDS and Foundation?• Migration issues• ONCONFIG parameters• SQL statement cache• Max Connect• Fuzzy Checkpoints• Smart large objects Dbspaces• Simple/complex datatypes and casting• User Defined Routines and Functions• Collections• DataBlades and the Blade Manager• Java in the engine

4


INFORMIX-Dynamic Server7.x

IDS - UDO9.1x

What Is IDS and Foundation?

• Parallelism built-in• Parallel Data Query• SQL92 Entry Level• Enterprise Replication• Many, many more...

• SQL 3 Support• DataBlade Support• DataBlade Developer Kit• User Defined Routines• User Defined Datatypes• User Defined Indexing• R-Tree Indexing• Extended B-Tree Support• Row Types• Collections

(sets, multiset, lists)• Inheritance• Polymorphism• Partitioning with new data types• Schema Knowledge

5


What Is IDS and Foundation?

INFORMIX-Dynamic Server7.x

IDS - UDO9.1x+ =

IDS 9.2 +

6


Then What Is Foundation?

Informix Dynamic ServerInformix Dynamic Server

WebDataBlade

ExcaliburText

Datablade

ObjectTranslator

OfficeSpigot

JavaVM

7


Which Version Is Loaded??(IDS)Odra: onstat -

Informix Dynamic Server 2000 Version 9.21.UC2

(Foundation)Ebro: onstat -

Informix Dynamic Server 2000 Version 9.21.UC2

(MSGPATH)Wed Oct 4 07:49:52 200007:49:52 Booting Language <builtin> from module <>07:49:52 Loading Module <BUILTINNULL>07:49:57 Informix Dynamic Server 2000 Version 9.21.UC2 Software Serial

Number AAB#J50070507:50:12 Informix Dynamic Server 2000 Initialized -- Shared Memory Initialized07:50:15 Physical Recovery Started07:50:25 Physical Recovery Completed: 0 pages restored

8


Which Version Is Loaded??Ebro: cd /usr/informix/9_21/extendEbro: ls -ldrwxr-xr-x 4 informix informix 1024 Jul 19 08:08 ETX.1.30.UC5drwxr-xr-x 4 informix informix 1024 Jul 19 08:07 TXT.1.10.UC5drwxr-xr-x 4 informix informix 1024 Jul 19 09:02 VTS.1.20.UC1drwxr-xr-x 2 informix informix 1024 Jul 19 07:42 ifxbuiltins.1.1drwxr-xr-x 2 informix informix 1024 Jul 19 07:42 ifxmngrdrwxr-xr-x 2 informix informix 1024 Jul 19 07:42 ifxrltree.2.00drwxr-xr-x 4 informix informix 1024 Jul 19 07:49 krakatoadrwxr-xr-x 4 informix informix 1024 Jul 19 07:42 LLD.1.20.UC2drwxr-xr-x 4 informix informix 1024 Aug 22 13:36

WEB.4.10.UC1

9


Migration Issues

10


Migration Issues: Things to Know• You can not directly migrate from < 7.x versions

• Informix STRONGLY suggests that you first move to the latest available 7.x version then move to 9.x

• While you may want to export/import your data, in-place upgrades have been quite successful -- just remember to wear your belt and suspenders

• HP-UX 11.0 - RUN_AS_ROOT.server script is wrongLinks created for one library goes into /usr/lib instead of /usr/lib/pa20_64; if left alone, engine will not come up (“library not found” error). A “defect” has been entered

11


Migration Issues: Things to Know• With IDS, CPU and user-defined VPs are run as

user “informix” while other VPs are still run as root. This prevents UDRs from having root access

• If you have auto-start scripts, you should modify them so they “su” to “informix” before starting the engine

• Install engine after version 2.2 client products for “finderr” to work. The order is:

• tools• network• engine

12


Migration Issues: Things to Know• Precision change in float / smallfloat

to decimal conversion smallfloat 8 -> 9float 16 -> 17

• Default behavior of “create index” is now to create detached indexes. 7.x attached indexes are supported by the upgrade process

• Use the “DEFAULT_ATTACH” variable to temporarily mimic 7.x behavior, it will be discontinued however in a future release

13


Migration Issues: Things to Know• If using DRDA Gateway version 7.31.UC1 with IDS

7.x, you must get the DRDA patch to avoid defect #132619

• HADR has changed. DRAUTO is no longer supported so after a HADR failure condition is declared, the secondary server goes to read-only mode

• You must manually convert the secondary server to “standard” mode “onmode -d standard”

• No changes to DRINTERVAL, DRTIMEOUT, DRLOSTFOUND

14


Migration Issues: Things to Know• 9.2 ER and HPL -- may not support named types.

This should be in place for 9.3

Test dbexport / dbimport to see if more complex named types work

• Shut down HADR / ER before upgrading

15


Migration Issues: Things to Know• System catalog changes - columns added, moved, data

types changed. New tables added and some dropped. Sysindexes and systables now a view rather than tables

• Long identifiers!!The 8/18 rule is dead (the crowd cheers!!)user ids - 32 characters identifiers - 128 characters

table name, server name, database name, column name, index name, synonym name, constraint name, procedure name, dbspace name, blobspace name, optical cluster name, trigger name, type name, routine language name, access method name, operator class name, trace message class name, trace message name, procedure variable names

16


Migration Issues: Things to Know• Each user connection uses about 5% more

shared memory. Watch your shared memory allocations - both instance and operating system

• This can be offset by using MaxConnect

17


Migration Issues: New Reserved Words• CACHE• COSTFUNC• ITEM• SELCONST• INNER• JOIN• LEFT• LOCKS• RETAIN

18


Migration Issues:New ONCONFIG Parameters• ALLOW_NEWLINE• BLOCKTIMEOUT• DD_HASHMAX• DD_HASHSIZE• PC_HASHSIZE• PC_POOLSIZE• SBSPACENAME• SYSSBSPACENAME• STMT_CACHE• STMT_CACHE_SIZE• VPCLASS

• JDKVERSION• JVPHOME• JVPLOGFILE• JVPPROPFILE• JVPJAVAVM• JVPJAVAHOME• JVPJAVALIB• JVPCLASSPATH• JVMTHREAD

19


Migration Issues:New ONCONFIG ParametersVPCLASS

• Enables you to designate and create uniquely named, user-defined, or system classes of VPs. User-defined VPs have the same functional power as CPU VPs

• User-defined VPs should be created to execute user- defined routines and/or DataBlades

• Some DataBlades (Verity, Excalibur Text) require their own VP as does the JVM

• User-defined VPs can be added/dropped on the fly with the “onmode -p class_name” command

20


Migration Issues:New ONCONFIG Parameters

VPCLASSSyntax tree:

VPCLASS name,num=X,[max=X,aff=(X-Y),noyield,noage]

• VP name is not case sensitive, options are order independent, no whitespace, 128 char max/entry

• Must declare multiple JVPs to execute Java UDRs in parallel (Unix)

• If using “noyield”, only declare “1” vp since the UDR will execute serially, causing others to queuefor their turn

21


Migration Issues:New ONCONFIG ParametersUse the VPCLASS parameter to replace the CPU and AIO ONCONFIG parameters

VPCLASS cpu,num=3,max=10,noage

Comment out AFF_SPROC, AFF_NPROCS, NOAGE, NUMCPUVPS, SINGLECPUVP parameters

VPCLASS aio,num=5,max=10

Comment out NUMAIOVPS parameter

22


Migration Issues: How do I Migrate?1. Stop database processing and force a change

to a new logical log

onmode -sy; onmode -l; onmode -c; onmode -ky;

2. Back up the instance(s) and critical configuration files in $INFORMIXDIR

etc/ /aaodir/adtcfg$ONCONFIG (s)** /dbssodir/adtmasks ONCONFIG.stdsm_versions sqlhosts**tctermcaptermcap

23


Migration Issues: How do I Migrate?3. Restart the instance(s) and put them into single-user

mode to ensure all open transactions are properly rolled back

oninit -sv

4. Verify data and index integrity with the oncheck utility(-cr, -ce -cc, -c [ I / D ] db_name)

5. Shut the instance(s) down again and create level 0 backup(s) and/or dbexports

6. Install the new IDS / Foundation software and change relevant scripts, global parameters, config files

24


/usr/informix

7_3

9_21

scripts

disks (also tapes)

ISA

logs

config_files

max_conn

koetari -- symbolic links

odra -- symbolic links

(current logs) /old_logsEnvironment Variables:

INFORMIXDIRONCONFIGINFORMIXSQLHOSTS

My Recommended Directory Structure

25


Migration Issues: How Do I Migrate?7. Change ALARMPROGRAM to [ null | no_log.sh ] if

using On-Bar or LTAPEDEV to /dev/null

8. Install any DataBlades

9. Restart each instance into quiescent mode and monitor the MSG_PATH file. Sysmaster and reserved pages conversion is automatic (see $INFORMIXDIR/etc/dummyupds7x.sql). When completed, a notice is written to MSG_PATH

10. Execute an “update statistics high” command on the sysmaster database. Standard “update stats” command on all others

26


Migration Issues: How Do I Migrate?11. Verify data integrity with the oncheck command

12. Change LTAPEDEV / ALARMPROGRAM back to original value

13. Create a level 0 back-up

14. Let the users back in

If there are any problems -- RESTORE from tape

27


SQL Statement Cache

28


SQL Statement Cache: IntroductionAs implied, prepared and optimized SQL statements are stored in a section of the virtual portion of shared memory. Users executing identical DML SQL statements can pull the pre-parsed query plan and even query data structures from the cache

More complex queries (multiple columns, many filters in the WHERE clause) benefit most from caching

Turned off by default, the STMT_CACHE $ONCONFIG parameter and the “SET STATEMENT CACHE” SQL command control how the caching process works. Can be overridden with the “onmode -e” command

29


SQL Statement Cache: How It Works1. Incoming SQL statements are hashed to a value

which is compared to other hash values in the SSC section of the virtual portion of shared memory

Caveats:• statements only optimized at the PREPARE phase• host variable names/placeholders are not included

in the hash algorithm• white spaces and case of the statement is significant• session-specific parameters (OPTCOMPIND,

OPT_GOAL) are significant and will affect hash value

30


SQL Statement Cache: How It Works2. If hash value matches an existing value, the

cached copy is used. The existing copy is NOT re-optimized

3. If a match does not exist, the statement is evaluated for inclusion in the SSC

Qualifications for inclusion in the SSC:• contains DML statements using built-in data types

and operators• no UDRs/UDFs• local database connections only• no explicit temporary tables

31


SQL Statement Cache: How It WorksQualifications for inclusion in the SSC (cont):

• “select *” statements can be fully expanded to include all columns in the listed tables

• is not a statement generated by a stored procedure

• no imbedded subselects in the SELECT statement

32


SQL Statement Cache:Enabling and Sizingonmode -e {ENABLE|ON|OFF|FLUSH}

• ENABLE - Allows statement caching to take place. This does not turn caching on, but allows it to be turned on

• ON - Turns on statement caching. Caching must first be enabled through the STMT_CACHE parameter or “onmode” command

• OFF - Turns off the SSC and immediately disables sharing. Cached statements that are in use remain in the SSC until their use count reaches zero

• FLUSH - Active statements are marked and subsequently flushed when they are released. When the SSC is turned off, sharing is immediately disabled

33


SQL Statement Cache:Enabling and SizingSTMT_CACHE $ONCONFIG parameter:

0 - SQL Statement Cache disabled (default)1 - SSC enabled though sessions must explicitly call

for its use2 - SSC enabled for all sessions unless explicitly denied

on a session-by-session basis

Set the STMT_CACHE environment variable to “0” (off)or “1” (on)

Execute a “set statement cache [ on | off ]”command

34


SQL Statement Cache:Enabling and SizingSTMT_CACHE_SIZE $ONCONFIG parameter:determines the size of the SSC in KBs, default = 524 kb

If SSC is full and all statements are active, if additional statements qualify for inclusion the SSC will expand to hold them. As a statement’s user counts drop to zero, the SSC will flush them and reduce back to STMT_CACHE_SIZE

35


SQL Statement Cache:Monitoring and Tuning

• “onstat -g cac stmt” displays size of the SSC the cached statements, who’s currently executing a statement, and the number of times it’s been used

• The “hash” column is not the statement’s hash value, rather the hash bucket in which the statement resides

• To tune, monitor the size of the SSC over time, then resize as appropriate

36


SQL Statement Cache:Monitoring and TuningUse the STMT_CACHE_DEBUG environment variable to trace what happens with any given SQL statement

export STMT_CACHE_DEBUG=debug_value:path_to_file

debug_value can be “1” (basic) or “2” (extended)

37


SSC identity (OK:status=1): select * from yoyoSSC link (semantic error): select * from yoyoSSC identity (stmt not found): create table yoyo (col1 int) in mustang_2SSC qualify (invalid stmt type): create table yoyo (col1 int) in mustang_2SSC identity (stmt not found): insert into yoyo values (1)SSC qualify (OK): insert into yoyo values (1)SSC key insert (OK): insert into yoyo values (1)SSC full insert (OK): insert into yoyo values (1)SSC identity (stmt not found): insert into yoyo values (2)SSC qualify (OK): insert into yoyo values (2)SSC key insert (OK): insert into yoyo values (2)SSC full insert (OK): insert into yoyo values (2)SSC identity (OK:status=1): insert into yoyo values (2)SSC link (OK): insert into yoyo values (2)SSC identity (stmt not found): drop table yoyoSSC qualify (invalid stmt type): drop table yoyo

SQL Statement Cache:Monitoring and Tuning

38


What Are FuzzyCheckpoints???

39


CheckpointsIDS introduced “fuzzy” operations and checkpoints to increase transactional throughput

Two types of checkpoints:

• Full or “sync”• Fuzzy

40


Steps of a Sync Checkpoint1. Engine blocks threads from entering “critical sections”

of code2. The page cleaner thread flushes the physical log buffer

to log on disk3. The page cleaner threads flush to disk all modified pages

in the buffer pool (chunk write)4. The checkpoint thread writes a checkpoint record

to the logical log buffer5. The logical log buffer is flushed to the current logical log

file on disk6. The physical log on disk is logically emptied

(current entries can be overwritten)7. The checkpoint thread updates the reserved pages

with the checkpoint record information

41


Good and bad:logically and physically data, interrupts user activity, expensive (I/O hit), must tune to reduce duration

What Causes Sync Checkpoints to Occur?• Physical log becomes 75% full• “onmode -c” or “-ky”• Administrative actions (adding dbspaces, altering tables)• A backup or restore operation using ontape or ON-Bar• End of fast recovery or full recovery• Reuse of a logical log containing the oldest fuzzy

operation not yet synced to disk• LTXHWM reached with fuzzy transactions• Through the onmonitor menu

42


Fuzzy CheckpointsRecord “results” of fuzzy operations in the logical log

Goal: reduce or eliminate checkpoint interruptions, eliminate the physical log (over 3 - 4 phases)

Cost: increased logical recovery time

Note: Fuzzy checkpoints are not used in HADR environments

43


Fuzzy CheckpointsThings to know:

• Inserts, updates, delete DML statements • Currently supports “built-in” data types only

(character, numeric values)• Logged databases• Not “old” pages. Determined by page timestamp

(4 byte, cycles 2 gb -> -2 gb -> 0)• No BEFORE images are generated in the physical

log• During “fuzzy” checkpoints, buffered information

on fuzzy operations is **not** flushed to disk• After a fuzzy checkpoint, disks are not

physically consistent

44


Steps of a Fuzzy Checkpoint1. Engine blocks threads from entering “critical sections”

of code2. The page cleaner threads flush to disk all **non-fuzzy**

modified pages in the buffer pool (chunk write) Note: MLRU queues still full

3. A Dirty Page Table (DPT) is constructed containing entries (buffer addresses) for fuzzy operations

4. The DPT is flushed to the current logical log on disk4A. The page cleaner thread flushes the physical and logical log buffers to disk

5. The checkpoint thread updates the reserved pages with the checkpoint information including the Log Sequence Number (LSN).

6. The physical log on disk is logically emptied (current entries can be overwritten)

45


Log Sequence Number (LSN)• A value representing a position in the logical log

(log # and log position)• Stored in each dirty buffer heading containing

fuzzy operation results along with a newly-stored timestamp

• The oldest LSN is tracked in shared memory because it contains the oldest fuzzy operation. Oldest LSN can not be further back than 1 logical log and 2 checkpoints

• Logs after the LSN can not be freed for re-use until a checkpoint has occurred

Net Net -- large logical logs are GOOD!!!

46


What Causes Fuzzy Checkpoints to Occur?• Checkpoint interval has elapsed• Physical log becomes 75% full• onmode -c fuzzy• Reuse of a logical log containing the last

checkpoint (either fuzzy or sync)

47


Recovery - Sync Checkpoint1. Physical log data used to return all disk pages

to original “synced” state (physical restore)2. The most recent checkpoint (sync) record is

located in the logical log files3. All subsequent logical log records

are rolled forward4. Uncommitted transactions are rolled back

48


Recovery - Fuzzy Checkpoint1. Physical restore proceeds as normal however

disk pages are not physically consistent since there are modified pages that were not logged in the physical log

2. Logical recovery begins in two phases:• Phase A -- reapply DPT records• Phase B -- post checkpoint records

49


Recovery - Fuzzy CheckpointPhase A - Logical Recovery:

1. Locate DPT created before last fuzzy checkpoint2. Selectively apply fuzzy operations from the

DPT/logical log from the oldest remaining logical log up to the most recent checkpoint

Conditional update based on page timestamp - if DPT record is “newer” than page timestamp, then the change is applied

Result: similar to end of physical restore when using sync checkpoints

50


Recovery - Fuzzy CheckpointPhase B - Logical Recovery:

1. Beginning with last checkpoint, roll forward subsequent logical log records.

2. Uncommitted transactions are rolled back

Result: similar to the logical restore when using sync checkpoints. Disks are physically and logically consistent to last recorded transaction

51


Fuzzy Checkpoints• Buffer structures, logical log records, and reserved

pages are automatically upgraded when migrating to 9.x

• Fuzzy checkpoints enabled by default. Set the NOFUZZYCKPT environment variable to true/on/1 to disable them prior to initializing shared memory

• Set the TRACEFUZZYCKPT environment variable to trace fuzzy checkpoints. It will affect instance performance but will log in the MSGLOG the number of buffers that remain dirty after a checkpoint is completed. It also lists the location of the old LSN

52


Large Objects, SBSpacesand

Dynamic Lock Allocation

53


Simple SmartI/O size configurable by

page sizeconfigurable by page size, the datais still stored in database serversized pages, but the I/O isconfigurable

Bufferpoolusage

partition blobs usethe buffer pool,blobspace blobsuse private buffers

you can choose between light I/O ornormal buffering when the smart-large-object is opened

Updating inserts and deletesonly ; there are nodirect updates

updates are done in place or movedas needed

Datarow/blobmapping

one large-objectcan be referencedby one data row

one large-object can be referencedby one or more sources

Simple Smart

Large Objects and SBSpacesIDS has 4 types of LOs:

• Simple - BYTE & TEXT• Smart - CLOB & BLOB

54


Simple Smart

LoggingMethod

logging dependson the table wherethe large-object isstored

logging can be turned on or off at thelarge-object level

Size 2 GB maximum 4 TB maximum

LockingGranularity

all or nothing can lock portions or all of the object

Reads andwrites

only the wholelarge-object canbe read

all or a portion of the large-object canbe read

Fragmentation all large-objectsfor a table arestored in the sameblobspace

ability to store the large-objects forone table in multiple sbspaces

Large Objects and SBSpaces

• Access to C/BLOBs is faster due to “portion-level” addressing• Default characteristics are easy to override• With sufficient resources, C/BLOBs are transactionally recoverable

55


header (reserved) pages

metadata areaUser Data

Size and location ofthe metadata area isconfigurable atsbspace and/or chunk creation

Metadata is alwayslogged regardless ofdatabase or sbspacelogging mode

Large Objects and SBSpacesSmart LOBs are stored in “smart” BLOBSpaces (sbspace) which contain meta-data as well asuser data

56


Large Objects and SBSpacesonspaces -S name -g pageunit -p pathname -o offset

-s size [ -m pathname offset ] -Ms mdsize -Mo mdoffset-Df default list

-Mo mdoffset offset, in kbs, into the disk partition where metadata will be stored

-Ms mdsize the size, in kbs, of the metadata area for the initial chunk, remainder is user data space -- not recommended though possible. Each SLO needs ~500 bytes of metadata space

-Df default list default specifications for objects stored in the sbspace. Comma separated, in double quotes (“ “)

57


Tag Values Default DescriptionACCESSTIME ON or

OFFOFF ON – instance tracks access time

of SLOs stored in the sbspace.AVG_LO_SIZE For

WindowsNT:4 to 2^31

For UNIX:2 to 2^31

8 Average size, in kbs, of SLOsstored in the sbspace. Used tocalculate metadata area size if -Msoption is not specified.Error 131 is returned if you run outof metadata area in the sbspace.To allocate additional chunks tothe sbspace that consist ofmetadata area only, use the -Msoption.

BUFFERING ON orOFF

ON ON - instance uses residentportion buffer pool for SLO I/Ooperations.OFF - instance uses light I/Obuffers in the virtual portion(lightweight I/O operations) –PREFERRED!!!!

LOCK_MODE RANGE orBLOB

BLOB RANGE - only a range of bytes inthe SLO is locked.BLOB - the entire SLO is locked.


58


Tag Values Default DescriptionLOGGING ON or

OFFOFF ON - instance logs changes to the

SLOs in the user data. Create alevel-0 backup of the sbspace aftercreating/changing

EXTENT_SIZE 4 to2^31

16 Size, in kbs, of the first diskallocation for SLO storage whenyou create the table.Let the system select theEXTENT_SIZE value.

MIN_EXT_SIZE 2 to2^31

4 Minimum amount of space, in kbs,to allocate for each SLO.

NEXT_SIZE 4 to2^31

16 Size, in kbs, of the next diskallocation for smart large objectswhen the initial extent becomesfull.Let the system select theNXT_SIZE value.

Create a level 0 backup after creating a sbspace


59


Large Objects and SBSpacesTo add a chunk, you can specify whether the chunk will contain only user data, metadata, or both:

onspaces -a sbspace -p pathname -o offset -s size

[-Ms mdsize -Mo mdoffset | -U ][ -m pathname offset ]

To drop the sbspace:onspaces -d sbspacename -f (force)

60


Creates a 72 byteCreates a 72 bytedescriptor “handle”descriptor “handle”as opposed to 56as opposed to 56bytes with “simple”bytes with “simple”LOsLOs

create table products (prod_sku int, prod_desc clob, prod_photo blob) put prod_desc in (sbspace_1, sbspace_2), prod_photo in (sbspace_9, sbspace_10), fragment by expression (prod_sku < 1000 in db_1, prod_sku >= 1000 in db_2);

Large Objects and SBSpacesSLOs can be distributed in round-robin form

among sbspaces like “regular” data

61


Large Objects and SBSpacesInformix recommends that you convert “simple” LOs to their “smart” counterparts as soon as feasible

You can, in many cases, use the alter table command to make the change(s):

alter table t_name modify col_name [ clob | blob ]put column_name in (sbspace(s));

62


Large Objects and SBSpacesMonitoring / tuning

oncheck -cs -cS / -ps -pS to print/view metadata-ce / -pe has additional sbspace information

onstat -g smb s | c | h | e | tsmb s sbspace summary informationsmb c sbspace chunk informationsmb fdd LO file descriptor tablesmb lod LO header tablesmb t timing statistics for enabled server,

not well documentedsmb e undocumented

onstat -k has additional columns to document SLO byte range locks

onstat -d has additional chunk-related columns for sbspaces

63


Chunk adjunct partition

Level 1 backup partition

Level 2 backup partition

LO header partition

Sbspace descriptor partition

user-data free-list partition

header (reserved) pages

metadata area

Large Objects and SBSpacesBacking up and restoring SLOs and sbspaces now operates like regular dbspaces -- at the page level. As a result, you can now have granular and truly incremental SLO/sbspace backups.

64


OtherOtherspacesspaces

Sbspace Sbspace headerheader

Metadata Metadata pagespages

Archive Archive descriptor pagesdescriptor pages

sbspace sbspace pagespages


Backup: • Metadata LO headers are read and a list made of pages to be

backed up• All individual page timestamps are compared too since updates

can occur without affecting the LO header timestamp• Qualifying data is sent to the backup program through large

buffer accesses

Restore:• Sbspace header and metadata information is restored• Archive desriptor pages contain page addresses and extent

sizes for the sbspace pages that follow. This is held in shared memory during the operation

• Sbspace pages are restored to their correct places

65


Dynamic Lock AllocationIn IDS, the locking mechanism changes. LOCKS is now a starting allocation (resident portion)

If the lock table fills, additional locks are allocated (x 16) minimum allocation: smaller of current lock table or 100,00 (virtual portion)

To prevent lock table flooding, range locks will be upgraded to the whole SLO if a single transaction uses > 10% of lock table

“Byte range” locks are what lock portions of a SLO. They are sorted lists of rowids, partition #s, and byte address ranges

66


Dynamic Lock Allocation• Range locks can be joined or split as needed if ranges to

be locked adjoin, overlap, or become separate

• Ranges locks are not stored in the lock hash table, rather an “intent” lock is placed for the LO header. Byte range locks are linked to associated hash intent lock then sorted by byte number

• Byte ranges locks allocated if:• < 50% of lock table is being used• 50% <= lock table <= 70%, whole SLO lock will be

attempted, if unsuccessful, byte lock allocated• > 70% whole SLO locks only, existing locks will be

upgraded automatically

67


MaxConnect

68


MaxConnectMax Connect is a middle-ware communications component that removes the overhead of managing user connections from the instance

Helps you get scale your user connections past the effective 2,000 user connection limit without any change or resource impact to the database server

It supports 7.3 and 9.2x+ Unix ports of the engine but accepts user connections from any platform

NO CHANGE to existing applications!!

69


MaxConnect PerformanceBaaN IVc World record:• 11,445 BRUs on HP N4000 (8CPU, 16GB) and IDS• 21% better than Oracle on identical HP system• Beat Oracle’s 64-CPU Sun E10000 result• 18,650 database connections using 98

transports !!

70


MaxConnect

MaxConnectIDS/Fnd

SQLClients

For large, hardware-partitionable SMP systems

MaxConnect

MaxConnect

SQLClients

Database server system

IDS/Fnd

MaxConnect Architecture

71


MaxConnect Configuration• A MaxConnect instance connects to only one

DB server instance• Connections are managed through the

SQLHOSTS file with a new connection protocol -- imc

• Clients change hostname and portaddress to point to the MaxConnect instance, instance name remains the same

• MaxConnect instance has an ADMIN entry, a client-facing entry, and a DB server-facing entry

• DB instance has a DBSERVERALIAS and a DBSERVERNAME entry

72


MaxConnect Configuration• Client SQLHOSTS file:

ebro_tcp ontlitcp maxcon_1 1526

• MaxConnect SQLHOSTS file:ebro_tcp ontlitcp maxcon_1 1526ebro_db ontliimc cronus 1575ebro_adm onsoctcp maxconn_1 1600

• Server SQLHOSTS file:ebro onipcshm cronus ebro_ph (DBSERVERNAME)ebro_tcp ontliimc cronus 1575

(DBSERVERALIAS)

73


NETTYPE entry changes:ipcshm,1,20,CPU# tlitcp,5,200,CPUtliimc,1,2000,CPU

MaxConnect instance environment variables:IMCONFIG - location of MaxConnect config file -

default is $INFORMIXDIR/etc/IMCconfigIMCSERVER - Client-side MaxConnect instance

nameIMCADMIN - Admin-side MaxConnect instance

name

MaxConnect Configuration

74


MaxConnect config file contains:IMCLOG - location of MaxConnect message logIMCWORKERTHREADS - # of worker threads, range = 1 -> 64IMCWORKERDELAY - amt of time MaxConnect will wait and pool messages to transmit to DB server default = 0, range = 0 -> 100,000 msIMCTRANSPORTS - # of transport connections to DB

server. Default = 2, range = 1 -> 64, ratio = 1/100 users

MaxConnect Configuration

75


MaxConnect Tuning• Use “onstat -g imc”, the imcadmin utility, or the

ISA to monitor and tune MaxConnect

• Tune the NETTYPE sessions parameter by monitoring the q-exceed and alloc/max values. May also need to tweak the IFX_NETBUF_PVPOOL_SIZE variable

• Tune IMCTRANSPORTS by monitoring blocked or partial writes

• Tune IMCWORKERTHREADS by monitoring AvgQlenIf > 5, increase number of threads

• Tune IMCWORKERDELAY by summing and comparing historgrams against AvgQlen

76


Complex and User-Defined Data Types

77


OpaqueOpaque DistinctDistinct

Row Data TypeRow Data Type

NamedNamed UnnamedUnnamed

CollectionCollection

MultisetMultiset ListList

SetSet

User-DefinedUser-DefinedComplexComplex

Extended DataExtended DataTypesTypes

BooleanBooleanInt8Int8

Serial8Serial8LvarcharLvarchar

New Built-inNew Built-inTypesTypes

Existing Built-inExisting Built-inTypesTypes

Data TypesData Types


78


Built-in Data TypesNew Built-In:

• int8 (8 bytes)• serial8 (8 bytes)

• Range is -9,223,372,036,854,775,807 to 9,223,372,036,854,775,807

• must add UC to serial8 to ensure uniqueness• one serial8 and serial per table

• boolean valid values: “t”, “f”, null. case sensitive

• lvarchar (variable length character data type; 32k maximum)

79







SetSet









80


No “alter type” statement, must drop and recreate

Sys Adm is more complexIntuitive

Not simple SQLRefers to a group of elements by a single name

More complexLess codingDISADVANTAGESDISADVANTAGESADVANTAGESADVANTAGES

Complex Data Types - RowRow DataTypes

analogous to C structure, come in two “kinds”NAMED - strongly typed, ID’ed by name, has inheritance, used

to build columns and tablesUNNAMED - weakly typed, ID’ed by structure, no inheritance,

used to build columnsCan contain built-in, collection, opaque, distinct, another row type data types

• Caveat: no serial or serial8

81


Row DataTypesNamed: create row type name_t (fname char(20), lname char(20));

create row type address_t (street_1 char(20), street_2 char(20), city char(20), state char(2), zip char(9));

create table student(student_id serial, name name_t, address address_t, company char(30));

Unnamed:ROW (a int, b char (10))

Note: is also equal to ROW(x int, y char(10))

create table part (part_id serial,cost decimal,part_dimensions row

(length decimal,width decimal, height decimal, weight decimal));

Complex Data Types - Row

82


Using Named Row Types in SQL statements:

Insert statement:insert into student values (1234, row (“John”,”Doe”)::name_t, row ("1234 Main Street","", "Anytown","TX","75022")::address_t, "Informix Software")

Select statement:select * from student where name.lname matches "Doe”;Result set:

student_id 1234name ROW('John ','Doe ')address ROW('1234 Main Street ',' ,'Anytown ','TX','75022 ')company Informix Software

Use of datatypeUse of datatypekeywordkeyword

Cast the row type!Cast the row type!

Access data withAccess data with‘‘dot’ notationdot’ notation

Complex Data Types - Row

83


Complex Data Types - RowTo drop a named row type:

drop row type address_t restrict;

84


Complex Data Types - CollectionsGrouping of elements of the same datatype (char, int), max size = 32 KB

Used when• The data is meaningless without the context of the

other members in the collection (e.g., golf scores, to do list, set of names)

• Individual data elements are not likely to be directly queried by position

• The maximum number of data elements is less than 32

Can be null

85


Complex Data Types - CollectionsThree kinds of collections:

Set - unordered, no duplicates allowed set {“apple”, ”orange”, ”grapefruit”, “plum”}

Multiset - unordered, duplicates allowedmultiset {“apple”, “orange”, “grapefruit”, “apple”, “plum”, “grapefruit”}

List - ordered, duplicates allowedlist {“apple”, “orange”, “grapefruit”, “apple”, “plum”, “grapefruit”}

86


Complex Data Types - Collectionscreate table class

(class_id serial, class_name varchar(60), description lvarchar, prereqs set(char(20) not null));

Insert syntax is similar to named row types:

insert into class values (300, “Performance and Tuning”, “Covers advanced information on tuning the Informix Dynamic Server”, (SET{“RDD”,”BSQL”}));

Use the “in” keyword to query values in a collectionselect * from class where (“ASQL”) in prereqs;

define xyz char(20)define set_var set(char(20))select prereqs into set_var from class where class_id = 300

foreach del_set_cursor forselect * into xyz from table(set_var)if xyz matches “RDD” then

delete from table(set_var) where current of del_set_cursorend if

end foreach

87


Complex Data Types - CollectionsYou can not update one element in a collection, you must replace the whole collection:

update class set prereqs = (set{“RDD”,”ASQL”,”BSQL”}) where class_id = 300;

update class set prereqs = set_char where class_id = 300;

88







SetSet









89


User-Defined Data Types - DistinctTwo user-defined data types (UDTs):

Distinct• data type modeled on an existing data type• has a unique name to distinguish it from other similar

“types”• inherits the internal structure from the source type• might have operations and casts defined over its

source type

Opaque• data type that is unknown to the database server• you must define the internal structure, functions,

and operations

90


User-Defined Data Types - Distinctcreate distinct type dollar as decimal;create distinct type aus_dollar as decimal;

create table sales ( sku int, sales_date date, us_sales dollar, aus_sales aus_dollar);

insert into sales values (1234, today, 15.0::dollar,0::aus_dollar);insert into sales values (5678, today, 0::dollar, 75.0::aus_dollar);

select sku, (sum(us_sales) + sum(aus_sales)) from sales where sales_date = today group by 1;

error: 674 - routine (plus) can not be resolved

91


User-Defined Data Types - DistinctNeed to create some UDFs that handle the type and value conversion for you:

create function usdlr_to_ausdlr(parm1 dollar) returning aus_dollar specific usd_to_ausd; return (parm1::decimal * 1.8)::aus_dollar;end function;

create function ausdlr_to_usdlr(parm1 aus_dollar) returning dollar specific ausd_to_usd; return (parm1::decimal / 1.8)::dollar;end function;

select sku, (sum(us_sales) + sum(ausdlr_to_usdlr(aus_sales))::dollar) from sales where sales_date = today group by 1;

92


User-Defined Data Types - OpaqueAn opaque data type stores a single “value” that cannot be divided into components by the engine.

Implemented as C or Java structures and manipulated by a set of routines written in C or Java

An opaque “value” is stored in its entirety by the engine without any interpretation of the contents or its structure

All access to an opaque type is through functions written by the user. You define the storage size of the data type and input and output routines

93


User-Defined Data Types - Opaque1. Create the C / Java data structure to represent the

internal data structure2. Write the support functions in C / Java3. Register the opaque data type with the “create

opaque type” statementcreate opaque type type_name ( internallength = length,

alignment = num_bytes);length is in bytes, alignment = 1,2,4,8 bytes (default = 4)

create opaque type type_name ( internallength = variable,maxlen = length);

length - default = 2 KB, max value = 32 kb

4. Register the support functions with the “create function” statement. If in Java, will be stored in sbspace(s)

94


User-Defined Data Types - Opaquecreate opaque type my_type(internallength=8, alignment=4);

create function support_in(lvarchar)returning my_type with (not variant);external name “/funcs/my_type.so”language C

end function;

create implict cast (lvarchar as my_type with support_in);

5. Grant access to the opaque data type and support functions

6. Write any user-defined functions needed to support the opaque data type - input, output, destroy, compare, aggregates, send, receive, etc.

7. Provide any customized secondary-access methods for creating indexes

95


Casts and Casting

96


Casts and Casting• Casts allow you to make comparisons between values of

different data types or substitute a value of one data type for a value of another data type.

create function ausdlr_to_usdlr(parm1 aus_dollar) returning dollar specific ausd_to_usd; return (parm1::decimal / 1.8)::dollar;end function;

• Engine provides a number of “built-in” casts (int to decimal, numeric to char, etc.) for most built-in datatypes

• Must create user-defined casts for user-defined types. Must be unique with respect to source and target data types

• Can not create casts for collections, LOBs, or unnamed row types

97


Casts and CastingTwo kinds of user-defined casts:

explicit. . . where price >= (cast aus_dollar as dollar) . . . .. . . return (parm1::decimal / 1.8)::dollar;create explicit cast (aus_dollar as dollar with aus_dlr_to_us_dlr);

implicitcreate implicit cast (aus_dollar as dollar);create implicit cast (aus_dollar as dollar with aus_dlr_to_us_dlr);

Automatically invoked when:one data type is passed to a user-defined routine whose

parameters are of another data type.expressions are evaluated that need to operate on two similar

data types.

98


Casts and Casting - Implicit Casts select sum(us_sales) + sum(aus_sales) from sales; # 674: Routine (plus) can not be resolved.

create implicit cast (aus_dollar as dollar);select sum(us_sales) + sum(aus_sales) from sales;

(expression) 120.00

drop cast (aus_dollar as dollar);create implicit cast (aus_dollar as dollar with ausdlr_to_usdlr);select sum(us_sales) + sum(aus_sales) from sales;

(expression) 80.00

99


Casts and Casting - Explicit Casts select sum(us_sales) + sum(aus_sales) from sales; # 674: Routine (plus) can not be resolved

create explicit cast (aus_dollar as dollar);select sum(us_sales) + sum(aus_sales) from sales; # 674: Routine (plus) can not be resolved

select sum(us_sales) + sum(aus_sales)::dollar from sales;(expression) 120.00

drop cast (aus_dollar as dollar);create explicit cast (aus_dollar as dollar with ausdlr_to_usdlr);select sum(us_sales) + sum(aus_sales)::dollar from sales;

(expression) 80.00

100


Casts and Casting Previous examples were “straight” casts. You can also create cast “functions” to cast types with dissimilar data structures

1. write the cast function in C / Java / SPLcreate function opaque_to_opaque (input_arg my_type_1)

returns my_type_2return cast(cast(input_arg as lvarchar) as my_type_2);

end function;

2. register the cast function with the“create function” command

3. register the cast with the “create cast” commandcreate explicit cast (my_type_1 as my_type_2 with

opaque_to_opaque);

101


User-defined Routines

102


UDRs There are UDFs and UDPs

Can be internal or external in nature. Internal are written in SPL, external can be written in C or Java.

If written in Java, compile into a class then .jar file.When registered in the engine, .jar file will be brought into a sbspace then executed when neededby the JVM in the engine

If written in C, create a shared object library in a directory owned by “informix” with 755 permissions

103


UDRs UDRs should be concise and precise. Don’t want engine wasting resources plowing though verbose code. The values returned should be relevant from a business perspective (e.g. proper domain)

Make sure you include error handling in the code

**ALWAYS** use a “specific” name, or alias, for every UDR to properly identify each one - used to drop, grant, or revoke permissions as well as to permit function overloading

104


UDRs Function overloading occurs when two or more functions have the same name but different signatures

signature = UDR name and parameter list

create function plus (in_1 dollar, in_2 aus_dollar) . .create function plus (in_1 aus_dollar, in_2 lira) . . .create function plus (in_1 lira, in_2 aus_dollar) . . .

{Note: these probably should be formalized into user-defined aggregates}

When properly registered, instance will use the data types passed to determine which UDR should be used

105


UDRs create function routine_name (param_list)

{ returns | returning } typename[ specific specific_name ][ with (internal | handlesnulls | [ [not] variant]

| class=“vp_class”) ]external name ‘full_path_name_of_file‘language language [ [not] variant ]

end function;

in the $ONCONFIG file:VPCLASS fince_vp ,num=2 # vps for finance UDRs

oronmode -p +2 fince_vp

USE UDVPs FOR UDRs!!!!

106


UDRs To drop UDRs, must either use full signature

drop function plus(lire, aus_dollar);

or the “specific” name

drop specific function lir_to_ausdlr;

107


UDRs Default behavior of UDRs is single-threaded, can be “paralellized” if conditions are met:

• only in DSS environments (PDQPRIORITY > 1)• parallelizable keyword is added to UDR

creation statement• external UDRs only (C / Java) that use PDQ

thread-safe SAPI calls (see DataBlade manual for list of SAPI calls)

• no complex types in input parameters or return values

• multiple fragments must be scanned

108


UDRs • Not directly called (singleton execution)• not an interator - called in loops to return multiple

values similar to with resume in SPL

• on some UDRs, you can add “selectivity” and “cost” modifiers that will affect optimizer activities and query plans used. Not really familiar with this

109


UDRs create function bigger(int,int) returns int

specific big_paralelwith(parallelizable,class=“test_vps”) external name“/path/plludr.bld(bigger)” language C;

alter function bigger (int,int) with (add parallelizable,add class=“test_vps”);

Can monitor the parallel execution of UDRs through• “set explain out” command• the ISA• onstat -g mgm• onstat -g ses sess_id• onstat -g ath

110


DataBlades and the Blade Manager

111


DataBlades • Datablades extend the functionality of the engine

by adding new UDTs, casts, interfaces, UDRs, error messages, clientcode (where necessary) to manipulate the UDTs

• Functionality available through SQL, SPL, API calls to external functions

• Can be mixed and matched to support any application

• While the code is loaded at an instance level, blade registration occurs at a database level. With registration information contained in the database system tables, if you drop the database, blades become unregistered

112


DataBlades Install the blade in $INFORMIXDIR/extend (tar, cpio, ?)

May have its own “install” script to brand binaries, use serial # / license key from engine install unless third-party requires unique key setup

Use the Blade Manager to manage blade registration graphical version available with NT and ISA command line only on Unix / Linux

Is case sensitive

113


DataBlades BladeManager supports the following commands:

list [database] - print list of registered DataBlade modulesset user [user]set server [server]show modules - print list of DataBlade modules on servershow databases - print list of databases on servershow servers - print list of serversshow client - print DataBlade modules with files on clientregister [module] [database]unregister [module] [database]add client [module] - install DataBlade module's client filesdel client [module] - remove DataBlade module's client filesinfo [module] - print information about a DataBlade moduledel logs - delete all the log filesshow log - prompt to display a logshow last log - display log from last actionhelp, ?bye, quit, exitset confirm on|off

114


DataBlades mustang>show databaseDatabases on server: agg1 dcs_demo dps_demo ireach kiosk media360 my_test_db visclass20

mustang>list media360DataBlade modules registered in database media360: lld.1.20.tc2 TXT.1.10.TC5 ifxrltree.2.00 ifxbuiltins.1.1 web.4.10.TC1 VTS.1.20.TC1 video.2.00.TC2

115


Introduction to Javain the Server

116


EJB

EJB

Java

JVM

Applet JavaBean JavaBean

RMI | Corba | IIOP

Web Server

Applet JavaBean JavaBean

Java DBDK

JavaBean EJB

Java in the Server J/Foundation is the marriage of the Java and SQL languages

Can write internal or external Java routines that are accessed natively by the engine

117


Java in the Server Informix provides a “Type 4” JDBC driver

Type 1 JDBC-ODBC bridge provides JDBC access via most ODBC drivers

Type 2 A native API and partly Java driver that converts JDBC calls into calls on the client API for Oracle, Sybase, Informix, DB2 etc.

Type 3 A net protocol, all Java driver that translates JDBC calls into a DBMS independent net protocol which is then translated to a DBMS protocol by a server

Type 4 A native protocol all-Java driver converts JDBC calls into the network protocol used by DBMSs directly. This allows a direct call from the client machine to the DBMS server

118


Java in the Server,How It Works Java UDRs execute on Java Virtual Processors

• Java VPs have the same capability as CPU VP• It can process any SQL query

• Avoids the inter-process communication overhead for executing Java UDRs

• The Java Virtual Machine (JVM) is embedded directly into the VP code

Java runs threads in parallelDynamically load Java VPs

• Increase JVPs to balance application load

119


Java in the Server,How It Works New ONCONFIG Parameters:

VPCLASS jvp,num=1JVPJAVAHOME /usr/javaJVPHOME /opt/informix/extend/krakatoaJVPPROPFILE /opt/informix/extend/krakatoa/.jvppropsJVPLOGFILE /opt/informix/jvp.logJDKVERSION 1.1

120


Questions???

The Why's and How's of Migrating to Foundation

Documents

Transcript of The Why's and How's of Migrating to Foundation