Leveraging Oracle ASM Cluster File System for Private Cloud Storage
Kai YuOracle Solutions EngineeringDell Inc
About AuthorKai Yu, Senior Principal Architect, Dell Database Engineering
21 years live in Austin and work on Oracle Technology
Specializing in Oracle RAC, Virtualization and Database
Infrastructure
Oracle ACE Director, authored 120 + presentations and 30 +
whitepapers
2011 OAUG Innovator of Year
2012 Oracle Excellence Award: Technologist of the Year: Cloud
Architect by Oracle Magazine
IOUG Spotlight for March 2016
Co-founder and VP of IOUG Cloud Computing SIG
My Oracle Blog: http://kyuoracleblog.wordpress.com/
Co-author Apress Book “Expert Oracle RAC 12c”
Tiered Storage Solution for Oracle 12c RAC database
Existing SAN
Flash based Storage
Oracle RAC Database
Rapidly deliver
Agenda
Oracle ACFS for Cloud Applications
Configuration of Oracle ACFS
Leverage ACFS Snapshot Feature
Oracle ACFS or ASM for Database
Oracle ACFS for Cloud Applications
Oracle ASM Cluster File System (ACFS):
• Initially introduced in Oracle 11gR2 to store non-Oracle Database files.
• ACFS mounted in OS and accessible through OS commands
• A cluster file system shared and accessible by single/cluster nodes with unified namespace.
• ACFS files stored in the Oracle ASM diskgroup with benefits from ASM
• IO load balancing through data stripping among the ASM disks
• Data reliability through mirroring within the ASM diskgroup
Oracle ACFS for Cloud Applications
What can be stored in Oracle ASM Cluster File System (ACFS)
• ACFS supports general purpose files, including applications files, data files, log files, etc ,across multiple OS and planforms,
• Starting with Oracle 12cR1, it support Oracle Database files except for data files and redologs for Oracle Restart
• All databases stored on Oracle ACFS can leverage Advanced Database services: snapshots, tagging, security, encrypting, and auditing
• ACFS doesn’t support Oracle Grid Infrastructure software , which should be stored in local disks
Oracle ACFS for Cloud Applications
Oracle Automatic Storage Management Stack• Oracle ASM provide the basic functionality of Automatic Storage
Management
• Oracle ASM files for Oracle Database supports diskgroups for database files
• Oracle ASM Dynamic Volume Manager (ADVM) : provides the logical volume management and standard device interface to files systems such as ACFS, OCFS2 and regular ext4 file systems
• Oracle RAC database can be stored in Oracle ASM files as well as in Oracle ACFS.
Oracle ACFS for Cloud Applications
Oracle ACFS Integration with Oracle ASM• Oracle ACFS is always configured with Oracle ASM storage.
• Traditional Storage Devices presented to Oracle ASM
• ASM diskgroup is built with the storage devices, presented to Oracle ADVM
• Oracle ADVM volumes are created with the ADVM in the ASM diskgroup and presented to OS as an OS logical block device.
• Oracle ACFS file system is bound to the Oracle ADVM volumes through the file system creation
ASM
Volume
ASM
Volume
ASM Diskgroup FG
ASM Disks
ADVM
ASM Cluster File System(ACFS)
/acfs1 /acfs2 ASM
Volume
ASM
Volume
ASM
Volume
/acfs3
Oracle ASM
ASM Diskgroup
DG
ASM Disks
RAC
DB1
RAC
DB2
Cluser
ware
Oracle ACFS for Cloud Applications
Oracle ACFS and Oracle Clusterware
• Oracle ACFS is based on Oracle ASM, and is tightly coupled with Oracle Clusterware , directly participate Clusterware cluster membership state transaction and resource based high availability (HA) management
• Oracle ACFS is a cluster enabled file system that can shared by multiple cluster nodes
• Oracle 12c RAC database files can stored in Oracle ASM diskgroups as well as Oracle ACFS.
Oracle ACFS for Cloud Applications
Oracle ACFS Resource-based Management
• Oracle clusterware resource support ACFS for enabling and disabling
volumes, loading drivers and mounting and unmounting file systems:
• When Oracle ASM instance starts, It starts the Oracle ACFS driver,
Oracle ADVM driver, Oracle Kernel Service drivers (OKS)
• Oracle ACFS file system can be automatically mounted or
dismounted based on Oracle clusterware resource action.
Oracle ACFS for Cloud Applications
ACFS on Standard ASM
• Each node has an ASM instance which costs CPU/memory
• Local ASM instance failure will cause DB instance failure
• ACFS on Standard ASM
.
Oracle ACFS for Cloud Applications
Oracle ACFS with Flex ASM for higher flexibility, scalability and availability
• Flex ASM is an option on Oracle 12c : enabled or disabled
• A small # of ASM instance. (default 3, specified by admin)
• DB instances connects to any ASM instance (local/remote)
Oracle ACFS for Cloud Applications
Two kinds of Oracle ASM configurations:• Local ASM clients connect to local ASM instance
• Flex ASM clients connect to a remote ASM instance
ASM network• Added for Flex ASM for communication between ASM clients and ASM
• On Oracle 12cR1, share the network with cluster private interconnect
Oracle ACFS for Cloud Applications
• Oracle ACFS with Flex ASM for higher flexibility, scalability and availability Database instance can connect to a remote ASM instance once the local ASM instance fails
• ADVM and Oracle ACFS file system leverage the Flex ASM for a better HA
• ADVM and ACFS utilize an Oracle ASM proxy to connect a remote
ASM instance
Configuring Oracle ACFS Prerequisites of creating Oracle ACFS
• Oracle Grid Infrastructure is installed/configured with Oracle ASM instance running:
[grid@inmem1 bin]$ /u01/app/12.1.0/grid/bin/crsctl check crsCRS-4638: Oracle High Availability Services is onlineCRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is onlineCRS-4533: Event Manager is online[grid@inmem1 bin]$ ps -ef | grep pmon | grep ASM
grid 31863 1 0 2015 ? 00:10:36 asm_pmon_+ASM1
• Required modules are loaded: These modules should be automatically loaded in Oracle 12c RAC.
[root@inmem1 ~]# lsmod | grep oracle
oracleacfs 3307969 3
oracleadvm 506254 8oracleoks 505749 2 oracleacfs,oracleadvm
Configuring Oracle ACFS Outline of ACFS creation:
• ACFS Creation Process:
• Create ASM diskgroup
• Create ASM volumes
• Create ACFS file system
• Tools for ACFS creation:
• ASMCA GUI tool
• Oracle Enterprise Manager
• ASMCMD command line tool
• SQLPLUS
Configuring Oracle ACFS
Example of creating a ACFS with ASMCMD command line tool:
• Create ASM volume “acfs_vm1” in ASM diskgroup “ACFSDG1”
ASMCMD> volcreate -G ACFSDG1 -s 1g acf_vm
• Create ACFS file system on ASM volume
mkfs -t acfs /dev/asm/acf_vm1-105
mkfs.acfs: version = 12.1.0.1.0
mkfs.acfs: on-disk version = 39.0
mkfs.acfs: volume = /dev/asm/acf_vm1-105
mkfs.acfs: volume size = 1073741824
mkfs.acfs: Format complete.
• Mount the acfs file to an OS mount /acfs_01#mkdir /acfs_vol1
#/sbin/acfsutil registry -a /dev/asm/acf_vm1-105 /acfs_vol1
acfsutil registry: mount point /acfs_vol1 successfully added to Oracle Registry
Now you can see the new ACFS file system mounted to /acfs_vol1:
# df -k | grep 'Filesystem \|asm'
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/asm/acf_vm1-105 1048576 119112 929464 12% /u01/acfs/asm_vol1
Configuring Oracle ACFS
Example of creating a ACFS with asmca GUI tool
• Create ASM volume “SSD_ACFS_VM” in ASM diskgroup “SSD_ACFS”
Configuring Oracle ACFS
Example of creating a ACFS with asmca GUI tool
• Create a ACFS file system on the ASM volume
[
Configuring Oracle ACFS
root@inmem1 ~]# df -k | grep 'Filesystem \|asm'Filesystem 1K-blocks Used Available Use% Mounted on/dev/asm/ssd_acfs_vm-329 5077204992 3978716232 1098488760 79% /SSD_ACFS
# /u01/app/grid/cfgtoollogs/asmca/scripts/acfs_script.shACFS file system /SSD_ACFS is mounted on nodes inmem1,inmem2
Configuring Oracle ACFS Creating ACFS for Oracle Database
• Use asmca GUI tool to create ACFS for Oracle Database as well as Oracle RAC Home
Configuring Oracle ACFS
Creating a ACFS for Oracle Database
• Create Oracle RAC database on ACFS with dbca: • ACFS supports Oracle 12c as well as pre-12c DBs
Configuring Oracle ACFS
Creating a ACFS for Oracle Database• After dbca, you can see the 12c RAC database configuration:
Leverage ACFS Snapshot feature
Oracle ACFS snapshot • An online, read-only or read-write, point in time copy of the Oracle
ACFS file system.• Immediately available, always online, can be resized.• Snapshot is stored in the file system . ACFS/snaps directory , can be
resized.• ACFS director not visible
Snapshot is in /SAN_ACFS/.ACFS/snaps/ro_dbsnap
Leverage ACFS Snapshot Feature
Creating an Oracle ACFS Snapshot
• Read-write snapshot can be read and written without impacting of ACFS system as a copy of the production system for testing without impacting the original production data.
• Read-write snapshot : the diskgroup compatiability attribute for ADVM set to 11.2.0.3.0 or higher.
• Take a snapshot for another snapshot, or convert a snapshot between read-only and read-write
• set diskgroup compatiability attribute for ADVM set to 12.1 or higher)
• The commands to create a snapshot :
#acfsutil snap create snapshot_name ACFS_name
• Check the snapshot status:
#acfsutil snap info ACFS_name
Leverage ACFS Snapshot Feature
Perform database backup with Oracle ACFS snapshot • Oracle ACFS in Oracle GI 12.1 supports all Oracle database files
(11.2.0.4 or higher) • Use AFCS snapshot to do a quick backup or clone the database.
Verify the database files stored in the ACFS file system:oracle@inmem1 ACFSDB]$ pwd/SAN_ACFS/db_acfs/ACFSDB[oracle@inmem1 ACFSDB]$ lsarchivelog controlfile datafile onlinelog
Leverage ACFS Snapshot Feature
Perform database backup with Oracle ACFS snapshot • Create an ACFS snapshot:
#acfsutil snap create ro_dbsnap /SAN_ACFS
#acfsutil snap create: Snapshot operation is complete.
• Check the snapshot status
# acfsutil snap info /SAN_ACFSsnapshot name: ro_dbsnapsnapshot location: /SAN_ACFS/.ACFS/snaps/ro_dbsnapRO snapshot or RW snapshot: ROparent name: /SAN_ACFSsnapshot creation time: Wed Jan 30 14:48:43 2016
Leverage ACFS Snapshot Feature
Perform database backup with Oracle ACFS snapshot • What the snapshot looks like:the database files stored in the ACFS file system:
database files in the Snapshot:
• Restore the database from the snapshot
The snapshot can be used to restore the database in case we lose the original database files in the ACFS system.
. Copy the files from the snapshot to the original directories with the cp command
cp /SAN_ACFS/.ACFS/snaps/ro_dbsnap/db_acfs/ACFSDB/onlinelog/*
/SAN_ACFS/db_acfs/ACFSDB/onlinelog
cp /SAN_ACFS/.ACFS/snaps/ro_dbsnap/db_acfs/ACFSDB/datafile/*
/SAN_ACFS/db_acfs/ACFSDB/datafile
Leverage ACFS Snapshot Feature
• Startup the database after restore.
. Create a read-write snapshot of the read snapshot as the database files (create OS softlinks from the original DB file location to the read-write snapshot
$acfsutil snap create -w -p ro_dbsnap rw_dbsnap /SAN_ACFS
# acfsutil snap info /SAN_ACFSsnapshot name: ro_dbsnapsnapshot location: /SAN_ACFS/.ACFS/snaps/ro_dbsnapRO snapshot or RW snapshot: ROparent name: /SAN_ACFSsnapshot creation time: Wed Jan 30 14:48:43 2016snapshot name: rw_dbsnap
snapshot location: /SAN_ACFS/.ACFS/snaps/rw_dbsnap
RO snapshot or RW snapshot: RW
parent name: /SAN_ACFS
snapshot creation time: Wed Jan 30 17:01:20 2016
……
Oracle ACFS Other Usage
High Availability NFS (HANFS) • Provide un-interrupted service of NFS exports
• Exports based on Highly Availability Virtual IPs(HAVIP)
• Oracle Clusterware agents ensure HAVIP and NFS
NFS export always online
• HAFS is configured using ACFS cluster
• If a cluster node fails, the HAVIP and NFS services are
are automatically taken over by a surviving
Oracle Database Appliance (ODA) • Use ACFS to store database files.
• Database Appliance Virtualized Platforms
use ACFS for shared storage access.
Oracle E-Business Suite uses OCFS2 instead of ACFS as the shared storage in the multi-node configuration.
Oracle ACFS or ASM for Database
Standalone database • Use a regular OS file system such as Linux and Windows file system• If use ASM, need to install Oracle Grid infrastructure for single node:
Oracle Restart and ASM • Oracle ACFS doesn’t support database files and redo files in Oracle
Restart configuration Oracle Database on Oracle Database Appliance
• Use ACFS to store the database files.
Oracle RAC Database • Need Oracle GI 12.1 to use Oracle ACFS for database• Pre-11.2.0.4 databases are not supported by ACFS, must use ASM• Pre-12.1.0.2 Oracle databases on Windows are not supported by
ACFS• Use ACFS for advanced features such as snapshot – for test and
development• ASM and ACFS offer the similar performance• ACFS is built on ASM and has all the benefits of ASM management• Oracle Databases Appliance uses ACFS for all databases
References
1. Oracle Automatic Storage Management Administrator's Guide, 12cR1,
Part III Oracle Automatic Storage Management Cluster File System
2. Oracle Whitepaper : Oracle ACFS, July 2014
3. Integrating ACFS Snapshots With RMAN, by Anju Garg, All things Oracle from
Redgate
4. Benefits of Oracle ACFS , Oracle Whitepaper 2Janurary 2015
5. Export Oracle RAC 12c, Apress , by Riyaj Shamsudeen, Syed Jaffar Hussain,
Tariq Farooq, Kai Yu , Chapter 5, Storage and ASM Practices
6. OCFS2 Certified with EBS 12.2 Shared File System Configurations
By Steven Chan - EBS-Oracle on Jun 06, 2016
Thank You and QAContact me at [email protected] or visit my Oracle Blog at http://kyuoracleblog.wordpress.com/
33
Top Related