Tag: Cluster File System

Troubleshooting not mounting ACFS File System

The ACSF /cloufs was created and registered on the CRS, following node reboot the file system was no longer mounting!

 

I logged on ASMCMD and checked the status of the ASM Volume:

[grid@rednodech07 ~]$ asmcmd
ASMCMD> volinfo -a
Diskgroup Name: FRA

 Volume Name: VOL_CLOUDFS
 Volume Device: ERROR
 State: DISABLED
 Size (MB): 20480
 Resize Unit (MB): 32
 Redundancy: MIRROR
 Stripe Columns: 4
 Stripe Width (K): 128
 Usage: ACFS
 Mountpath: /cloudfs

The output of the command above shows that the volume VOL_CLOUDFS is DISABLED. I tried manually  to restart it, but I got the following error:

ASMCMD> volenable -a
ORA-15032: not all alterations performed
ORA-15477: cannot communicate with the volume driver (DBD ERROR: OCIStmtExecute)
ASMCMD>

 

Then I checked if the ACFS kernel module where loaded into the Linux kernel:

  • oracleacfs (oracleacfs.ko): manages all ACFS filesystem operations.
  •  oracleavdm (oracleavdm.ko): AVDM module enabling direct interface with the filesystem
  • oracleoks (oracleoks.ko): provides memory management, lock and cluster synchronization
[root@rednodech07 ~]# /sbin/lsmod | grep oracle

Because the kernel modules were not loaded, I tried to manally load with the command

/bin/acfsload start

But it didn’t work, so I stopped the Grid Instrastruceure on the local node and I have reinstalled the ACFS drivers:

[root@rednodech07 asm]# /u01/GRID/11.2.0.4/bin/crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rednodech07'
CRS-2673: Attempting to stop 'ora.crsd' on 'rednodech07'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rednodech07'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rednodech07'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rednodech07'
CRS-2673: Attempting to stop 'ora.cvu' on 'rednodech07'
CRS-2673: Attempting to stop 'ora.oc4j' on 'rednodech07'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'rednodech07'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'rednodech07'
CRS-2673: Attempting to stop 'ora.GRID.dg' on 'rednodech07'
CRS-2673: Attempting to stop 'ora.tvdtst.db' on 'rednodech07'
CRS-2677: Stop of 'ora.cvu' on 'rednodech07' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rednodech07' succeeded
CRS-2673: Attempting to stop 'ora.rednodech07.vip' on 'rednodech07'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rednodech07' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rednodech07'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'rednodech07' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rednodech07'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'rednodech07' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rednodech07'
CRS-2677: Stop of 'ora.tvdtst.db' on 'rednodech07' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rednodech07'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'rednodech07'
CRS-2677: Stop of 'ora.DATA.dg' on 'rednodech07' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'rednodech07' succeeded
CRS-2677: Stop of 'ora.rednodech07.vip' on 'rednodech07' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'rednodech07' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'rednodech07' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'rednodech07' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'rednodech07' succeeded
CRS-2677: Stop of 'ora.GRID.dg' on 'rednodech07' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rednodech07'
CRS-2677: Stop of 'ora.asm' on 'rednodech07' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rednodech07'
CRS-2677: Stop of 'ora.ons' on 'rednodech07' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rednodech07'
CRS-2677: Stop of 'ora.net1.network' on 'rednodech07' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rednodech07' has completed
CRS-2677: Stop of 'ora.crsd' on 'rednodech07' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rednodech07'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rednodech07'
CRS-2673: Attempting to stop 'ora.evmd' on 'rednodech07'
CRS-2673: Attempting to stop 'ora.asm' on 'rednodech07'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rednodech07'
CRS-2677: Stop of 'ora.crf' on 'rednodech07' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rednodech07' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rednodech07' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rednodech07' succeeded
CRS-2677: Stop of 'ora.asm' on 'rednodech07' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rednodech07'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rednodech07' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rednodech07'
CRS-2677: Stop of 'ora.cssd' on 'rednodech07' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rednodech07'
CRS-2677: Stop of 'ora.gipcd' on 'rednodech07' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rednodech07'
CRS-2677: Stop of 'ora.gpnpd' on 'rednodech07' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rednodech07' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@rednodech07 asm]#



[root@rednodech07 ~]# /u01/GRID/11.2.0.4/bin/acfsroot install
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9312: Existing ADVM/ACFS installation detected.
ACFS-9314: Removing previous ADVM/ACFS installation.
ACFS-9315: Previous ADVM/ACFS components successfully removed.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9154: Loading 'oracleoks.ko' driver.
ACFS-9154: Loading 'oracleadvm.ko' driver.
ACFS-9154: Loading 'oracleacfs.ko' driver.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'.
ACFS-9156: Detecting control device '/dev/ofsctl'.
ACFS-9309: ADVM/ACFS installation correctness verified.
[root@rednodech07 ~]#

At this point I have restarted the Grid Infrastructure, the ACFS kernel modules got loaded  and the ASM Volume State become ENABLED.

[root@rednodech07 ~]# /u01/GRID/11.2.0.4/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.


[root@rednodech07 ~]# /sbin/lsmod | grep oracle
oracleacfs 1994567 0
oracleadvm 243254 0
oracleoks 460313 2 oracleacfs,oracleadvm

[grid@rednodech07 ~]$ asmcmd
ASMCMD> volinfo -a
Diskgroup Name: FRA

 Volume Name: VOL_CLOUDFS
 Volume Device: /dev/asm/vol_cloudfs-390
 State: ENABLED
 Size (MB): 20480
 Resize Unit (MB): 32
 Redundancy: MIRROR
 Stripe Columns: 4
 Stripe Width (K): 128
 Usage: ACFS
 Mountpath: /cloudfs

ASMCMD>

It remained to restart the ACFS File system with the following command:

[root@rednodech07 /]# /bin/mount -t acfs /dev/asm/vol_cloudfs-390 /cloudfs

[oracle@rednodech07 duplicate_tcswu]$ mount
/dev/mapper/vg_rednodech07-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sdbc1 on /boot type ext4 (rw)
/dev/mapper/vg_rednodech07-lv_home on /home type ext4 (rw)
/dev/mapper/vg_rednodech07-lv_tmp on /tmp type ext4 (rw)
/dev/mapper/vg_rednodech07-lv_u01 on /u01 type ext4 (rw)
/dev/mapper/vg_rednodech07-lv_var on /var type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
/dev/asm/vol_cloudfs-390 on /cloudfs type acfs (rw)

ASM 11gR2 Create ACFS Cluster FS

#####################################################
##           Step by step how to create Oracle ACFS Cluster Filesystem       ##
#####################################################

[grid@lnxcld02 trace]$ asmcmd


  Type "help [command]" to get help on a specific ASMCMD command.

        commands:
        --------

        md_backup, md_restore

        lsattr, setattr

        cd, cp, du, find, help, ls, lsct, lsdg, lsof, mkalias
        mkdir, pwd, rm, rmalias

        chdg, chkdg, dropdg, iostat, lsdsk, lsod, mkdg, mount
        offline, online, rebal, remap, umount

        dsget, dsset, lsop, shutdown, spbackup, spcopy, spget
        spmove, spset, startup

        chtmpl, lstmpl, mktmpl, rmtmpl

        chgrp, chmod, chown, groups, grpmod, lsgrp, lspwusr, lsusr
        mkgrp, mkusr, orapwusr, passwd, rmgrp, rmusr

        volcreate, voldelete, voldisable, volenable, volinfo
        volresize, volset, volstat


ASMCMD>     
ASMCMD> volcreate -G FRA1 -s 5G Vol_ACFS01
ASMCMD> volinfo -a
Diskgroup Name: FRA1

         Volume Name: VOL_ACFS01
         Volume Device: /dev/asm/vol_acfs01-199
         State: ENABLED
         Size (MB): 5120
         Resize Unit (MB): 32
         Redundancy: UNPROT
         Stripe Columns: 4
         Stripe Width (K): 128
         Usage:
         Mountpath:

ASMCMD> volenable -a
ASMCMD>
ASMCMD> exit


[grid@lnxcld02 trace]$ acfsdriverstate version
ACFS-9325:     Driver OS kernel version = 2.6.18-8.el5(i386).
ACFS-9326:     Driver Oracle version = 110803.1.
[grid@lnxcld02 trace]$ acfsdriverstate loaded
ACFS-9203: true



SQL> SELECT volume_name, volume_device FROM V$ASM_VOLUME;

VOLUME_NAME                    VOLUME_DEVICE
------------------------------ ----------------------------------------
VOL_ACFS01                     /dev/asm/vol_acfs01-199

1 row selected.

---------------------------------------------------------------------------------

[root@lnxcld02 adump]# ls -la /dev/asm/vol_acfs01-199
brwxrwx--- 1 root asmadmin 252, 101889 Nov  1 20:03 /dev/asm/vol_acfs01-199

[root@lnxcld02 adump]# mkdir /cloud_FS
[root@lnxcld01 adump]# mkdir /cloud_FS


[root@lnxcld02 adump]# mkfs -t acfs /dev/asm/vol_acfs01-199
mkfs.acfs: version                   = 11.2.0.3.0
mkfs.acfs: on-disk version           = 39.0
mkfs.acfs: volume                    = /dev/asm/vol_acfs01-199
mkfs.acfs: volume size               = 5368709120
mkfs.acfs: Format complete.


[root@lnxcld02 adump]# acfsutil registry -a -f /dev/asm/vol_acfs01-199 /cloud_FS
acfsutil registry: mount point /cloud_FS successfully added to Oracle Registry


[root@lnxcld02 adump]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/hda1              11G  3.6G  6.0G  38% /
/dev/hdb1              12G  7.2G  3.9G  66% /home
tmpfs                 1.5G  634M  867M  43% /dev/shm
Oracle_Software       293G  180G  114G  62% /media/sf_Oracle_Software
/dev/hdc               40G   18G    22  45% /u01
/dev/asm/vol_acfs01-199
                      5.0G   75M  5.0G   2% /cloud_FS

                      
                      
SQL> select * from v$asm_volume;

GROUP_NUMBER VOLUME_NAME                    COMPOUND_INDEX    SIZE_MB VOLUME_NUMBER REDUND STRIPE_COLUMNS STRIPE_WIDTH_K STATE            FILE_NUMBER
------------ ------------------------------ -------------- ---------- ------------- ------ -------------- -------------- ---------------- -----------
INCARNATION DRL_FILE_NUMBER RESIZE_UNIT_MB USAGE                          VOLUME_DEVICE                            MOUNTPATH
----------- --------------- -------------- ------------------------------ ---------------------------------------- --------------------
           2 VOL_ACFS01                           33554433       5120             1 UNPROT              4            128 ENABLED                  270   
766094623               0             32    ACFS                           /dev/asm/vol_acfs01-199                  /cloud_FS


1 row selected.