Bulk Exadata Patching

After more than 11 year from the launch Oracle Exadata Machine has become popular on many companies across industries, making administrators, developers and final users almost unanimously satisfied about performance and availability.

But also, on Exadata there are cumbersome maintenance activities like patching.

Most of my Exadata customers have acquired 2 non-full RACKs, which makes the patching effort quite reasonable; but recently I started working on a project with multiple full RACKs, with tens of Storage Servers, Compute Nodes and hundreds of Virtual Machines…

A very challenging environment, especially when it came to patching…

Patching all the systems using the standard patchmgr utiliy was not acceptable, therefore I had to replace my standard patching procedure with a new one offering automation and scalability.

At this subject Oracle provides few handy options:

Patching Exadata Infrastructure

  • Storage Server Patching via http/https server: starting with Oracle Exadata System Software release 18.1.0.0.0, it is possible to patch the Storage Servers using an external http server hosting the new software image. The activitiy can be scheduled up to one week before the installation, allowing on each Cell the Management Server (MS) to start downloading and run pre-checks in advance. MS interrupts the software upgrade and generates an alert if the Cell does not comply with all pre-requisites.
  • Unbreakable Linux Network: ULN offers software patches, updates, and fixes for Oracle Linux and Oracle VM. The implementation of a local YUM repository leverages the patch automation of the bare metal OS or dom0/domU.
  • InfiniBand Switch: standard rolling upgrade patching procedure using patchmgr.

Patching Grid Infrastructure & RDBMS

  • GI & RDBMS: those components are patched using the standard Oracle tools common to all platforms, but the entire process has been parallalized using OS tools like dcli commands.

Overview Bulk Exadata Patching


Main Patching Commands

Storage Server – Scheduling Automated Storage Server Update via HTTP/HTTPS

On the Storage Cell set the local Apache location hosting the cell software

[root@efucndb01-a ~]# dcli -l root -g ~/cells cellcli -e 'alter softwareUpdate store=\"http://uln-yum.emilianofusaglia.net/cellsw\"'
efucncel01-a: Software Update successfully altered.
efucncel02-a: Software Update successfully altered.
efucncel03-a: Software Update successfully altered.
efucncel04-a: Software Update successfully altered.
efucncel05-a: Software Update successfully altered.
efucncel06-a: Software Update successfully altered.
efucncel07-a: Software Update successfully altered.
efucncel08-a: Software Update successfully altered.
efucncel09-a: Software Update successfully altered.
efucncel10-a: Software Update successfully altered.
efucncel11-a: Software Update successfully altered.
efucncel12-a: Software Update successfully altered.
efucncel13-a: Software Update successfully altered.
efucncel14-a: Software Update successfully altered.
[root@efucndb01-a ~]#

Schedule the update

[root@efucndb01-a ~]# dcli -l root -g ~/cells cellcli -e 'alter softwareUpdate time=\"03:20 AM WEDNESDAY\"'
efucncel01-a: Software update is scheduled to begin at: 2020-02-05T03:20:00+01:00.
efucncel02-a: Software update is scheduled to begin at: 2020-02-05T03:20:00+01:00.
efucncel03-a: Software update is scheduled to begin at: 2020-02-05T03:20:00+01:00.
efucncel04-a: Software update is scheduled to begin at: 2020-02-05T03:20:00+01:00.
efucncel05-a: Software update is scheduled to begin at: 2020-02-05T03:20:00+01:00.
efucncel06-a: Software update is scheduled to begin at: 2020-02-05T03:20:00+01:00.
efucncel07-a: Software update is scheduled to begin at: 2020-02-05T03:20:00+01:00.
efucncel08-a: Software update is scheduled to begin at: 2020-02-05T03:20:00+01:00.
efucncel09-a: Software update is scheduled to begin at: 2020-02-05T03:20:00+01:00.
efucncel10-a: Software update is scheduled to begin at: 2020-02-05T03:20:00+01:00.
efucncel11-a: Software update is scheduled to begin at: 2020-02-05T03:20:00+01:00.
efucncel12-a: Software update is scheduled to begin at: 2020-02-05T03:20:00+01:00.
efucncel13-a: Software update is scheduled to begin at: 2020-02-05T03:20:00+01:00.
efucncel14-a: Software update is scheduled to begin at: 2020-02-05T03:20:00+01:00.
[root@efucndb01-a ~]#

Verify the scheduled upgrade

[root@efucndb01-a ~]# dcli -l root -g ~/cells cellcli -e 'list softwareupdate detail'
efucncel01-a: name: 19.3.4.0.0.200130
efucncel01-a: status: PreReq OK. Ready to update at: 2020-02-05T03:20:00+01:00
efucncel01-a: store: http://uln-yum.emilianofusaglia.net/cellsw
efucncel01-a: time: 2020-02-05T03:20:00+01:00
efucncel02-a: name: 19.3.4.0.0.200130
efucncel02-a: status: PreReq OK. Ready to update at: 2020-02-05T03:20:00+01:00
efucncel02-a: store: http://uln-yum.emilianofusaglia.net/cellsw
efucncel02-a: time: 2020-02-05T03:20:00+01:00
efucncel03-a: name: 19.3.4.0.0.200130
efucncel03-a: status: PreReq OK. Ready to update at: 2020-02-05T03:20:00+01:00
efucncel03-a: store: http://uln-yum.emilianofusaglia.net/cellsw
efucncel03-a: time: 2020-02-05T03:20:00+01:00
efucncel04-a: name: 19.3.4.0.0.200130
efucncel04-a: status: PreReq OK. Ready to update at: 2020-02-05T03:20:00+01:00
efucncel04-a: store: http://uln-yum.emilianofusaglia.net/cellsw
efucncel04-a: time: 2020-02-05T03:20:00+01:00
efucncel05-a: name: 19.3.4.0.0.200130
efucncel05-a: status: PreReq OK. Ready to update at: 2020-02-05T03:20:00+01:00
efucncel05-a: store: http://uln-yum.emilianofusaglia.net/cellsw
efucncel05-a: time: 2020-02-05T03:20:00+01:00
efucncel06-a: name: 19.3.4.0.0.200130
efucncel06-a: status: PreReq OK. Ready to update at: 2020-02-05T03:20:00+01:00
efucncel06-a: store: http://uln-yum.emilianofusaglia.net/cellsw
efucncel06-a: time: 2020-02-05T03:20:00+01:00
efucncel07-a: name: 19.3.4.0.0.200130
efucncel07-a: status: PreReq OK. Ready to update at: 2020-02-05T03:20:00+01:00
efucncel07-a: store: http://uln-yum.emilianofusaglia.net/cellsw
efucncel07-a: time: 2020-02-05T03:20:00+01:00
efucncel08-a: name: 19.3.4.0.0.200130
efucncel08-a: status: PreReq OK. Ready to update at: 2020-02-05T03:20:00+01:00
efucncel08-a: store: http://uln-yum.emilianofusaglia.net/cellsw
efucncel08-a: time: 2020-02-05T03:20:00+01:00
efucncel09-a: name: 19.3.4.0.0.200130
efucncel09-a: status: PreReq OK. Ready to update at: 2020-02-05T03:20:00+01:00
efucncel09-a: store: http://uln-yum.emilianofusaglia.net/cellsw
efucncel09-a: time: 2020-02-05T03:20:00+01:00
efucncel10-a: name: 19.3.4.0.0.200130
efucncel10-a: status: PreReq OK. Ready to update at: 2020-02-05T03:20:00+01:00
efucncel10-a: store: http://uln-yum.emilianofusaglia.net/cellsw
efucncel10-a: time: 2020-02-05T03:20:00+01:00
efucncel11-a: name: 19.3.4.0.0.200130
efucncel11-a: status: PreReq OK. Ready to update at: 2020-02-05T03:20:00+01:00
efucncel11-a: store: http://uln-yum.emilianofusaglia.net/cellsw
efucncel11-a: time: 2020-02-05T03:20:00+01:00
efucncel12-a: name: 19.3.4.0.0.200130
efucncel12-a: status: PreReq OK. Ready to update at: 2020-02-05T03:20:00+01:00
efucncel12-a: store: http://uln-yum.emilianofusaglia.net/cellsw
efucncel12-a: time: 2020-02-05T03:20:00+01:00
efucncel13-a: name: 19.3.4.0.0.200130
efucncel13-a: status: PreReq OK. Ready to update at: 2020-02-05T03:20:00+01:00
efucncel13-a: store: http://uln-yum.emilianofusaglia.net/cellsw
efucncel13-a: time: 2020-02-05T03:20:00+01:00
efucncel14-a: name: 19.3.4.0.0.200130
efucncel14-a: status: PreReq OK. Ready to update at: 2020-02-05T03:20:00+01:00
efucncel14-a: store: http://uln-yum.emilianofusaglia.net/cellsw
efucncel14-a: time: 2020-02-05T03:20:00+01:00
[root@efucndb01-a ~]#

Unbreakable Linux Network

dom0 checks

[root@efuconsole dbserver_patch_19.200120]# ./patchmgr -dbnodes ~/dom0 -precheck -yum_repo http://uln-yum.emilianofusaglia.net/yum/EngineeredSystems/exadata/dbserver/dom0/19.3.4.0.0/base/x86_64 -target_version 19.3.4.0.0.200130

NOTE patchmgr release: 19.200120 (always check MOS 1553103.1 for the latest release of dbserver.patch.zip)
NOTE
WARNING Do not interrupt the patchmgr session.
WARNING Do not resize the screen. It may disturb the screen layout.
WARNING Do not reboot database nodes during update or rollback.
WARNING Do not open logfiles in write mode and do not try to alter them.

2020-02-06 14:06:17 +0100 :Working: Verify SSH equivalence for the root user to node(s)
2020-02-06 14:06:19 +0100 :SUCCESS: Verify SSH equivalence for the root user to node(s)
2020-02-06 14:06:22 +0100 :Working: Initiate precheck on 8 node(s)
2020-02-06 14:07:36 +0100 :Working: Check free space on node(s)
2020-02-06 14:07:42 +0100 :SUCCESS: Check free space on node(s)
2020-02-06 14:08:07 +0100 :Working: dbnodeupdate.sh running a precheck on node(s).
2020-02-06 14:09:43 +0100 :SUCCESS: Initiate precheck on node(s).
2020-02-06 14:09:45 +0100 :SUCCESS: Completed run of command: ./patchmgr -dbnodes /root/dom0 -precheck -yum_repo http://uln-yum.emilianofusaglia.net/yum/EngineeredSystems/exadata/dbserver/dom0/19.3.4.0.0/base/x86_64 -target_version 19.3.4.0.0.200130
2020-02-06 14:09:45 +0100 :INFO : Precheck attempted on nodes in file /root/dom0: [efucndb01-a efucndb02-a efucndb03-a efucndb04-a efucndb05-a efucndb06-a efucndb07-a efucndb08-a]
2020-02-06 14:09:45 +0100 :INFO : Current image version on dbnode(s) is:
2020-02-06 14:09:45 +0100 :INFO : efucndb01-a: 19.2.6.0.0.190911.1
2020-02-06 14:09:45 +0100 :INFO : efucndb02-a: 19.2.6.0.0.190911.1
2020-02-06 14:09:45 +0100 :INFO : efucndb03-a: 19.2.6.0.0.190911.1
2020-02-06 14:09:45 +0100 :INFO : efucndb04-a: 19.2.6.0.0.190911.1
2020-02-06 14:09:45 +0100 :INFO : efucndb05-a: 19.2.6.0.0.190911.1
2020-02-06 14:09:45 +0100 :INFO : efucndb06-a: 19.2.6.0.0.190911.1
2020-02-06 14:09:45 +0100 :INFO : efucndb07-a: 19.2.6.0.0.190911.1
2020-02-06 14:09:45 +0100 :INFO : efucndb08-a: 19.2.6.0.0.190911.1
2020-02-06 14:09:45 +0100 :INFO : For details, check the following files in /EXAVMIMAGES/Patch/patchmgr_DBSERVER/dbserver_patch_19.200120:
2020-02-06 14:09:45 +0100 :INFO : - _dbnodeupdate.log
2020-02-06 14:09:45 +0100 :INFO : - patchmgr.log
2020-02-06 14:09:45 +0100 :INFO : - patchmgr.trc
2020-02-06 14:09:45 +0100 :INFO : Exit status:0
2020-02-06 14:09:45 +0100 :INFO : Exiting.
[root@efucndb01-a dbserver_patch_19.200120]#

dom0 upgrade

[root@efuconsole dbserver_patch_19.200120]# ./patchmgr -dbnodes ~/dom0 -upgrade -yum_repo http://uln-yum.emilianofusaglia.net/yum/EngineeredSystems/exadata/dbserver/dom0/19.3.4.0.0/base/x86_64 -target_version 19.3.4.0.0.200130

NOTE patchmgr release: 19.200120 (always check MOS 1553103.1 for the latest release of dbserver.patch.zip)
NOTE
NOTE Database nodes will reboot during the update process.
NOTE
WARNING Do not interrupt the patchmgr session.
WARNING Do not resize the screen. It may disturb the screen layout.
WARNING Do not reboot database nodes during update or rollback.
WARNING Do not open logfiles in write mode and do not try to alter them.

2020-02-06 14:29:11 +0100 :Working: Verify SSH equivalence for the root user to node(s)
2020-02-06 14:29:13 +0100 :SUCCESS: Verify SSH equivalence for the root user to node(s)
2020-02-06 14:29:15 +0100 :Working: Initiate prepare steps on node(s).
2020-02-06 14:29:17 +0100 :Working: Check free space on node(s)
2020-02-06 14:29:23 +0100 :SUCCESS: Check free space on node(s)
2020-02-06 14:29:59 +0100 :SUCCESS: Initiate prepare steps on node(s).
2020-02-06 14:29:59 +0100 :Working: Initiate update on 8 node(s).
2020-02-06 14:29:59 +0100 :Working: dbnodeupdate.sh running a backup on 8 node(s).
2020-02-06 14:36:16 +0100 :SUCCESS: dbnodeupdate.sh running a backup on 8 node(s).
2020-02-06 14:36:16 +0100 :Working: Initiate update on node(s)
2020-02-06 14:36:16 +0100 :Working: Get information about any required OS upgrades from node(s).
2020-02-06 14:36:28 +0100 :SUCCESS: Get information about any required OS upgrades from node(s).
2020-02-06 14:36:28 +0100 :Working: dbnodeupdate.sh running an update step on all nodes.
2020-02-06 14:56:38 +0100 :INFO : efucndb01-a is ready to reboot.
2020-02-06 14:56:38 +0100 :INFO : efucndb02-a is ready to reboot.
2020-02-06 14:56:38 +0100 :INFO : efucndb03-a is ready to reboot.
2020-02-06 14:56:38 +0100 :INFO : efucndb04-a is ready to reboot.
2020-02-06 14:56:38 +0100 :INFO : efucndb05-a is ready to reboot.
2020-02-06 14:56:38 +0100 :INFO : efucndb06-a is ready to reboot.
2020-02-06 14:56:38 +0100 :INFO : efucndb07-a is ready to reboot.
2020-02-06 14:56:38 +0100 :INFO : efucndb08-a is ready to reboot.
2020-02-06 14:56:39 +0100 :SUCCESS: dbnodeupdate.sh running an update step on all nodes.
2020-02-06 14:56:51 +0100 :Working: Initiate reboot on node(s)
2020-02-06 14:56:55 +0100 :SUCCESS: Initiate reboot on node(s)
2020-02-06 14:56:55 +0100 :Working: Waiting to ensure node(s) is down before reboot.
2020-02-06 14:58:20 +0100 :SUCCESS: Waiting to ensure node(s) is down before reboot.
2020-02-06 14:58:20 +0100 :Working: Waiting to ensure node(s) is up after reboot.
2020-02-06 15:04:23 +0100 :SUCCESS: Waiting to ensure node(s) is up after reboot.
2020-02-06 15:04:23 +0100 :Working: Waiting to connect to node(s) with SSH. During Linux upgrades this can take some time.
2020-02-06 15:27:46 +0100 :SUCCESS: Waiting to connect to node(s) with SSH. During Linux upgrades this can take some time.
2020-02-06 15:27:46 +0100 :Working: Wait for node(s) is ready for the completion step of update.
2020-02-06 15:31:29 +0100 :SUCCESS: Wait for node(s) is ready for the completion step of update.
2020-02-06 15:31:30 +0100 :Working: Initiate completion step from dbnodeupdate.sh on node(s)
2020-02-06 15:48:10 +0100 :SUCCESS: Initiate completion step from dbnodeupdate.sh on efucndb01-a
2020-02-06 15:48:14 +0100 :SUCCESS: Initiate completion step from dbnodeupdate.sh on efucndb02-a
2020-02-06 15:48:19 +0100 :SUCCESS: Initiate completion step from dbnodeupdate.sh on efucndb03-a
2020-02-06 15:48:30 +0100 :SUCCESS: Initiate completion step from dbnodeupdate.sh on efucndb04-a
2020-02-06 15:48:35 +0100 :SUCCESS: Initiate completion step from dbnodeupdate.sh on efucndb05-a
2020-02-06 15:48:46 +0100 :SUCCESS: Initiate completion step from dbnodeupdate.sh on efucndb06-a
2020-02-06 15:48:50 +0100 :SUCCESS: Initiate completion step from dbnodeupdate.sh on efucndb07-a
2020-02-06 15:49:02 +0100 :SUCCESS: Initiate completion step from dbnodeupdate.sh on efucndb08-a
2020-02-06 15:49:18 +0100 :SUCCESS: Initiate update on node(s).
2020-02-06 15:49:18 +0100 :SUCCESS: Initiate update on 0 node(s).
[INFO ] Collected dbnodeupdate diag in file: Diag_patchmgr_dbnode_upgrade_060220142909.tbz
-rw-r--r-- 1 root root 6381043 Feb 6 15:49 Diag_patchmgr_dbnode_upgrade_060220142909.tbz
2020-02-06 15:49:22 +0100 :SUCCESS: Completed run of command: ./patchmgr -dbnodes /root/dom0 -upgrade -yum_repo http://uln-yum.emilianofusaglia.net/yum/EngineeredSystems/exadata/dbserver/dom0/19.3.4.0.0/base/x86_64 -target_version 19.3.4.0.0.200130
2020-02-06 15:49:22 +0100 :INFO : Upgrade attempted on nodes in file /root/dom0: [efucndb01-a efucndb02-a efucndb03-a efucndb04-a efucndb05-a efucndb06-a efucndb07-a efucndb08-a]
2020-02-06 15:49:22 +0100 :INFO : Current image version on dbnode(s) is:
2020-02-06 15:49:22 +0100 :INFO : efucndb01-a: 19.3.4.0.0.200130
2020-02-06 15:49:22 +0100 :INFO : efucndb02-a: 19.3.4.0.0.200130
2020-02-06 15:49:22 +0100 :INFO : efucndb03-a: 19.3.4.0.0.200130
2020-02-06 15:49:22 +0100 :INFO : efucndb04-a: 19.3.4.0.0.200130
2020-02-06 15:49:22 +0100 :INFO : efucndb05-a: 19.3.4.0.0.200130
2020-02-06 15:49:22 +0100 :INFO : efucndb06-a: 19.3.4.0.0.200130
2020-02-06 15:49:22 +0100 :INFO : efucndb07-a: 19.3.4.0.0.200130
2020-02-06 15:49:22 +0100 :INFO : efucndb08-a: 19.3.4.0.0.200130
2020-02-06 15:49:22 +0100 :INFO : For details, check the following files in /EXAVMIMAGES/Patch/patchmgr_DBSERVER/dbserver_patch_19.200120:
2020-02-06 15:49:22 +0100 :INFO : - _dbnodeupdate.log
2020-02-06 15:49:22 +0100 :INFO : - patchmgr.log
2020-02-06 15:49:22 +0100 :INFO : - patchmgr.trc
2020-02-06 15:49:22 +0100 :INFO : Exit status:0
2020-02-06 15:49:22 +0100 :INFO : Exiting.

InfiniBand Switch

IB switch checks

[root@efucndb01-a patch_switch_19.3.4.0.0.200130]# ./patchmgr -ibswitches ~/ibs -upgrade -ibswitch_precheck
2020-02-10 07:57:44 +0100 :Working: Verify SSH equivalence for the root user to node(s)
2020-02-10 07:57:46 +0100 :SUCCESS: Verify SSH equivalence for the root user to node(s)
2020-02-10 07:57:47 +0100 1 of 1 :Working: Initiate pre-upgrade validation check on InfiniBand switch(es).
----- InfiniBand switch update process started 2020-02-10 07:57:48 +0100 -----
[NOTE ] Log file at /EXAVMIMAGES/Patch/IBs_19.3.4.0.0/patch_switch_19.3.4.0.0.200130/upgradeIBSwitch.log
[INFO ] List of InfiniBand switches for upgrade: ( efucnsw-iba01-a efucnsw-ibb01-a )
[SUCCESS ] Verifying Network connectivity to efucnsw-iba01-a
[SUCCESS ] Verifying Network connectivity to efucnsw-ibb01-a
[SUCCESS ] Validating verify-topology output
[INFO ] Master Subnet Manager is set to "efucnsw-iba01-a" in all Switches
[INFO ] ---------- Starting with InfiniBand Switch efucnsw-iba01-a
[WARNING ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.13-2 with the current package.
To downgrade to other versions:
Manually download the InfiniBand switch firmware package to the patch directory
Set export variable "EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION" to the appropriate version
Run patchmgr command to initiate downgrade.
[SUCCESS ] Verify SSH access to the patchmgr host efucndb01-a.emilianofusaglia.net from the InfiniBand Switch efucnsw-iba01-a.
[INFO ] Starting pre-update validation on efucnsw-iba01-a
[SUCCESS ] Verifying that /tmp has 150M in efucnsw-iba01-a, found 492M
[SUCCESS ] Verifying that / has 20M in efucnsw-iba01-a, found 28M
[SUCCESS ] NTP daemon is running on efucnsw-iba01-a.
[INFO ] Manually validate the following entries Date:(YYYY-aM-DD) 2020-02-10 Time:(HH:MM:SS) 07:58:05
[INFO ] Validating the current firmware on the InfiniBand Switch
[SUCCESS ] Firmware verification on InfiniBand switch efucnsw-iba01-a
[SUCCESS ] Verifying that the patchmgr host efucndb01-a.emilianofusaglia.net is recognized on the InfiniBand Switch efucnsw-iba01-a through getHostByName
[SUCCESS ] Execute plugin check for Patch Check Prereq on efucnsw-iba01-a
[INFO ] Finished pre-update validation on efucnsw-iba01-a
[SUCCESS ] Pre-update validation on efucnsw-iba01-a
[SUCCESS ] Prereq check on efucnsw-iba01-a
[INFO ] ---------- Starting with InfiniBand Switch efucnsw-ibb01-a
[WARNING ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.13-2 with the current package.
To downgrade to other versions:
Manually download the InfiniBand switch firmware package to the patch directory
Set export variable "EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION" to the appropriate version
Run patchmgr command to initiate downgrade.
[SUCCESS ] Verify SSH access to the patchmgr host efucndb01-a.emilianofusaglia.net from the InfiniBand Switch efucnsw-ibb01-a.
[INFO ] Starting pre-update validation on efucnsw-ibb01-a
[SUCCESS ] Verifying that /tmp has 150M in efucnsw-ibb01-a, found 492M
[SUCCESS ] Verifying that / has 20M in efucnsw-ibb01-a, found 28M
[SUCCESS ] NTP daemon is running on efucnsw-ibb01-a.
[INFO ] Manually validate the following entries Date:(YYYY-aM-DD) 2020-02-10 Time:(HH:MM:SS) 07:58:25
[INFO ] Validating the current firmware on the InfiniBand Switch
[SUCCESS ] Firmware verification on InfiniBand switch efucnsw-ibb01-a
[SUCCESS ] Verifying that the patchmgr host efucndb01-a.emilianofusaglia.net is recognized on the InfiniBand Switch efucnsw-ibb01-a through getHostByName
[SUCCESS ] Execute plugin check for Patch Check Prereq on efucnsw-ibb01-a
[INFO ] Finished pre-update validation on efucnsw-ibb01-a
[SUCCESS ] Pre-update validation on efucnsw-ibb01-a
[SUCCESS ] Prereq check on efucnsw-ibb01-a
[SUCCESS ] Overall status
----- InfiniBand switch update process ended 2020-02-10 07:58:42 +0100 -----
2020-02-10 07:58:42 +0100 1 of 1 :SUCCESS: Initiate pre-upgrade validation check on InfiniBand switch(es).
2020-02-10 07:58:42 +0100 :SUCCESS: Completed run of command: ./patchmgr -ibswitches /root/ibs -upgrade -ibswitch_precheck
2020-02-10 07:58:42 +0100 :INFO : upgrade attempted on nodes in file /root/ibs: [efucnsw-iba01-a efucnsw-ibb01-a]
2020-02-10 07:58:42 +0100 :INFO : For details, check the following files in /EXAVMIMAGES/Patch/IBs_19.3.4.0.0/patch_switch_19.3.4.0.0.200130:
2020-02-10 07:58:42 +0100 :INFO : - upgradeIBSwitch.log
2020-02-10 07:58:42 +0100 :INFO : - upgradeIBSwitch.trc
2020-02-10 07:58:42 +0100 :INFO : - patchmgr.stdout
2020-02-10 07:58:42 +0100 :INFO : - patchmgr.stderr
2020-02-10 07:58:42 +0100 :INFO : - patchmgr.log
2020-02-10 07:58:42 +0100 :INFO : - patchmgr.trc
2020-02-10 07:58:42 +0100 :INFO : Exit status:0
2020-02-10 07:58:42 +0100 :INFO : Exiting.
[root@efucndb01-a patch_switch_19.3.4.0.0.200130]#

IB switch upgrade

[root@efucndb01-a patch_switch_19.3.4.0.0.200130]# ./patchmgr -ibswitches ~/ibs -upgrade
2020-02-10 07:59:22 +0100 :Working: Verify SSH equivalence for the root user to node(s)
2020-02-10 07:59:24 +0100 :SUCCESS: Verify SSH equivalence for the root user to node(s)
2020-02-10 07:59:25 +0100 1 of 1 :Working: Initiate upgrade of InfiniBand switches to 2.2.14-1. Expect up to 40 minutes for each switch
----- InfiniBand switch update process started 2020-02-10 07:59:25 +0100 -----
[NOTE ] Log file at /EXAVMIMAGES/Patch/IBs_19.3.4.0.0/patch_switch_19.3.4.0.0.200130/upgradeIBSwitch.log
[INFO ] List of InfiniBand switches for upgrade: ( efucnsw-iba01-a efucnsw-ibb01-a )
[SUCCESS ] Verifying Network connectivity to efucnsw-iba01-a
[SUCCESS ] Verifying Network connectivity to efucnsw-ibb01-a
[SUCCESS ] Validating verify-topology output
[INFO ] Proceeding with upgrade of InfiniBand switches to version 2.2.14_1
[INFO ] Master Subnet Manager is set to "efucnsw-iba01-a" in all Switches
[INFO ] ---------- Starting with InfiniBand Switch efucnsw-iba01-a
[WARNING ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.13-2 with the current package.
To downgrade to other versions:
Manually download the InfiniBand switch firmware package to the patch directory
Set export variable "EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION" to the appropriate version
Run patchmgr command to initiate downgrade.
[SUCCESS ] Verify SSH access to the patchmgr host efucndb01-a.emilianofusaglia.net from the InfiniBand Switch efucnsw-iba01-a.
[INFO ] Starting pre-update validation on efucnsw-iba01-a
[SUCCESS ] Verifying that /tmp has 150M in efucnsw-iba01-a, found 492M
[SUCCESS ] Verifying that / has 20M in efucnsw-iba01-a, found 26M
[SUCCESS ] Service opensmd is running on InfiniBand Switch efucnsw-iba01-a
[SUCCESS ] NTP daemon is running on efucnsw-iba01-a.
[INFO ] Manually validate the following entries Date:(YYYY-aM-DD) 2020-02-10 Time:(HH:MM:SS) 07:59:41
[INFO ] Validating the current firmware on the InfiniBand Switch
[SUCCESS ] Firmware verification on InfiniBand switch efucnsw-iba01-a
[SUCCESS ] Verifying that the patchmgr host efucndb01-a.emilianofusaglia.net is recognized on the InfiniBand Switch efucnsw-iba01-a through getHostByName
[SUCCESS ] Execute plugin check for Patch Check Prereq on efucnsw-iba01-a
[INFO ] Finished pre-update validation on efucnsw-iba01-a
[SUCCESS ] Pre-update validation on efucnsw-iba01-a
[INFO ] Package will be downloaded at firmware update time via scp
[SUCCESS ] Execute plugin check for Patching on efucnsw-iba01-a
[INFO ] Starting upgrade on efucnsw-iba01-a to 2.2.14_1. Please give upto 15 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C during the upgrade
[INFO ] Rebooting efucnsw-iba01-a to complete the firmware update. Wait for 15 minutes before continuing. DO NOT MANUALLY REBOOT THE INFINIBAND SWITCH
Connection to efucndb01-a closed by remote host.
Connection to efucndb01-a closed.
2020-02-10 08:27:49 +0100 :Working: Verify SSH equivalence for the root user to node(s)
2020-02-10 08:27:51 +0100 :SUCCESS: Verify SSH equivalence for the root user to node(s)
2020-02-10 08:27:52 +0100 1 of 1 :Working: Initiate upgrade of InfiniBand switches to 2.2.14-1. Expect up to 40 minutes for each switch
----- InfiniBand switch update process started 2020-02-10 08:27:52 +0100 -----
[NOTE ] Log file at /EXAVMIMAGES/Patch/IBs_19.3.4.0.0/patch_switch_19.3.4.0.0.200130/upgradeIBSwitch.log
[INFO ] List of InfiniBand switches for upgrade: ( efucnsw-iba01-a efucnsw-ibb01-a )
[SUCCESS ] Verifying Network connectivity to efucnsw-iba01-a
[SUCCESS ] Verifying Network connectivity to efucnsw-ibb01-a
[INFO ] InfiniBand switch efucnsw-iba01-a is already at target version.
[SUCCESS ] Validating verify-topology output
[INFO ] Proceeding with upgrade of InfiniBand switches to version 2.2.14_1
[INFO ] Master Subnet Manager is set to "efucnsw-ibb01-a" in all Switches
[INFO ] ---------- Starting with InfiniBand Switch efucnsw-ibb01-a
[WARNING ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.13-2 with the current package.
To downgrade to other versions:
Manually download the InfiniBand switch firmware package to the patch directory
Set export variable "EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION" to the appropriate version
Run patchmgr command to initiate downgrade.
[SUCCESS ] Verify SSH access to the patchmgr host efucndb01-a.emilianofusaglia.net from the InfiniBand Switch efucnsw-ibb01-a.
[INFO ] Starting pre-update validation on efucnsw-ibb01-a
[SUCCESS ] Verifying that /tmp has 150M in efucnsw-ibb01-a, found 492M
[SUCCESS ] Verifying that / has 20M in efucnsw-ibb01-a, found 26M
[SUCCESS ] Service opensmd is running on InfiniBand Switch efucnsw-ibb01-a
[SUCCESS ] NTP daemon is running on efucnsw-ibb01-a.
[INFO ] Manually validate the following entries Date:(YYYY-aM-DD) 2020-02-10 Time:(HH:MM:SS) 08:28:07
[INFO ] Validating the current firmware on the InfiniBand Switch
[SUCCESS ] Firmware verification on InfiniBand switch efucnsw-ibb01-a
[SUCCESS ] Verifying that the patchmgr host efucndb01-a.emilianofusaglia.net is recognized on the InfiniBand Switch efucnsw-ibb01-a through getHostByName
[SUCCESS ] Execute plugin check for Patch Check Prereq on efucnsw-ibb01-a
[INFO ] Finished pre-update validation on efucnsw-ibb01-a
[SUCCESS ] Pre-update validation on efucnsw-ibb01-a
[INFO ] Package will be downloaded at firmware update time via scp
[SUCCESS ] Execute plugin check for Patching on efucnsw-ibb01-a
[INFO ] Starting upgrade on efucnsw-ibb01-a to 2.2.14_1. Please give upto 15 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C during the upgrade
[INFO ] Rebooting efucnsw-ibb01-a to complete the firmware update. Wait for 15 minutes before continuing. DO NOT MANUALLY REBOOT THE INFINIBAND SWITCH
Connection to efucndb01-a closed by remote host.
Connection to efucndb01-a closed.

Exadata X8M introduces BIG Architectural Changes

Launched at the 2019 Oracle Open Word Conference the Exadata X8M introduces few big architectural changes which make the leading Oracle Database Machine even more attractive.

Among the most relevant changes there are:

  • InfiniBand Network replacement with an Ethernet network fabric
  • Intel Optane DC Persistent Memory inside the Storage Cell
  • New Remote Direct Memory Access (RDMA) functionalities
  • Replacement of the XEN Hypervisor with KVM

InfiniBand Network replacement with an Ethernet network fabric

The characteristic 40Gbit/second InfiniBand network used for all private network communications among database nodes and storage cells has been replaced by a new 100Gbit/second RDMA over Converged Ethernet Fabric (RoCE) based on the Cisco Switch 9336c RoCE .

The new network not only increase 2.5x the throughput but also reduces the communication latancy.

The schema below highlight the network architecture change for all private communications

Intel Optane DC Persistent Memory inside the Storage Cell

Oracle has introduced 1.5TB of Intel Optane DC Persistent Memory as additional storage device inside all Exadata X8M Storage Cell, (no matter if equipped with HC and EF devices), and it is used as accelerator in front of the Flash Memory Cards. In term of speed this new type of ultra fast storage device is located between the DRAM and the Flash Memory, bringing to three the number of multi-tiered storage devices present inside the Storage Cell.

The Exadata unique software is than capable to extract the maximum performance from this HW configuration, automatically detecting and placing the hottest data on the Persistent Memory, reducing the I/O latency of the most critical tasks.

Below is described the list of Storage Cell’s devices ordered by speed.

New Remote Direct Memory Access (RDMA) functionalities

Until now the RDMA was used among database nodes for exchanging Exafusion messages or for Smart Fusion Block Transfer. Starting with Exadata X8M, the RDMA technology is also used to perform direct I/O access to the Persistent Memory of the Storage Cells, bypassing the network and I/O stack and eliminating expensive CPU interrupts and context switches. This optimization reduces the latency by 10x, from 200μs to less than 19μs.

The picture below highlight the “Database Node to Database Node” and the new “Database Node to Storage Cell” communication using RDMA functionalities.

Replacement of the XEN Hypervisor with KVM

Oracle virtualization technology is called Oracle Virtual Machine (OVM) and in a productive invironment it can be implemented with one of this two different products:

  • Xen
  • KVM

Starting with Exadata X8M-2 the virtualization technology in use is KVM instead of Xen. Oracle started replacing Xen with KVM few years ago, for example on the smaller engineered system ODA X7-2M & X7-2S, but for the Exadata took longer, and I think the root cause was the InfiniBand network. Infact KVM is not fully integrated with InfiniBand, and it does not support bridging.

Exadata Deployment with Elastic Configuration

Recently, for one of my customers, I had the chance to install a couples of Exadata X7-2 using the new Elastic Configuration. The major benefits of using Elastic Configuration consists in the possibility to acquire the Exadata Machine with almost any possible combination of Database Nodes and Storage Cells.

In the past we were used to standard Oracle pre-defined Exadata Machine configurations: Eighth Rack, Quarter Rack, Half Rack and Full Rack, which is still possible, but not flexible enough.

The pictures below highlight the differences between the two configurations:

Edadats_Classiv_vs_Elastic

source: Oracle Data Sheet Exadata Database Machine X7-2

 

Deployment Exadata Elactic Configuration

The elastic configuration process automates the initial IP address allocations to databasenodes and storage cells, regardless the ordered configuration.  The Exadata Machine is connected to the InfiniBand switches using a standard cabling methodology which allows to determinate the node’s location in the rack. This information is therefore used when the nodes are powered up for the first time in order to assign the initial default IPs.

[root@exatest-iba0 ~]# ibhosts
Ca : 0x579b0123796ba0 ports 2 "node10 elasticNode 192.168.10.17,192.168.10.18 ETH0"
Ca : 0x579b01237966e0 ports 2 "node8 elasticNode 192.168.10.15,192.168.10.16 ETH0"
Ca : 0x579b0123844ab0 ports 2 "node6 elasticNode 192.168.10.11,192.168.10.12 ETH0"
Ca : 0x579b0123845e50 ports 2 "node5 elasticNode 192.168.10.7,192.168.10.8 ETH0"
Ca : 0x579b0123845fe0 ports 2 "node4 elasticNode 192.168.10.40,172.16.2.40 ETH0"
Ca : 0x579b0123845ea0 ports 2 "node3 elasticNode 192.168.10.9,192.168.10.10 ETH0"
Ca : 0x579b0123812b90 ports 2 "node2 elasticNode 192.168.10.1,192.168.10.2 ETH0"
Ca : 0x579b0123812970 ports 2 "node1 elasticNode 192.168.10.3,192.168.10.4 ETH0"
[root@exatest-iba0 ~]#

 

 

Because the Virtualization option was required,  it has to be activated at this stage:

[root@node8 ~]# /opt/oracle.SupportTools/switch_to_ovm.sh
2019-03-07 01:05:22 -0800 [INFO] Switch to DOM0 system partition /dev/VGExaDb/LVDbSys3 (/dev/mapper/VGExaDb-LVDbSys3)
2019-03-07 01:05:22 -0800 [INFO] Active system device: /dev/mapper/VGExaDb-LVDbSys1
2019-03-07 01:05:22 -0800 [INFO] Active system device in boot area: /dev/mapper/VGExaDb-LVDbSys1
2019-03-07 01:05:23 -0800 [INFO] Set active system device to /dev/VGExaDb/LVDbSys3 in /boot/I_am_hd_boot
2019-03-07 01:05:23 -0800 [INFO] Creating /.elasticConfig on DOM0 boot partition /boot
2019-03-07 01:05:34 -0800 [INFO] Reboot has been initiated to switch to the DOM0 system partition
Connection to 192.168.1.8 closed by remote host.
Connection to 192.168.1.8 closed.
✘

 

After the switch to OVM command it is time to reclaim the space initially used by the Linux bare metal Logical Volumes:

[root@node8 ~]# /opt/oracle.SupportTools/reclaimdisks.sh -free -reclaim
Model is ORACLE SERVER X7-2
Number of LSI controllers: 1
Physical disks found: 4 (252:0 252:1 252:2 252:3)
Logical drives found: 1
Linux logical drive: 0
RAID Level for the Linux logical drive: 5
Physical disks in the Linux logical drive: 4 (252:0 252:1 252:2 252:3)
Dedicated Hot Spares for the Linux logical drive: 0
Global Hot Spares: 0
[INFO ] Check for DOM0 with inactive Linux system disk
[INFO ] Valid DOM0 with inactive Linux system disk is detected
[INFO ] Number of partitions on the system device /dev/sda: 3
[INFO ] Higher partition number on the system device /dev/sda: 3
[INFO ] Last sector on the system device /dev/sda: 3509760000
[INFO ] End sector of the last partition on the system device /dev/sda: 3509759966
[INFO ] Remove inactive system logical volume /dev/VGExaDb/LVDbSys1
[INFO ] Remove logical volume /dev/VGExaDb/LVDbOra1
[INFO ] Extend logical volume /dev/VGExaDb/LVDbExaVMImages
[INFO ] Resize ocfs2 on logical volume /dev/VGExaDb/LVDbExaVMImages
[INFO ] XEN boot version and rpm versions are in sync
[INFO ] XEN EFI files will not be updated
[INFO ] Force setup grub
[root@node8 ~]#

 

Check the success of the reclaim disks procedure:

[root@node8 ~]# /opt/oracle.SupportTools/reclaimdisks.sh -check
Model is ORACLE SERVER X7-2
Number of LSI controllers: 1
Physical disks found: 4 (252:0 252:1 252:2 252:3)
Logical drives found: 1
Linux logical drive: 0
RAID Level for the Linux logical drive: 5
Physical disks in the Linux logical drive: 4 (252:0 252:1 252:2 252:3)
Dedicated Hot Spares for the Linux logical drive: 0
Global Hot Spares: 0
Valid. Disks configuration: RAID5 from 4 disks with no global and dedicated hot spare disks.
Valid. Booted: DOM0. Layout: DOM0.
[root@node8 ~]#

 

Upload the Oracle Exadata Database Machine Deployment Assistant configuration files to the database server, together with all software images, and run the One command procedure.

List of all Steps

[root@exatestdbadm01 linux-x64]# ./install.sh -cf TVD-exatest.xml -l
Initializing

1. Validate Configuration File
2. Update Nodes for Eighth Rack
3. Create Virtual Machine
4. Create Users
5. Setup Cell Connectivity
6. Calibrate Cells
7. Create Cell Disks
8. Create Grid Disks
9. Install Cluster Software
10. Initialize Cluster Software
11. Install Database Software
12. Relink Database with RDS
13. Create ASM Diskgroups
14. Create Databases
15. Apply Security Fixes
16. Install Exachk
17. Create Installation Summary
18. Resecure Machine
[root@exatestdbadm01 linux-x64]#

 

Run Step One to validate the setup

This example includes the creation of three different Clusters.

[root@exatestdbadm01 linux-x64]# ./install.sh -cf TVD-exatest.xml -s 1
Initializing
Executing Validate Configuration File
Validating cluster: Cluster-EFU
Locating machines...
Verifying operating systems...
Validating cluster networks...
Validating network connectivity...
Validating private ips on virtual cluster
Validating NTP setup...
Validating physical disks on storage cells...
Validating users...
Validating cluster: Cluster-PR1
Locating machines...
Verifying operating systems...
Validating cluster networks...
Validating network connectivity...
Validating private ips on virtual cluster
Validating NTP setup...
Validating physical disks on storage cells...
Validating users...
Validating cluster: Cluster-VAL
Locating machines...
Verifying operating systems...
Validating cluster networks...
Validating network connectivity...
Validating private ips on virtual cluster
Validating NTP setup...
Validating physical disks on storage cells...
Validating users...
Validating platinum...
Validating switches...
Checking disk reclaim status...
Checking Disk Tests Status....
Completed validation...

SUCCESS: Ip address: 10.x8.xx.40 is configured correctly
SUCCESS: Ip address: 10.x9.xx.55 is configured correctly
SUCCESS: Ip address: 10.x8.xx.41 is configured correctly
SUCCESS: Ip address: 10.x9.xx.56 is configured correctly
SUCCESS: Ip address: 10.x8.xx.45 is configured correctly
SUCCESS: Ip address: 10.x8.xx.46 is configured correctly
SUCCESS: Ip address: 10.x8.xx.44 is configured correctly
SUCCESS: Ip address: 10.x8.xx.43 is configured correctly
SUCCESS: Ip address: 10.x8.xx.42 is configured correctly
SUCCESS: 10.x8.xx.40 configured correctly on exatestceladm01.my.domain.com
SUCCESS: 10.x9.xx.55 configured correctly on exatestceladm01.my.domain.com
SUCCESS: 10.x8.xx.41 configured correctly on exatestceladm01.my.domain.com
SUCCESS: 10.x9.xx.56 configured correctly on exatestceladm01.my.domain.com
SUCCESS: 10.x8.xx.45 configured correctly on exatestceladm01.my.domain.com
SUCCESS: 10.x8.xx.46 configured correctly on exatestceladm01.my.domain.com
SUCCESS: 10.x8.xx.44 configured correctly on exatestceladm01.my.domain.com
SUCCESS: 10.x8.xx.43 configured correctly on exatestceladm01.my.domain.com
SUCCESS: 10.x8.xx.42 configured correctly on exatestceladm01.my.domain.com
SUCCESS: 10.x8.xx.40 configured correctly on exatestceladm02.my.domain.com
SUCCESS: 10.x9.xx.55 configured correctly on exatestceladm02.my.domain.com
SUCCESS: 10.x8.xx.41 configured correctly on exatestceladm02.my.domain.com
SUCCESS: 10.x9.xx.56 configured correctly on exatestceladm02.my.domain.com
SUCCESS: 10.x8.xx.45 configured correctly on exatestceladm02.my.domain.com
SUCCESS: 10.x8.xx.46 configured correctly on exatestceladm02.my.domain.com
SUCCESS: 10.x8.xx.44 configured correctly on exatestceladm02.my.domain.com
SUCCESS: 10.x8.xx.43 configured correctly on exatestceladm02.my.domain.com
SUCCESS: 10.x8.xx.42 configured correctly on exatestceladm02.my.domain.com
SUCCESS: 10.x8.xx.40 configured correctly on exatestceladm03.my.domain.com
SUCCESS: 10.x9.xx.55 configured correctly on exatestceladm03.my.domain.com
SUCCESS: 10.x8.xx.41 configured correctly on exatestceladm03.my.domain.com
SUCCESS: 10.x9.xx.56 configured correctly on exatestceladm03.my.domain.com
SUCCESS: 10.x8.xx.45 configured correctly on exatestceladm03.my.domain.com
SUCCESS: 10.x8.xx.46 configured correctly on exatestceladm03.my.domain.com
SUCCESS: 10.x8.xx.44 configured correctly on exatestceladm03.my.domain.com
SUCCESS: 10.x8.xx.43 configured correctly on exatestceladm03.my.domain.com
SUCCESS: 10.x8.xx.42 configured correctly on exatestceladm03.my.domain.com
SUCCESS: Ip address: 10.x8.xx.47 is configured correctly
SUCCESS: Ip address: 10.x9.xx.57 is configured correctly
SUCCESS: Ip address: 10.x8.xx.48 is configured correctly
SUCCESS: Ip address: 10.x9.xx.58 is configured correctly
SUCCESS: Ip address: 10.x8.xx.52 is configured correctly
SUCCESS: Ip address: 10.x8.xx.51 is configured correctly
SUCCESS: Ip address: 10.x8.xx.53 is configured correctly
SUCCESS: Ip address: 10.x8.xx.50 is configured correctly
SUCCESS: Ip address: 10.x8.xx.49 is configured correctly
SUCCESS: 10.x8.xx.47 configured correctly on exatestceladm01.my.domain.com
SUCCESS: 10.x9.xx.57 configured correctly on exatestceladm01.my.domain.com
SUCCESS: 10.x8.xx.48 configured correctly on exatestceladm01.my.domain.com
SUCCESS: 10.x9.xx.58 configured correctly on exatestceladm01.my.domain.com
SUCCESS: 10.x8.xx.52 configured correctly on exatestceladm01.my.domain.com
SUCCESS: 10.x8.xx.51 configured correctly on exatestceladm01.my.domain.com
SUCCESS: 10.x8.xx.53 configured correctly on exatestceladm01.my.domain.com
SUCCESS: 10.x8.xx.50 configured correctly on exatestceladm01.my.domain.com
SUCCESS: 10.x8.xx.49 configured correctly on exatestceladm01.my.domain.com
SUCCESS: 10.x8.xx.47 configured correctly on exatestceladm02.my.domain.com
SUCCESS: 10.x9.xx.57 configured correctly on exatestceladm02.my.domain.com
SUCCESS: 10.x8.xx.48 configured correctly on exatestceladm02.my.domain.com
SUCCESS: 10.x9.xx.58 configured correctly on exatestceladm02.my.domain.com
SUCCESS: 10.x8.xx.52 configured correctly on exatestceladm02.my.domain.com
SUCCESS: 10.x8.xx.51 configured correctly on exatestceladm02.my.domain.com
SUCCESS: 10.x8.xx.53 configured correctly on exatestceladm02.my.domain.com
SUCCESS: 10.x8.xx.50 configured correctly on exatestceladm02.my.domain.com
SUCCESS: 10.x8.xx.49 configured correctly on exatestceladm02.my.domain.com
SUCCESS: 10.x8.xx.47 configured correctly on exatestceladm03.my.domain.com
SUCCESS: 10.x9.xx.57 configured correctly on exatestceladm03.my.domain.com
SUCCESS: 10.x8.xx.48 configured correctly on exatestceladm03.my.domain.com
SUCCESS: 10.x9.xx.58 configured correctly on exatestceladm03.my.domain.com
SUCCESS: 10.x8.xx.52 configured correctly on exatestceladm03.my.domain.com
SUCCESS: 10.x8.xx.51 configured correctly on exatestceladm03.my.domain.com
SUCCESS: 10.x8.xx.53 configured correctly on exatestceladm03.my.domain.com
SUCCESS: 10.x8.xx.50 configured correctly on exatestceladm03.my.domain.com
SUCCESS: 10.x8.xx.49 configured correctly on exatestceladm03.my.domain.com
SUCCESS: Ip address: 10.x8.xx.54 is configured correctly
SUCCESS: Ip address: 10.x9.xx.59 is configured correctly
SUCCESS: Ip address: 10.x8.xx.55 is configured correctly
SUCCESS: Ip address: 10.x9.xx.60 is configured correctly
SUCCESS: Ip address: 10.x8.xx.58 is configured correctly
SUCCESS: Ip address: 10.x8.xx.60 is configured correctly
SUCCESS: Ip address: 10.x8.xx.59 is configured correctly
SUCCESS: Ip address: 10.x8.xx.57 is configured correctly
SUCCESS: Ip address: 10.x8.xx.56 is configured correctly
SUCCESS: 10.x8.xx.54 configured correctly on exatestceladm01.my.domain.com
SUCCESS: 10.x9.xx.59 configured correctly on exatestceladm01.my.domain.com
SUCCESS: 10.x8.xx.55 configured correctly on exatestceladm01.my.domain.com
SUCCESS: 10.x9.xx.60 configured correctly on exatestceladm01.my.domain.com
SUCCESS: 10.x8.xx.58 configured correctly on exatestceladm01.my.domain.com
SUCCESS: 10.x8.xx.60 configured correctly on exatestceladm01.my.domain.com
SUCCESS: 10.x8.xx.59 configured correctly on exatestceladm01.my.domain.com
SUCCESS: 10.x8.xx.57 configured correctly on exatestceladm01.my.domain.com
SUCCESS: 10.x8.xx.56 configured correctly on exatestceladm01.my.domain.com
SUCCESS: 10.x8.xx.54 configured correctly on exatestceladm02.my.domain.com
SUCCESS: 10.x9.xx.59 configured correctly on exatestceladm02.my.domain.com
SUCCESS: 10.x8.xx.55 configured correctly on exatestceladm02.my.domain.com
SUCCESS: 10.x9.xx.60 configured correctly on exatestceladm02.my.domain.com
SUCCESS: 10.x8.xx.58 configured correctly on exatestceladm02.my.domain.com
SUCCESS: 10.x8.xx.60 configured correctly on exatestceladm02.my.domain.com
SUCCESS: 10.x8.xx.59 configured correctly on exatestceladm02.my.domain.com
SUCCESS: 10.x8.xx.57 configured correctly on exatestceladm02.my.domain.com
SUCCESS: 10.x8.xx.56 configured correctly on exatestceladm02.my.domain.com
SUCCESS: 10.x8.xx.54 configured correctly on exatestceladm03.my.domain.com
SUCCESS: 10.x9.xx.59 configured correctly on exatestceladm03.my.domain.com
SUCCESS: 10.x8.xx.55 configured correctly on exatestceladm03.my.domain.com
SUCCESS: 10.x9.xx.60 configured correctly on exatestceladm03.my.domain.com
SUCCESS: 10.x8.xx.58 configured correctly on exatestceladm03.my.domain.com
SUCCESS: 10.x8.xx.60 configured correctly on exatestceladm03.my.domain.com
SUCCESS: 10.x8.xx.59 configured correctly on exatestceladm03.my.domain.com
SUCCESS: 10.x8.xx.57 configured correctly on exatestceladm03.my.domain.com
SUCCESS: 10.x8.xx.56 configured correctly on exatestceladm03.my.domain.com
SUCCESS: Validated NTP server 10.x3.xx.xx0
SUCCESS: Validated NTP server 10.x3.xx.xx1
SUCCESS: Required file /EXAVMIMAGES/onecommand/linux-x64/WorkDir/p28514222_122118_Linux-x86-64.zip exists...
SUCCESS: Required file /EXAVMIMAGES/onecommand/linux-x64/WorkDir/p28762988_12201181016GIOCT2018RU_Linux-x86-64.zip exists...
SUCCESS: Required file /EXAVMIMAGES/onecommand/linux-x64/WorkDir/p28762989_12201181016DBOCT2018RU_Linux-x86-64.zip exists...
SUCCESS: Required file config/exachk.zip exists...
SUCCESS: Found Operating system LinuxPhysical and configuration file expects LinuxPhysical on machine exatestceladm03.my.domain.com, machine type: storage
SUCCESS: Found Operating system LinuxPhysical and configuration file expects LinuxPhysical on machine exatestceladm02.my.domain.com, machine type: storage
SUCCESS: Found Operating system LinuxPhysical and configuration file expects LinuxPhysical on machine exatestceladm01.my.domain.com, machine type: storage
SUCCESS: Expected machine exatestdbadm01.my.domain.com to have OS Type of Linux Dom0, and found OsType LinuxDom0
SUCCESS: Expected machine exatestdbadm02.my.domain.com to have OS Type of Linux Dom0, and found OsType LinuxDom0
SUCCESS: NTP servers on machine exatestceladm02.my.domain.com verified successfully
SUCCESS: NTP servers on machine exatestceladm01.my.domain.com verified successfully
SUCCESS: NTP servers on machine exatestceladm03.my.domain.com verified successfully
SUCCESS: NTP servers on machine exatestdbadm01.my.domain.com verified successfully
SUCCESS: NTP servers on machine exatestdbadm02.my.domain.com verified successfully
SUCCESS: Sufficient memory for all the guests on database node exatestdbadm02.my.domain.com
SUCCESS: Sufficient memory for all the guests on database node exatestdbadm01.my.domain.com
SUCCESS: Expected machine exatestdbadm02.my.domain.com to have OS Type of Linux Dom0, and found OsType LinuxDom0
SUCCESS: Found Operating system LinuxPhysical and configuration file expects LinuxPhysical on machine exatestceladm01.my.domain.com, machine type: storage
SUCCESS: Found Operating system LinuxPhysical and configuration file expects LinuxPhysical on machine exatestceladm02.my.domain.com, machine type: storage
SUCCESS: Expected machine exatestdbadm01.my.domain.com to have OS Type of Linux Dom0, and found OsType LinuxDom0
SUCCESS: Found Operating system LinuxPhysical and configuration file expects LinuxPhysical on machine exatestceladm03.my.domain.com, machine type: storage
SUCCESS: NTP servers on machine exatestceladm03.my.domain.com verified successfully
SUCCESS: NTP servers on machine exatestceladm01.my.domain.com verified successfully
SUCCESS: NTP servers on machine exatestceladm02.my.domain.com verified successfully
SUCCESS: NTP servers on machine exatestdbadm02.my.domain.com verified successfully
SUCCESS: NTP servers on machine exatestdbadm01.my.domain.com verified successfully
SUCCESS: Sufficient memory for all the guests on database node exatestdbadm02.my.domain.com
SUCCESS: Sufficient memory for all the guests on database node exatestdbadm01.my.domain.com
SUCCESS: Found Operating system LinuxPhysical and configuration file expects LinuxPhysical on machine exatestceladm03.my.domain.com, machine type: storage
SUCCESS: Found Operating system LinuxPhysical and configuration file expects LinuxPhysical on machine exatestceladm02.my.domain.com, machine type: storage
SUCCESS: Found Operating system LinuxPhysical and configuration file expects LinuxPhysical on machine exatestceladm01.my.domain.com, machine type: storage
SUCCESS: Expected machine exatestdbadm02.my.domain.com to have OS Type of Linux Dom0, and found OsType LinuxDom0
SUCCESS: Expected machine exatestdbadm01.my.domain.com to have OS Type of Linux Dom0, and found OsType LinuxDom0
SUCCESS: NTP servers on machine exatestceladm03.my.domain.com verified successfully
SUCCESS: NTP servers on machine exatestceladm02.my.domain.com verified successfully
SUCCESS: NTP servers on machine exatestceladm01.my.domain.com verified successfully
SUCCESS: NTP servers on machine exatestdbadm01.my.domain.com verified successfully
SUCCESS: NTP servers on machine exatestdbadm02.my.domain.com verified successfully
SUCCESS: Sufficient memory for all the guests on database node exatestdbadm02.my.domain.com
SUCCESS: Sufficient memory for all the guests on database node exatestdbadm01.my.domain.com
SUCCESS: Switch IP 10.x9.xx.51 resolves successfully to host exatest-iba0.my.domain.com on node exatestceladm03.my.domain.com
SUCCESS:
SUCCESS: Switch IP 10.x9.xx.51 resolves successfully to host exatest-iba0.my.domain.com on node exatestceladm02.my.domain.com
SUCCESS: Switch IP 10.x9.xx.52 resolves successfully to host exatest-ibb0.my.domain.com on node exatestceladm03.my.domain.com
SUCCESS:
SUCCESS:
SUCCESS:
SUCCESS: Switch IP 10.x9.xx.52 resolves successfully to host exatest-ibb0.my.domain.com on node exatestceladm02.my.domain.com
SUCCESS:
SUCCESS: Switch IP 10.x9.xx.51 resolves successfully to host exatest-iba0.my.domain.com on node exatestceladm01.my.domain.com
SUCCESS: Switch IP 10.x9.xx.52 resolves successfully to host exatest-ibb0.my.domain.com on node exatestceladm01.my.domain.com
SUCCESS:
SUCCESS: X7 compute node exatestdbadm01.my.domain.com has updated Broadcom firmware
SUCCESS: X7 compute node exatestdbadm02.my.domain.com has updated Broadcom firmware
SUCCESS: Disk Tests are not running/active on any of the Storage Servers.
SUCCESS: Cluster Version 12.2.0.1.181016 is compatible with OL7 on exatestdbadm01
SUCCESS: Cluster Version 12.2.0.1.181016 is compatible with OL7 on exatestdbadm02
SUCCESS: Cluster Version 12.2.0.1.181016 is compatible with OL7 on exatestdbadm01
SUCCESS: Cluster Version 12.2.0.1.181016 is compatible with OL7 on exatestdbadm02
SUCCESS: Cluster Version 12.2.0.1.181016 is compatible with OL7 on exatestdbadm01
SUCCESS: Cluster Version 12.2.0.1.181016 is compatible with OL7 on exatestdbadm02
SUCCESS: Disk size 10000GB on cell exatestceladm01.my.domain.com matches the value specified in the OEDA configuration file
SUCCESS: Disk size 10000GB on cell exatestceladm02.my.domain.com matches the value specified in the OEDA configuration file
SUCCESS: Disk size 10000GB on cell exatestceladm03.my.domain.com matches the value specified in the OEDA configuration file
SUCCESS: Disk size 10000GB on cell exatestceladm04.my.domain.com matches the value specified in the OEDA configuration file
SUCCESS: Disk size 10000GB on cell exatestceladm05.my.domain.com matches the value specified in the OEDA configuration file
SUCCESS: Disk size 10000GB on cell exatestceladm06.my.domain.com matches the value specified in the OEDA configuration file
Successfully completed execution of step Validate Configuration File [elapsed Time [Elapsed = 250301 mS [4.0 minutes] Thu Mar 07 12:35:31 CET 2019]]
[root@exatestdbadm01 linux-x64]#

 

 

Execution of all remaining steps

Than, because we felt confident, we decide to invoke all remaining steps together:

root@exatestdbadm01 linux-x64]# ./install.sh -cf TVD-exatest.xml -r 1-18
...
..

 

The final result is the Exadata Machine installed with six Oracle VMs, and three Grid Infrastructure clusters each one running a test RAC database.

 

 

My OOW18 Summary

 

For those who are interested here my major takeaways from the OOW18

 

As we all know, since few years the HOTTEST topic advertized at the OOW is “Cloud Computing”, but this time Oracle Cloud was no longer alone!

In fact the focus was divided between the new Oracle OCI Cloud, also named by Larry as Second Generation of Cloud and the Autonomous Database.

 

OCI Second Gen of Cloud

Here a summary of the major advantages compared to the previous version:

– Security, guaranteed by robots which scan the network for any malicious attack.  

– The cutting edge virtual network, which brings up to 50GB speed and extreme flexibility.

– Bare Metal Infrastructure based on Exadata Machines.

– Aggressive pricing, compared to the competitors.

 

Autonomous Database.

The Autonomous Database option is now available for OLTP and DWH databases and includes new capabilities like automatic index creation and column stored table conversion. In version 19 it will manage online memory increase and additional tuning options.

As announces during Larry’s keynote, the  Autonomous database will be also available with the Cloud @ Customer option (on Exadata only), ant it will no longer require human labor (DBA and Sys Admin intervention), because Self Provisioning, Self Driving, Self Tuning and Self Repairing.

For non-technical people it looks magic, but it is few steps from what we already use in a standard Oracle 12c Database. In fact Autonomous Database leverages a bunch of database advisors and tuning options, now orchestrated by an Artificial Intelligence and Machine Learning software, in order to provide data-driven predictions and decisions.

Over the next few years, Autonomous Database will be enriched with several new options, improving the quality of live of many DBAs, which will be relieved of the majority of the tedious and recurring tasks, leaving the most added value tasks under their own responsibility.

Last but not least, the Autonomous Database runs in a very high end configurations (Oracle guarantees 99,995% of availability), which is quite expensive to acquire due to the list of mandatory requirements: Exadata, RAC, Active DG, Multitenant, Tuning Pack, Diagnostic Pack etc..

 

Exadata Machine

Several interesting features are coming next year with the introduction of the INTEL Optane DC Persistent Memory for even faster OLTP.

This new type of memory will be installed on the Storage Cell and used as accelerator in front of Flash memory.

The database node will  access to the Persistent Memory via RDMA with a gain up to 20 x faster access latency.

Oracle is developing the more and more Remote Direct Memory Access (RDMA) instructions for Cache Fusion and Storage Cell operations in order to offload the database nodes and increase the overall performance.

Stay tuned on Exadata Machine because the next generation will also include BIG architectural change…

 

Oracle Virtual Machine (OVM)

One curiosity directly collected at Linux Virtualization booth is that even though the next generation of hypervisor will be based on KVM, Oracle will keep calling it OVM and of course the current OVM product based on XEN (OVS, OVM) will still be in use by many companies.

How possibly the customers can get confused ?!?

 

With this I finished, although there would be much more to write.

 


 

Exadata How Safely Erase All Data

When the time arrives to decommission an environment with sesitive data, we are frequently confronted to the problem how to certify to our customer or management the erase of all data and logs.

On Exadata Machine starting from the software release 12.2.1.1.0, this problem has been elegantly solved by Oracle introducing a new utility called Secure Eraser; which securely erases data on hard drives, flash devices, internal USBs, and resets ILOM to factory default.

 

In earlier software versions, the Exadata Storage Software includes CellCli commands to securely erase the user data:

CellCLI> DROP GRIDDISK ALL FLASHDISK PREFIX=DATA, ERASE=7pass
CellCLI> DROP GRIDDISK ALL PREFIX=DATA, ERASE=3pass

and

CellCLI> DROP CELLDISK ALL FLASHDISK ERASE=7pass 
CellCLI> DROP CELL ERASE=3pass

Unfortunatly those commands only cover the user data stored on the Storage Cell, and none of them produces an official certificate with the summary of the actions taken to guarantee the wipe of the data. While all this is done by Secure Eraser on all Compute and Storage nodes, sanitizing on all type of devices: user data, OS logs and network configurations.

 

Depending from the Exadata model, a subset of all of available options to execute Secure Eraser is possible:

  • Automatic Secure Eraser Ethrough PXE Boot
  • Interactive Secure Eraser through PXE Boot
  • Interactive Secure Eraser through Network Boot
  • Interactive Secure Eraser through External USB

 


 

Recently I used Secure Eraser through External USB on one Exadata X7-2 Machine and here are reported the different steps.

 

Copy the Secure Eraser Diagnostic image from MOS 2180963.1 to a USB stick.

 # dd if=image_diagnostics_18.1.4.0.0_LINUX.X64_180125.3-1.x86_64.usb of=/dev/sdb

 

Boot the server using the USB device with the Secure Eraser Diagnostic image

Exa_BootList.jpg

 

After login, start the Secure Erase process

/usr/sbin/secureeraser --erase --all --flash_erasure_method=7pass --hdd_erasure_method=3pass --technician=Emiliano_Fusaglia --witness=Mario_Bros --output=/mnt/iso

 

 

At the end of the erase process a Data Erasure Certificate similar to the one on the example below will be available in TXT, HTML and PDF format.

Exa_SecureErase_Report


 

 

 

Exadata Storage Snapshots

This post describes how to implement Oracle Database Snapshot Technology on Exadata Machine.

Because Exadata Storage Cell Smart Features, Storage Indexes, IORM and Network Resource Manager work at level of ASM Volume Manager only, (and they don’t work on top of ACFS Cluster File System), the implementation of the snapshot technology is different compared to any other non-Exadata environment.

At this purpuse Oracle has developed a new type of ASM Disk Group called SPARSE Disk Group. It uses ASM SPARSE Grid Disk based on Thin Provisioning to save the database snapshot copies and the associated metadata, and it supports non-CDB and PDB snapshot copy.

The implementation requires the following minimal software versions :

  • Exadata Storage Software version 12.1.2.1.0.
  • Oracle Database version 12.1.0.2 with bundle patch 5.
One major restriction applies to Exadata Storage Sanpshot compared to ACFS;
the source database must be a shared copy open on read only and called Test Master. The Test Master Database can not be modified or deleted as long the latest child snapshot is in use.
This restriction exists because Exadata Snapshot technology uses “allocate on first write”, and not “copy on write” (like for ACFS), and the snapshot is per-database-datafile.
When a child snapshot issue a write, the write goes to a private copy of that block inside the snapshot, preserving the original block value which can be accessed by other child snapshots of the same Test Master.

How to Implement Exadata Storage Snapshots in a PDB Environment

Check the celldisks for available free space to allocate to a new SPARSE Disk Group

[root@strgceladm01 ~]# cellcli -e list celldisk attributes name,freespace
 CD_00_strgceladm01 853.34375G
 CD_01_strgceladm01 853.34375G
 CD_02_strgceladm01 853.34375G
 CD_03_strgceladm01 853.34375G
 CD_04_strgceladm01 853.34375G
 CD_05_strgceladm01 853.34375G
 CD_06_strgceladm01 853.34375G
 CD_07_strgceladm01 853.34375G
 CD_08_strgceladm01 853.34375G
 CD_09_strgceladm01 853.34375G
 CD_10_strgceladm01 853.34375G
 CD_11_strgceladm01 853.34375G
 FD_00_strgceladm01 0
 FD_01_strgceladm01 0
 FD_02_strgceladm01 0
 FD_03_strgceladm01 0
[root@strgceladm01 ~]#


[root@strgceladm02 ~]# cellcli -e list celldisk attributes name,freespace
 CD_00_strgceladm02 853.34375G
 CD_01_strgceladm02 853.34375G
 CD_02_strgceladm02 853.34375G
 CD_03_strgceladm02 853.34375G
 CD_04_strgceladm02 853.34375G
 CD_05_strgceladm02 853.34375G
 CD_06_strgceladm02 853.34375G
 CD_07_strgceladm02 853.34375G
 CD_08_strgceladm02 853.34375G
 CD_09_strgceladm02 853.34375G
 CD_10_strgceladm02 853.34375G
 CD_11_strgceladm02 853.34375G
 FD_00_strgceladm02 0
 FD_01_strgceladm02 0
 FD_02_strgceladm02 0
 FD_03_strgceladm02 0
[root@strgceladm02 ~]#


[root@strgceladm03 ~]# cellcli -e list celldisk attributes name,freespace
 CD_00_strgceladm03 853.34375G
 CD_01_strgceladm03 853.34375G
 CD_02_strgceladm03 853.34375G
 CD_03_strgceladm03 853.34375G
 CD_04_strgceladm03 853.34375G
 CD_05_strgceladm03 853.34375G
 CD_06_strgceladm03 853.34375G
 CD_07_strgceladm03 853.34375G
 CD_08_strgceladm03 853.34375G
 CD_09_strgceladm03 853.34375G
 CD_10_strgceladm03 853.34375G
 CD_11_strgceladm03 853.34375G
 FD_00_strgceladm03 0
 FD_01_strgceladm03 0
 FD_02_strgceladm03 0
 FD_03_strgceladm03 0
[root@strgceladm03 ~]#

For each Storage Cell Create a SPARSE Grid Disks as described below

[root@strgceladm01 ~]# cellcli -e CREATE GRIDDISK ALL PREFIX=SPARSE, sparse=true, SIZE=853.34375G
Cell disks were skipped because they had no freespace for grid disks: FD_00_strgceladm01, FD_01_strgceladm01, FD_02_strgceladm01, FD_03_strgceladm01.
GridDisk SPARSE_CD_00_strgceladm01 successfully created
GridDisk SPARSE_CD_01_strgceladm01 successfully created
GridDisk SPARSE_CD_02_strgceladm01 successfully created
GridDisk SPARSE_CD_03_strgceladm01 successfully created
GridDisk SPARSE_CD_04_strgceladm01 successfully created
GridDisk SPARSE_CD_05_strgceladm01 successfully created
GridDisk SPARSE_CD_06_strgceladm01 successfully created
GridDisk SPARSE_CD_07_strgceladm01 successfully created
GridDisk SPARSE_CD_08_strgceladm01 successfully created
GridDisk SPARSE_CD_09_strgceladm01 successfully created
GridDisk SPARSE_CD_10_strgceladm01 successfully created
GridDisk SPARSE_CD_11_strgceladm01 successfully created
[root@strgceladm01 ~]#

For each Storage Cell List all Grid Disks

[root@strgceladm01 ~]# cellcli -e list griddisk attributes name,size
 DATAC1_CD_00_strgceladm01 6.294586181640625T
 DATAC1_CD_01_strgceladm01 6.294586181640625T
 DATAC1_CD_02_strgceladm01 6.294586181640625T
 DATAC1_CD_03_strgceladm01 6.294586181640625T
 DATAC1_CD_04_strgceladm01 6.294586181640625T
 DATAC1_CD_05_strgceladm01 6.294586181640625T
 DATAC1_CD_06_strgceladm01 6.294586181640625T
 DATAC1_CD_07_strgceladm01 6.294586181640625T
 DATAC1_CD_08_strgceladm01 6.294586181640625T
 DATAC1_CD_09_strgceladm01 6.294586181640625T
 DATAC1_CD_10_strgceladm01 6.294586181640625T
 DATAC1_CD_11_strgceladm01 6.294586181640625T
 FGRID_FD_00_strgceladm01 2.0717315673828125T
 FGRID_FD_01_strgceladm01 2.0717315673828125T
 FGRID_FD_02_strgceladm01 2.0717315673828125T
 FGRID_FD_03_strgceladm01 2.0717315673828125T
 RECOC1_CD_00_strgceladm01 1.78143310546875T
 RECOC1_CD_01_strgceladm01 1.78143310546875T
 RECOC1_CD_02_strgceladm01 1.78143310546875T
 RECOC1_CD_03_strgceladm01 1.78143310546875T
 RECOC1_CD_04_strgceladm01 1.78143310546875T
 RECOC1_CD_05_strgceladm01 1.78143310546875T
 RECOC1_CD_06_strgceladm01 1.78143310546875T
 RECOC1_CD_07_strgceladm01 1.78143310546875T
 RECOC1_CD_08_strgceladm01 1.78143310546875T
 RECOC1_CD_09_strgceladm01 1.78143310546875T
 RECOC1_CD_10_strgceladm01 1.78143310546875T
 RECOC1_CD_11_strgceladm01 1.78143310546875T
 SPARSE_CD_00_strgceladm01 853.34375G
 SPARSE_CD_01_strgceladm01 853.34375G
 SPARSE_CD_02_strgceladm01 853.34375G
 SPARSE_CD_03_strgceladm01 853.34375G
 SPARSE_CD_04_strgceladm01 853.34375G
 SPARSE_CD_05_strgceladm01 853.34375G
 SPARSE_CD_06_strgceladm01 853.34375G
 SPARSE_CD_07_strgceladm01 853.34375G
 SPARSE_CD_08_strgceladm01 853.34375G
 SPARSE_CD_09_strgceladm01 853.34375G
 SPARSE_CD_10_strgceladm01 853.34375G
 SPARSE_CD_11_strgceladm01 853.34375G
[root@strgceladm01 ~]#

From ASM Instance Create a SPARSE Disk Group

SQL> CREATE DISKGROUP SPARSEC1 EXTERNAL REDUNDANCY DISK 'o/*/SPARSE_CD_*'
ATTRIBUTE
'compatible.asm' = '12.2.0.1',
'compatible.rdbms' = '12.2.0.1',
'cell.smart_scan_capable'='TRUE',
'cell.sparse_dg' = 'allsparse',
'AU_SIZE' = '4M';

Diskgroup created.

Set the following ASM attributes on the Disk Group hosting the Test Master Database

ALTER DISKGROUP DATAC1 SET ATTRIBUTE 'access_control.enabled' = 'true';

Grant access to the OS RDBMS user used to access to the Disk Group

ALTER DISKGROUP DATAC1 ADD USER 'oracle';

From an ASM Instance Set ownership permissions for every file that belongs solely to the PDB being snapped cloned as per example below

alter diskgroup DATAC1 set ownership owner='oracle' for file '+DATAC1/CDBT/<xxxxxxxxxxxxxxxxxxx>/DATAFILE/system.xxx.xxxxxxx';
alter diskgroup DATAC1 set ownership owner='oracle' for file '+DATAC1/CDBT/<xxxxxxxxxxxxxxxxxxx>/DATAFILE/sysaux.xxx.xxxxxxx';
alter diskgroup DATAC1 set ownership owner='oracle' for file '+DATAC1/CDBT/<xxxxxxxxxxxxxxxxxxx>/DATAFILE/users.xxx.xxxxxxx';
...
..

Restart the Master Test PDB in Read Only

alter pluggable database PDBTESTMASTER close immediate instances=all;
alter pluggable database PDBTESTMASTER open read only;

Create the first PDB Snapshot Copy on Exadata SPARSE Disk Group

Create pluggable database PDBDEV01 from PDBTESTMASTER tempfile reuse create_file_dest='+SPARSEC1' snapshot copy;

Feedback of the Exadata Storage Snapshots

The ability to create storage efficient database copies in a few seconds, independently from the size of the Test Master is very useful for today IT departments; but such extreme velocity and flexibility is not entirely free. In fact performance tests on a I/O bound workload have highlighted important performance degradation. This reminds us that as defined by Oracle Corporation, the Snapshot Technology, included on Exadata Machine remains a non-production option.

Oracle DB stored on ASM vs ACFS

Nowadays a new Oracle database environment with Grid Infrastructure has three main storage options:

  1. Third party clustered file system
  2. ASM Disk Groups
  3. ACFS File System

While the first option was not in scope, this blog compares the result of the tests between ASM and ACFS, highlighting when to use one or the other to store 12c NON-CDB or CDB Databases.

The tests conducted on different environments using Oracle version 12.1.0.2 July PSU have shown controversial results compared to what Oracle  is promoting for the Oracle Database Appliance (ODA) in the following paper: “Frequently Asked Questions Storing Database Files in ACFS on Oracle Database Appliance

 

Outcome of the tests

ASM remains the preferred option to achieve the best I/O performance, while ACFS introduces interesting features like DB snapshot to quickly and space efficiently provision new databases.

The performance gap between the two solutions is not negligible as reported below by the  AWR – TOP Timed Events sections of two PDBs, sharing the same infrastructure, executing the same workload but respectively using ASM and ACFS storage:

  • PDBASM: Pluggable Database stored on  ASM Disk Group
  • PDBACFS:Pluggable Database stored on ACFS File System

 

 

PDBASM AWR – TOP Timed Events and Other Stats

topevents_asm

fg_asm

 

 

PDBACFS AWR – TOP Timed Events and Other Stats

TopEvents_ACFS.png

fg_acfs

 

Due to the different characteristics and results when ASM or ACFS is in use, it is not possible to give a generic recommendation. But case by case the choise should be driven by business needs like maximum performance versus fast and efficient database clone.

 

 

 

 

ODA X5-2 how to cap the number of active CPU Cores

I recently had to cap the number of active CPUs on a bare metal ODA X5-2, and I noticed that the procedure is slightly different from what I used in the past (link to initial post).

 

Perform the following steps to generate the Core Key:

  • Login to My Oracle Support (MOS) and click the submenu Systems.
  • Select the serial number of the appliance and click on “Core Configuration”in the Asset Details Screen
  • Select Manage Key
  • From the Combo list select the number of cores to activate  and click Generate Key to generate the key.
  • Click Copy Key to Clipboard to copy the key to the clipboard.
  • Paste the key into an empty text file and save the file to a location on the Oracle Database Appliance.

 

ODA X5-2 initial number of CPU Cores

[root@odax5-2n0 ~]# cat /proc/cpuinfo | grep -i processor
processor : 0
processor : 1
processor : 2
processor : 3
...
...
..
.
processor : 70
processor : 71

[root@odax5-2n0 ~]# cat /proc/cpuinfo | grep -i processor |wc -l
72
[root@odax5-2n0 ~]#

 

Checks before enforcing the CPU restriction:

[root@odax5-2n0 ~]# oakcli show server

Power State : On
 Open Problems : 0
 Model : ODA X5-2
 Type : Rack Mount
 Part Number : xxxxxxxxxxx
 Serial Number : nnnnXXXXnnX <<<<<<<<<<<< This serial MUST match on BOTH of the ODA servers
 Primary OS : Not Available
 ILOM Address : 192.168.21.35
 ILOM MAC Address : xx:xx:xx:xx:xx:xx
 Description : Oracle Database Appliance X5-2 nnnnXXXXnnX
 Locator Light : Off
 Actual Power Consumption : 345 watts
 Ambient Temperature : 21.250 degree C
 Open Problems Report : System is healthy

[root@odax5-2n0 ~]#


[root@odax5-2n1 /]# oakcli show server

Power State : On
 Open Problems : 0
 Model : ODA X5-2
 Type : Rack Mount
 Part Number : xxxxxxxxxxx
 Serial Number : nnnnXXXXnnX <<<<<<<<<<<< This serial MUST match on BOTH of the ODA servers 
 Primary OS : Not Available
 ILOM Address : 192.168.21.36
 ILOM MAC Address : xx:xx:xx:xx:xx:xx
 Description : Oracle Database Appliance X5-2 nnnnXXXXnnX
 Locator Light : Off
 Actual Power Consumption : 342 watts
 Ambient Temperature : 21.750 degree C
 Open Problems Report : System is healthy

[root@odax5-2n1 /]#

[root@odax5-2n0 ~]# oakcli show env_hw
BM ODA X5-2
Public interface : COPPER
[root@odax5-2n0 ~]#


[root@odax5-2n1 /]# oakcli show env_hw
BM ODA X5-2
Public interface : COPPER
[root@odax5-2n1 /]#


[root@odax5-2n0 ~]# ipmitool -I open sunoem getval /X/system_identifier
Target Value: Oracle Database Appliance X5-2 nnnnXXXXnnX
[root@odax5-2n0 ~]# fwupdate list sp_bios
==================================================
SP + BIOS
==================================================
ID Product Name ILOM Version BIOS/OBP Version XML Support
---------------------------------------------------------------------------------------------------------------
sp_bios ORACLE SERVER X5-2 v3.2.4.52 r101649 30050100 N/A
[root@odax5-2n0 ~]#

[root@odax5-2n1 /]# ipmitool -I open sunoem getval /X/system_identifier
Target Value: Oracle Database Appliance X5-2 nnnnXXXXnnX
[root@odax5-2n1 /]# fwupdate list sp_bios
==================================================
SP + BIOS
==================================================
ID Product Name ILOM Version BIOS/OBP Version XML Support
---------------------------------------------------------------------------------------------------------------
sp_bios ORACLE SERVER X5-2 v3.2.4.52 r101649 30050100 N/A
[root@odax5-2n1 /]#

 

Apply the CPU Key form the first ODA node

[root@odax5-2n0 ~]# /opt/oracle/oak/bin/oakcli apply core_config_key /root/ODA_PROD_CPU_KEY_SerialNumber_NumberofCores_Configkey.txt
INFO: Both nodes will be rebooted automatically after applying the license
Do you want to continue: [Y/N]?:
Y
INFO: User has confirmed for reboot


Please enter the root password:

............Completed

INFO: Applying core_config_key on '192.168.16.25'
... 
INFO : Running as root: /usr/bin/ssh -l root 192.168.16.25 /tmp/tmp_lic_exec.pl
INFO : Running as root: /usr/bin/ssh -l root 192.168.16.25 /opt/oracle/oak/bin/oakcli enforce core_config_key /tmp/.lic_file
Waiting for the Node '192.168.16.25' to reboot..................................
Node '192.168.16.25' is rebooted
Waiting for the Node '192.168.16.25' to be up before applying the license on the node '192.168.16.24'.
INFO: Applying core_config_key on '192.168.16.24'
...
INFO : Running as root: /usr/bin/ssh -l root 192.168.16.24 /tmp/tmp_lic_exec.pl
INFO : Running as root: /usr/bin/ssh -l root 192.168.16.24 /opt/oracle/oak/bin/oakcli enforce core_config_key /tmp/.lic_file

Broadcast message from root@odax5-2n0
 (unknown) at 11:03 ...

The system is going down for reboot NOW!
[root@odax5-2n0 ~]#

 

New CPU cores configuration

[root@odax5-2n0 ~]# /opt/oracle/oak/bin/oakcli show core_config_key

Host's serialnumber = nnnnXXXXnnX
Enabled Cores (per server) = 6
Total Enabled Cores (on two servers) = 12
Server type = X5-2 -> Oracle Server X5-2
Hyperthreading is enabled. Each core has 2 threads. Operating system displays 12 processors per server
[root@odax5-2n0 ~]#

[root@odax5-2n1 ~]# /opt/oracle/oak/bin/oakcli show core_config_key

Host's serialnumber = nnnnXXXXnnX
Enabled Cores (per server) = 6
Total Enabled Cores (on two servers) = 12
Server type = X5-2 -> Oracle Server X5-2
Hyperthreading is enabled. Each core has 2 threads. Operating system displays 12 processors per server
[root@odax5-2n1 ~]#

The “Great” ODA overwhelming the Exadata

Introduction

This article try to explain the technical reasons of the Oracle Database Appliance success, a well-known appliance with whom Oracle targets small and medium businesses, or specific departments of big companies looking for privacy and isolation from the rest of the IT. Nowadays this small and relatively cheap appliance (around 65’000$ price list) has evolved a lot, the storage has reached an important capacity 128TB raw expansible to 256TB, and the two X5-2 servers are the same used on the database node of the Exadata machine. Many customers, while defining the new database architecture evaluate the pros and cons of acquiring an ODA compared to the smallest Exadata configuration (one eight of a Rack). If the customer is not looking for a system with extreme performance and horizontal scalability beyond the two X5-2 servers, the Oracle Database Appliance is frequently the retained option.

Some of the ODA major features are:

  • High Availability: no single point of failure on all hardware and software components.
  • Performance: each server is equipped with 2×18-core Intel Xeon and 256GB of RAM extensible up to 768GB, cluster communication over InfiniBand. The shared storage offers a multi-tiers configuration with HDDs at 7.2K rpm and two type of SSDs for frequently accessed data and for database redo logs.
  • Flexibility & Scalability: running RAC, RAC One node and Single Instance databases.
  • Virtualized configuration: designed for offering Solution in-a-box, with high available virtual machines.
  • Optimized licensing model: pay-as-you-grow model activating a crescendo number of CPU-cores on demand, with the Bare Metal configuration; or capping the resources combining Oracle VM with the Hard Partitioning setup.
  • Time-to-market: no-matter if the ODA has to be installed bare metal or virtualized, this is a standardized and automated process generally completed in one or two day of work.
  • Price: the ODA is very competitive when comparing the cost to an equivalent commodity architecture; which in addition, must be engineered, integrated and maintained by the customer.

 

At the time of the writing of this article, the latest hardware model is ODA X5-2 and 12.1.2.6.0 is the software version. This HW and SW combination offers unique features, few of them not even available on the Exadata machine, like the possibility to host databases and applications in one single box, or the possibility to rapidly and space efficiently clone an 11gR2 and 12c database using ACFS Snapshot.

 

 

ODA HW & SW Architecture

Oracle Database Appliance is composed by two X5-2 servers and a shared storage shelf, which optionally can be doubled. Each Server disposes of: two 18-core Intel Xeon E5-2699 v3; 256GB RAM (optionally upgradable to 768GB) and two 600GB 10k rpm internal disks in RAID 1 for OS and software binaries.

This appliance is equipped with redundant networking connectivity up to 10Gb, redundant SAS HBAs and Storage I/O modules, redundant InfiniBand interconnect for cluster communication enabling 40 Gb/second server-to-server communication.

The software components are all part of Oracle “Red Stack” with Oracle Linux 6 UEK or OVM 3, Grid Infrastructure 12c, Oracle RDBMS 12c & 11gR2 and Oracle Appliance Manager.

 

 

ODA Front view

Components number 1 & 2 are the X5-2 Servers. Components 3 & 4 are the Storage and the optionally Storage extension.

ODA_Front

 

ODA Rear view

Highlight of the multiple redundant connections, including InfiniBand for Oracle Clusterware, ASM and RAC communications. No single point of HW or SW failure.

ODA_Back

 

 

Storage Organization

With 16x8TB SAS HDDs a total raw space of 128TB is available on each storage self (64TB in configuration ASM double-mirrored and 42.7TB with ASM triple-mirrored). To offer better I/O performance without exploding the price, Oracle has implemented the following SSD devices: 4x400GB ASM double-mirrored, for frequently accessed data, and 4x200GB ASM triple-mirrored, for database redo logs.

As shown on the picture aside, each rotating disk has two slices, the external, and more performant partition assigned to the +DATA ASM disk group, and the internal one allocated to +RECO ASM disk group.

 

ODA_Disk

This storage optimization allows the ODA to achieve competitive I/O performance. In a production-like environment, using the three type of disks, as per ODA Database template odb-24 (https://docs.oracle.com/cd/E22693_01/doc.12/e55580/sizing.htm), Trivadis has measured 12k I/O per second and a throughput of 2300 MB/s with an average latency of 10ms. As per Oracle documentation, the maximum number of I/O per second of the rotating disks, with a single storage shelf is 3300; but this value increases significantly relocating the hottest data files to +FLASH disk group created on SSD devices.

 

ACFS becomes the default database storage of ODA

Starting from the ODA software version 12.1.0.2, any fresh installation enforces ASM Cluster File System (ACFS) as lonely type of database storage support, restricting the supported database versions to 11.2.0.4 and greater. In case of ODA upgrade from previous release, all pre-existing databases are not automatically migrated to ACFS, but Oracle provides a tool called acfs_mig.pl for executing this mandatory step on all Non-CDB databases of version >= 11.2.0.4.

Oracle has decided to promote ACFS as default database storage on ODA environment for the following reasons:

  • ACFS provides almost equivalent performance than Oracle ASM disk groups.
  • Additional functionalities on industry standard POSIX file system.
  • Database snapshot copy of PDBs, and NON-CDB of version 11.2.0.4 or greater.
  • Advanced functionality for general-purpose files such as replication, tagging, encryption, security, and auditing.

Database created on ACFS follows the same Oracle Managed Files (OMF) standard used by ASM.

As in the past, the database provisioning requires the utilization of the command line interface oakcli and the selection of a database template, which defines several characteristics including the amount of space to allocate on each file system. Container and Non-Container databases can coexist on the same Oracle Database Appliance.

The ACFS file systems are created during the database provisioning process on top of the ASM disk groups +DATA, +RECO, +REDO, and optionally +FLASH. The file systems have two possible setups, depending on the database type Container or Non-Container.

  • Container database: for each CDB the ODA database-provisioning job creates dedicated ACFS file systems with the following characteristics:
Disk Characteristics ASM Disk group ACFS Mount Point
SAS Disk external partition +DATA /u02/app/oracle/oradata/datc<db_unique_name>
SAS Disk internal partition +RECO /u01/app/oracle/fast_recovery_area/rcoc<db_unique_name>
SSD Triple-mirrored +REDO /u01/app/oracle/oradata/rdoc<db_unique_name>
SSD Double-mirrored +FLASH (*) /u02/app/oracle/oradata/flashdata

 

  • Non-Container database: in case of Non-CDB the ODA database-provisioning job creates or resizes the following shared ACFS file systems:
Disk Characteristics ASM Disk group ACFS Mount Point
SAS Disk external partition +DATA /u02/app/oracle/oradata/datastore
SAS Disk internal partition +RECO /u01/app/oracle/fast_recovery_area/datastore
SSD Triple-mirrored +REDO /u01/app/oracle/oradata/datastore
SSD Double-mirrored +FLASH (*) /u02/app/oracle/oradata/flashdata

(*) Optionally used by the databases as Smart Flash Cache (extension of the SGA buffer cache), or allocated to store the hottest data files leveraging the I/O performance of the SSD disks.

 

Oracle Database Appliance Bare Metal

The bare metal configuration has been available since version one of the appliance, and nowadays it remains the default option proposed by Oracle, which pre-install the OS Linux on any new system. Very simple and intuitive to install thanks to the pre-built bundle software, which automates most of the steps. At the end of the installation, the architecture is very similar to any other two node RAC setup based on commodity hardware; but even from an operation point of view there are great advantages, because the Oracle Appliance Manager framework simplifies and accelerates the execution of almost any system and database administrator task.

Here below is depicted the ODA architecture when the bare metal configuration is in use:

ODA_Bare_Metal

 

Oracle Database Appliance Virtualized

When the ODA is deployed with the virtualization, both servers run Oracle VM Server, also called Dom0. Each Dom0 hosts in a local dedicated repository the ODA Base (or Dom Base), a privileged virtual machine where it is installed the Appliance Manager, Grid Infrastructure and RDBMS binaries. The ODA Base takes advantage of the Xen PCI Pass-through technology to provide direct access to the ODA shared disks presented and managed by ASM. This configuration reduces the VM flexibility; in fact, no VM migration is allowed for the two ODA Base, but it guarantees almost no I/O penalty in term of performance. With the Dom Base setup, the basic installation is completed and it is possible to start provisioning databases using Oracle Appliance Manager.

At the same time, the administrator can create new-shared repositories hosted on ACFS and NFS exported to the hypervisor for hosting the application virtual machines. Those application virtual machines are also identified with the name of Domain U.  The Domain U and the templates can be stored on a local or shared Oracle VM Server repository, but to enable the functionality to migrate between the two Oracle VM Servers a shared repository on the ACFS file system should be used.

Even when the virtualization is in use, Oracle Appliance Manager is the only framework for system and database administration tasks like repository creation, import of template, deployment of virtual machine, network configuration, database provisioning and so on, relieving the administrator from all complexity.

The implementation of the Solution-in-a-box guarantees the maximum Return on Investment of the ODA; in fact, while restricting the virtual CPUs to license on the Dom Base it allows relocating spare resources to the application virtual machines as showed on the picture below.

ODA_Virtualized

 

 

ODA compared to Exadata Machine and Commodity Hardware

As described on the previous sections, Oracle Database Appliance offers unique features such as pay-as-you-grow, solution-in-a-box and so on, which can heavily influence the decision for a new database architecture. The aim of the table below is to list the main architecture characteristics to evaluate while defining a new database infrastructure, comparing the result between Oracle Database Appliance, Exadata Machine and a Commodity Architecture based on Intel Linux engineered to run RAC databases.

Table_Architectures

As shown by the different scores of the three architectures, each solution comes with points of strength and weakness; about the Oracle Database Appliance, it is evident that due to its characteristics, the smallest Oracle Engineered System remains a great option for small, medium database environments.

 

Conclusion

I hope this article keep the initial promise to explain the technical reasons of the Oracle Database Appliance success, and it has highlighted the great work done by Oracle, engineering this solution on the edge of the technology keeping the price under control.

One last summary of what in my opinion are the major benefits offered by the ODA:

  • Time-to-market: Thanks to automated processes and pre-build software images, the deployment phase is extremely rapid.
  • Simplicity: The use of standard software components, combined to the appliance orchestrator Oracle Appliance Manager makes the ODA very simple to operate.
  • Standardization & Automation: The Appliance Manager encapsulates and automatizes all repeatable and error-prone tasks like provisioning, decommissioning, patching and so on.
  • Vendor certified platform: Oracle validates and certifies the compatibility among all HW & SW components.
  • Evolution: Over the time, the ODA benefits of specific bug fixing and software evolution (introduced by Oracle though the quarterly patch sets); keeping the system on the edge for longer time when compared to a commodity architecture.

EXADATA: How to enable Flash Cache WriteBack on a running system

In a recent tuning activity it was necessary to change the Exadata Smart Flash Cache from “WriteThrough” to “WriteBack“. Because the system was used in a 24/7 environment we had to implement the change in a Rolling Upgrade Fashion.

Here below are described the different steps.

 

From one DB node using dcli check the currest status of the storage cells:

[root@efudbadm02 ~]# dcli -g ~/cell_group -l root cellcli -e "list cell attributes flashcachemode"
efuceladm01: WriteThrough
efuceladm02: WriteThrough
efuceladm03: WriteThrough
efuceladm04: WriteThrough
efuceladm05: WriteThrough
efuceladm06: WriteThrough
efuceladm07: WriteThrough
efuceladm08: WriteThrough
efuceladm09: WriteThrough
efuceladm10: WriteThrough
efuceladm11: WriteThrough

From one DB node using dcli check that the properties asmdeactivationoutcome and asmmodestatus of all griddisks are respectively “Yes” and “ONLINE” before continuing with the change.

[root@efudbadm02 ~]# dcli -g cell_group -l root cellcli -e list griddisk attributes asmdeactivationoutcome, asmmodestatus
efuceladm01: Yes ONLINE
efuceladm01: Yes ONLINE
efuceladm01: Yes ONLINE
efuceladm01: Yes ONLINE
efuceladm01: Yes ONLINE
efuceladm01: Yes ONLINE
efuceladm01: Yes ONLINE
efuceladm01: Yes ONLINE
efuceladm01: Yes ONLINE
efuceladm01: Yes ONLINE
efuceladm01: Yes ONLINE
efuceladm01: Yes ONLINE
efuceladm01: Yes ONLINE
efuceladm01: Yes ONLINE
efuceladm01: Yes ONLINE
efuceladm01: Yes ONLINE
efuceladm02: Yes ONLINE
efuceladm02: Yes ONLINE
efuceladm02: Yes ONLINE
efuceladm02: Yes ONLINE
efuceladm02: Yes ONLINE
efuceladm02: Yes ONLINE
efuceladm02: Yes ONLINE
efuceladm02: Yes ONLINE
efuceladm02: Yes ONLINE
efuceladm02: Yes ONLINE
efuceladm02: Yes ONLINE
efuceladm02: Yes ONLINE
efuceladm02: Yes ONLINE
efuceladm02: Yes ONLINE
efuceladm02: Yes ONLINE
efuceladm02: Yes ONLINE
...
..
.

From one DB node using dcli check that all flashcache modules are in “normal” state and no flash disk is in degraded or critical state.

[root@efudbadm02 ~]# dcli -g cell_group -l root cellcli -e list flashcache detail
efuceladm01: name: efuceladm01_FLASHCACHE
efuceladm01: cellDisk: FD_00_efuceladm01,FD_07_efuceladm01,FD_06_efuceladm01,FD_03_efuceladm01,FD_05_efuceladm01,FD_01_efuceladm01,FD_02_efuceladm01,FD_04_efuceladm01
efuceladm01: creationTime: 2013-06-18T15:21:13+02:00
efuceladm01: degradedCelldisks:
efuceladm01: effectiveCacheSize: 744.125G
efuceladm01: id: 35b61001-438f-4d66-8ce9-40704f758d3f
efuceladm01: size: 744.125G
efuceladm01: status: normal
efuceladm02: name: efuceladm02_FLASHCACHE
efuceladm02: cellDisk: FD_06_efuceladm02,FD_05_efuceladm02,FD_00_efuceladm02,FD_02_efuceladm02,FD_01_efuceladm02,FD_07_efuceladm02,FD_03_efuceladm02,FD_04_efuceladm02
efuceladm02: creationTime: 2013-06-18T15:21:12+02:00
efuceladm02: degradedCelldisks:
efuceladm02: effectiveCacheSize: 744.125G
efuceladm02: id: 2f7eedd6-cda2-496e-98ec-417b94fb8ee7
efuceladm02: size: 744.125G
efuceladm02: status: normal
efuceladm03: name: efuceladm03_FLASHCACHE
efuceladm03: cellDisk: FD_00_efuceladm03,FD_04_efuceladm03,FD_01_efuceladm03,FD_02_efuceladm03,FD_03_efuceladm03,FD_06_efuceladm03,FD_05_efuceladm03,FD_07_efuceladm03
efuceladm03: creationTime: 2013-06-18T15:21:10+02:00
efuceladm03: degradedCelldisks:
efuceladm03: effectiveCacheSize: 744.125G
efuceladm03: id: c271cdb8-dc70-4009-ba97-dfc4c26b00ef
efuceladm03: size: 744.125G
efuceladm03: status: normal
...
..
.

Logon on the first Cell Storage and using CellCli interface perform the following procedure to enable the WriteBack Flash Cache in a rolling upgrade fashion.

 

Drop the existing flash cache

CellCLI> drop flashcache
Flash cache efuceladm01_FLASHCACHE successfully dropped

Inactivate the griddisk on the cell

CellCLI> alter griddisk all inactive
GridDisk DATA_CD_00_efuceladm01 successfully altered
GridDisk DATA_CD_01_efuceladm01 successfully altered
GridDisk DATA_CD_02_efuceladm01 successfully altered
GridDisk DATA_CD_03_efuceladm01 successfully altered
GridDisk DATA_CD_04_efuceladm01 successfully altered
GridDisk DATA_CD_05_efuceladm01 successfully altered
GridDisk DBFS_DG_CD_02_efuceladm01 successfully altered
GridDisk DBFS_DG_CD_03_efuceladm01 successfully altered
GridDisk DBFS_DG_CD_04_efuceladm01 successfully altered
GridDisk DBFS_DG_CD_05_efuceladm01 successfully altered
GridDisk RECO_CD_00_efuceladm01 successfully altered
GridDisk RECO_CD_01_efuceladm01 successfully altered
GridDisk RECO_CD_02_efuceladm01 successfully altered
GridDisk RECO_CD_03_efuceladm01 successfully altered
GridDisk RECO_CD_04_efuceladm01 successfully altered
GridDisk RECO_CD_05_efuceladm01 successfully altered

Shut down cellsrv service

CellCLI> alter cell shutdown services cellsrv

Stopping CELLSRV services...
The SHUTDOWN of CELLSRV services was successful.

Enable the Smart Flash Cache WriteBack

CellCLI> alter cell flashCacheMode=writeback
Cell efuceladm01 successfully altered

Restart the cellsrv service

CellCLI> alter cell startup services cellsrv

Starting CELLSRV services...
The STARTUP of CELLSRV services was successful.

Reactivate the griddisk on the cell

CellCLI> alter griddisk all active
GridDisk DATA_CD_00_efuceladm01 successfully altered
GridDisk DATA_CD_01_efuceladm01 successfully altered
GridDisk DATA_CD_02_efuceladm01 successfully altered
GridDisk DATA_CD_03_efuceladm01 successfully altered
GridDisk DATA_CD_04_efuceladm01 successfully altered
GridDisk DATA_CD_05_efuceladm01 successfully altered
GridDisk DBFS_DG_CD_02_efuceladm01 successfully altered
GridDisk DBFS_DG_CD_03_efuceladm01 successfully altered
GridDisk DBFS_DG_CD_04_efuceladm01 successfully altered
GridDisk DBFS_DG_CD_05_efuceladm01 successfully altered
GridDisk RECO_CD_00_efuceladm01 successfully altered
GridDisk RECO_CD_01_efuceladm01 successfully altered
GridDisk RECO_CD_02_efuceladm01 successfully altered
GridDisk RECO_CD_03_efuceladm01 successfully altered
GridDisk RECO_CD_04_efuceladm01 successfully altered
GridDisk RECO_CD_05_efuceladm01 successfully altered

Recreate the flash cache

CellCLI> create flashcache all
Flash cache efuceladm01_FLASHCACHE successfully created

 


Verify that the Smart Flash Cache WriteBackWriteBack option is enabled

[root@efuceladm01 ~]# cellcli -e list cell detail | grep flashCacheMode
 flashCacheMode: writeback

Before applying the change to the next Exadata Storage Server  wait that all griddisk are synronized and online.

[root@efuceladm01 ~]# cellcli -e list griddisk attributes name,asmmodestatus,asmdeactivationoutcome
 DATA_CD_00_efuceladm01 SYNCING Yes
 DATA_CD_01_efuceladm01 SYNCING Yes
 DATA_CD_02_efuceladm01 SYNCING Yes
 DATA_CD_03_efuceladm01 SYNCING Yes
 DATA_CD_04_efuceladm01 SYNCING Yes
 DATA_CD_05_efuceladm01 SYNCING Yes
 DBFS_DG_CD_02_efuceladm01 ONLINE Yes
 DBFS_DG_CD_03_efuceladm01 ONLINE Yes
 DBFS_DG_CD_04_efuceladm01 ONLINE Yes
 DBFS_DG_CD_05_efuceladm01 ONLINE Yes
 RECO_CD_00_efuceladm01 OFFLINE Yes
 RECO_CD_01_efuceladm01 OFFLINE Yes
 RECO_CD_02_efuceladm01 OFFLINE Yes
 RECO_CD_03_efuceladm01 OFFLINE Yes
 RECO_CD_04_efuceladm01 OFFLINE Yes
 RECO_CD_05_efuceladm01 OFFLINE Yes

Once the asmmodestatus is ONLINE on all griddisks it is safe to move to the next Storage Server.


 

At the end of the procedure all Storage Servers are configured with Smart Flash Cache WriteBach option:

[root@efudbadm02 ~]# dcli -g ~/cell_group -l root cellcli -e "list cell attributes flashcachemode"
efuceladm01: writeback
efuceladm02: writeback
efuceladm03: writeback
efuceladm04: writeback
efuceladm05: writeback
efuceladm06: writeback
efuceladm07: writeback
efuceladm08: writeback
efuceladm09: writeback
efuceladm10: writeback
efuceladm11: writeback


	

Patching ODA X5-2 Virtualized to version 12.1.2.6

Here is described the procedure to upgrade the ODA to the Bundle Patch 12.1.2.6.0.

This Bundle contains a BIG change because it replaces Oracle Enterprise Linux 5.11 with the version 6.7.

One critical requirement: this patch can only be installed on top of 12.1.2.5.0, to check the exisitng ODA version run:

# /opt/oracle/oak/bin/oakcli show version
Version
12.1.2.5.0

The patch can be downloaded from MOS selecting the following note: 22328442 ORACLE DATABASE APPLIANCE PATCH BUNDLE 12.1.2.6.0 (Patch)

 

And now let’s start with the installation:

  • Upload the patch on both ODA_Base (Dom1)  on /tmp
  • Remove any Extra RPM installed by the user on the ODA_Base
  • Unpack both ZIP files of the patch on both ODA_Base using the following oakcli command:
[root@oda_base01 / ] # cd /tmp/Patch_12.1.2.6.0
[root@oda_base01 patch]# oakcli unpack -package /tmp/patch/p22328442_121260_Linux-x86-64_1of2.zip
Unpacking takes a while, pls wait....
Successfully unpacked the files to repository.
[root@oda_base01 patch]#
[root@oda_base01 patch]#
[root@oda_base01 patch]# oakcli unpack -package /tmp/patch/p22328442_121260_Linux-x86-64_2of2.zip
Unpacking takes a while, pls wait....
Successfully unpacked the files to repository.
[root@oda_base01 patch]#


Verify the patch compatibility on both ODA_Base with the following check:

[root@oda_base01 patch]# oakcli update -patch 12.1.2.6.0 -verify

INFO: 2016-03-31 17:07:29: Reading the metadata file now...
 Component Name Installed Version Proposed Patch Version
 --------------- ------------------ -----------------
 Controller_INT     4.230.40-3739       Up-to-date
 Controller_EXT     06.00.02.00         Up-to-date
 Expander           0018                Up-to-date
 SSD_SHARED {
 [ c1d20,c1d21,c1d22, A29A              Up-to-date
 c1d23 ]
 [ c1d16,c1d17,c1d18, A29A              Up-to-date
 c1d19 ]
 }
 HDD_LOCAL            A720              Up-to-date
 HDD_SHARED           P554              Up-to-date
 ILOM             3.2.4.42 r99377     3.2.4.52 r101649
 BIOS               30040200              30050100
 IPMI               1.8.12.0              1.8.12.4
 HMP                2.3.2.4.1             2.3.4.0.1
 OAK               12.1.2.5.0            12.1.2.6.0
 OL                    5.11                  6.7
 OVM                  3.2.9              Up-to-date
 GI_HOME           12.1.0.2.5(21359755, 12.1.0.2.160119(2194
                              21359758) 8354,21948344)
 DB_HOME {
 [ OraDb11204_home1 ] 11.2.0.4.8(21352635, 11.2.0.4.160119(2194
 21352649) 8347,21948348)
 [ OraDb12102_home2,O 12.1.0.2.5(21359755, 12.1.0.2.160119(2194
 raDb12102_home1 ] 21359758) 8354,21948344)
 }
[root@oda_base01 patch]#

Validate the Upgrade to OEL6 checking:

  • The minimum required version
  • The space requirement
  • The list of valid ol5 rpms.
[root@oda_base01 patch]# oakcli validate -c ol6upgrade -prechecks
INFO: Validating the OL6 upgrade -prechecks
INFO: 2016-04-09 17:11:41: Checking for minimum compatible version
SUCCESS: 2016-04-09 17:11:41: Minimum compatible version check passed

INFO: 2016-04-09 17:11:41: Checking available free space on /u01
INFO: 2016-04-09 17:11:41: Free space on /u01 is 39734588 1K-blocks
SUCCESS: 2016-04-09 17:11:41: Check for available free space passed

INFO: 2016-04-09 17:11:42: Checking for additional RPMs
SUCCESS: 2016-04-09 17:11:42: Check for additional RPMs passed

INFO: 2016-04-09 17:11:42: Checking for expected RPMs installed
INFO: 2016-04-09 17:11:42: Please take backup of ODA_BASE. Ensure ODA_BASE, Share Repos and all the VMs are shutdown cleanly before taking backup.
INFO: 2016-04-09 17:11:42: You may use eg tar -cvzf oakDom1.<node>.tar.gz /OVS/Repositories/odabaseRepo/VirtualMachines/oakDom1.
SUCCESS: 2016-04-09 17:11:42: All the expected ol5 RPMs are installed
SUCCESS: Node is ready for upgrade
[root@oda_base01 patch]#

Apply the patch to the first node using the flag -local

[root@oda_base01 patch]# /opt/oracle/oak/bin/oakcli update -patch 12.1.2.6.0 --infra -local
INFO: Local patch is running on the Node <0>
INFO: ***************************************************
INFO: ** Please do not patch both nodes simultaneously **
INFO: ***************************************************
INFO: DB, ASM, Clusterware may be stopped during the patch if required
INFO: Local Node may get rebooted automatically during the patch if necessary
Do you want to continue: [Y/N]?: Y
INFO: User has confirmed for the reboot
INFO: 2016-04-09 17:14:22: Checking for minimum compatible version
SUCCESS: 2016-04-09 17:14:22: Minimum compatible version check passed

INFO: 2016-04-09 17:14:22: Checking available free space on /u01
INFO: 2016-04-09 17:14:22: Free space on /u01 is 39733684 1K-blocks
SUCCESS: 2016-04-09 17:14:22: Check for available free space passed

INFO: 2016-04-09 17:14:22: Checking for additional RPMs
SUCCESS: 2016-04-09 17:14:22: Check for additional RPMs passed

INFO: 2016-04-09 17:14:22: Checking for expected RPMs installed
INFO: 2016-04-09 17:14:22: Please take backup of ODA_BASE. Ensure ODA_BASE, Share Repos and all the VMs are shutdown cleanly before taking backup.
INFO: 2016-04-09 17:14:22: You may use eg tar -cvzf oakDom1.<node>.tar.gz /OVS/Repositories/odabaseRepo/VirtualMachines/oakDom1.
SUCCESS: 2016-04-09 17:14:22: All the expected ol5 RPMs are installed
INFO: All the VMs except the ODABASE will be shutdown forcefully if needed
Do you want to continue : [Y/N]? : Y
INFO: Running pre-install scripts
INFO: Running prepatching on local node
INFO: Completed pre-install scripts
INFO: local patching code START
INFO: Stopping local VMs, repos and oakd...
INFO: Shutdown of local VM, Repo and OAKD on node <0>.
INFO: Stopping OAKD on the local node.
INFO: Stopped Oakd on local node
INFO: Waiting for processes to sync up...
INFO: Oakd running on remote node
INFO: Stopping local VMs...
INFO: Stopping local shared repos...
INFO: Patching Dom0 components

INFO: Patching dom0 components on Local Node... <12.1.2.6.0>
INFO: 2016-04-09 17:27:02: Attempting to patch the HMP on Dom0...
SUCCESS: 2016-04-09 17:27:08: Successfully updated the device HMP to the version 2.3.4.0.1 on Dom0
INFO: 2016-04-09 17:27:08: Attempting to patch the IPMI on Dom0...
INFO: 2016-04-09 17:27:08: Successfully updated the IPMI on Dom0
INFO: 2016-04-09 17:27:08: Attempting to patch OS on Dom0...
INFO: 2016-04-09 17:27:18: Clusterware is running on local node
INFO: 2016-04-09 17:27:18: Attempting to stop clusterware and its resources locally
SUCCESS: 2016-04-09 17:29:12: Successfully stopped the clusterware on local node

SUCCESS: 2016-04-09 17:31:36: Successfully updated the device OVM to 3.2.9

INFO: Patching ODABASE components

INFO: Patching Infrastructure on the Local Node...

INFO: 2016-04-09 17:31:38: ------------------Patching OS-------------------------
INFO: 2016-04-09 17:31:38: OSPatching : Patching will start from step 0
INFO: 2016-04-09 17:31:38: OSPatching : Performing the step 0
INFO: 2016-04-09 17:31:39: OSPatching : step 0 completed
==================================================================================
INFO: 2016-04-09 17:31:39: OSPatching : Performing the step 1
INFO: 2016-04-09 17:31:39: OSPatching : step 1 completed
==================================================================================
INFO: 2016-04-09 17:31:39: OSPatching : Performing the step 2
INFO: 2016-04-09 17:31:42: OSPatching : step 2 completed.
==================================================================================
INFO: 2016-04-09 17:31:42: OSPatching : Performing the step 3
INFO: 2016-04-09 17:31:51: OSPatching : step 3 completed
==================================================================================
INFO: 2016-04-09 17:31:51: OSPatching : Performing the step 4
INFO: 2016-04-09 17:31:51: OSPatching : step 4 completed.
==================================================================================
INFO: 2016-04-09 17:31:51: OSPatching : Performing the step 5
INFO: 2016-04-09 17:31:52: OSPatching : step 5 completed
==================================================================================
INFO: 2016-04-09 17:31:52: OSPatching : Performing the step 6
INFO: 2016-04-09 17:31:52: OSPatching : Installing OL6 RPMs. Please wait...
INFO: 2016-04-09 17:35:05: OSPatching : step 6 completed
==================================================================================
INFO: 2016-04-09 17:35:05: OSPatching : Performing the step 7
INFO: 2016-04-09 17:37:36: OSPatching : step 7 completed
==================================================================================
INFO: 2016-04-09 17:37:36: OSPatching : Performing the step 8
INFO: 2016-04-09 17:37:37: OSPatching : step 8 completed
==================================================================================
INFO: 2016-04-09 17:37:37: OSPatching : Performing the step 9
INFO: 2016-04-09 17:38:14: OSPatching : step 9 completed
==================================================================================
INFO: 2016-04-09 17:38:14: OSPatching : Performing the step 10
INFO: 2016-04-09 17:38:50: OSPatching : step 10 completed
==================================================================================
INFO: 2016-04-09 17:38:50: OSPatching : Performing the step 11
INFO: 2016-04-09 17:38:50: OSPatching : step 11 completed
==================================================================================
INFO: 2016-04-09 17:38:50: OSPatching : Performing the step 12
INFO: 2016-04-09 17:38:50: Checking for expected RPMs installed
SUCCESS: 2016-04-09 17:38:51: All the expected ol6 RPMs are installed
INFO: 2016-04-09 17:38:51: OSPatching : step 12 completed
==================================================================================
SUCCESS: 2016-04-09 17:38:51: Successfully upgraded the OS

INFO: 2016-04-09 17:38:52: ----------------------Patching IPMI---------------------
INFO: 2016-04-09 17:38:52: IPMI is already upgraded or running with the latest version

INFO: 2016-04-09 17:38:52: ------------------Patching HMP-------------------------
INFO: 2016-04-09 17:38:53: HMP is already Up-to-date
INFO: 2016-04-09 17:38:53: /usr/lib64/sun-ssm already exists.

INFO: 2016-04-09 17:38:53: ----------------------Patching OAK---------------------
SUCCESS: 2016-04-09 17:39:27: Successfully upgraded OAK

INFO: 2016-04-09 17:39:31: ----------------------Patching JDK---------------------
SUCCESS: 2016-04-09 17:39:36: Successfully upgraded JDK

INFO: local patching code END

INFO: patching summary on local node
SUCCESS: 2016-04-09 17:39:39: Successfully upgraded the HMP on Dom0
SUCCESS: 2016-04-09 17:39:39: Successfully updated the device OVM
SUCCESS: 2016-04-09 17:39:39: Successfully upgraded the OS
INFO: 2016-04-09 17:39:39: IPMI is already upgraded
INFO: 2016-04-09 17:39:39: HMP is already updated
SUCCESS: 2016-04-09 17:39:39: Successfully updated the OAK
SUCCESS: 2016-04-09 17:39:39: Successfully updated the JDK

INFO: Running post-install scripts
INFO: Running postpatch on local node
INFO: Dom0 Needs to be rebooted, will be rebooting the Dom0

Broadcast message from root@oda_base01
 (unknown) at 17:40 ...

The system is going down for power off NOW!

Validate the steps with the  infrastructure post patch checks:

[root@oda_base01 ~]# /u01/app/12.1.0.2/grid/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

[root@oda_base01 ~]# /opt/oracle/oak/bin/oakcli validate -c ol6upgrade -postchecks
INFO: Validating the OL6 upgrade -postchecks

INFO: 2016-04-09 19:50:40: Current kernel is OL6
INFO: 2016-04-09 19:50:43: Checking for expected RPMs installed
SUCCESS: 2016-04-09 19:50:43: All the expected ol6 RPMs are installed

Apply the patch to the second node using the flag -local

[root@oda_base02 patch]# /opt/oracle/oak/bin/oakcli update -patch 12.1.2.6.0 --infra -local
INFO: Local patch is running on the Node <1>
INFO: ***************************************************
INFO: ** Please do not patch both nodes simultaneously **
INFO: ***************************************************
INFO: DB, ASM, Clusterware may be stopped during the patch if required
INFO: Local Node may get rebooted automatically during the patch if necessary
Do you want to continue: [Y/N]?: Y
INFO: User has confirmed for the reboot
INFO: 2016-04-09 19:58:07: Checking for minimum compatible version
SUCCESS: 2016-04-09 19:58:07: Minimum compatible version check passed

INFO: 2016-04-09 19:58:07: Checking available free space on /u01
INFO: 2016-04-09 19:58:07: Free space on /u01 is 45790328 1K-blocks
SUCCESS: 2016-04-09 19:58:07: Check for available free space passed

INFO: 2016-04-09 19:58:07: Checking for additional RPMs
SUCCESS: 2016-04-09 19:58:07: Check for additional RPMs passed

INFO: 2016-04-09 19:58:07: Checking for expected RPMs installed
INFO: 2016-04-09 19:58:08: Please take backup of ODA_BASE. Ensure ODA_BASE, Share Repos and all the VMs are shutdown cleanly before taking backup.
INFO: 2016-04-09 19:58:08: You may use eg tar -cvzf oakDom1.<node>.tar.gz /OVS/Repositories/odabaseRepo/VirtualMachines/oakDom1.
SUCCESS: 2016-04-09 19:58:08: All the expected ol5 RPMs are installed
INFO: All the VMs except the ODABASE will be shutdown forcefully if needed
Do you want to continue : [Y/N]? : Y
INFO: Running pre-install scripts
INFO: Running prepatching on local node
INFO: Completed pre-install scripts
INFO: local patching code START
INFO: Stopping local VMs, repos and oakd...
INFO: Shutdown of local VM, Repo and OAKD on node <1>.
INFO: Stopping OAKD on the local node.
INFO: Stopped Oakd on local node
INFO: Waiting for processes to sync up...
INFO: Oakd running on remote node
INFO: Stopping local VMs...
INFO: Stopping local shared repos...
INFO: Patching Dom0 components

INFO: Patching dom0 components on Local Node... <12.1.2.6.0>
INFO: 2016-04-09 20:04:26: Attempting to patch the HMP on Dom0...
SUCCESS: 2016-04-09 20:04:33: Successfully updated the device HMP to the version 2.3.4.0.1 on Dom0
INFO: 2016-04-09 20:04:33: Attempting to patch the IPMI on Dom0...
INFO: 2016-04-09 20:04:33: Successfully updated the IPMI on Dom0
INFO: 2016-04-09 20:04:33: Attempting to patch OS on Dom0...
INFO: 2016-04-09 20:04:43: Clusterware is running on local node
INFO: 2016-04-09 20:04:43: Attempting to stop clusterware and its resources locally
SUCCESS: 2016-04-09 20:08:20: Successfully stopped the clusterware on local node

SUCCESS: 2016-04-09 20:10:44: Successfully updated the device OVM to 3.2.9

INFO: Patching ODABASE components

INFO: Patching Infrastructure on the Local Node...

INFO: 2016-04-09 20:10:48: ------------------Patching OS-------------------------
INFO: 2016-04-09 20:10:48: OSPatching : Patching will start from step 0
INFO: 2016-04-09 20:10:48: OSPatching : Performing the step 0
INFO: 2016-04-09 20:10:51: OSPatching : step 0 completed
==================================================================================
INFO: 2016-04-09 20:10:51: OSPatching : Performing the step 1
INFO: 2016-04-09 20:10:51: OSPatching : step 1 completed
==================================================================================
INFO: 2016-04-09 20:10:51: OSPatching : Performing the step 2
INFO: 2016-04-09 20:10:53: OSPatching : step 2 completed.
==================================================================================
INFO: 2016-04-09 20:10:53: OSPatching : Performing the step 3
INFO: 2016-04-09 20:11:00: OSPatching : step 3 completed
==================================================================================
INFO: 2016-04-09 20:11:00: OSPatching : Performing the step 4
INFO: 2016-04-09 20:11:00: OSPatching : step 4 completed.
==================================================================================
INFO: 2016-04-09 20:11:00: OSPatching : Performing the step 5
INFO: 2016-04-09 20:11:00: OSPatching : step 5 completed
==================================================================================
INFO: 2016-04-09 20:11:00: OSPatching : Performing the step 6
INFO: 2016-04-09 20:11:00: OSPatching : Installing OL6 RPMs. Please wait...
INFO: 2016-04-09 20:14:25: OSPatching : step 6 completed
==================================================================================
INFO: 2016-04-09 20:14:25: OSPatching : Performing the step 7
INFO: 2016-04-09 20:16:58: OSPatching : step 7 completed
==================================================================================
INFO: 2016-04-09 20:16:58: OSPatching : Performing the step 8
INFO: 2016-04-09 20:16:59: OSPatching : step 8 completed
==================================================================================
INFO: 2016-04-09 20:16:59: OSPatching : Performing the step 9
INFO: 2016-04-09 20:17:35: OSPatching : step 9 completed
==================================================================================
INFO: 2016-04-09 20:17:35: OSPatching : Performing the step 10
INFO: 2016-04-09 20:18:11: OSPatching : step 10 completed
==================================================================================
INFO: 2016-04-09 20:18:11: OSPatching : Performing the step 11
INFO: 2016-04-09 20:18:11: OSPatching : step 11 completed
==================================================================================
INFO: 2016-04-09 20:18:11: OSPatching : Performing the step 12
INFO: 2016-04-09 20:18:12: Checking for expected RPMs installed
SUCCESS: 2016-04-09 20:18:12: All the expected ol6 RPMs are installed
INFO: 2016-04-09 20:18:12: OSPatching : step 12 completed
==================================================================================
SUCCESS: 2016-04-09 20:18:12: Successfully upgraded the OS

INFO: 2016-04-09 20:18:12: ----------------------Patching IPMI---------------------
INFO: 2016-04-09 20:18:13: IPMI is already upgraded or running with the latest version

INFO: 2016-04-09 20:18:13: ------------------Patching HMP-------------------------
INFO: 2016-04-09 20:18:15: HMP is already Up-to-date
INFO: 2016-04-09 20:18:15: /usr/lib64/sun-ssm already exists.

INFO: 2016-04-09 20:18:15: ----------------------Patching OAK---------------------
SUCCESS: 2016-04-09 20:18:53: Successfully upgraded OAK

INFO: 2016-04-09 20:18:56: ----------------------Patching JDK---------------------
SUCCESS: 2016-04-09 20:19:02: Successfully upgraded JDK

INFO: local patching code END

INFO: patching summary on local node
SUCCESS: 2016-04-09 20:19:06: Successfully upgraded the HMP on Dom0
SUCCESS: 2016-04-09 20:19:06: Successfully updated the device OVM
SUCCESS: 2016-04-09 20:19:06: Successfully upgraded the OS
INFO: 2016-04-09 20:19:06: IPMI is already upgraded
INFO: 2016-04-09 20:19:06: HMP is already updated
SUCCESS: 2016-04-09 20:19:06: Successfully updated the OAK
SUCCESS: 2016-04-09 20:19:06: Successfully updated the JDK

INFO: Running post-install scripts
INFO: Running postpatch on local node
INFO: Dom0 Needs to be rebooted, will be rebooting the Dom0

Broadcast message from root@oda_base02
 (unknown) at 20:20 ...

The system is going down for power off NOW!

From the first ODA_Base apply the fix to the InfiniBand connection:

[root@oda_base01 ~]# python /opt/oracle/oak/bin/infiniFixSetup.py
IB Fix requires nodes reboot. Do you want to continue? [Y/N] : Y
INFO: Checking version for IB Fix setup
INFO: Checking whether IB Fix setup is already done or not
INFO: Checking default HAVIP for IB Fix setup
INFO: Setting up IB fix
INFO: Enabling IB fix and rebooting all nodes....
[root@oda_base01 ~]#
Broadcast message from root@oda_base01
 (unknown) at 20:40 ...

The system is going down for power off NOW!

Check the correct application of the InfiniBand patch, the value of the file below should be 1

[root@oda_base01 ~]#  view /opt/oracle/oak/conf/ib_fix
1

Installation of the Grid Infrastructure patch, two available methods:

  • Full Downtime
  • Rolling Upgrade

The example below show the first method

[root@oda_base01 ~]# oakcli update -patch 12.1.2.6.0 --gi

Please enter the 'SYSASM' password : (During deployment we set the SYSASM password to 'welcome1'):
Please re-enter the 'SYSASM' password:
INFO: Running pre-install scripts
INFO: Running prepatching on node 0
INFO: Running prepatching on node 1
INFO: Completed pre-install scripts
...
...
INFO: Stopped Oakd
...
...

......
SUCCESS: All nodes in /opt/oracle/oak/temp_clunodes.txt are pingable and alive.
INFO: 2016-04-09 22:32:16: Setting up SSH for grid User
......
SUCCESS: All nodes in /opt/oracle/oak/temp_clunodes.txt are pingable and alive.
INFO: 2016-04-09 22:32:34: Patching the GI Home on the Node oda_base01 ...
INFO: 2016-04-09 22:32:34: Updating OPATCH...
INFO: 2016-04-09 22:32:36: Rolling back GI on oda_base01 (if necessary)...
INFO: 2016-04-09 22:32:39: Rolling back GI on oda_base02 (if necessary)...
INFO: 2016-04-09 22:32:46: Patching the GI Home on the Node oda_base01
INFO: 2016-04-09 22:34:02: Performing the conflict checks...
SUCCESS: 2016-04-09 22:34:16: Conflict checks passed for all the Homes
INFO: 2016-04-09 22:34:16: Checking if the patch is already applied on any of the Homes
INFO: 2016-04-09 22:34:28: Home is not Up-to-date
SUCCESS: 2016-04-09 22:37:01: Successfully stopped the Database consoles
SUCCESS: 2016-04-09 22:37:18: Successfully stopped the EM agents
INFO: 2016-04-09 22:37:23: Applying patch on /u01/app/12.1.0.2/grid Homes
INFO: 2016-04-09 22:37:23: It may take upto 15 mins. Please wait...
SUCCESS: 2016-04-09 22:50:57: Successfully applied the patch on the Home : /u01/app/12.1.0.2/grid
SUCCESS: 2016-04-09 22:51:24: Successfully started the Database consoles
SUCCESS: 2016-04-09 22:51:40: Successfully started the EM Agents
INFO: 2016-04-09 22:51:41: Patching the GI Home on the Node oda_base02
...
INFO: 2016-04-09 23:16:27: ASM is running in Flex mode


INFO: GI patching summary on node: oda_base01
SUCCESS: 2016-04-09 23:16:28: Successfully applied the patch on the Home /u01/app/12.1.0.2/grid

INFO: GI patching summary on node: oda_base02
SUCCESS: 2016-04-09 23:16:28: Successfully applied the patch on the Home /u01/app/12.1.0.2/grid

INFO: GI versions: installed <12.1.0.2.160119> expected <12.1.0.2.160119>
INFO: Running post-install scripts
INFO: Running postpatch on node 1...
INFO: Running postpatch on node 0...
...
...
INFO: Started Oakd

Installation of the RDBMS patch, two available methods:

  • Full Downtime
  • Rolling Upgrade

The example below show the first method

[root@oda_base01 ~]# oakcli update -patch 12.1.2.6.0 --database
INFO: Running pre-install scripts
INFO: Running prepatching on node 0
INFO: Running prepatching on node 1
INFO: Completed pre-install scripts
...
...

......
SUCCESS: All nodes in /opt/oracle/oak/temp_clunodes.txt are pingable and alive.
INFO: 2016-04-09 23:27:31: Getting all the possible Database Homes for patching
...
INFO: 2016-04-09 23:27:42: Patching 11.2.0.4 Database Homes on the Node oda_base01

Found the following 11.2.0.4 homes possible for patching:

HOME_NAME HOME_LOCATION
--------- -------------
OraDb11204_home1 /u01/app/oracle/product/11.2.0.4/dbhome_1

[Please note that few of the above Database Homes may be already up-to-date. They will be automatically ignored]

Would you like to patch all the above homes: Y | N ? : Y
INFO: 2016-04-09 23:29:17: Setting up SSH for the User oracle
......
SUCCESS: All nodes in /opt/oracle/oak/temp_clunodes.txt are pingable and alive.
INFO: 2016-04-09 23:29:35: Updating OPATCH
Fixing home : /u01/app/oracle/product/11.2.0.4/dbhome_1...done
INFO: 2016-04-09 23:30:33: Performing the conflict checks...
SUCCESS: 2016-04-09 23:30:43: Conflict checks passed for all the Homes
INFO: 2016-04-09 23:30:43: Checking if the patch is already applied on any of the Homes
INFO: 2016-04-09 23:30:47: Home is not Up-to-date
SUCCESS: 2016-04-09 23:31:13: Successfully stopped the Database consoles
SUCCESS: 2016-04-09 23:31:31: Successfully stopped the EM agents
INFO: 2016-04-09 23:31:36: Applying the patch on oracle home : /u01/app/oracle/product/11.2.0.4/dbhome_1 ...
SUCCESS: 2016-04-09 23:32:52: Successfully applied the patch on the Home : /u01/app/oracle/product/11.2.0.4/dbhome_1
SUCCESS: 2016-04-09 23:32:52: Successfully started the Database consoles
SUCCESS: 2016-04-09 23:33:08: Successfully started the EM Agents
INFO: 2016-04-09 23:33:17: Patching 11.2.0.4 Database Homes on the Node oda_base02
INFO: 2016-04-09 23:40:45: Running the catbundle.sql
INFO: 2016-04-09 23:40:52: Running catbundle.sql on the Database XXXXXXX
INFO: 2016-04-09 23:41:29: Running catbundle.sql on the Database YYYYYYY
INFO: 2016-04-09 23:42:07: Running catbundle.sql on the Database ZZZZZZZ
INFO: 2016-04-09 23:42:42: Running catbundle.sql on the Database WWWWWWW
...
INFO: 2016-04-09 23:47:56: Patching 12.1.0.2 Database Homes on the Node oda_base01

Found the following 12.1.0.2 homes possible for patching:

HOME_NAME HOME_LOCATION
--------- -------------
OraDb12102_home1 /u01/app/oracle/product/12.1.0.2/dbhome_1
OraDb12102_home2 /u01/app/oracle/product/12.1.0.2/dbhome_2

[Please note that few of the above Database Homes may be already up-to-date. They will be automatically ignored]

Would you like to patch all the above homes: Y | N ? : Y
INFO: 2016-04-09 23:49:11: Updating OPATCH
INFO: 2016-04-09 23:49:55: Performing the conflict checks...
SUCCESS: 2016-04-09 23:50:21: Conflict checks passed for all the Homes
INFO: 2016-04-09 23:50:21: Checking if the patch is already applied on any of the Homes
INFO: 2016-04-09 23:50:28: Home is not Up-to-date
SUCCESS: 2016-04-09 23:50:47: Successfully stopped the Database consoles
SUCCESS: 2016-04-09 23:51:04: Successfully stopped the EM agents
INFO: 2016-04-09 23:51:10: Applying patch on /u01/app/oracle/product/12.1.0.2/dbhome_1,/u01/app/oracle/product/12.1.0.2/dbhome_2 Homes
INFO: 2016-04-09 23:51:10: It may take upto 30 mins. Please wait...
SUCCESS: 2016-04-09 23:54:20: Successfully applied the patch on the Home : /u01/app/oracle/product/12.1.0.2/dbhome_1,/u01/app/oracle/product/12.1.0.2/dbhome_2
SUCCESS: 2016-04-09 23:54:20: Successfully started the Database consoles
SUCCESS: 2016-04-09 23:54:37: Successfully started the EM Agents
INFO: 2016-04-09 23:54:47: Patching 12.1.0.2 Database Homes on the Node oda_base02


INFO: DB patching summary on node: oda_base01
SUCCESS: 2016-04-01 00:03:19: Successfully applied the patch on the Home /u01/app/oracle/product/11.2.0.4/dbhome_1
SUCCESS: 2016-04-01 00:03:19: Successfully applied the patch on the Home /u01/app/oracle/product/12.1.0.2/dbhome_1,/u01/app/oracle/product/12.1.0.2/dbhome_2

INFO: DB patching summary on node: oda_base02
SUCCESS: 2016-04-01 00:03:20: Successfully applied the patch on the Home /u01/app/oracle/product/11.2.0.4/dbhome_1
SUCCESS: 2016-04-01 00:03:20: Successfully applied the patch on the Home /u01/app/oracle/product/12.1.0.2/dbhome_1,/u01/app/oracle/product/12.1.0.2/dbhome_2

Post patching validation:

[root@oda_base01 ~]# /opt/oracle/oak/bin/oakcli validate -d
INFO: oak system information and Validations
RESULT: System Software inventory details
 Reading the metadata. It takes a while...
 System Version Component Name Installed Version Supported Version
 -------------- --------------- ------------------ -----------------
 12.1.2.6.0
                  Controller_INT   4.230.40-3739     Up-to-date
                  Controller_EXT   06.00.02.00       Up-to-date
                  Expander         0018              Up-to-date
 SSD_SHARED {
 [ c1d20,c1d21,c1d22,              A29A               Up-to-date
 c1d23 ]
 [ c1d16,c1d17,c1d18,              A29A               Up-to-date
 c1d19 ]
 }
 HDD_LOCAL                         A720               Up-to-date
 HDD_SHARED                        P554               Up-to-date
 ILOM                              3.2.4.42 r99377    Up-to-date
 BIOS                              30040200           Up-to-date
 IPMI                              1.8.12.4           Up-to-date
 HMP                               2.3.4.0.1          Up-to-date
 OAK                               12.1.2.6.0         Up-to-date
 OL                                6.7                Up-to-date
 OVM                               3.2.9              Up-to-date
 GI_HOME                         12.1.0.2.160119(2194 Up-to-date
                                 8354,21948344)
 DB_HOME {
 [ OraDb11204_home1 ]            11.2.0.4.160119(2194 Up-to-date
                                 8347,21948348)
 [ OraDb12102_home2,O            12.1.0.2.160119(2194 Up-to-date
 raDb12102_home1 ]               8354,21948344)
 }
RESULT: System Information:-
 Manufacturer:Oracle Corporation
 Product Name:ORACLE SERVER X5-2
 Serial Number:1548NM102F
RESULT: BIOS Information:-
 Vendor:American Megatrends Inc.
 Version:30040200
 Release Date:04/29/2015
 BIOS Revision:4.2
 Firmware Revision:3.2
SUCCESS: Controller p1 has the IR Bypass mode set correctly
SUCCESS: Controller p2 has the IR Bypass mode set correctly
INFO: Reading ilom data, may take short while..
INFO: Read the ilom data. Doing Validations
RESULT: System ILOM Version: 3.2.4.42 r99377
RESULT: System BMC firmware version 3.02
RESULT: Powersupply PS0 V_IN=230 Volts IN_POWER=180 Watts OUT_POWER=170 Watts
RESULT: Powersupply PS1 V_IN=230 Volts IN_POWER=190 Watts OUT_POWER=160 Watts
SUCCESS: Both the powersupply are ok and functioning
RESULT: Cooling Unit FM0 fan speed F0=5000 RPM F1=4500 RPM
RESULT: Cooling Unit FM1 fan speed F0=9100 RPM F1=8000 RPM
SUCCESS: Both the cooling unit are present
RESULT: Processor P0 present Details:-
 Version:Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
 Current Speed:2300 MHz Core Enabled:18 Thread Count:36
SUCCESS: All 4 memory modules of CPU P0 ok, each module is of Size:32767 MB Type:Other Speed:2133 MHz manufacturer:Samsung
RESULT: Processor P1 present Details:-
 Version:Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
 Current Speed:2300 MHz Core Enabled:18 Thread Count:36
SUCCESS: All 4 memory modules of CPU P1 ok, each module is of Size:32767 MB Type:Other Speed:2133 MHz manufacturer:Samsung
RESULT: Total Physical System Memory is 132037124 kB
SUCCESS: All OS Disks are present and in ok state
RESULT: Power Supply=24 degrees C
INFO: Checking Operating System Storage
SUCCESS: The OS disks have the boot stamp
RESULT: Device /dev/xvda2 is mounted on / of type ext3 in (rw)
RESULT: Device /dev/xvda1 is mounted on /boot of type ext3 in (rw)
RESULT: Device /dev/xvdb1 is mounted on /u01 of type ext3 in (rw)
RESULT: / has 19218 MB free out of total 55852 MB
RESULT: /boot has 384 MB free out of total 460 MB
RESULT: /u01 has 34501 MB free out of total 93868 MB
INFO: Checking Shared Storage
RESULT: Disk HDD_E0_S00_993971920 path1 status active device sdy with status active path2 status active device sda with status active
SUCCESS: HDD_E0_S00_993971920 has both the paths up and active
RESULT: Disk HDD_E0_S01_993379760 path1 status active device sdz with status active path2 status active device sdb with status active
SUCCESS: HDD_E0_S01_993379760 has both the paths up and active
RESULT: Disk HDD_E0_S02_993993052 path1 status active device sdaa with status active path2 status active device sdc with status active
SUCCESS: HDD_E0_S02_993993052 has both the paths up and active
RESULT: Disk HDD_E0_S03_993310956 path1 status active device sdab with status active path2 status active device sdd with status active
SUCCESS: HDD_E0_S03_993310956 has both the paths up and active
RESULT: Disk HDD_E0_S04_993385276 path1 status active device sdac with status active path2 status active device sde with status active
SUCCESS: HDD_E0_S04_993385276 has both the paths up and active
RESULT: Disk HDD_E0_S05_993388928 path1 status active device sdf with status active path2 status active device sdad with status active
SUCCESS: HDD_E0_S05_993388928 has both the paths up and active
RESULT: Disk HDD_E0_S06_993310572 path1 status active device sdae with status active path2 status active device sdg with status active
SUCCESS: HDD_E0_S06_993310572 has both the paths up and active
RESULT: Disk HDD_E0_S07_991849548 path1 status active device sdh with status active path2 status active device sdaf with status active
SUCCESS: HDD_E0_S07_991849548 has both the paths up and active
RESULT: Disk HDD_E0_S08_992415004 path1 status active device sdag with status active path2 status active device sdi with status active
SUCCESS: HDD_E0_S08_992415004 has both the paths up and active
RESULT: Disk HDD_E0_S09_992392444 path1 status active device sdj with status active path2 status active device sdah with status active
SUCCESS: HDD_E0_S09_992392444 has both the paths up and active
RESULT: Disk HDD_E0_S10_992233592 path1 status active device sdai with status active path2 status active device sdk with status active
SUCCESS: HDD_E0_S10_992233592 has both the paths up and active
RESULT: Disk HDD_E0_S11_992337644 path1 status active device sdl with status active path2 status active device sdaj with status active
SUCCESS: HDD_E0_S11_992337644 has both the paths up and active
RESULT: Disk HDD_E0_S12_993363524 path1 status active device sdm with status active path2 status active device sdak with status active
SUCCESS: HDD_E0_S12_993363524 has both the paths up and active
RESULT: Disk HDD_E0_S13_992394252 path1 status active device sdn with status active path2 status active device sdal with status active
SUCCESS: HDD_E0_S13_992394252 has both the paths up and active
RESULT: Disk HDD_E0_S14_993366344 path1 status active device sdam with status active path2 status active device sdo with status active
SUCCESS: HDD_E0_S14_993366344 has both the paths up and active
RESULT: Disk HDD_E0_S15_993407552 path1 status active device sdp with status active path2 status active device sdan with status active
SUCCESS: HDD_E0_S15_993407552 has both the paths up and active
RESULT: Disk SSD_E0_S16_1313537708 path1 status active device sdq with status active path2 status active device sdao with status active
SUCCESS: SSD_E0_S16_1313537708 has both the paths up and active
RESULT: Disk SSD_E0_S17_1313522352 path1 status active device sdr with status active path2 status active device sdap with status active
SUCCESS: SSD_E0_S17_1313522352 has both the paths up and active
RESULT: Disk SSD_E0_S18_1313531936 path1 status active device sds with status active path2 status active device sdaq with status active
SUCCESS: SSD_E0_S18_1313531936 has both the paths up and active
RESULT: Disk SSD_E0_S19_1313534520 path1 status active device sdt with status active path2 status active device sdar with status active
SUCCESS: SSD_E0_S19_1313534520 has both the paths up and active
RESULT: Disk SSD_E0_S20_1313568492 path1 status active device sdu with status active path2 status active device sdas with status active
SUCCESS: SSD_E0_S20_1313568492 has both the paths up and active
RESULT: Disk SSD_E0_S21_1313571440 path1 status active device sdv with status active path2 status active device sdat with status active
SUCCESS: SSD_E0_S21_1313571440 has both the paths up and active
RESULT: Disk SSD_E0_S22_1313568380 path1 status active device sdw with status active path2 status active device sdau with status active
SUCCESS: SSD_E0_S22_1313568380 has both the paths up and active
RESULT: Disk SSD_E0_S23_1313568480 path1 status active device sdx with status active path2 status active device sdav with status active
SUCCESS: SSD_E0_S23_1313568480 has both the paths up and active
INFO: Doing oak network checks
RESULT: Detected active link for interface eth0 with link speed 10000Mb/s and cable type as TwistedPair
RESULT: Detected active link for interface eth1 with link speed 10000Mb/s and cable type as TwistedPair
WARNING: No Link detected for interface eth2 with cable type as TwistedPair
WARNING: No Link detected for interface eth3 with cable type as TwistedPair
INFO: Checking bonding interface status
RESULT: No Bond Interface Found
SUCCESS: ibbond0 is running 192.168.16.27
 It may take a while. Please wait...
 INFO : ODA Topology Verification
 INFO : Running on Node0
 INFO : Check hardware type
 SUCCESS : Type of hardware found : X5-2
 INFO : Check for Environment(Bare Metal or Virtual Machine)
 SUCCESS : Type of environment found : Virtual Machine(ODA BASE)
 SUCCESS : Number of External SCSI controllers found : 2
 INFO : Check for Controllers correct PCIe slot address
 SUCCESS : External LSI SAS controller 0 : 00:04.0
 SUCCESS : External LSI SAS controller 1 : 00:05.0
 INFO : Check if JBOD powered on
 SUCCESS : 1JBOD : Powered-on
 INFO : Check for correct number of EBODS(2 or 4)
 SUCCESS : EBOD found : 2
 INFO : Check for External Controller 0
 SUCCESS : Controller connected to correct EBOD number
 SUCCESS : Controller port connected to correct EBOD port
 SUCCESS : Overall Cable check for controller 0
 INFO : Check for External Controller 1
 SUCCESS : Controller connected to correct EBOD number
 SUCCESS : Controller port connected to correct EBOD port
 SUCCESS : Overall Cable check for Controller 1
 INFO : Check for overall status of cable validation on Node0
 SUCCESS : Overall Cable Validation on Node0
 INFO : Check Node Identification status
 SUCCESS : Node Identification
 SUCCESS : Node name based on cable configuration found : NODE0
 INFO : Check JBOD Nickname
 SUCCESS : JBOD Nickname set correctly : Oracle Database Appliance - E0
 INFO : The details for Storage Topology Validation can also be found in the log file=/opt/oracle/oak/log/oda_base01/storagetopology/StorageTopology-2016-04-01-00:06:34_28446_1789.log

 One takeaway

Despite the fact that patching an Oracle Engineered system should be a straight forward task, it is recommended to carefully read the instructions (README), and the MOS notes continuously updated with bug, known issues and other related information.


 

Patching Exadata Machine

################################################################
##    EXADATA MACHINE  INFRASTRUCTURE PATCHING of 1/8 RACK     ##
################################################################

This post describe step-by-step how to patch the infrastructure components of an Exadata Machine

———————————————————–
— Cell Storage Pre-requisites
———————————————————–

--Stop CRS using dcli
[root@ch01db01 oracle]# dcli -g /home/oracle/dbhosts -l root '/u01/app/12.1.0.2/grid/bin/crsctl stop crs'
 [root@ch01db01 oracle]# dcli -g /home/oracle/dbhosts -l root '/u01/app/12.1.0.2/grid/bin/crsctl stat res -t -init'
ch01db01: CRS-4639: Could not contact Oracle High Availability Services
ch01db01: CRS-4000: Command Status failed, or completed with errors.
ch01db02: CRS-4639: Could not contact Oracle High Availability Services
ch01db02: CRS-4000: Command Status failed, or completed with errors.
--Stop All Cell Storage Services
 [root@ch01db01 oracle]# dcli -g /home/oracle/cellhosts_ALL -l root "cellcli -e alter cell shutdown services all"
ch01celadm01:
ch01celadm01: Stopping the RS, CELLSRV, and MS services...
 ch01celadm01: The SHUTDOWN of services was successful.
 ch01celadm02:
 ch01celadm02: Stopping the RS, CELLSRV, and MS services...
 ch01celadm02: The SHUTDOWN of services was successful.
 ch01celadm03:
 ch01celadm03: Stopping the RS, CELLSRV, and MS services...
 ch01celadm03: The SHUTDOWN of services was successful.

[root@ch01db01 oracle]#

 

———————————————————–
–Cell Storage patching
———————————————————–

[root@ch01db01 patch_12.1.2.1.0.141206.1]# ./patchmgr -cells /home/oracle/cellhosts -reset_force
2016-02-05 11:17:07 +0100 :DONE: reset_force
[root@ch01db01 patch_12.1.2.1.0.141206.1]# ./patchmgr -cells /home/oracle/cellhosts -cleanup
2016-02-05 11:19:19 +0100        :Working: DO: Cleanup ...
2016-02-05 11:19:20 +0100        :SUCCESS: DONE: Cleanup
[root@ch01db01 patch_12.1.2.1.0.141206.1]# ./patchmgr -cells /home/oracle/cellhosts -patch_check_prereq
2016-02-05 11:20:56 +0100        :Working: DO: Check cells have ssh equivalence for root user. Up to 10 seconds per cell ...
 2016-02-05 11:20:57 +0100        :SUCCESS: DONE: Check cells have ssh equivalence for root user.
 2016-02-05 11:20:59 +0100        :Working: DO: Initialize files, check space and state of cell services. Up to 1 minute ...
 2016-02-05 11:21:01 +0100        :SUCCESS: DONE: Initialize files, check space and state of cell services.
 2016-02-05 11:22:19 +0100        :SUCCESS: DONE: Initialize files, check space and state of cell services.
 2016-02-05 11:22:19 +0100        :Working: DO: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction. Up to 40 minutes ...
 2016-02-05 11:22:33 +0100 Wait correction of degraded md11 due to md partner size mismatch. Up to 30 minutes.
2016-02-05 11:22:34 +0100        :SUCCESS: DONE: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction.
 2016-02-05 11:22:34 +0100        :Working: DO: Check prerequisites on all cells. Up to 2 minutes ...
 2016-02-05 11:23:38 +0100        :SUCCESS: DONE: Check prerequisites on all cells.
 2016-02-05 11:23:38 +0100        :Working: DO: Execute plugin check for Patch Check Prereq ...
 2016-02-05 11:23:38 +0100        :INFO: Patchmgr plugin start: Prereq check for exposure to bug 17854520 v1.3. Details in logfile /u02/p17885582_121210_Linux-x86-64/patch_12.1.2.1.0.141206.1/patchmgr.stdout.
 2016-02-05 11:23:38 +0100        :SUCCESS: No exposure to bug 17854520 with non-rolling patching
 2016-02-05 11:23:39 +0100        :SUCCESS: DONE: Execute plugin check for Patch Check Prereq.
[root@ch01db01 patch_12.1.2.1.0.141206.1]#
 [root@ch01db01 patch_12.1.2.1.0.141206.1]# ./patchmgr -cells /home/oracle/cellhosts -patch
********************************************************************************
 NOTE Cells will reboot during the patch or rollback process.
 NOTE For non-rolling patch or rollback, ensure all ASM instances using
 NOTE the cells are shut down for the duration of the patch or rollback.
 NOTE For rolling patch or rollback, ensure all ASM instances using
 NOTE the cells are up for the duration of the patch or rollback.
WARNING Do not start more than one instance of patchmgr.
 WARNING Do not interrupt the patchmgr session.
 WARNING Do not alter state of ASM instances during patch or rollback.
 WARNING Do not resize the screen. It may disturb the screen layout.
 WARNING Do not reboot cells or alter cell services during patch or rollback.
 WARNING Do not open log files in editor in write mode or try to alter them.
NOTE All time estimates are approximate.
 NOTE You may interrupt this patchmgr run in next 60 seconds with CONTROL-c.
********************************************************************************
2016-02-05 11:27:08 +0100        :Working: DO: Check cells have ssh equivalence for root user. Up to 10 seconds per cell ...
 2016-02-05 11:27:09 +0100        :SUCCESS: DONE: Check cells have ssh equivalence for root user.
 2016-02-05 11:27:12 +0100        :Working: DO: Initialize files, check space and state of cell services. Up to 1 minute ...
 2016-02-05 11:27:32 +0100        :SUCCESS: DONE: Initialize files, check space and state of cell services.
 2016-02-05 11:27:32 +0100        :Working: DO: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction. Up to 40 minutes ...
 2016-02-05 11:27:45 +0100 Wait correction of degraded md11 due to md partner size mismatch. Up to 30 minutes.
2016-02-05 11:27:46 +0100        :SUCCESS: DONE: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction.
 2016-02-05 11:27:46 +0100        :Working: DO: Check prerequisites on all cells. Up to 2 minutes ...
 2016-02-05 11:28:50 +0100        :SUCCESS: DONE: Check prerequisites on all cells.
 2016-02-05 11:28:50 +0100        :Working: DO: Copy the patch to all cells. Up to 3 minutes ...
 2016-02-05 11:29:22 +0100        :SUCCESS: DONE: Copy the patch to all cells.
 2016-02-05 11:29:24 +0100        :Working: DO: Execute plugin check for Patch Check Prereq ...
 2016-02-05 11:29:24 +0100        :INFO: Patchmgr plugin start: Prereq check for exposure to bug 17854520 v1.3. Details in logfile /u02/p17885582_121210_Linux-x86-64/patch_12.1.2.1.0.141206.1/patchmgr.stdout.
 2016-02-05 11:29:24 +0100        :SUCCESS: No exposure to bug 17854520 with non-rolling patching
 2016-02-05 11:29:25 +0100        :SUCCESS: DONE: Execute plugin check for Patch Check Prereq.
 2016-02-05 11:29:25 +0100 1 of 5 :Working: DO: Initiate patch on cells. Cells will remain up. Up to 5 minutes ...
 2016-02-05 11:29:37 +0100 1 of 5 :SUCCESS: DONE: Initiate patch on cells.
 2016-02-05 11:29:37 +0100 2 of 5 :Working: DO: Waiting to finish pre-reboot patch actions. Cells will remain up. Up to 45 minutes ...
 2016-02-05 11:30:37 +0100 Wait for patch pre-reboot procedures
2016-02-05 11:44:56 +0100 2 of 5 :SUCCESS: DONE: Waiting to finish pre-reboot patch actions.
 2016-02-05 11:44:56 +0100        :Working: DO: Execute plugin check for Patching ...
 2016-02-05 11:44:56 +0100        :SUCCESS: DONE: Execute plugin check for Patching.
 2016-02-05 11:44:56 +0100 3 of 5 :Working: DO: Finalize patch on cells. Cells will reboot. Up to 5 minutes ...
 2016-02-05 11:45:17 +0100 3 of 5 :SUCCESS: DONE: Finalize patch on cells.
 2016-02-05 11:45:17 +0100 4 of 5 :Working: DO: Wait for cells to reboot and come online. Up to 120 minutes ...
 2016-02-05 11:46:17 +0100 Wait for patch finalization and reboot
2016-02-05 13:09:24 +0100 4 of 5 :SUCCESS: DONE: Wait for cells to reboot and come online.
 2016-02-05 13:09:24 +0100 5 of 5 :Working: DO: Check the state of patch on cells. Up to 5 minutes ...
 2016-02-05 13:10:09 +0100 5 of 5 :SUCCESS: DONE: Check the state of patch on cells.
 2016-02-05 13:10:09 +0100        :Working: DO: Execute plugin check for Post Patch ...
 2016-02-05 13:10:10 +0100        :SUCCESS: DONE: Execute plugin check for Post Patch.
[root@ch01db01 patch_12.1.2.1.0.141206.1]#
[root@ch01db01 patch_12.1.2.1.0.141206.1]# dcli -c ch01celadm01 -l root 'imageinfo'
 ch01celadm01:
 ch01celadm01: Kernel version: 2.6.39-400.243.1.el6uek.x86_64 #1 SMP Wed Nov 26 09:15:35 PST 2014 x86_64
 ch01celadm01: Cell version: OSS_12.1.2.1.0_LINUX.X64_141206.1
 ch01celadm01: Cell rpm version: cell-12.1.2.1.0_LINUX.X64_141206.1-1.x86_64
 ch01celadm01:
 ch01celadm01: Active image version: 12.1.2.1.0.141206.1
 ch01celadm01: Active image activated: 2016-02-05 20:14:52 +0100
 ch01celadm01: Active image status: success
 ch01celadm01: Active system partition on device: /dev/md5
 ch01celadm01: Active software partition on device: /dev/md7
 ch01celadm01:
 ch01celadm01: Cell boot usb partition: /dev/sdac1
 ch01celadm01: Cell boot usb version: 12.1.2.1.0.141206.1
 ch01celadm01:
 ch01celadm01: Inactive image version: 12.1.1.1.1.140712
 ch01celadm01: Inactive image activated: 2014-08-06 11:50:09 +0200
 ch01celadm01: Inactive image status: success
 ch01celadm01: Inactive system partition on device: /dev/md6
 ch01celadm01: Inactive software partition on device: /dev/md8
 ch01celadm01:
 ch01celadm01: Inactive marker for the rollback: /boot/I_am_hd_boot.inactive
 ch01celadm01: Inactive grub config for the rollback: /boot/grub/grub.conf.inactive
 ch01celadm01: Inactive kernel version for the rollback: 2.6.39-400.128.17.el5uek
 ch01celadm01: Rollback to the inactive partitions: Possible
 [root@ch01db01 patch_12.1.2.1.0.141206.1]#

-----------------------------------------------------------
-- DB Server Patching
-----------------------------------------------------------

[root@ch01db02 dbnodeupdate]# ./dbnodeupdate.sh -h

Usage: dbnodeupdate.sh [ -u | -r | -c ] -l <baseurl|zip file> [-p] <phase> [-n] [-s] [-q] [-v] [-t] [-a] <alert.sh> [-b] [-m] | [-V] | [-h]
-u                       Upgrade
 -r                       Rollback
 -c                       Complete post actions (verify image status, cleanup, apply fixes, relink all homes, enable GI to start/start all domU's)
 -l <baseurl|zip file>    Baseurl (http or zipped iso file for the repository)
 -s                       Shutdown stack (domU's for VM) before upgrading/rolling back
 -p                       Bootstrap phase (1 or 2) only to be used when instructed by dbnodeupdate.sh
 -q                       Quiet mode (no prompting) only be used in combination with -t
 -n                       No backup will be created (Option disabled for systems being updated from Oracle Linux 5 to Oracle Linux 6)
 -t                       'to release' - used when in quiet mode or used when updating to one-offs/releases via 'latest' channel (requires 11.2.3.2.1)
 -v                       Verify prereqs only. Only to be used with -u and -l option
 -b                       Perform backup only
 -a <alert.sh>            Full path to shell script used for alert trapping
 -m                       Install / update-to exadata-sun/hp-computenode-minimum only (11.2.3.3.0 and later)
 -i                       Ignore /etc/oratab - relinking will be disabled. Only possible in combination with -c.
 -V                       Print version
 -h                       Print usage
For upgrading from releases 11.2.2.4.2 and later:
 Example using iso  : ./dbnodeupdate.sh -u -l /u01/p16432033_112321_Linux-x86-64.zip
 Example using http : ./dbnodeupdate.sh -u -l http://my-yum-repo.my-domain.com/yum/unknown/EXADATA/dbserver/11.2.3.3.0/base/x86_64/
 Example: ./dbnodeupdate.sh -u -l http://my-yum-repo.my-domain.com/yum/unknown/EXADATA/dbserver/11.2.3.2.1/base/x86_64/
 Example: ./dbnodeupdate.sh -u -l http://my-yum-repo.my-domain.com/yum/unknown/EXADATA/dbserver/11.2.3.3.0/base/x86_64/
For upgrading from releases 11.2.2.4.2 and later in quiet mode:
 Example: ./dbnodeupdate.sh -u -l /u01/p16432033_112321_Linux-x86-64.zip -q -t 11.2.3.2.1.130302
For completion steps:
 Example: ./dbnodeupdate.sh -c
For rollback:
 Example: ./dbnodeupdate.sh -r
For pre-req checks only:
 Example using iso  : ./dbnodeupdate.sh -u -l /u01/p16432033_112321_Linux-x86-64.zip -v
 Example using http : ./dbnodeupdate.sh -u -l http://my-yum-repo.my-domain.com/yum/unknown/EXADATA/dbserver/11.2.3.3.0/base/x86_64/ -v
For backup only:
 Example: ./dbnodeupdate.sh -u -l /u01/p16432033_112321_Linux-x86-64.zip -b
See MOS 1553103.1 for more examples
[root@ch01db02 dbnodeupdate]#

———————————– –DB Server patching Verification ———————————–

[root@ch01db02 dbnodeupdate]# ./dbnodeupdate.sh -u -l /u01/exapatch/p20170913_121210_Linux-x86-64/p20170913_121210_Linux-x86-64.zip -v
##########################################################################################################################
 #                                                                                                                        #
 # Guidelines for using dbnodeupdate.sh (rel. 4.18):                                                                      #
 #                                                                                                                        #
 # - Prerequisites for usage:                                                                                             #
 #         1. Refer to dbnodeupdate.sh options. See MOS 1553103.1                                                         #
 #         2. Use the latest release of dbnodeupdate.sh. See patch 16486998                                               #
 #         3. Run the prereq check with the '-v' option.                                                                  #
 #                                                                                                                        #
 #   I.e.:  ./dbnodeupdate.sh -u -l /u01/my-iso-repo.zip -v                                                               #
 #          ./dbnodeupdate.sh -u -l http://my-yum-repo -v                                                                 #
 #                                                                                                                        #
 # - Prerequisite dependency check failures can happen due to customization:                                              #
 #     - The prereq check detects dependency issues that need to be addressed prior to running a successful update.       #
 #     - Customized rpm packages may fail the built-in dependency check and system updates cannot proceed until resolved. #
 #                                                                                                                        #
 #   When upgrading from releases later than 11.2.2.4.2 to releases before 11.2.3.3.0:                                    #
 #      - Conflicting packages should be removed before proceeding the update.                                            #
 #                                                                                                                        #
 #   When upgrading to releases 11.2.3.3.0 or later:                                                                      #
 #      - When the 'exact' package dependency check fails 'minimum' package dependency check will be tried.               #
 #      - When the 'minimum' package dependency check also fails,                                                         #
 #        the conflicting packages should be removed before proceeding.                                                   #
 #                                                                                                                        #
 # - As part of the prereq checks and as part of the update, a number of rpms will be removed.                            #
 #   This removal is required to preserve Exadata functioning. This should not be confused with obsolete packages.        #
 #      - See /var/log/cellos/packages_to_be_removed.txt for details on what packages will be removed.                    #
 #                                                                                                                        #
 # - In case of any problem when filing an SR, upload the following:                                                      #
 #      - /var/log/cellos/dbnodeupdate.log                                                                                #
 #      - /var/log/cellos/dbnodeupdate.<runid>.diag                                                                       #
 #      - where <runid> is the unique number of the failing run.                                                          #
 #                                                                                                                        #
 ##########################################################################################################################
Continue ? [y/n]
 y
(*) 2016-02-05 17:06:43: Unzipping helpers (/u01/exapatch/dbnodeupdate/dbupdate-helpers.zip) to /opt/oracle.SupportTools/dbnodeupdate_helpers
 (*) 2016-02-05 17:06:43: Initializing logfile /var/log/cellos/dbnodeupdate.log
 (*) 2016-02-05 17:06:44: Collecting system configuration settings. This may take a while...
 (*) 2016-02-05 17:07:10: Validating system settings for known issues and best practices. This may take a while...
 (*) 2016-02-05 17:07:10: Checking free space in /u01/exapatch/p20170913_121210_Linux-x86-64/iso.stage.050215170641
 (*) 2016-02-05 17:07:10: Unzipping /u01/exapatch/p20170913_121210_Linux-x86-64/p20170913_121210_Linux-x86-64.zip to /u01/exapatch/p20170913_121210_Linux-x86-64/iso.stage.050215170641, this may take a while
 (*) 2016-02-05 17:07:23: Original /etc/yum.conf moved to /etc/yum.conf.050215170641, generating new yum.conf
 (*) 2016-02-05 17:07:23: Generating Exadata repository file /etc/yum.repos.d/Exadata-computenode.repo
 (*) 2016-02-05 17:07:56: Validating the specified source location.
 (*) 2016-02-05 17:07:57: Cleaning up the yum cache.

—————————————————————————————————————————–
Running in prereq check mode
—————————————————————————————————————————–

Active Image version   : 12.1.1.1.1.140712
 Active Kernel version  : 2.6.39-400.128.17.el5uek
 Active LVM Name        : /dev/mapper/VGExaDb-LVDbSys1
 Inactive Image version : 12.1.1.1.0.131219
 Inactive LVM Name      : /dev/mapper/VGExaDb-LVDbSys2
 Current user id        : root
 Action                 : upgrade
 Upgrading to           : 12.1.2.1.0.141206.1 - Oracle Linux 5->6 upgrade
 Baseurl                : file:///var/www/html/yum/unknown/EXADATA/dbserver/050215170641/x86_64/ (iso)
 Iso file               : /u01/exapatch/p20170913_121210_Linux-x86-64/iso.stage.050215170641/exadata_ol6_base_repo_141206.1.iso
 Create a backup        : Yes (backup at update mandatory when updating from OL5 to OL6)
 Shutdown stack         : No (Currently stack is down)
 RPM exclusion list     : Function not available for OL5->OL6 upgrades
 RPM obsolete list      : Function not available for OL5->OL6 upgrades
 Exact dependencies     : Function not available for OL5->OL6 upgrades
 Minimum dependencies   : Function not available for OL5->OL6 upgrades
 Logfile                : /var/log/cellos/dbnodeupdate.log (runid: 050215170641)
 Diagfile               : /var/log/cellos/dbnodeupdate.050215170641.diag
 Server model           : SUN FIRE X4170 M3
 dbnodeupdate.sh rel.   : 4.18 (always check MOS 1553103.1 for the latest release of dbnodeupdate.sh)
 Note                   : After upgrading and rebooting run './dbnodeupdate.sh -c' to finish post steps.
The following known issues will be checked for and automatically corrected by dbnodeupdate.sh:
 (*) - Issue - Fix for CVE-2014-9295 AND ELSA-2014-1974
The following known issues will be checked for but require manual follow-up:
 (*) - Issue - Yum rolling update requires fix for 11768055 when Grid Infrastructure is below 11.2.0.2 BP12
 (*) - Issue - Exafusion silently enabled for database 12.1.0.2.0 with kernel 2.6.39-400.200 and later. See MOS 1947476.1 for more details.
---------------------------------------------------------------------------------------------------------------------
 NOTE:
 When upgrading to Oracle Linux 6 a backup is required for systems configured with logical volume manager (lvm).
 It appears no backup of the current image exist on the inactive lvm.
 This means a mandatory backup will be made using dbnodeupdate.sh before the actual update starts.
 ---------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------

-------------------------------------------
 Prereq check finished successfully, check the above report for next steps.
 -----------------------------------------------------------------------------------------------------------------------------
(*) 2016-02-05 17:08:01: Cleaning up iso and temp mount points
[root@ch01db02 dbnodeupdate]#

———————————–

–DB Server patching Execution

———————————–

[root@ch01db02 dbnodeupdate]# ./dbnodeupdate.sh -u -l /u01/exapatch/p20170913_121210_Linux-x86-64/p20170913_121210_Linux-x86-64.zip
##########################################################################################################################
 #                                                                                                                        #
 # Guidelines for using dbnodeupdate.sh (rel. 4.18):                                                                      #
 #                                                                                                                        #
 # - Prerequisites for usage:                                                                                             #
 #         1. Refer to dbnodeupdate.sh options. See MOS 1553103.1                                                         #
 #         2. Use the latest release of dbnodeupdate.sh. See patch 16486998                                               #
 #         3. Run the prereq check with the '-v' option.                                                                  #
 #                                                                                                                        #
 #   I.e.:  ./dbnodeupdate.sh -u -l /u01/my-iso-repo.zip -v                                                               #
 #          ./dbnodeupdate.sh -u -l http://my-yum-repo -v                                                                 #
 #                                                                                                                        #
 # - Prerequisite dependency check failures can happen due to customization:                                              #
 #     - The prereq check detects dependency issues that need to be addressed prior to running a successful update.       #
 #     - Customized rpm packages may fail the built-in dependency check and system updates cannot proceed until resolved. #
 #                                                                                                                        #
 #   When upgrading from releases later than 11.2.2.4.2 to releases before 11.2.3.3.0:                                    #
 #      - Conflicting packages should be removed before proceeding the update.                                            #
 #                                                                                                                        #
 #   When upgrading to releases 11.2.3.3.0 or later:                                                                      #
 #      - When the 'exact' package dependency check fails 'minimum' package dependency check will be tried.               #
 #      - When the 'minimum' package dependency check also fails,                                                         #
 #        the conflicting packages should be removed before proceeding.                                                   #
 #                                                                                                                        #
 # - As part of the prereq checks and as part of the update, a number of rpms will be removed.                            #
 #   This removal is required to preserve Exadata functioning. This should not be confused with obsolete packages.        #
 #      - See /var/log/cellos/packages_to_be_removed.txt for details on what packages will be removed.                    #
 #                                                                                                                        #
 # - In case of any problem when filing an SR, upload the following:                                                      #
 #      - /var/log/cellos/dbnodeupdate.log                                                                                #
 #      - /var/log/cellos/dbnodeupdate.<runid>.diag                                                                       #
 #      - where <runid> is the unique number of the failing run.                                                          #
 #                                                                                                                        #
 ##########################################################################################################################
Continue ? [y/n]
 y
(*) 2016-02-05 17:09:38: Unzipping helpers (/u01/exapatch/dbnodeupdate/dbupdate-helpers.zip) to /opt/oracle.SupportTools/dbnodeupdate_helpers
 (*) 2016-02-05 17:09:38: Initializing logfile /var/log/cellos/dbnodeupdate.log
 (*) 2016-02-05 17:09:39: Collecting system configuration settings. This may take a while...
 (*) 2016-02-05 17:10:07: Validating system settings for known issues and best practices. This may take a while...
 (*) 2016-02-05 17:10:07: Checking free space in /u01/exapatch/p20170913_121210_Linux-x86-64/iso.stage.050215170936
 (*) 2016-02-05 17:10:07: Unzipping /u01/exapatch/p20170913_121210_Linux-x86-64/p20170913_121210_Linux-x86-64.zip to /u01/exapatch/p20170913_121210_Linux-x86-64/iso.stage.050215170936, this may take a while
 (*) 2016-02-05 17:10:19: Original /etc/yum.conf moved to /etc/yum.conf.050215170936, generating new yum.conf
 (*) 2016-02-05 17:10:19: Generating Exadata repository file /etc/yum.repos.d/Exadata-computenode.repo
 (*) 2016-02-05 17:10:42: Validating the specified source location.
 (*) 2016-02-05 17:10:43: Cleaning up the yum cache.
Active Image version   : 12.1.1.1.1.140712
 Active Kernel version  : 2.6.39-400.128.17.el5uek
 Active LVM Name        : /dev/mapper/VGExaDb-LVDbSys1
 Inactive Image version : 12.1.1.1.0.131219
 Inactive LVM Name      : /dev/mapper/VGExaDb-LVDbSys2
 Current user id        : root
 Action                 : upgrade
 Upgrading to           : 12.1.2.1.0.141206.1 - Oracle Linux 5->6 upgrade
 Baseurl                : file:///var/www/html/yum/unknown/EXADATA/dbserver/050215170936/x86_64/ (iso)
 Iso file               : /u01/exapatch/p20170913_121210_Linux-x86-64/iso.stage.050215170936/exadata_ol6_base_repo_141206.1.iso
 Create a backup        : Yes (backup at update mandatory when updating from OL5 to OL6)
 Shutdown stack         : No (Currently stack is down)
 RPM exclusion list     : Function not available for OL5->OL6 upgrades
 RPM obsolete list      : Function not available for OL5->OL6 upgrades
 Exact dependencies     : Function not available for OL5->OL6 upgrades
 Minimum dependencies   : Function not available for OL5->OL6 upgrade
 Logfile                : /var/log/cellos/dbnodeupdate.log (runid: 050215170936)
 Diagfile               : /var/log/cellos/dbnodeupdate.050215170936.diag
 Server model           : SUN FIRE X4170 M3
 dbnodeupdate.sh rel.   : 4.18 (always check MOS 1553103.1 for the latest release of dbnodeupdate.sh)
 Note                   : After upgrading and rebooting run './dbnodeupdate.sh -c' to finish post steps.
The following known issues will be checked for and automatically corrected by dbnodeupdate.sh:
 (*) - Issue - Fix for CVE-2014-9295 AND ELSA-2014-1974
The following known issues will be checked for but require manual follow-up:
 (*) - Issue - Yum rolling update requires fix for 11768055 when Grid Infrastructure is below 11.2.0.2 BP12
 (*) - Issue - Exafusion silently enabled for database 12.1.0.2.0 with kernel 2.6.39-400.200 and later. See MOS 1947476.1 for more details.
Continue ? [y/n]
 y
(*) 2016-02-05 17:11:59: Verifying GI and DB''s are shutdown
 (*) 2016-02-05 17:12:00: Collecting console history for diag purposes
 (*) 2016-02-05 17:12:32: Unmount of /boot successful
 (*) 2016-02-05 17:12:32: Check for /dev/sda1 successful
 (*) 2016-02-05 17:12:32: Mount of /boot successful
 (*) 2016-02-05 17:12:32: Disabling stack from starting
 (*) 2016-02-05 17:12:33: Performing filesystem backup to /dev/mapper/VGExaDb-LVDbSys2. Avg. 30 minutes (maximum 120) depends per environment.......
 (*) 2016-02-05 17:18:44: Backup successful
 (*) 2016-02-05 17:18:47: ExaWatcher stopped successful
 (*) 2016-02-05 17:19:07: EM Agent (in /u01/app/oracle/product/agent12c/core/12.1.0.4.0) stopped successfully
 (*) 2016-02-05 17:19:07: Capturing service status and file attributes. This may take a while...
 (*) 2016-02-05 17:19:12: Service status and file attribute report in: /etc/exadata/reports
 (*) 2016-02-05 17:19:12: Validating the specified source location.
 (*) 2016-02-05 17:19:13: Cleaning up the yum cache.
 (*) 2016-02-05 17:19:14: Executing OL5->OL6 upgrade steps, system is expected to reboot multiple times.
 (*) 2016-02-05 17:21:37: Initialize of Oracle Linux 6 Upgrade successful. Rebooting now...
Broadcast message from root (pts/0) (Thu Feb  5 17:21:37 2015):
The system is going down for reboot NOW!
[root@ch01db02 dbnodeupdate]#
[root@ch01db02 dbnodeupdate]# ./dbnodeupdate.sh -c

———————————–
–Output new Image Version
———————————–

[root@ch01db01 ibdiagtools]# imageinfo
Kernel version: 2.6.39-400.243.1.el6uek.x86_64 #1 SMP Wed Nov 26 09:15:35 PST 2014 x86_64
 Image version: 12.1.2.1.0.141206.1
 Image activated: 2016-02-05 18:24:46 +0100
 Image status: success
 System partition on device: /dev/mapper/VGExaDb-LVDbSys1

ODA CPU Capping

####################################################################
# How to reduce the number of active CPU cores on ODA system
####################################################################

--Find the ODA Serial Number
 [root@odanode1 ~]# /usr/sbin/dmidecode -t1 |grep Serial
 Serial Number: 1xxxxxXXXXxG
--Login to MOS and generate the CPU Key using the ODA Serial Number.
####################################################################
 ## ODA CPU Capping
 ####################################################################
------------------------------------
 -- Target  active CPU cores --
 ------------------------------------
   HOSTNAME    |   CPU COUNT  
 ---------------|----------------
 odanode1        |      6
 ---------------|----------------
 odanode2        |      6
 --------------------------------
-------------------------------------------------------------------------------
 --Reduce the CPU cores running the following command from the first node only!
 -------------------------------------------------------------------------------
 /opt/oracle/oak/bin/oakcli show core_config_key
 /opt/oracle/oak/bin/oakcli apply core_config_key /tmp/CPU_KEY
------------------------------------------
 --Activity Log
 ------------------------------------------
 [root@odanode1 tmp]# vi CPU_KEY  <--- Store the CPU key generated on MOS
 [root@odanode1 tmp]# /opt/oracle/oak/bin/oakcli show core_config_key
 Optional core_config_key is not applied on this machine yet !
 [root@odanode1 tmp]# pwd
 /tmp
 [root@odanode1 tmp]# /opt/oracle/oak/bin/oakcli apply core_config_key /tmp/CPU_KEY
 INFO: Both nodes get rebooted automatically after applying the license
 Do you want to continue: [Y/N]?:
 Y
 INFO: User has confirmed the reboot
Please enter the root password:
............done
INFO: Applying core_config_key on '192.168.16.25'
 ...
 INFO   : Running as root: /usr/bin/ssh -l root 192.168.16.25 /tmp/tmp_lic_exec.pl
 INFO   : Running as root: /usr/bin/ssh -l root 192.168.16.25 /opt/oracle/oak/bin/oakcli enforce core_config_key /tmp/.lic_file
 Waiting for the Node '192.168.16.25' to reboot...........................
 Node '192.168.16.25' is  rebooted
 Waiting for the Node '192.168.16.25' to be up before applying the license on the node '192.168.16.24'.
 .............................................
 INFO: Applying core_config_key on '192.168.16.24'
 ...
 INFO   : Running as root: /usr/bin/ssh -l root 192.168.16.24 /tmp/tmp_lic_exec.pl
 INFO   : Running as root: /usr/bin/ssh -l root 192.168.16.24 /opt/oracle/oak/bin/oakcli enforce core_config_key /tmp/.lic_file
Broadcast message from root (Fri Jun  7 15:18:34 2013):
The system is going down for reboot NOW!
 [root@odanode1 tmp]#
-------------------------------------------------------------
 --Check the new Number of active cores
 -------------------------------------------------------------
[root@odanode1 ~]# /opt/oracle/oak/bin/oakcli show core_config_key
Host's serialnumber                    =                     1xxxxxXXXXxG
 Enabled Cores (per server)             =                                   6
 Total Enabled Cores (on two servers)   =                             12
 Server type                            =        V1 -> SUN FIRE X4370 M2
 Hyperthreading is enabled.  Each core has 2 threads. Operating system displays 12 processors per server
 [root@odanode1 ~]#

	

Oracle Database Appliance Bundle 2.6

##################################################################
# Installation Oracle Database Appliance (ODA) bundle patch 2.6.0.0
##################################################################

--Path where all ODA logs are stored:
 /opt/oracle/oak/log/odanode1/patch/2.6.0.0.0
-------------------------------------------------
 --ODA Software version before patching
 -------------------------------------------------
 [root@odanode1 bin]# /opt/oracle/oak/bin/oakcli show version -detail
 reporting the metadata. It takes a while...
 System Version          Component Name                Installed Version               Supported Version
 -------------------         ---------------------    ------------------            -----------------
 2.4.0.0.0
 Controller                      11.05.02.00                     Up-to-date
 Expander                       0342                              Up-to-date
 SSD_SHARED                   E125                              Up-to-date
 HDD_LOCAL                     5G08                              Up-to-date
 HDD_SHARED                   A700                              A6C0
 ILOM                              3.0.16.22.a r75629           Up-to-date
 BIOS                              12010310                        Up-to-date
 IPMI                               1.8.10.5                          Up-to-date
 HMP                               2.2.4                              Up-to-date
 OAK                               2.4.0.0.0                         Up-to-date
 OEL                                5.8                                Up-to-date
 TFA                                2.4                                Up-to-date
 GI_HOME                         1.2.0.3.4(14275605,          Up-to-date
 14275572)
 DB_HOME                       11.2.0.3.4(14275605,         Up-to-date
 14275572)
 ASR                                Unknown                         3.9
 [root@odanode1 bin]#
####################################################################################################
-------------------------------------------------
 --Unzip the patch bundle 2.6.0.0.
 -------------------------------------------------
 --ODA Node 1
 [root@odanode1 u01]# cd /opt/oracle/oak/bin
 [root@odanode1 bin]# ./oakcli unpack -package /u01/ODA_patches/bundle_2600/p16744915_26000_Linux-x86-64.zip
 Unpacking takes a while,  pls wait....
 Successfully unpacked the files to repository.
 [root@odanode1 bin]#
--ODA Node 2
 [root@odanode2 u01]# cd /opt/oracle/oak/bin
 [root@odanode2 bin]# ./oakcli unpack -package /u01/ODA_patches/bundle_2600/p16744915_26000_Linux-x86-64.zip
 Unpacking takes a while,  pls wait....
 Successfully unpacked the files to repository.
 [root@odanode2 bin]#
-------------------------------------------------
 --Apply the Patch to the Infrastructure
 -------------------------------------------------
--ODA Node 1 ONLY
 [root@odanode1 bin]# cd /opt/oracle/oak/bin
 [root@odanode1 bin]# ./oakcli update -patch 2.6.0.0.0 --infra
 INFO: DB, ASM, Clusterware may be stopped during the patch if required
 INFO: Both nodes may get rebooted automatically during the patch if required
 Do you want to continue: [Y/N]?: Y
 INFO: User has confirmed the reboot
 INFO: Patch bundle must be unpacked on the second node also before applying this patch
 Did you unpack the patch bundle on the second node?: [Y/N]?: Y
Please enter the 'root' user password:
 Please re-enter the 'root' user password:
 INFO: Setting up the SSH
 ..........done
 INFO: Running pre-install scripts
 ..........done
 INFO: 2013-05-15 11:04:11: Running pre patch script for 2.6.0.0.0
 INFO: 2013-05-15 11:04:14: Completed pre patch script for 2.6.0.0.0
INFO: 2013-05-15 11:04:19: ------------------Patching HMP-------------------------
 SUCCESS: 2013-05-15 11:04:50: Successfully upgraded the HMP
INFO: 2013-05-15 11:04:50: ----------------------Patching OAK---------------------
 SUCCESS: 2013-05-15 11:05:13: Succesfully upgraded OAK
INFO: 2013-05-15 11:05:15: -----------------Installing / Patching  TFA-----------------
 SUCCESS: 2013-05-15 11:06:55: Successfully updated / installed the TFA
 ...
INFO: 2013-05-15 11:06:56: ------------------Patching OS-------------------------
 INFO: 2013-05-15 11:07:05: Clusterware is running on one or more nodes of the cluster
 INFO: 2013-05-15 11:07:05: Attempting to stop clusterware and its resources across the cluster
 SUCCESS: 2013-05-15 11:09:08: Successfully stopped the clusterware
SUCCESS: 2013-05-15 11:09:49: Successfully upgraded the OS
INFO: 2013-05-15 11:09:53: ----------------------Patching IPMI---------------------
 SUCCESS: 2013-05-15 11:09:55: Succesfully upgraded IPMI
INFO: 2013-05-15 11:10:02: ----------------Patching the Storage-------------------
 INFO: 2013-05-15 11:10:02: ....................Patching SSDs...............
 INFO: 2013-05-15 11:10:02: Updating the  Disk : d20 with the firmware : ZeusIOPs G3 E12B
 SUCCESS: 2013-05-15 11:10:27: Successfully updated the firmware on  Disk : d20 to ZeusIOPs G3 E12B
 INFO: 2013-05-15 11:10:27: Updating the  Disk : d21 with the firmware : ZeusIOPs G3 E12B
 SUCCESS: 2013-05-15 11:10:48: Successfully updated the firmware on  Disk : d21 to ZeusIOPs G3 E12B
 INFO: 2013-05-15 11:10:48: Updating the  Disk : d22 with the firmware : ZeusIOPs G3 E12B
 SUCCESS: 2013-05-15 11:11:10: Successfully updated the firmware on  Disk : d22 to ZeusIOPs G3 E12B
 INFO: 2013-05-15 11:11:11: Updating the  Disk : d23 with the firmware : ZeusIOPs G3 E12B
 SUCCESS: 2013-05-15 11:11:34: Successfully updated the firmware on  Disk : d23 to ZeusIOPs G3 E12B
 INFO: 2013-05-15 11:11:34: ....................Patching shared HDDs...............
 INFO: 2013-05-15 11:11:34: Disk : d0  is alreporty running with : HUS1560SCSUN600G A700
 INFO: 2013-05-15 11:11:34: Disk : d1  is alreporty running with : HUS1560SCSUN600G A700
 INFO: 2013-05-15 11:11:34: Disk : d2  is alreporty running with : HUS1560SCSUN600G A700
 INFO: 2013-05-15 11:11:35: Disk : d3  is alreporty running with : HUS1560SCSUN600G A700
 INFO: 2013-05-15 11:11:35: Disk : d4  is alreporty running with : HUS1560SCSUN600G A700
 INFO: 2013-05-15 11:11:35: Disk : d5  is alreporty running with : HUS1560SCSUN600G A700
 INFO: 2013-05-15 11:11:35: Disk : d6  is alreporty running with : HUS1560SCSUN600G A700
 INFO: 2013-05-15 11:11:35: Disk : d7  is alreporty running with : HUS1560SCSUN600G A700
 INFO: 2013-05-15 11:11:36: Disk : d8  is alreporty running with : HUS1560SCSUN600G A700
 INFO: 2013-05-15 11:11:36: Disk : d9  is alreporty running with : HUS1560SCSUN600G A700
 INFO: 2013-05-15 11:11:36: Disk : d10  is alreporty running with : HUS1560SCSUN600G A700
 INFO: 2013-05-15 11:11:36: Disk : d11  is alreporty running with : HUS1560SCSUN600G A700
 INFO: 2013-05-15 11:11:37: Disk : d12  is alreporty running with : HUS1560SCSUN600G A700
 INFO: 2013-05-15 11:11:37: Disk : d13  is alreporty running with : HUS1560SCSUN600G A700
 INFO: 2013-05-15 11:11:37: Disk : d14  is alreporty running with : HUS1560SCSUN600G A700
 INFO: 2013-05-15 11:11:37: Disk : d15  is alreporty running with : HUS1560SCSUN600G A700
 INFO: 2013-05-15 11:11:38: Disk : d16  is alreporty running with : HUS1560SCSUN600G A700
 INFO: 2013-05-15 11:11:38: Disk : d17  is alreporty running with : HUS1560SCSUN600G A700
 INFO: 2013-05-15 11:11:38: Disk : d18  is alreporty running with : HUS1560SCSUN600G A700
 INFO: 2013-05-15 11:11:38: Disk : d19  is alreporty running with : HUS1560SCSUN600G A700
 INFO: 2013-05-15 11:11:38: ....................Patching local HDDs...............
 INFO: 2013-05-15 11:11:38: Disk : c0d0  is alreporty running with : WD500BLHXSUN 5G08
 INFO: 2013-05-15 11:11:39: Disk : c0d1  is alreporty running with : WD500BLHXSUN 5G08
 INFO: 2013-05-15 11:11:39: ....................Patching Expanders...............
 INFO: 2013-05-15 11:11:39: Expander : x0  is alreporty running with : T4 Storage 0342
 INFO: 2013-05-15 11:11:39: Expander : x1  is alreporty running with : T4 Storage 0342
 INFO: 2013-05-15 11:11:39: ....................Patching Controllers...............
 INFO: 2013-05-15 11:11:39: No-update for the Controller: c0
 INFO: 2013-05-15 11:11:39: Controller : c1  is alreporty running with : 0x0072 11.05.02.00
 INFO: 2013-05-15 11:11:39: Controller : c2  is alreporty running with : 0x0072 11.05.02.00
 INFO: 2013-05-15 11:11:39: ------------Finished the storage Patching------------
INFO: 2013-05-15 11:11:40: -----------------Patching Ilom & Bios-----------------
 INFO: 2013-05-15 11:11:41: Getting the ILOM Ip address
 INFO: 2013-05-15 11:11:42: Updating the Ilom using LAN+ protocol
 INFO: 2013-05-15 11:11:43: Updating the ILOM. It takes a while
 INFO: 2013-05-15 11:16:24: Verifying the updated Ilom Version, it may take a while if ServiceProcessor is booting
 INFO: 2013-05-15 11:16:25: Waiting for the service processor to be up
 SUCCESS: 2013-05-15 11:20:09: Successfully updated the ILOM with the firmware 3.0.16.22.b r78329
INFO: Patching the infrastructure on node: odanode2 , it may take upto 30 minutes. Please wait
 ...
 ............done
INFO: Infrastructure patching summary on node: 192.168.16.24
 SUCCESS: 2013-05-15 11:31:05:  Successfully upgraded the HMP
 SUCCESS: 2013-05-15 11:31:05:  Succesfully updated the OAK
 SUCCESS: 2013-05-15 11:31:05:  Successfully updated the TFA
 SUCCESS: 2013-05-15 11:31:05:  Successfully upgraded the OS
 SUCCESS: 2013-05-15 11:31:05:  Succesfully updated the IPMI
 INFO: 2013-05-15 11:31:05:  Storage patching summary
 SUCCESS: 2013-05-15 11:31:05:  No failures during storage upgrade
 SUCCESS: 2013-05-15 11:31:05:  Successfully updated the ILOM & Bios
INFO: Infrastructure patching summary on node: 192.168.16.25
 SUCCESS: 2013-05-15 11:31:05:  Successfully upgraded the HMP
 SUCCESS: 2013-05-15 11:31:05:  Succesfully updated the OAK
 SUCCESS: 2013-05-15 11:31:05:  Successfully upgraded the OS
 SUCCESS: 2013-05-15 11:31:05:  Succesfully updated the IPMI
 INFO: 2013-05-15 11:31:05:  Storage patching summary
 SUCCESS: 2013-05-15 11:31:05:  No failures during storage upgrade
 SUCCESS: 2013-05-15 11:31:05:  Successfully updated the ILOM & Bios
INFO: Running post-install scripts
 ............done
 INFO: Some of the patched components require node reboot. Rebooting the nodes
 INFO: Setting up the SSH
 ............done
Broadcast message from root (Wed May 15 11:35:50 2013):
The system is going down for system halt NOW!
-------------------------------------------------
 --Apply the Patch to the Grid Infrastructure
 -------------------------------------------------
--ODA on BOTH Nodes
 [oracle@odanode1 OPatch]$ /u01/app/oracle/product/agent12c/agent_inst/bin/emctl stop agent
 Oracle Enterprise Manager Cloud Control 12c Release 2
 Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
 Stopping agent ..... stopped.
--ODA Node 1 ONLY
 [root@odanode1 bin]# cd /opt/oracle/oak/bin
 [root@odanode1 bin]# ./oakcli update -patch 2.6.0.0.0 --gi
Please enter the 'root' user password:
 Please re-enter the 'root' user password:
Please enter the 'grid' user password:
 Please re-enter the 'grid' user password:
 INFO: Setting up the SSH
 ..........done
 ...
 ...
..........done
 ...
 SUCCESS: All nodes in /opt/oracle/oak/temp_clunodes.txt are pingable and alive.
 INFO: 2013-05-15 11:56:10: Setting up the ssh for grid user
 ..........done
 ...
 SUCCESS: All nodes in /opt/oracle/oak/temp_clunodes.txt are pingable and alive.
 INFO: 2013-05-15 11:56:30: Patching the GI home on node odanode1
 INFO: 2013-05-15 11:56:30: Updating the opatch
 INFO: 2013-05-15 11:56:56: Performing the conflict checks
 SUCCESS: 2013-05-15 11:57:07: Conflict checks passed for all the homes
 INFO: 2013-05-15 11:57:07: Checking if the patch is alreporty applied on any of the homes
 INFO: 2013-05-15 11:57:10: No home is alreporty up-to-date
 SUCCESS: 2013-05-15 11:57:21: Successfully stopped the dbconsoles
 SUCCESS: 2013-05-15 11:57:36: Successfully stopped the EM agents
 INFO: 2013-05-15 11:57:41: Applying patch on the homes: /u01/app/11.2.0.3/grid
 INFO: 2013-05-15 11:57:41: It may take upto 15 mins
 SUCCESS: 2013-05-15 12:08:27: Successfully applied the patch on home: /u01/app/11.2.0.3/grid
 SUCCESS: 2013-05-15 12:08:27: Successfully started the dbconsoles
 SUCCESS: 2013-05-15 12:08:38: Successfully started the EM Agents
 INFO: 2013-05-15 12:08:39: Patching the GI home on node odanode2
 ...
..........done
INFO: GI patching summary on node: odanode1
 SUCCESS: 2013-05-15 12:22:58:  Successfully applied the patch on home /u01/app/11.2.0.3/grid
INFO: GI patching summary on node: odanode2
 SUCCESS: 2013-05-15 12:22:58:  Successfully applied the patch on home /u01/app/11.2.0.3/grid
INFO: Running post-install scripts
 ..........done
 INFO: Setting up the SSH
 ..........done
[root@odanode1 bin]#
[root@odanode2 ~]# su - grid
 [grid@odanode2 ~]$ cd /u01/app/11.2.0.3/grid/OPatch/
 [grid@odanode2 OPatch]$ ./opatch lsinv
 Oracle Interim Patch Installer version 11.2.0.3.4
 Copyright (c) 2012, Oracle Corporation.  All rights reserved.
Oracle Home       : /u01/app/11.2.0.3/grid
 Central Inventory : /u01/app/oraInventory
 from           : /u01/app/11.2.0.3/grid/oraInst.loc
 OPatch version    : 11.2.0.3.4
 OUI version       : 11.2.0.3.0
 Log file location : /u01/app/11.2.0.3/grid/cfgtoollogs/opatch/opatch2013-05-15_12-33-15PM_1.log
Lsinventory Output file location : /u01/app/11.2.0.3/grid/cfgtoollogs/opatch/lsinv/lsinventory2013-05-15_12-33-15PM.txt
--------------------------------------------------------------------------------
 Installed Top-level Products (1):
Oracle Grid Infrastructure                                           11.2.0.3.0
 There are 1 products installed in this Oracle Home.
Interim patches (2) :
Patch  16056266     : applied on Wed May 15 12:18:38 CEST 2013
 Unique Patch ID:  15962803
 Patch description:  "Database Patch Set Update : 11.2.0.3.6 (16056266)"
 Created on 12 Mar 2013, 02:14:47 hrs PST8PDT
 Sub-patch  14727310; "Database Patch Set Update : 11.2.0.3.5 (14727310)"
 Sub-patch  14275605; "Database Patch Set Update : 11.2.0.3.4 (14275605)"
 Sub-patch  13923374; "Database Patch Set Update : 11.2.0.3.3 (13923374)"
 Sub-patch  13696216; "Database Patch Set Update : 11.2.0.3.2 (13696216)"
 Sub-patch  13343438; "Database Patch Set Update : 11.2.0.3.1 (13343438)"
 Bugs fixed:
 13566938, 13593999, 10350832, 14138130, 12919564, 13561951, 13624984
 13588248, 13080778, 13914613, 13804294, 14258925, 12873183, 13645875
 14472647, 12880299, 14664355, 14409183, 12998795, 14469008, 13719081
 13492735, 13496884, 12857027, 14263036, 14263073, 13732226, 13742433
 16368108, 16314469, 12905058, 13742434, 12849688, 12950644, 13742435
 13464002, 13534412, 12879027, 13958038, 14613900, 12585543, 12535346
 12588744, 11877623, 13786142, 12847466, 13649031, 13981051, 12582664
 12797765, 14262913, 12923168, 13384182, 13612575, 13466801, 13484963
 14207163, 11063191, 13772618, 13070939, 12797420, 13041324, 16314467
 16314468, 12976376, 11708510, 13680405, 14589750, 13026410, 13742437
 13737746, 14644185, 13742438, 13326736, 13596521, 13001379, 16344871
 13099577, 9873405, 14275605, 13742436, 9858539, 14841812, 11715084
 16231699, 14040433, 12662040, 9703627, 12617123, 12845115, 12764337
 13354082, 14459552, 13397104, 13913630, 12964067, 12983611, 13550185
 13810393, 12780983, 12583611, 14546575, 13476583, 15862016, 11840910
 13903046, 15862017, 13572659, 16294378, 13718279, 14088346, 13657605
 13448206, 16314466, 14480676, 13419660, 13632717, 14063281, 14110275
 13430938, 13467683, 13420224, 13812031, 14548763, 16299830, 12646784
 13616375, 14035825, 12861463, 12834027, 15862021, 13632809, 13377816
 13036331, 14727310, 13685544, 15862018, 13499128, 16175381, 13584130
 12829021, 15862019, 12794305, 14546673, 12791981, 13787482, 13503598
 10133521, 12718090, 13399435, 14023636, 13860201, 12401111, 13257247
 13362079, 14176879, 12917230, 13923374, 14220725, 14480675, 13524899
 13559697, 9706792, 14480674, 13916709, 13098318, 13773133, 14076523
 13340388, 13366202, 13528551, 12894807, 13454210, 13343438, 12748240
 14205448, 13385346, 15853081, 14273397, 12971775, 13582702, 10242202
 13035804, 13544396, 16382353, 8547978, 14226599, 14062795, 13035360
 12693626, 13332439, 14038787, 14062796, 12913474, 14841409, 14390252
 16314470, 13370330, 13059165, 14062797, 14062794, 12959852, 13358781
 12345082, 12960925, 9659614, 13699124, 14546638, 13936424, 13338048
 12938841, 12658411, 12620823, 12656535, 14062793, 12678920, 13038684
 14062792, 13807411, 13250244, 12594032, 15862022, 9761357, 12612118
 13742464, 14052474, 13911821, 13457582, 13527323, 15862020, 13910420
 13502183, 12780098, 13705338, 13696216, 14841558, 10263668, 15862023
 16056266, 15862024, 13554409, 13645917, 13103913, 13011409, 14063280
Patch  16315641     : applied on Wed May 15 12:17:13 CEST 2013
 Unique Patch ID:  15966967
 Patch description:  "Grid Infrastructure Patch Set Update : 11.2.0.3.6 (16083653)"
 Created on 1 Apr 2013, 03:41:20 hrs PST8PDT
 Bugs fixed:
 16315641, 15876003, 14275572, 13919095, 13696251, 13348650, 12659561
 14305980, 14277586, 13987807, 14625969, 13825231, 12794268, 13000491
 13498267, 11675721, 14082976, 12771830, 14515980, 14085018, 13943175
 14102704, 14171552, 12594616, 13879428, 12897902, 12726222, 12829429
 13079948, 13090686, 12995950, 13251796, 13582411, 12990582, 13857364
 13082238, 12947871, 13256955, 13037709, 14535011, 12878750, 14048512
 11772838, 13058611, 13001955, 13440962, 13727853, 13425727, 12885323
 12870400, 14212634, 14407395, 13332363, 13430626, 13811209, 12709476
 14168708, 14096821, 14626717, 13460353, 13694885, 12857064, 12899169
 13111013, 12558569, 13323698, 10260842, 13085732, 10317921, 13869978
 12914824, 13789135, 12730342, 12950823, 13355963, 13531373, 14268365
 13776758, 12720728, 13620816, 13023609, 13024624, 13039908, 13036424
 13938166, 13011520, 13569812, 12758736, 13001901, 13077654, 13430715
 13550689, 13806545, 13634583, 14271305, 12538907, 13947200, 12996428
 13066371, 13483672, 12897651, 13540563, 12896850, 13241779, 12728585
 12876314, 12925041, 12650672, 12398492, 12848480, 13652088, 16307750
 12917897, 12975811, 13653178, 13371153, 14800989, 10114953, 14001941
 11836951, 14179376, 12965049, 14773530, 12765467, 13339443, 13965075
 16210540, 14307855, 12784559, 14242977, 13955385, 12704789, 13745317
 13074261, 12971251, 13993634, 13523527, 13719731, 13396284, 12639013
 12867511, 12959140, 14748254, 12829917, 12349553, 12849377, 12934171
 13843080, 14496536, 13924431, 12680491, 13334158, 10418841, 12832204
 13838047, 13002015, 12791719, 13886023, 13821454, 12782756, 14100232
 14186070, 14569263, 12873909, 13845120, 14214257, 12914722, 12842804
 12772345, 12663376, 14059576, 13889047, 12695029, 13924910, 13146560
 14070200, 13820621, 14304758, 12996572, 13941934, 14711358, 13019958
 13888719, 16463033, 12823838, 13877508, 12823042, 14494305, 13582706
 13617861, 12825835, 13025879, 13853089, 13410987, 13570879, 13247273
 13255295, 14152875, 13912373, 13011182, 13243172, 13045518, 12765868
 11825850, 15986571, 13345868, 13683090, 12932852, 13038806, 14588629
 14251904, 13396356, 13697828, 12834777, 13258062, 14371335, 13657366
 12810890, 15917085, 13502441, 14637577, 13880925, 13726162, 14153867
 13506114, 12820045, 13604057, 13263435, 14009845, 12827493, 13637590, 13068077
Rac system comprising of multiple nodes
 Local node = odanode2
 Remote node = odanode1
--------------------------------------------------------------------------------
OPatch succeeded.
 [grid@odanode2 OPatch]$
--Stop CRS on both Nodes
 [root@odanode1 2.6.0.0.0]# /u01/app/11.2.0.3/grid/bin/crsctl stop crs
 CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'odanode1'
 CRS-2673: Attempting to stop 'ora.crsd' on 'odanode1'
 CRS-2790: Starting shutdown of Cluster reporty Services-managed resources on 'odanode1'
 CRS-2673: Attempting to stop 'ora.cvu' on 'odanode1'
 CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'odanode1'
 CRS-2673: Attempting to stop 'ora.efboeur.efbo_applb.efow.com.svc' on 'odanode1'
 CRS-2673: Attempting to stop 'ora.efcteur.efct_applb.efow.com.svc' on 'odanode1'
 CRS-2673: Attempting to stop 'ora.efpheur.efph_applb.efow.com.svc' on 'odanode1'
 CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'odanode1'
 CRS-2677: Stop of 'ora.efboeur.efbo_applb.efow.com.svc' on 'odanode1' succeeded
 CRS-2677: Stop of 'ora.efpheur.efph_applb.efow.com.svc' on 'odanode1' succeeded
 CRS-2677: Stop of 'ora.efcteur.efct_applb.efow.com.svc' on 'odanode1' succeeded
 CRS-2673: Attempting to stop 'ora.efboeur.db' on 'odanode1'
 CRS-2673: Attempting to stop 'ora.efpheur.db' on 'odanode1'
 CRS-2673: Attempting to stop 'ora.efcteur.db' on 'odanode1'
 CRS-2673: Attempting to stop 'ora.registry.acfs' on 'odanode1'
 CRS-2677: Stop of 'ora.cvu' on 'odanode1' succeeded
 CRS-2672: Attempting to start 'ora.cvu' on 'odanode2'
 CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'odanode1' succeeded
 CRS-2673: Attempting to stop 'ora.odanode1.vip' on 'odanode1'
 CRS-2676: Start of 'ora.cvu' on 'odanode2' succeeded
 CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'odanode1' succeeded
 CRS-2673: Attempting to stop 'ora.scan2.vip' on 'odanode1'
 CRS-2677: Stop of 'ora.registry.acfs' on 'odanode1' succeeded
 CRS-2677: Stop of 'ora.efboeur.db' on 'odanode1' succeeded
 CRS-2677: Stop of 'ora.efpheur.db' on 'odanode1' succeeded
 CRS-2677: Stop of 'ora.efcteur.db' on 'odanode1' succeeded
 CRS-2673: Attempting to stop 'ora.RECO.dg' on 'odanode1'
 CRS-2673: Attempting to stop 'ora.REDO.dg' on 'odanode1'
 CRS-2673: Attempting to stop 'ora.DATA.dg' on 'odanode1'
 CRS-2677: Stop of 'ora.odanode1.vip' on 'odanode1' succeeded
 CRS-2672: Attempting to start 'ora.odanode1.vip' on 'odanode2'
 CRS-2677: Stop of 'ora.REDO.dg' on 'odanode1' succeeded
 CRS-2677: Stop of 'ora.scan2.vip' on 'odanode1' succeeded
 CRS-2672: Attempting to start 'ora.scan2.vip' on 'odanode2'
 CRS-2677: Stop of 'ora.RECO.dg' on 'odanode1' succeeded
 CRS-2676: Start of 'ora.odanode1.vip' on 'odanode2' succeeded
 CRS-2676: Start of 'ora.scan2.vip' on 'odanode2' succeeded
 CRS-2677: Stop of 'ora.DATA.dg' on 'odanode1' succeeded
 CRS-2673: Attempting to stop 'ora.asm' on 'odanode1'
 CRS-2677: Stop of 'ora.asm' on 'odanode1' succeeded
 CRS-2673: Attempting to stop 'ora.ons' on 'odanode1'
 CRS-2677: Stop of 'ora.ons' on 'odanode1' succeeded
 CRS-2673: Attempting to stop 'ora.net1.network' on 'odanode1'
 CRS-2677: Stop of 'ora.net1.network' on 'odanode1' succeeded
 CRS-2792: Shutdown of Cluster reporty Services-managed resources on 'odanode1' has completed
 CRS-2677: Stop of 'ora.crsd' on 'odanode1' succeeded
 CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'odanode1'
 CRS-2673: Attempting to stop 'ora.ctssd' on 'odanode1'
 CRS-2673: Attempting to stop 'ora.evmd' on 'odanode1'
 CRS-2673: Attempting to stop 'ora.asm' on 'odanode1'
 CRS-2673: Attempting to stop 'ora.mdnsd' on 'odanode1'
 CRS-2677: Stop of 'ora.evmd' on 'odanode1' succeeded
 CRS-2677: Stop of 'ora.mdnsd' on 'odanode1' succeeded
 CRS-2677: Stop of 'ora.ctssd' on 'odanode1' succeeded
 CRS-2677: Stop of 'ora.drivers.acfs' on 'odanode1' succeeded
 CRS-2677: Stop of 'ora.asm' on 'odanode1' succeeded
 CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'odanode1'
 CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'odanode1' succeeded
 CRS-2673: Attempting to stop 'ora.cssd' on 'odanode1'
 CRS-2677: Stop of 'ora.cssd' on 'odanode1' succeeded
 CRS-2673: Attempting to stop 'ora.crf' on 'odanode1'
 CRS-2677: Stop of 'ora.crf' on 'odanode1' succeeded
 CRS-2673: Attempting to stop 'ora.gipcd' on 'odanode1'
 CRS-2677: Stop of 'ora.gipcd' on 'odanode1' succeeded
 CRS-2673: Attempting to stop 'ora.gpnpd' on 'odanode1'
 CRS-2677: Stop of 'ora.gpnpd' on 'odanode1' succeeded
 CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'odanode1' has completed
 CRS-4133: Oracle High Availability Services has been stopped.
 [root@odanode1 2.6.0.0.0]#
--Start CRS on both Nodes
 [root@odanode1 2.6.0.0.0]# /u01/app/11.2.0.3/grid/bin/crsctl start crs
 CRS-4123: Oracle High Availability Services has been started.
--Check GI status
 [root@odanode1 2.6.0.0.0]# /u01/app/11.2.0.3/grid/bin/crsctl status res -t
 --------------------------------------------------------------------------------
 NAME           TARGET  STATE        SERVER                   STATE_DETAILS
 --------------------------------------------------------------------------------
 Local Resources
 --------------------------------------------------------------------------------
 ora.DATA.dg
 ONLINE  ONLINE       odanode1
 ONLINE  ONLINE       odanode2
 ora.LISTENER.lsnr
 ONLINE  ONLINE       odanode1
 ONLINE  ONLINE       odanode2
 ora.RECO.dg
 ONLINE  ONLINE       odanode1
 ONLINE  ONLINE       odanode2
 ora.REDO.dg
 ONLINE  ONLINE       odanode1
 ONLINE  ONLINE       odanode2
 ora.asm
 ONLINE  ONLINE       odanode1               Started
 ONLINE  ONLINE       odanode2               Started
 ora.gsd
 OFFLINE OFFLINE      odanode1
 OFFLINE OFFLINE      odanode2
 ora.net1.network
 ONLINE  ONLINE       odanode1
 ONLINE  ONLINE       odanode2
 ora.ons
 ONLINE  ONLINE       odanode1
 ONLINE  ONLINE       odanode2
 ora.registry.acfs
 ONLINE  ONLINE       odanode1
 ONLINE  ONLINE       odanode2
 --------------------------------------------------------------------------------
 Cluster Resources
 --------------------------------------------------------------------------------
 ora.LISTENER_SCAN1.lsnr
 1        ONLINE  ONLINE       odanode1
 ora.LISTENER_SCAN2.lsnr
 1        ONLINE  ONLINE       odanode2
 ora.cvu
 1        ONLINE  ONLINE       odanode2
 ora.efboeur.db
 1        ONLINE  ONLINE       odanode1               Open
 2        ONLINE  ONLINE       odanode2               Open
 ora.efboeur.efbo_applb.efow.com.svc
 1        ONLINE  ONLINE       odanode1
 2        ONLINE  ONLINE       odanode2
 ora.efboeur.efbo_report.efow.com.svc
 1        OFFLINE OFFLINE
 2        OFFLINE OFFLINE
 ora.efcteur.db
 1        ONLINE  ONLINE       odanode1               Open
 2        ONLINE  ONLINE       odanode2               Open
 ora.efcteur.efct_applb.efow.com.svc
 1        ONLINE  ONLINE       odanode1
 2        ONLINE  ONLINE       odanode2
 ora.efcteur.efct_report.efow.com.svc
 1        OFFLINE OFFLINE
 2        OFFLINE OFFLINE
 ora.odanode1.vip
 1        ONLINE  ONLINE       odanode1
 ora.odanode2.vip
 1        ONLINE  ONLINE       odanode2
 ora.efpheur.db
 1        ONLINE  ONLINE       odanode1               Open
 2        ONLINE  ONLINE       odanode2               Open
 ora.efpheur.efph_applb.efow.com.svc
 1        ONLINE  ONLINE       odanode1
 2        ONLINE  ONLINE       odanode2
 ora.efpheur.efph_report.efow.com.svc
 1        OFFLINE OFFLINE
 2        OFFLINE OFFLINE
 ora.oc4j
 1        ONLINE  ONLINE       odanode2
 ora.scan1.vip
 1        ONLINE  ONLINE       odanode1
 ora.scan2.vip
 1        ONLINE  ONLINE       odanode2
 [root@odanode1 2.6.0.0.0]#
-------------------------------------------------
 --Apply the Patch to the RDBMS
 -------------------------------------------------
 --Check the RDBMS patch level before applying the PSU
 [root@odanode1 bin]# /opt/oracle/oak/bin/oakcli show databases
 Database Name    Database Type   Database HomeName    Database HomeLocation                                        Database Version
 ----------------       -----------            ----------------                   ---------------------------------------                           ---------------------
 efboeur                RAC                   OraDb11203_home1      /u01/app/oracle/product/11.2.0.3/dbhome_1         11.2.0.3.4(14275605,14275572)
 efcteur                 RAC                  OraDb11203_home1      /u01/app/oracle/product/11.2.0.3/dbhome_1          11.2.0.3.4(14275605,14275572)
 efpheur                RAC                  OraDb11203_home1      /u01/app/oracle/product/11.2.0.3/dbhome_1          11.2.0.3.4(14275605,14275572)
 [root@odanode1 bin]#
--ODA on BOTH Nodes
 [oracle@odanode1 OPatch]$ /u01/app/oracle/product/agent12c/agent_inst/bin/emctl stop agent
 Oracle Enterprise Manager Cloud Control 12c Release 2
 Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
 Stopping agent ..... stopped.
--In addition to avoind issue while patching the RDBMS:
 --ODA on BOTH Nodes
 [root@efoda01n1 ~]# /sbin/fuser /u01/app/oracle/product/11.2.0.3/dbhome_1/lib/libclntsh.so.11.1
 /u01/app/oracle/product/11.2.0.3/dbhome_1/lib/libclntsh.so.11.1: 18877m 18911m
[root@efoda01n1 ~]# ps -ef|grep 18877
 oracle   18877 18791  0 10:06 ?        00:00:22 /u01/app/oracle/product/11.2.0.3/dbhome_1/jdk/bin/java -server -Xmx384M -XX:MaxPermSize=400M -XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=40 -DORACLE_HOME=/u01/app/oracle/product/11.2.0.3/dbhome_1 -Doracle.home=/u01/app/oracle/product/11.2.0.3/dbhome_1/oc4j -Doracle.oc4j.localhome=/u01/app/oracle/product/11.2.0.3/dbhome_1/efoda01n1_test/sysman -DEMSTATE=/u01/app/oracle/product/11.2.0.3/dbhome_1/efoda01n1_test -Doracle.j2ee.dont.use.memory.archive=true -Djava.protocol.handler.pkgs=HTTPClient -Doracle.security.jazn.config=/u01/app/oracle/product/11.2.0.3/dbhome_1/oc4j/j2ee/OC4J_DBConsole_efoda01n1_test/config/jazn.xml -Djava.security.policy=/u01/app/oracle/product/11.2.0.3/dbhome_1/oc4j/j2ee/OC4J_DBConsole_efoda01n1_test/config/java2.policy -Djavax.net.ssl.KeyStore=/u01/app/oracle/product/11.2.0.3/dbhome_1/sysman/config/OCMTrustedCerts.txt-Djava.security.properties=/u01/app/oracle/product/11.2.0.3/dbhome_1/oc4j/j2ee/home/config/jazn.security.props -DEMDROOT=/u01/app/oracle/product/11.2.0.3/dbhome_1/efoda01n1_test -Dsysman.md5password=true -Drepapi.oracle.home=/u01/app/oracle/product/11.2.0.3/dbhome_1 -Ddisable.checkForUpdate=true -Doracle.sysman.ccr.ocmSDK.websvc.keystore=/u01/app/oracle/product/11.2.0.3/dbhome_1/jlib/emocmclnt.ks -Dice.pilots.html4.ignoreNonGenericFonts=true -Djava.awt.headless=true -jar /u01/app/oracle/product/11.2.0.3/dbhome_1/oc4j/j2ee/home/oc4j.jar -config /u01/app/oracle/product/11.2.0.3/dbhome_1/oc4j/j2ee/OC4J_DBConsole_efoda01n1_test/config/server.xml
[root@efoda01n1 ~]# kill -9 18877 18911
--ODA Node 1 ONLY
 [root@odanode1 bin]# ./oakcli update -patch 2.6.0.0.0 --database
Please enter the 'root' user password:
 Please re-enter the 'root' user password:
Please enter the 'oracle' user password:
 Please re-enter the 'oracle' user password:
 INFO: Setting up the SSH
 ..........done
 ...
 ...
..........done
 ...
 SUCCESS: All nodes in /opt/oracle/oak/temp_clunodes.txt are pingable and alive.
 INFO: 2013-05-15 15:10:16: Getting the possible database homes for patching
 ...
 INFO: 2013-05-15 15:10:21: Patching 11.2.0.3 Database homes on node odanode1
Found the following 11.2.0.3 homes possible for patching:
HOME_NAME                      HOME_LOCATION
 ---------                      -------------
 OraDb11203_home1               /u01/app/oracle/product/11.2.0.3/dbhome_1
[Please note that few of the above database homes may be alreporty up-to-date. They will be automatically ignored]
Would you like to patch all the above homes: Y | N ? :Y
 INFO: 2013-05-15 15:15:48: Setting up ssh for the user oracle
 ..........done
 ...
 SUCCESS: All nodes in /opt/oracle/oak/temp_clunodes.txt are pingable and alive.
 INFO: 2013-05-15 15:16:07: Updating the opatch
 INFO: 2013-05-15 15:16:24: Performing the conflict checks
 SUCCESS: 2013-05-15 15:16:42: Conflict checks passed for all the homes
 INFO: 2013-05-15 15:16:42: Checking if the patch is alreporty applied on any of the homes
 INFO: 2013-05-15 15:16:46: No home is alreporty up-to-date
 SUCCESS: 2013-05-15 15:16:52: Successfully stopped the dbconsoles
 SUCCESS: 2013-05-15 15:16:58: Successfully stopped the EM agents
 INFO: 2013-05-15 15:17:03: Applying patch on the homes: /u01/app/oracle/product/11.2.0.3/dbhome_1
 INFO: 2013-05-15 15:17:03: It may take upto 15 mins
 SUCCESS: 2013-05-15 15:21:35: Successfully applied the patch on home: /u01/app/oracle/product/11.2.0.3/dbhome_1
 SUCCESS: 2013-05-15 15:21:35: Successfully started the dbconsoles
 SUCCESS: 2013-05-15 15:21:35: Successfully started the EM Agents
 INFO: 2013-05-15 15:21:37: Patching 11.2.0.3 Database homes on node odanode2
 INFO: 2013-05-15 15:22:11: Running the catbundle.sql
 INFO: 2013-05-15 15:22:18: Running catbundle.sql on the database efboeur
 INFO: 2013-05-15 15:22:26: Running catbundle.sql on the database efcteur
 INFO: 2013-05-15 15:22:35: Running catbundle.sql on the database efpheur
..........done
INFO: DB patching summary on node: odanode1
 SUCCESS: 2013-05-15 15:22:57:  Successfully applied the patch on home /u01/app/oracle/product/11.2.0.3/dbhome_1
INFO: DB patching summary on node: odanode2
 INFO: 2013-05-15 15:22:57:  Homes /u01/app/oracle/product/11.2.0.3/dbhome_1 are alreporty up-to-date
INFO: Setting up the SSH
 ..........done
[root@odanode1 bin]#
[root@odanode1 2.6.0.0.0]# /opt/oracle/oak/bin/oakcli show databases
 Database Name    Database Type   Database HomeName    Database HomeLocation                                        Database Version
 ----------------       -----------            ----------------                   ---------------------------------------                           ---------------------
 efboeur                RAC                   OraDb11203_home1      /u01/app/oracle/product/11.2.0.3/dbhome_1         11.2.0.3.6(16056266,16083653)
 efcteur                 RAC                  OraDb11203_home1      /u01/app/oracle/product/11.2.0.3/dbhome_1          11.2.0.3.6(16056266,16083653)
 efpheur                RAC                  OraDb11203_home1      /u01/app/oracle/product/11.2.0.3/dbhome_1          11.2.0.3.6(16056266,16083653)
 [root@odanode1 bin]#
[oracle@odanode2 OPatch]$ ./opatch lsinv
 Oracle Interim Patch Installer version 11.2.0.3.4
 Copyright (c) 2012, Oracle Corporation.  All rights reserved.
Oracle Home       : /u01/app/oracle/product/11.2.0.3/dbhome_1
 Central Inventory : /u01/app/oraInventory
 from           : /u01/app/oracle/product/11.2.0.3/dbhome_1/oraInst.loc
 OPatch version    : 11.2.0.3.4
 OUI version       : 11.2.0.3.0
 Log file location : /u01/app/oracle/product/11.2.0.3/dbhome_1/cfgtoollogs/opatch/opatch2013-05-15_15-49-48PM_1.log
Lsinventory Output file location : /u01/app/oracle/product/11.2.0.3/dbhome_1/cfgtoollogs/opatch/lsinv/lsinventory2013-05-15_15-49-48PM.txt
--------------------------------------------------------------------------------
 Installed Top-level Products (1):
Oracle Database 11g                                                  11.2.0.3.0
 There are 1 products installed in this Oracle Home.
Interim patches (2) :
Patch  16056266     : applied on Wed May 15 15:00:15 CEST 2013
 Unique Patch ID:  15962803
 Patch description:  "Database Patch Set Update : 11.2.0.3.6 (16056266)"
 Created on 12 Mar 2013, 02:14:47 hrs PST8PDT
 Sub-patch  14727310; "Database Patch Set Update : 11.2.0.3.5 (14727310)"
 Sub-patch  14275605; "Database Patch Set Update : 11.2.0.3.4 (14275605)"
 Sub-patch  13923374; "Database Patch Set Update : 11.2.0.3.3 (13923374)"
 Sub-patch  13696216; "Database Patch Set Update : 11.2.0.3.2 (13696216)"
 Sub-patch  13343438; "Database Patch Set Update : 11.2.0.3.1 (13343438)"
 Bugs fixed:
 13566938, 13593999, 10350832, 14138130, 12919564, 13561951, 13624984
 13588248, 13080778, 13914613, 13804294, 14258925, 12873183, 13645875
 14472647, 12880299, 14664355, 14409183, 12998795, 14469008, 13719081
 13492735, 13496884, 12857027, 14263036, 14263073, 13732226, 13742433
 16368108, 16314469, 12905058, 13742434, 12849688, 12950644, 13742435
 13464002, 13534412, 12879027, 13958038, 14613900, 12585543, 12535346
 12588744, 11877623, 13786142, 12847466, 13649031, 13981051, 12582664
 12797765, 14262913, 12923168, 13384182, 13612575, 13466801, 13484963
 14207163, 11063191, 13772618, 13070939, 12797420, 13041324, 16314467
 16314468, 12976376, 11708510, 13680405, 14589750, 13026410, 13742437
 13737746, 14644185, 13742438, 13326736, 13596521, 13001379, 16344871
 13099577, 9873405, 14275605, 13742436, 9858539, 14841812, 11715084
 16231699, 14040433, 12662040, 9703627, 12617123, 12845115, 12764337
 13354082, 14459552, 13397104, 13913630, 12964067, 12983611, 13550185
 13810393, 12780983, 12583611, 14546575, 13476583, 15862016, 11840910
 13903046, 15862017, 13572659, 16294378, 13718279, 14088346, 13657605
 13448206, 16314466, 14480676, 13419660, 13632717, 14063281, 14110275
 13430938, 13467683, 13420224, 13812031, 14548763, 16299830, 12646784
 13616375, 14035825, 12861463, 12834027, 15862021, 13632809, 13377816
 13036331, 14727310, 13685544, 15862018, 13499128, 16175381, 13584130
 12829021, 15862019, 12794305, 14546673, 12791981, 13787482, 13503598
 10133521, 12718090, 13399435, 14023636, 13860201, 12401111, 13257247
 13362079, 14176879, 12917230, 13923374, 14220725, 14480675, 13524899
 13559697, 9706792, 14480674, 13916709, 13098318, 13773133, 14076523
 13340388, 13366202, 13528551, 12894807, 13454210, 13343438, 12748240
 14205448, 13385346, 15853081, 14273397, 12971775, 13582702, 10242202
 13035804, 13544396, 16382353, 8547978, 14226599, 14062795, 13035360
 12693626, 13332439, 14038787, 14062796, 12913474, 14841409, 14390252
 16314470, 13370330, 13059165, 14062797, 14062794, 12959852, 13358781
 12345082, 12960925, 9659614, 13699124, 14546638, 13936424, 13338048
 12938841, 12658411, 12620823, 12656535, 14062793, 12678920, 13038684
 14062792, 13807411, 13250244, 12594032, 15862022, 9761357, 12612118
 13742464, 14052474, 13911821, 13457582, 13527323, 15862020, 13910420
 13502183, 12780098, 13705338, 13696216, 14841558, 10263668, 15862023
 16056266, 15862024, 13554409, 13645917, 13103913, 13011409, 14063280
Patch  16315641     : applied on Wed May 15 13:58:54 CEST 2013
 Unique Patch ID:  15966967
 Patch description:  "Grid Infrastructure Patch Set Update : 11.2.0.3.6 (16083653)"
 Created on 1 Apr 2013, 03:41:20 hrs PST8PDT
 Bugs fixed:
 16315641, 15876003, 14275572, 13919095, 13696251, 13348650, 12659561
 14305980, 14277586, 13987807, 14625969, 13825231, 12794268, 13000491
 13498267, 11675721, 14082976, 12771830, 14515980, 14085018, 13943175
 14102704, 14171552, 12594616, 13879428, 12897902, 12726222, 12829429
 13079948, 13090686, 12995950, 13251796, 13582411, 12990582, 13857364
 13082238, 12947871, 13256955, 13037709, 14535011, 12878750, 14048512
 11772838, 13058611, 13001955, 13440962, 13727853, 13425727, 12885323
 12870400, 14212634, 14407395, 13332363, 13430626, 13811209, 12709476
 14168708, 14096821, 14626717, 13460353, 13694885, 12857064, 12899169
 13111013, 12558569, 13323698, 10260842, 13085732, 10317921, 13869978
 12914824, 13789135, 12730342, 12950823, 13355963, 13531373, 14268365
 13776758, 12720728, 13620816, 13023609, 13024624, 13039908, 13036424
 13938166, 13011520, 13569812, 12758736, 13001901, 13077654, 13430715
 13550689, 13806545, 13634583, 14271305, 12538907, 13947200, 12996428
 13066371, 13483672, 12897651, 13540563, 12896850, 13241779, 12728585
 12876314, 12925041, 12650672, 12398492, 12848480, 13652088, 16307750
 12917897, 12975811, 13653178, 13371153, 14800989, 10114953, 14001941
 11836951, 14179376, 12965049, 14773530, 12765467, 13339443, 13965075
 16210540, 14307855, 12784559, 14242977, 13955385, 12704789, 13745317
 13074261, 12971251, 13993634, 13523527, 13719731, 13396284, 12639013
 12867511, 12959140, 14748254, 12829917, 12349553, 12849377, 12934171
 13843080, 14496536, 13924431, 12680491, 13334158, 10418841, 12832204
 13838047, 13002015, 12791719
Rac system comprising of multiple nodes
 Local node = odanode2
 Remote node = odanode1
--------------------------------------------------------------------------------
OPatch succeeded.
 [oracle@odanode2 OPatch]$
-------------------------------------------------
 --ODA Software version after patching
 -------------------------------------------------
 [root@odanode1 bin]# /opt/oracle/oak/bin/oakcli show version -detail
 reporting the metadata. It takes a while...
 System Version         Component Name            Installed Version                Supported Version
 --------------          ---------------              ---------------------         ----------------------
 2.6.0.0.0
 Controller                        11.05.02.00                      Up-to-date
 Expander                         0342                               Up-to-date
 SSD_SHARED                    E12B                               Up-to-date
 HDD_LOCAL                      5G08                               Up-to-date
 HDD_SHARED                    A700                               Up-to-date
 ILOM                               3.0.16.22.b r78329            Up-to-date
 BIOS                               12010310                         Up-to-date
 IPMI                                1.8.10.5                          Up-to-date
 HMP                                2.2.6.1                            Up-to-date
 OAK                                2.6.0.0.0                          Up-to-date
 OEL                                5.8                                  Up-to-date
 TFA                                2.5.1.4                            Up-to-date
 GI_HOME                         11.2.0.3.6(16056266,         Up-to-date
 16083653)
 DB_HOME                         11.2.0.3.6(16056266,          Up-to-date
 16083653)
 ASR                                  Unknown                          4.4
 [root@odanode1 bin]#
[grid@odanode2 ~]$ crsctl stat res -t
 --------------------------------------------------------------------------------
 NAME           TARGET  STATE        SERVER                   STATE_DETAILS
 --------------------------------------------------------------------------------
 Local Resources
 --------------------------------------------------------------------------------
 ora.DATA.dg
 ONLINE  ONLINE       odanode1
 ONLINE  ONLINE       odanode2
 ora.LISTENER.lsnr
 ONLINE  ONLINE       odanode1
 ONLINE  ONLINE       odanode2
 ora.RECO.dg
 ONLINE  ONLINE       odanode1
 ONLINE  ONLINE       odanode2
 ora.REDO.dg
 ONLINE  ONLINE       odanode1
 ONLINE  ONLINE       odanode2
 ora.asm
 ONLINE  ONLINE       odanode1               Started
 ONLINE  ONLINE       odanode2               Started
 ora.gsd
 OFFLINE OFFLINE      odanode1
 OFFLINE OFFLINE      odanode2
 ora.net1.network
 ONLINE  ONLINE       odanode1
 ONLINE  ONLINE       odanode2
 ora.ons
 ONLINE  ONLINE       odanode1
 ONLINE  ONLINE       odanode2
 ora.registry.acfs
 ONLINE  ONLINE       odanode1
 ONLINE  ONLINE       odanode2
 --------------------------------------------------------------------------------
 Cluster Resources
 --------------------------------------------------------------------------------
 ora.LISTENER_SCAN1.lsnr
 1        ONLINE  ONLINE       odanode2
 ora.LISTENER_SCAN2.lsnr
 1        ONLINE  ONLINE       odanode1
 ora.cvu
 1        ONLINE  ONLINE       odanode1
 ora.efboeur.db
 1        ONLINE  ONLINE       odanode1               Open
 2        ONLINE  ONLINE       odanode2               Open
 ora.efboeur.efbo_applb.efow.com.svc
 1        ONLINE  ONLINE       odanode2
 2        ONLINE  ONLINE       odanode1
 ora.efboeur.efbo_report.efow.com.svc
 1        OFFLINE OFFLINE
 2        OFFLINE OFFLINE
 ora.efcteur.db
 1        ONLINE  ONLINE       odanode1               Open
 2        ONLINE  ONLINE       odanode2               Open
 ora.efcteur.efct_applb.efow.com.svc
 1        ONLINE  ONLINE       odanode1
 2        ONLINE  ONLINE       odanode2
 ora.efcteur.efct_report.efow.com.svc
 1        OFFLINE OFFLINE
 2        OFFLINE OFFLINE
 ora.odanode1.vip
 1        ONLINE  ONLINE       odanode1
 ora.odanode2.vip
 1        ONLINE  ONLINE       odanode2
 ora.efpheur.db
 1        ONLINE  ONLINE       odanode1               Open
 2        ONLINE  ONLINE       odanode2               Open
 ora.efpheur.efph_applb.efow.com.svc
 1        ONLINE  ONLINE       odanode2
 2        ONLINE  ONLINE       odanode1
 ora.efpheur.efph_report.efow.com.svc
 1        OFFLINE OFFLINE
 2        OFFLINE OFFLINE
 ora.oc4j
 1        ONLINE  ONLINE       odanode1
 ora.scan1.vip
 1        ONLINE  ONLINE       odanode2
 ora.scan2.vip
 1        ONLINE  ONLINE       odanode1
 [grid@odanode2 ~]$