Oracle VM Server 3.4.5 – Kernel Memory Leak

Oracle VM Server instability on version 3.4.5 due to a Oracle Unbreakable Enterprise Kernel (UEK) bug.

The kernel version 4.1.12-124.14.5.el6uek.x86_64 has introduced a memory leak of the network module i40e.

Here is reported the backtraces collected after the problem from the /var/log/messages:

Dec 12 07:06:30 efuovs02 kernel: [1508192.885203] ntpd invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
Dec 12 07:06:30 efuovs02 kernel: [1508192.885208] ntpd cpuset=/ mems_allowed=0
Dec 12 07:06:30 efuovs02 kernel: [1508192.885217] CPU: 3 PID: 4751 Comm: ntpd Not tainted 4.1.12-124.14.5.el6uek.x86_64 #2
Dec 12 07:06:30 efuovs02 kernel: [1508192.885221] Hardware name: HPE ProLiant DL360 Gen10/ProLiant DL360 Gen10, BIOS U32 02/14/2018
Dec 12 07:06:30 efuovs02 kernel: [1508192.885224]  0000000000000000 ffff8804484cf678 ffffffff816e4bdb ffff88044d44aa00
Dec 12 07:06:30 efuovs02 kernel: [1508192.885230]  0000000000000000 ffff8804484cf708 ffffffff816e32d1 01ff8804484cf688
Dec 12 07:06:30 efuovs02 kernel: [1508192.885235]  ffff8804484cf718 ffff8804484cf6c8 ffffffff811fc561 ffff8804484cf800
Dec 12 07:06:30 efuovs02 kernel: [1508192.885241] Call Trace:
Dec 12 07:06:30 efuovs02 kernel: [1508192.885251]  [<ffffffff816e4bdb>] dump_stack+0x63/0x81
Dec 12 07:06:30 efuovs02 kernel: [1508192.885256]  [<ffffffff816e32d1>] dump_header+0x7f/0x1f3
Dec 12 07:06:30 efuovs02 kernel: [1508192.885264]  [<ffffffff811fc561>] ? vmpressure+0x21/0x90
Dec 12 07:06:30 efuovs02 kernel: [1508192.885272]  [<ffffffff8118e53c>] oom_kill_process+0x1cc/0x3c0
Dec 12 07:06:30 efuovs02 kernel: [1508192.885283]  [<ffffffff8108de0e>] ? has_capability_noaudit+0x1e/0x30
Dec 12 07:06:31 efuovs02 kernel: [1508192.885288]  [<ffffffff8118eaab>] __out_of_memory+0x31b/0x530
Dec 12 07:06:31 efuovs02 kernel: [1508192.885294]  [<ffffffff8118ee5b>] out_of_memory+0x5b/0x80
Dec 12 07:06:31 efuovs02 kernel: [1508192.885300]  [<ffffffff81194d42>] __alloc_pages_nodemask+0x952/0xab0
Dec 12 07:06:31 efuovs02 kernel: [1508192.885307]  [<ffffffff811de28d>] alloc_pages_vma+0xbd/0x260
Dec 12 07:06:31 efuovs02 kernel: [1508192.885311]  [<ffffffff8118a59e>] ? find_get_entry+0x1e/0xc0
Dec 12 07:06:31 efuovs02 kernel: [1508192.885317]  [<ffffffff811ce6bd>] read_swap_cache_async+0xed/0x170
Dec 12 07:06:31 efuovs02 kernel: [1508192.885322]  [<ffffffff811ce82d>] swapin_readahead+0xed/0x190
Dec 12 07:06:31 efuovs02 kernel: [1508192.885328]  [<ffffffff811bbfe0>] handle_mm_fault+0x12d0/0x1770
Dec 12 07:06:31 efuovs02 kernel: [1508192.885335]  [<ffffffff8121d910>] ? poll_select_copy_remaining+0x130/0x130
Dec 12 07:06:31 efuovs02 kernel: [1508192.885340]  [<ffffffff8106d57f>] __do_page_fault+0x1af/0x480
Dec 12 07:06:31 efuovs02 kernel: [1508192.885346]  [<ffffffff816f2c1c>] ? page_fault+0xcc/0x120
Dec 12 07:06:31 efuovs02 kernel: [1508192.885350]  [<ffffffff8106d87f>] do_page_fault+0x2f/0x80
Dec 12 07:06:31 efuovs02 kernel: [1508192.885354]  [<ffffffff816f2be4>] ? page_fault+0x94/0x120
Dec 12 07:06:31 efuovs02 kernel: [1508192.885359]  [<ffffffff816f2bdd>] ? page_fault+0x8d/0x120
Dec 12 07:06:31 efuovs02 kernel: [1508192.885363]  [<ffffffff816f2bd6>] ? page_fault+0x86/0x120
Dec 12 07:06:31 efuovs02 kernel: [1508192.885367]  [<ffffffff816f2c5f>] page_fault+0x10f/0x120
Dec 12 07:06:31 efuovs02 kernel: [1508192.885375]  [<ffffffff813316c5>] ? copy_user_enhanced_fast_string+0x5/0x10
Dec 12 07:06:31 efuovs02 kernel: [1508192.885379]  [<ffffffff8121d7d1>] ? set_fd_set+0x21/0x30
Dec 12 07:06:31 efuovs02 kernel: [1508192.885384]  [<ffffffff8121e5aa>] core_sys_select+0x1fa/0x2f0
Dec 12 07:06:31 efuovs02 kernel: [1508192.885392]  [<ffffffff810f8fc3>] ? ntp_notify_cmos_timer+0x23/0x30
Dec 12 07:06:31 efuovs02 kernel: [1508192.885396]  [<ffffffff810f8a1d>] ? do_adjtimex+0xed/0x100
Dec 12 07:06:31 efuovs02 kernel: [1508192.885402]  [<ffffffff810ed3ac>] ? SYSC_adjtimex+0x4c/0x80
Dec 12 07:06:31 efuovs02 kernel: [1508192.885410]  [<ffffffff810209e9>] ? read_tsc+0x9/0x10
Dec 12 07:06:31 efuovs02 kernel: [1508192.885414]  [<ffffffff810f68cb>] ? ktime_get_ts64+0x4b/0x110
Dec 12 07:06:31 efuovs02 kernel: [1508192.885419]  [<ffffffff8121e74b>] SyS_select+0xab/0x100
Dec 12 07:06:31 efuovs02 kernel: [1508192.885424]  [<ffffffff816ed451>] ? system_call_after_swapgs+0xdb/0x18c
Dec 12 07:06:31 efuovs02 kernel: [1508192.885428]  [<ffffffff816ed51a>] system_call_fastpath+0x18/0xd4
Dec 12 07:06:31 efuovs02 kernel: [1508192.885457] Mem-Info:
Dec 12 07:06:31 efuovs02 kernel: [1508192.885469] active_anon:1452 inactive_anon:1426 isolated_anon:65
Dec 12 07:06:31 efuovs02 kernel: [1508192.885469]  active_file:4559 inactive_file:873 isolated_file:0
Dec 12 07:06:31 efuovs02 kernel: [1508192.885469]  unevictable:1547 dirty:20 writeback:31 unstable:0
Dec 12 07:06:31 efuovs02 kernel: [1508192.885469]  slab_reclaimable:6776 slab_unreclaimable:8649
Dec 12 07:06:31 efuovs02 kernel: [1508192.885469]  mapped:3007 shmem:0 pagetables:1705 bounce:0
Dec 12 07:06:31 efuovs02 kernel: [1508192.885469]  free:33536 free_pcp:918 free_cma:0
Dec 12 07:06:31 efuovs02 kernel: [1508192.885483] Node 0 DMA free:15740kB min:60kB low:72kB high:84kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15988kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Dec 12 07:06:31 efuovs02 kernel: [1508192.885499] lowmem_reserve[]: 0 2661 15921 15921
Dec 12 07:06:31 efuovs02 kernel: [1508192.885508] Node 0 DMA32 free:64064kB min:11076kB low:13844kB high:16612kB active_anon:5912kB inactive_anon:5876kB active_file:112kB inactive_file:52kB unevictable:836kB isolated(anon):256kB isolated(file):0kB present:2781336kB managed:2751088kB mlocked:836kB dirty:0kB writeback:0kB mapped:668kB shmem:0kB slab_reclaimable:4792kB slab_unreclaimable:6452kB kernel_stack:912kB pagetables:1692kB unstable:0kB bounce:0kB free_pcp:1156kB local_pcp:248kB free_cma:0kB writeback_tmp:0kB pages_scanned:619296 all_unreclaimable? yes
Dec 12 07:06:31 efuovs02 kernel: [1508192.885524] lowmem_reserve[]: 0 0 13260 13260
Dec 12 07:06:31 efuovs02 kernel: [1508192.885532] Node 0 Normal free:54340kB min:54392kB low:67988kB high:81584kB active_anon:0kB inactive_anon:0kB active_file:18124kB inactive_file:3440kB unevictable:5352kB isolated(anon):4kB isolated(file):0kB present:13979888kB managed:13534768kB mlocked:5352kB dirty:80kB writeback:124kB mapped:11360kB shmem:0kB slab_reclaimable:22312kB slab_unreclaimable:28144kB kernel_stack:2880kB pagetables:5128kB unstable:0kB bounce:0kB free_pcp:2516kB local_pcp:572kB free_cma:0kB writeback_tmp:0kB pages_scanned:129384 all_unreclaimable? yes
Dec 12 07:06:31 efuovs02 kernel: [1508192.885546] lowmem_reserve[]: 0 0 0 0
Dec 12 07:06:31 efuovs02 kernel: [1508192.885551] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 1*32kB (U) 1*64kB (U) 0*128kB 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15740kB
Dec 12 07:06:31 efuovs02 kernel: [1508192.885573] Node 0 DMA32: 143*4kB (UE) 111*8kB (UEM) 209*16kB (UE) 140*32kB (UE) 94*64kB (UEM) 57*128kB (UEM) 28*256kB (UEM) 7*512kB (UEM) 4*1024kB (EM) 9*2048kB (MR) 2*4096kB (MR) = 64068kB
Dec 12 07:06:31 efuovs02 kernel: [1508192.885596] Node 0 Normal: 8736*4kB (UEM) 1360*8kB (UEM) 208*16kB (UEM) 32*32kB (UE) 2*64kB (UE) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 54400kB
Dec 12 07:06:31 efuovs02 kernel: [1508192.885615] 8002 total pagecache pages
Dec 12 07:06:31 efuovs02 kernel: [1508192.885618] 1494 pages in swap cache
Dec 12 07:06:31 efuovs02 kernel: [1508192.885621] Swap cache stats: add 3717250313, delete 3717248819, find 2895172777/5168362256
Dec 12 07:06:31 efuovs02 kernel: [1508192.885624] Free swap  = 4129656kB
Dec 12 07:06:31 efuovs02 kernel: [1508192.885626] Total swap = 4194300kB
Dec 12 07:06:31 efuovs02 kernel: [1508192.885628] 4194303 pages RAM
Dec 12 07:06:31 efuovs02 kernel: [1508192.885630] 0 pages HighMem/MovableOnly
Dec 12 07:06:31 efuovs02 kernel: [1508192.885632] 118864 pages reserved
Dec 12 07:06:31 efuovs02 kernel: [1508192.885634] 0 pages cma reserved
Dec 12 07:06:31 efuovs02 kernel: [1508192.885636] 0 pages hwpoisoned
Dec 12 07:06:31 efuovs02 kernel: [1508192.885638] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Dec 12 07:06:31 efuovs02 kernel: [1508192.885650] [  983]     0   983     2677      266      11       3      111         -1000 udevd
Dec 12 07:06:31 efuovs02 kernel: [1508192.885657] [ 3783]     0  3783   125771     1414      48       5        0         -1000 multipathd
Dec 12 07:06:31 efuovs02 kernel: [1508192.885662] [ 4334]     0  4334     6944      399      15       3      108         -1000 auditd
Dec 12 07:06:31 efuovs02 kernel: [1508192.885667] [ 4368]     0  4368    61281      438      23       3      377             0 rsyslogd
Dec 12 07:06:31 efuovs02 kernel: [1508192.885671] [ 4383]     0  4383     2832      352      11       3      164             0 irqbalance
Dec 12 07:06:31 efuovs02 kernel: [1508192.885675] [ 4412]    32  4412     4760      397      16       3       74             0 rpcbind
Dec 12 07:06:31 efuovs02 kernel: [1508192.885680] [ 4436]    29  4436     5853      354      17       3      112             0 rpc.statd
Dec 12 07:06:31 efuovs02 kernel: [1508192.885684] [ 4481]     0  4481     5790        0      15       3       50             0 rpc.idmapd
Dec 12 07:06:31 efuovs02 kernel: [1508192.885689] [ 4522]     0  4522     2106      268      12       5       28             0 fcoemon
Dec 12 07:06:31 efuovs02 kernel: [1508192.885694] [ 4537]    81  4537     5373        0      15       3       62             0 dbus-daemon
Dec 12 07:06:31 efuovs02 kernel: [1508192.885698] [ 4612]     0  4612     1030      310       9       5       41             0 o2hbmonitor
Dec 12 07:06:31 efuovs02 kernel: [1508192.885702] [ 4632]     0  4632    47286      410      51       3      221             0 cupsd
Dec 12 07:06:31 efuovs02 kernel: [1508192.885706] [ 4692]     0  4692     1039      323       9       5       30             0 acpid
Dec 12 07:06:31 efuovs02 kernel: [1508192.885711] [ 4718]     0  4718     1580      211       8       5       27             0 mcelog
Dec 12 07:06:31 efuovs02 kernel: [1508192.885715] [ 4738]     0  4738    16579      304      34       3      185         -1000 sshd
Dec 12 07:06:32 efuovs02 kernel: [1508192.885720] [ 4751]    38  4751     6644      567      18       3      162             0 ntpd
Dec 12 07:06:32 efuovs02 kernel: [1508192.885724] [ 4796]     0  4796     3235      331      15       6      143             0 xenstored
Dec 12 07:06:32 efuovs02 kernel: [1508192.885729] [ 4803]     0  4803    21126      307      21       6       69             0 xenconsoled
Dec 12 07:06:32 efuovs02 kernel: [1508192.885733] [ 4807]     0  4807    53362      393      65       3      525             0 qemu-system-i38
Dec 12 07:06:32 efuovs02 kernel: [1508192.885737] [ 4910]     0  4910    20252      478      45       3      239             0 master
Dec 12 07:06:32 efuovs02 kernel: [1508192.885742] [ 4922]    89  4922    20315      486      46       3      238             0 qmgr
Dec 12 07:06:32 efuovs02 kernel: [1508192.885746] [ 4930]     0  4930    29223      395      16       3      171             0 crond
Dec 12 07:06:32 efuovs02 kernel: [1508192.885750] [ 5036]     0  5036     5291      283      15       3       67             0 atd
Dec 12 07:06:32 efuovs02 kernel: [1508192.885755] [ 5345]     0  5345    38468      249      14       5       30             0 osmdaemon
Dec 12 07:06:32 efuovs02 kernel: [1508192.885759] [ 5366]     0  5366    85597      650      61       7     1514             0 python
Dec 12 07:06:32 efuovs02 kernel: [1508192.885764] [ 5378]     0  5378    24079      467      27       6      113             0 ovmport
Dec 12 07:06:32 efuovs02 kernel: [1508192.885768] [ 5390]     0  5390    60521      410      65       6      920             0 ovmwatch
Dec 12 07:06:32 efuovs02 kernel: [1508192.885772] [ 5405]     0  5405   208969      656      87       7     1558             0 python
Dec 12 07:06:32 efuovs02 kernel: [1508192.885777] [ 5772]     0  5772   177327     1015      89       6     1775             0 python
Dec 12 07:06:32 efuovs02 kernel: [1508192.885782] [ 5789]     0  5789    49154      741      71       7     1366             0 python
Dec 12 07:06:32 efuovs02 kernel: [1508192.885786] [ 5831]     0  5831    82559      555      70       6     1491             0 devmon
Dec 12 07:06:32 efuovs02 kernel: [1508192.885790] [ 5901]     0  5901     1031      292       9       5       18             0 mingetty
Dec 12 07:06:32 efuovs02 kernel: [1508192.885794] [ 5903]     0  5903     1031      292       8       5       19             0 mingetty
Dec 12 07:06:32 efuovs02 kernel: [1508192.885798] [ 5905]     0  5905     1031      292       9       5       19             0 mingetty
Dec 12 07:06:32 efuovs02 kernel: [1508192.885802] [ 5907]     0  5907     1031      292       9       5       19             0 mingetty
Dec 12 07:06:32 efuovs02 kernel: [1508192.885806] [ 5909]     0  5909     1031      292       9       5       19             0 mingetty
Dec 12 07:06:32 efuovs02 kernel: [1508192.885812] [26455]     0 26455    11091      458      44       5      108             0 socat
Dec 12 07:06:32 efuovs02 kernel: [1508192.885816] [27087]     0 27087    11091      458      45       5      108             0 socat
Dec 12 07:06:32 efuovs02 kernel: [1508192.885820] [27845]     0 27845    11091      458      44       5      109             0 socat
Dec 12 07:06:32 efuovs02 kernel: [1508192.885825] [27996]     0 27996    11091      458      44       5      107             0 socat
Dec 12 07:06:32 efuovs02 kernel: [1508192.885829] [14189]     0 14189    11091      458      44       5      109             0 socat
Dec 12 07:06:32 efuovs02 kernel: [1508192.885833] [16371]     0 16371    11091      458      44       5      109             0 socat
Dec 12 07:06:32 efuovs02 kernel: [1508192.885838] [14238]     0 14238     2676      256      11       3      129         -1000 udevd
Dec 12 07:06:32 efuovs02 kernel: [1508192.885842] [14374]     0 14374     2676      240      11       3      119         -1000 udevd
Dec 12 07:06:32 efuovs02 kernel: [1508192.885846] [15869]     0 15869    22957      931      62       6     1730             0 python
Dec 12 07:06:32 efuovs02 kernel: [1508192.885851] [16935]     0 16935    28695     2029      16       5       64             0 OSWatcher
Dec 12 07:06:32 efuovs02 kernel: [1508192.885855] [ 5867]    89  5867    20272     1250      45       3      229             0 pickup
Dec 12 07:06:32 efuovs02 kernel: [1508192.885860] [ 8948]     0  8948    27070      682      17       5       71             0 vmsub
Dec 12 07:06:32 efuovs02 kernel: [1508192.885864] [ 8951]     0  8951    27070      675      17       5       77             0 mpsub
Dec 12 07:06:32 efuovs02 kernel: [1508192.885868] [ 8953]     0  8953     1581      328      11       5       42             0 vmstat
Dec 12 07:06:32 efuovs02 kernel: [1508192.885872] [ 8958]     0  8958    27070      659      17       6       41             0 iosub
Dec 12 07:06:32 efuovs02 kernel: [1508192.885876] [ 8959]     0  8959    25258      441      13       5       46             0 mpstat
Dec 12 07:06:32 efuovs02 kernel: [1508192.885880] [ 8966]     0  8966    25261      420      11       5       19             0 iostat
Dec 12 07:06:32 efuovs02 kernel: [1508192.885884] [ 8971]     0  8971    27070      679      17       5        0             0 xtop
Dec 12 07:06:32 efuovs02 kernel: [1508192.885888] [ 8976]     0  8976    27070      695      17       5        0             0 psmemsub
Dec 12 07:06:32 efuovs02 kernel: [1508192.885892] [ 8977]     0  8977     3771      483      20       5        3             0 top
Dec 12 07:06:32 efuovs02 kernel: [1508192.885896] [ 8980]     0  8980    27070      680      17       5        0             0 oswsub
Dec 12 07:06:32 efuovs02 kernel: [1508192.885901] [ 8985]     0  8985    28695     1794      15       5      131             0 OSWatcher
Dec 12 07:06:32 efuovs02 kernel: [1508192.885905] [ 8986]     0  8986    27564      523      19       5        8             0 ps
Dec 12 07:06:32 efuovs02 kernel: [1508192.885909] [ 8987]     0  8987    27070       54      12       5        0             0 psmemsub
Dec 12 07:06:32 efuovs02 kernel: [1508192.885913] [ 8988]     0  8988    27070       52      11       5        0             0 oswsub
Dec 12 07:06:32 efuovs02 kernel: [1508192.885917] Out of memory: Kill process 5772 (python) score 0 or sacrifice child
Dec 12 07:06:32 efuovs02 kernel: [1508192.886216] Killed process 5772 (python) total-vm:709308kB, anon-rss:0kB, file-rss:4060kB

How to fix the OVS Kernel Memory Leak

Download the following kernel version which includes the memoy leak fix for the i40e module: link to Oracle RPM repository

kernel-uek-4.1.12-124.21.1.el6uek.x86_64.rpm
kernel-uek-firmware-4.1.12-124.21.1.el6uek.noarch.rpm


[root@efuovs02 new_Kernel]# rpm -qp --changelog kernel-uek-4.1.12-124.21.1.el6uek.x86_64.rpm | grep -B 3 28228724
warning: kernel-uek-4.1.12-124.21.1.el6uek.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID ec551f03: NOKEY
* Tue Oct 30 2018 Brian Maly <brian.maly@oracle.com> [4.1.12-124.20.8.el6uek] 
- scsi: lpfc: devloss timeout race condition caused null pointer reference (James Smart) [Orabug: 27994179] 
- scsi: qla2xxx: Fix race condition between iocb timeout and initialisation (Ben Hutchings) [Orabug: 28013813] 
- i40e: Add programming descriptors to cleaned_count (Alexander Duyck) [Orabug: 28228724] 
- i40e: Fix memory leak related filter programming status (Alexander Duyck) [Orabug: 28228724]

Install the new OVS Kernel

Using the steps reported below, the new kernel has been installed on all OVS servers of the farm.

[root@efuovs02 new_Kernel]# rpm -ivh kernel*
warning: kernel-uek-4.1.12-124.21.1.el6uek.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID ec551f03: NOKEY
Preparing... ########################################### [100%]
1:kernel-uek-firmware ########################################### [ 50%]
2:kernel-uek ########################################### [100%]
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.1.12-124.21.1.el6uek.x86_64
Found linux image: /boot/vmlinuz-4.1.12-124.14.5.el6uek.x86_64
Found initrd image: /boot/initramfs-4.1.12-124.14.5.el6uek.x86_64.img
done
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.1.12-124.21.1.el6uek.x86_64
Found initrd image: /boot/initramfs-4.1.12-124.21.1.el6uek.x86_64.img
Found linux image: /boot/vmlinuz-4.1.12-124.14.5.el6uek.x86_64
Found initrd image: /boot/initramfs-4.1.12-124.14.5.el6uek.x86_64.img
done
[root@efuovs02 new_Kernel]#

....

[root@efuovs02 new_Kernel]# reboot

[root@efuovs02 ~]# uname -a
Linux efuovs02 4.1.12-124.21.1.el6uek.x86_64 #2 SMP Tue Nov 6 13:31:13 PST 2018 x86_64 x86_64 x86_64 GNU/Linux

Patching ODA X5-2 Virtualized to version 12.1.2.6

Here is described the procedure to upgrade the ODA to the Bundle Patch 12.1.2.6.0.

This Bundle contains a BIG change because it replaces Oracle Enterprise Linux 5.11 with the version 6.7.

One critical requirement: this patch can only be installed on top of 12.1.2.5.0, to check the exisitng ODA version run:

# /opt/oracle/oak/bin/oakcli show version
Version
12.1.2.5.0

The patch can be downloaded from MOS selecting the following note: 22328442 ORACLE DATABASE APPLIANCE PATCH BUNDLE 12.1.2.6.0 (Patch)

And now let’s start with the installation:

Upload the patch on both ODA_Base (Dom1) on /tmp
Remove any Extra RPM installed by the user on the ODA_Base
Unpack both ZIP files of the patch on both ODA_Base using the following oakcli command:

[root@oda_base01 / ] # cd /tmp/Patch_12.1.2.6.0
[root@oda_base01 patch]# oakcli unpack -package /tmp/patch/p22328442_121260_Linux-x86-64_1of2.zip
Unpacking takes a while, pls wait....
Successfully unpacked the files to repository.
[root@oda_base01 patch]#
[root@oda_base01 patch]#
[root@oda_base01 patch]# oakcli unpack -package /tmp/patch/p22328442_121260_Linux-x86-64_2of2.zip
Unpacking takes a while, pls wait....
Successfully unpacked the files to repository.
[root@oda_base01 patch]#

Verify the patch compatibility on both ODA_Base with the following check:

[root@oda_base01 patch]# oakcli update -patch 12.1.2.6.0 -verify

INFO: 2016-03-31 17:07:29: Reading the metadata file now...
 Component Name Installed Version Proposed Patch Version
 --------------- ------------------ -----------------
 Controller_INT     4.230.40-3739       Up-to-date
 Controller_EXT     06.00.02.00         Up-to-date
 Expander           0018                Up-to-date
 SSD_SHARED {
 [ c1d20,c1d21,c1d22, A29A              Up-to-date
 c1d23 ]
 [ c1d16,c1d17,c1d18, A29A              Up-to-date
 c1d19 ]
 }
 HDD_LOCAL            A720              Up-to-date
 HDD_SHARED           P554              Up-to-date
 ILOM             3.2.4.42 r99377     3.2.4.52 r101649
 BIOS               30040200              30050100
 IPMI               1.8.12.0              1.8.12.4
 HMP                2.3.2.4.1             2.3.4.0.1
 OAK               12.1.2.5.0            12.1.2.6.0
 OL                    5.11                  6.7
 OVM                  3.2.9              Up-to-date
 GI_HOME           12.1.0.2.5(21359755, 12.1.0.2.160119(2194
                              21359758) 8354,21948344)
 DB_HOME {
 [ OraDb11204_home1 ] 11.2.0.4.8(21352635, 11.2.0.4.160119(2194
 21352649) 8347,21948348)
 [ OraDb12102_home2,O 12.1.0.2.5(21359755, 12.1.0.2.160119(2194
 raDb12102_home1 ] 21359758) 8354,21948344)
 }
[root@oda_base01 patch]#

Validate the Upgrade to OEL6 checking:

The minimum required version
The space requirement
The list of valid ol5 rpms.

[root@oda_base01 patch]# oakcli validate -c ol6upgrade -prechecks
INFO: Validating the OL6 upgrade -prechecks
INFO: 2016-04-09 17:11:41: Checking for minimum compatible version
SUCCESS: 2016-04-09 17:11:41: Minimum compatible version check passed

INFO: 2016-04-09 17:11:41: Checking available free space on /u01
INFO: 2016-04-09 17:11:41: Free space on /u01 is 39734588 1K-blocks
SUCCESS: 2016-04-09 17:11:41: Check for available free space passed

INFO: 2016-04-09 17:11:42: Checking for additional RPMs
SUCCESS: 2016-04-09 17:11:42: Check for additional RPMs passed

INFO: 2016-04-09 17:11:42: Checking for expected RPMs installed
INFO: 2016-04-09 17:11:42: Please take backup of ODA_BASE. Ensure ODA_BASE, Share Repos and all the VMs are shutdown cleanly before taking backup.
INFO: 2016-04-09 17:11:42: You may use eg tar -cvzf oakDom1.<node>.tar.gz /OVS/Repositories/odabaseRepo/VirtualMachines/oakDom1.
SUCCESS: 2016-04-09 17:11:42: All the expected ol5 RPMs are installed
SUCCESS: Node is ready for upgrade
[root@oda_base01 patch]#

Apply the patch to the first node using the flag -local

[root@oda_base01 patch]# /opt/oracle/oak/bin/oakcli update -patch 12.1.2.6.0 --infra -local
INFO: Local patch is running on the Node <0>
INFO: ***************************************************
INFO: ** Please do not patch both nodes simultaneously **
INFO: ***************************************************
INFO: DB, ASM, Clusterware may be stopped during the patch if required
INFO: Local Node may get rebooted automatically during the patch if necessary
Do you want to continue: [Y/N]?: Y
INFO: User has confirmed for the reboot
INFO: 2016-04-09 17:14:22: Checking for minimum compatible version
SUCCESS: 2016-04-09 17:14:22: Minimum compatible version check passed

INFO: 2016-04-09 17:14:22: Checking available free space on /u01
INFO: 2016-04-09 17:14:22: Free space on /u01 is 39733684 1K-blocks
SUCCESS: 2016-04-09 17:14:22: Check for available free space passed

INFO: 2016-04-09 17:14:22: Checking for additional RPMs
SUCCESS: 2016-04-09 17:14:22: Check for additional RPMs passed

INFO: 2016-04-09 17:14:22: Checking for expected RPMs installed
INFO: 2016-04-09 17:14:22: Please take backup of ODA_BASE. Ensure ODA_BASE, Share Repos and all the VMs are shutdown cleanly before taking backup.
INFO: 2016-04-09 17:14:22: You may use eg tar -cvzf oakDom1.<node>.tar.gz /OVS/Repositories/odabaseRepo/VirtualMachines/oakDom1.
SUCCESS: 2016-04-09 17:14:22: All the expected ol5 RPMs are installed
INFO: All the VMs except the ODABASE will be shutdown forcefully if needed
Do you want to continue : [Y/N]? : Y
INFO: Running pre-install scripts
INFO: Running prepatching on local node
INFO: Completed pre-install scripts
INFO: local patching code START
INFO: Stopping local VMs, repos and oakd...
INFO: Shutdown of local VM, Repo and OAKD on node <0>.
INFO: Stopping OAKD on the local node.
INFO: Stopped Oakd on local node
INFO: Waiting for processes to sync up...
INFO: Oakd running on remote node
INFO: Stopping local VMs...
INFO: Stopping local shared repos...
INFO: Patching Dom0 components

INFO: Patching dom0 components on Local Node... <12.1.2.6.0>
INFO: 2016-04-09 17:27:02: Attempting to patch the HMP on Dom0...
SUCCESS: 2016-04-09 17:27:08: Successfully updated the device HMP to the version 2.3.4.0.1 on Dom0
INFO: 2016-04-09 17:27:08: Attempting to patch the IPMI on Dom0...
INFO: 2016-04-09 17:27:08: Successfully updated the IPMI on Dom0
INFO: 2016-04-09 17:27:08: Attempting to patch OS on Dom0...
INFO: 2016-04-09 17:27:18: Clusterware is running on local node
INFO: 2016-04-09 17:27:18: Attempting to stop clusterware and its resources locally
SUCCESS: 2016-04-09 17:29:12: Successfully stopped the clusterware on local node

SUCCESS: 2016-04-09 17:31:36: Successfully updated the device OVM to 3.2.9

INFO: Patching ODABASE components

INFO: Patching Infrastructure on the Local Node...

INFO: 2016-04-09 17:31:38: ------------------Patching OS-------------------------
INFO: 2016-04-09 17:31:38: OSPatching : Patching will start from step 0
INFO: 2016-04-09 17:31:38: OSPatching : Performing the step 0
INFO: 2016-04-09 17:31:39: OSPatching : step 0 completed
==================================================================================
INFO: 2016-04-09 17:31:39: OSPatching : Performing the step 1
INFO: 2016-04-09 17:31:39: OSPatching : step 1 completed
==================================================================================
INFO: 2016-04-09 17:31:39: OSPatching : Performing the step 2
INFO: 2016-04-09 17:31:42: OSPatching : step 2 completed.
==================================================================================
INFO: 2016-04-09 17:31:42: OSPatching : Performing the step 3
INFO: 2016-04-09 17:31:51: OSPatching : step 3 completed
==================================================================================
INFO: 2016-04-09 17:31:51: OSPatching : Performing the step 4
INFO: 2016-04-09 17:31:51: OSPatching : step 4 completed.
==================================================================================
INFO: 2016-04-09 17:31:51: OSPatching : Performing the step 5
INFO: 2016-04-09 17:31:52: OSPatching : step 5 completed
==================================================================================
INFO: 2016-04-09 17:31:52: OSPatching : Performing the step 6
INFO: 2016-04-09 17:31:52: OSPatching : Installing OL6 RPMs. Please wait...
INFO: 2016-04-09 17:35:05: OSPatching : step 6 completed
==================================================================================
INFO: 2016-04-09 17:35:05: OSPatching : Performing the step 7
INFO: 2016-04-09 17:37:36: OSPatching : step 7 completed
==================================================================================
INFO: 2016-04-09 17:37:36: OSPatching : Performing the step 8
INFO: 2016-04-09 17:37:37: OSPatching : step 8 completed
==================================================================================
INFO: 2016-04-09 17:37:37: OSPatching : Performing the step 9
INFO: 2016-04-09 17:38:14: OSPatching : step 9 completed
==================================================================================
INFO: 2016-04-09 17:38:14: OSPatching : Performing the step 10
INFO: 2016-04-09 17:38:50: OSPatching : step 10 completed
==================================================================================
INFO: 2016-04-09 17:38:50: OSPatching : Performing the step 11
INFO: 2016-04-09 17:38:50: OSPatching : step 11 completed
==================================================================================
INFO: 2016-04-09 17:38:50: OSPatching : Performing the step 12
INFO: 2016-04-09 17:38:50: Checking for expected RPMs installed
SUCCESS: 2016-04-09 17:38:51: All the expected ol6 RPMs are installed
INFO: 2016-04-09 17:38:51: OSPatching : step 12 completed
==================================================================================
SUCCESS: 2016-04-09 17:38:51: Successfully upgraded the OS

INFO: 2016-04-09 17:38:52: ----------------------Patching IPMI---------------------
INFO: 2016-04-09 17:38:52: IPMI is already upgraded or running with the latest version

INFO: 2016-04-09 17:38:52: ------------------Patching HMP-------------------------
INFO: 2016-04-09 17:38:53: HMP is already Up-to-date
INFO: 2016-04-09 17:38:53: /usr/lib64/sun-ssm already exists.

INFO: 2016-04-09 17:38:53: ----------------------Patching OAK---------------------
SUCCESS: 2016-04-09 17:39:27: Successfully upgraded OAK

INFO: 2016-04-09 17:39:31: ----------------------Patching JDK---------------------
SUCCESS: 2016-04-09 17:39:36: Successfully upgraded JDK

INFO: local patching code END

INFO: patching summary on local node
SUCCESS: 2016-04-09 17:39:39: Successfully upgraded the HMP on Dom0
SUCCESS: 2016-04-09 17:39:39: Successfully updated the device OVM
SUCCESS: 2016-04-09 17:39:39: Successfully upgraded the OS
INFO: 2016-04-09 17:39:39: IPMI is already upgraded
INFO: 2016-04-09 17:39:39: HMP is already updated
SUCCESS: 2016-04-09 17:39:39: Successfully updated the OAK
SUCCESS: 2016-04-09 17:39:39: Successfully updated the JDK

INFO: Running post-install scripts
INFO: Running postpatch on local node
INFO: Dom0 Needs to be rebooted, will be rebooting the Dom0

Broadcast message from root@oda_base01
 (unknown) at 17:40 ...

The system is going down for power off NOW!

Validate the steps with the infrastructure post patch checks:

[root@oda_base01 ~]# /u01/app/12.1.0.2/grid/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

[root@oda_base01 ~]# /opt/oracle/oak/bin/oakcli validate -c ol6upgrade -postchecks
INFO: Validating the OL6 upgrade -postchecks

INFO: 2016-04-09 19:50:40: Current kernel is OL6
INFO: 2016-04-09 19:50:43: Checking for expected RPMs installed
SUCCESS: 2016-04-09 19:50:43: All the expected ol6 RPMs are installed

Apply the patch to the second node using the flag -local

[root@oda_base02 patch]# /opt/oracle/oak/bin/oakcli update -patch 12.1.2.6.0 --infra -local
INFO: Local patch is running on the Node <1>
INFO: ***************************************************
INFO: ** Please do not patch both nodes simultaneously **
INFO: ***************************************************
INFO: DB, ASM, Clusterware may be stopped during the patch if required
INFO: Local Node may get rebooted automatically during the patch if necessary
Do you want to continue: [Y/N]?: Y
INFO: User has confirmed for the reboot
INFO: 2016-04-09 19:58:07: Checking for minimum compatible version
SUCCESS: 2016-04-09 19:58:07: Minimum compatible version check passed

INFO: 2016-04-09 19:58:07: Checking available free space on /u01
INFO: 2016-04-09 19:58:07: Free space on /u01 is 45790328 1K-blocks
SUCCESS: 2016-04-09 19:58:07: Check for available free space passed

INFO: 2016-04-09 19:58:07: Checking for additional RPMs
SUCCESS: 2016-04-09 19:58:07: Check for additional RPMs passed

INFO: 2016-04-09 19:58:07: Checking for expected RPMs installed
INFO: 2016-04-09 19:58:08: Please take backup of ODA_BASE. Ensure ODA_BASE, Share Repos and all the VMs are shutdown cleanly before taking backup.
INFO: 2016-04-09 19:58:08: You may use eg tar -cvzf oakDom1.<node>.tar.gz /OVS/Repositories/odabaseRepo/VirtualMachines/oakDom1.
SUCCESS: 2016-04-09 19:58:08: All the expected ol5 RPMs are installed
INFO: All the VMs except the ODABASE will be shutdown forcefully if needed
Do you want to continue : [Y/N]? : Y
INFO: Running pre-install scripts
INFO: Running prepatching on local node
INFO: Completed pre-install scripts
INFO: local patching code START
INFO: Stopping local VMs, repos and oakd...
INFO: Shutdown of local VM, Repo and OAKD on node <1>.
INFO: Stopping OAKD on the local node.
INFO: Stopped Oakd on local node
INFO: Waiting for processes to sync up...
INFO: Oakd running on remote node
INFO: Stopping local VMs...
INFO: Stopping local shared repos...
INFO: Patching Dom0 components

INFO: Patching dom0 components on Local Node... <12.1.2.6.0>
INFO: 2016-04-09 20:04:26: Attempting to patch the HMP on Dom0...
SUCCESS: 2016-04-09 20:04:33: Successfully updated the device HMP to the version 2.3.4.0.1 on Dom0
INFO: 2016-04-09 20:04:33: Attempting to patch the IPMI on Dom0...
INFO: 2016-04-09 20:04:33: Successfully updated the IPMI on Dom0
INFO: 2016-04-09 20:04:33: Attempting to patch OS on Dom0...
INFO: 2016-04-09 20:04:43: Clusterware is running on local node
INFO: 2016-04-09 20:04:43: Attempting to stop clusterware and its resources locally
SUCCESS: 2016-04-09 20:08:20: Successfully stopped the clusterware on local node

SUCCESS: 2016-04-09 20:10:44: Successfully updated the device OVM to 3.2.9

INFO: Patching ODABASE components

INFO: Patching Infrastructure on the Local Node...

INFO: 2016-04-09 20:10:48: ------------------Patching OS-------------------------
INFO: 2016-04-09 20:10:48: OSPatching : Patching will start from step 0
INFO: 2016-04-09 20:10:48: OSPatching : Performing the step 0
INFO: 2016-04-09 20:10:51: OSPatching : step 0 completed
==================================================================================
INFO: 2016-04-09 20:10:51: OSPatching : Performing the step 1
INFO: 2016-04-09 20:10:51: OSPatching : step 1 completed
==================================================================================
INFO: 2016-04-09 20:10:51: OSPatching : Performing the step 2
INFO: 2016-04-09 20:10:53: OSPatching : step 2 completed.
==================================================================================
INFO: 2016-04-09 20:10:53: OSPatching : Performing the step 3
INFO: 2016-04-09 20:11:00: OSPatching : step 3 completed
==================================================================================
INFO: 2016-04-09 20:11:00: OSPatching : Performing the step 4
INFO: 2016-04-09 20:11:00: OSPatching : step 4 completed.
==================================================================================
INFO: 2016-04-09 20:11:00: OSPatching : Performing the step 5
INFO: 2016-04-09 20:11:00: OSPatching : step 5 completed
==================================================================================
INFO: 2016-04-09 20:11:00: OSPatching : Performing the step 6
INFO: 2016-04-09 20:11:00: OSPatching : Installing OL6 RPMs. Please wait...
INFO: 2016-04-09 20:14:25: OSPatching : step 6 completed
==================================================================================
INFO: 2016-04-09 20:14:25: OSPatching : Performing the step 7
INFO: 2016-04-09 20:16:58: OSPatching : step 7 completed
==================================================================================
INFO: 2016-04-09 20:16:58: OSPatching : Performing the step 8
INFO: 2016-04-09 20:16:59: OSPatching : step 8 completed
==================================================================================
INFO: 2016-04-09 20:16:59: OSPatching : Performing the step 9
INFO: 2016-04-09 20:17:35: OSPatching : step 9 completed
==================================================================================
INFO: 2016-04-09 20:17:35: OSPatching : Performing the step 10
INFO: 2016-04-09 20:18:11: OSPatching : step 10 completed
==================================================================================
INFO: 2016-04-09 20:18:11: OSPatching : Performing the step 11
INFO: 2016-04-09 20:18:11: OSPatching : step 11 completed
==================================================================================
INFO: 2016-04-09 20:18:11: OSPatching : Performing the step 12
INFO: 2016-04-09 20:18:12: Checking for expected RPMs installed
SUCCESS: 2016-04-09 20:18:12: All the expected ol6 RPMs are installed
INFO: 2016-04-09 20:18:12: OSPatching : step 12 completed
==================================================================================
SUCCESS: 2016-04-09 20:18:12: Successfully upgraded the OS

INFO: 2016-04-09 20:18:12: ----------------------Patching IPMI---------------------
INFO: 2016-04-09 20:18:13: IPMI is already upgraded or running with the latest version

INFO: 2016-04-09 20:18:13: ------------------Patching HMP-------------------------
INFO: 2016-04-09 20:18:15: HMP is already Up-to-date
INFO: 2016-04-09 20:18:15: /usr/lib64/sun-ssm already exists.

INFO: 2016-04-09 20:18:15: ----------------------Patching OAK---------------------
SUCCESS: 2016-04-09 20:18:53: Successfully upgraded OAK

INFO: 2016-04-09 20:18:56: ----------------------Patching JDK---------------------
SUCCESS: 2016-04-09 20:19:02: Successfully upgraded JDK

INFO: local patching code END

INFO: patching summary on local node
SUCCESS: 2016-04-09 20:19:06: Successfully upgraded the HMP on Dom0
SUCCESS: 2016-04-09 20:19:06: Successfully updated the device OVM
SUCCESS: 2016-04-09 20:19:06: Successfully upgraded the OS
INFO: 2016-04-09 20:19:06: IPMI is already upgraded
INFO: 2016-04-09 20:19:06: HMP is already updated
SUCCESS: 2016-04-09 20:19:06: Successfully updated the OAK
SUCCESS: 2016-04-09 20:19:06: Successfully updated the JDK

INFO: Running post-install scripts
INFO: Running postpatch on local node
INFO: Dom0 Needs to be rebooted, will be rebooting the Dom0

Broadcast message from root@oda_base02
 (unknown) at 20:20 ...

The system is going down for power off NOW!

From the first ODA_Base apply the fix to the InfiniBand connection:

[root@oda_base01 ~]# python /opt/oracle/oak/bin/infiniFixSetup.py
IB Fix requires nodes reboot. Do you want to continue? [Y/N] : Y
INFO: Checking version for IB Fix setup
INFO: Checking whether IB Fix setup is already done or not
INFO: Checking default HAVIP for IB Fix setup
INFO: Setting up IB fix
INFO: Enabling IB fix and rebooting all nodes....
[root@oda_base01 ~]#
Broadcast message from root@oda_base01
 (unknown) at 20:40 ...

The system is going down for power off NOW!

Check the correct application of the InfiniBand patch, the value of the file below should be 1

[root@oda_base01 ~]#  view /opt/oracle/oak/conf/ib_fix
1

Installation of the Grid Infrastructure patch, two available methods:

Full Downtime
Rolling Upgrade

The example below show the first method

[root@oda_base01 ~]# oakcli update -patch 12.1.2.6.0 --gi

Please enter the 'SYSASM' password : (During deployment we set the SYSASM password to 'welcome1'):
Please re-enter the 'SYSASM' password:
INFO: Running pre-install scripts
INFO: Running prepatching on node 0
INFO: Running prepatching on node 1
INFO: Completed pre-install scripts
...
...
INFO: Stopped Oakd
...
...

......
SUCCESS: All nodes in /opt/oracle/oak/temp_clunodes.txt are pingable and alive.
INFO: 2016-04-09 22:32:16: Setting up SSH for grid User
......
SUCCESS: All nodes in /opt/oracle/oak/temp_clunodes.txt are pingable and alive.
INFO: 2016-04-09 22:32:34: Patching the GI Home on the Node oda_base01 ...
INFO: 2016-04-09 22:32:34: Updating OPATCH...
INFO: 2016-04-09 22:32:36: Rolling back GI on oda_base01 (if necessary)...
INFO: 2016-04-09 22:32:39: Rolling back GI on oda_base02 (if necessary)...
INFO: 2016-04-09 22:32:46: Patching the GI Home on the Node oda_base01
INFO: 2016-04-09 22:34:02: Performing the conflict checks...
SUCCESS: 2016-04-09 22:34:16: Conflict checks passed for all the Homes
INFO: 2016-04-09 22:34:16: Checking if the patch is already applied on any of the Homes
INFO: 2016-04-09 22:34:28: Home is not Up-to-date
SUCCESS: 2016-04-09 22:37:01: Successfully stopped the Database consoles
SUCCESS: 2016-04-09 22:37:18: Successfully stopped the EM agents
INFO: 2016-04-09 22:37:23: Applying patch on /u01/app/12.1.0.2/grid Homes
INFO: 2016-04-09 22:37:23: It may take upto 15 mins. Please wait...
SUCCESS: 2016-04-09 22:50:57: Successfully applied the patch on the Home : /u01/app/12.1.0.2/grid
SUCCESS: 2016-04-09 22:51:24: Successfully started the Database consoles
SUCCESS: 2016-04-09 22:51:40: Successfully started the EM Agents
INFO: 2016-04-09 22:51:41: Patching the GI Home on the Node oda_base02
...
INFO: 2016-04-09 23:16:27: ASM is running in Flex mode


INFO: GI patching summary on node: oda_base01
SUCCESS: 2016-04-09 23:16:28: Successfully applied the patch on the Home /u01/app/12.1.0.2/grid

INFO: GI patching summary on node: oda_base02
SUCCESS: 2016-04-09 23:16:28: Successfully applied the patch on the Home /u01/app/12.1.0.2/grid

INFO: GI versions: installed <12.1.0.2.160119> expected <12.1.0.2.160119>
INFO: Running post-install scripts
INFO: Running postpatch on node 1...
INFO: Running postpatch on node 0...
...
...
INFO: Started Oakd

Installation of the RDBMS patch, two available methods:

Full Downtime
Rolling Upgrade

The example below show the first method

[root@oda_base01 ~]# oakcli update -patch 12.1.2.6.0 --database
INFO: Running pre-install scripts
INFO: Running prepatching on node 0
INFO: Running prepatching on node 1
INFO: Completed pre-install scripts
...
...

......
SUCCESS: All nodes in /opt/oracle/oak/temp_clunodes.txt are pingable and alive.
INFO: 2016-04-09 23:27:31: Getting all the possible Database Homes for patching
...
INFO: 2016-04-09 23:27:42: Patching 11.2.0.4 Database Homes on the Node oda_base01

Found the following 11.2.0.4 homes possible for patching:

HOME_NAME HOME_LOCATION
--------- -------------
OraDb11204_home1 /u01/app/oracle/product/11.2.0.4/dbhome_1

[Please note that few of the above Database Homes may be already up-to-date. They will be automatically ignored]

Would you like to patch all the above homes: Y | N ? : Y
INFO: 2016-04-09 23:29:17: Setting up SSH for the User oracle
......
SUCCESS: All nodes in /opt/oracle/oak/temp_clunodes.txt are pingable and alive.
INFO: 2016-04-09 23:29:35: Updating OPATCH
Fixing home : /u01/app/oracle/product/11.2.0.4/dbhome_1...done
INFO: 2016-04-09 23:30:33: Performing the conflict checks...
SUCCESS: 2016-04-09 23:30:43: Conflict checks passed for all the Homes
INFO: 2016-04-09 23:30:43: Checking if the patch is already applied on any of the Homes
INFO: 2016-04-09 23:30:47: Home is not Up-to-date
SUCCESS: 2016-04-09 23:31:13: Successfully stopped the Database consoles
SUCCESS: 2016-04-09 23:31:31: Successfully stopped the EM agents
INFO: 2016-04-09 23:31:36: Applying the patch on oracle home : /u01/app/oracle/product/11.2.0.4/dbhome_1 ...
SUCCESS: 2016-04-09 23:32:52: Successfully applied the patch on the Home : /u01/app/oracle/product/11.2.0.4/dbhome_1
SUCCESS: 2016-04-09 23:32:52: Successfully started the Database consoles
SUCCESS: 2016-04-09 23:33:08: Successfully started the EM Agents
INFO: 2016-04-09 23:33:17: Patching 11.2.0.4 Database Homes on the Node oda_base02
INFO: 2016-04-09 23:40:45: Running the catbundle.sql
INFO: 2016-04-09 23:40:52: Running catbundle.sql on the Database XXXXXXX
INFO: 2016-04-09 23:41:29: Running catbundle.sql on the Database YYYYYYY
INFO: 2016-04-09 23:42:07: Running catbundle.sql on the Database ZZZZZZZ
INFO: 2016-04-09 23:42:42: Running catbundle.sql on the Database WWWWWWW
...
INFO: 2016-04-09 23:47:56: Patching 12.1.0.2 Database Homes on the Node oda_base01

Found the following 12.1.0.2 homes possible for patching:

HOME_NAME HOME_LOCATION
--------- -------------
OraDb12102_home1 /u01/app/oracle/product/12.1.0.2/dbhome_1
OraDb12102_home2 /u01/app/oracle/product/12.1.0.2/dbhome_2

[Please note that few of the above Database Homes may be already up-to-date. They will be automatically ignored]

Would you like to patch all the above homes: Y | N ? : Y
INFO: 2016-04-09 23:49:11: Updating OPATCH
INFO: 2016-04-09 23:49:55: Performing the conflict checks...
SUCCESS: 2016-04-09 23:50:21: Conflict checks passed for all the Homes
INFO: 2016-04-09 23:50:21: Checking if the patch is already applied on any of the Homes
INFO: 2016-04-09 23:50:28: Home is not Up-to-date
SUCCESS: 2016-04-09 23:50:47: Successfully stopped the Database consoles
SUCCESS: 2016-04-09 23:51:04: Successfully stopped the EM agents
INFO: 2016-04-09 23:51:10: Applying patch on /u01/app/oracle/product/12.1.0.2/dbhome_1,/u01/app/oracle/product/12.1.0.2/dbhome_2 Homes
INFO: 2016-04-09 23:51:10: It may take upto 30 mins. Please wait...
SUCCESS: 2016-04-09 23:54:20: Successfully applied the patch on the Home : /u01/app/oracle/product/12.1.0.2/dbhome_1,/u01/app/oracle/product/12.1.0.2/dbhome_2
SUCCESS: 2016-04-09 23:54:20: Successfully started the Database consoles
SUCCESS: 2016-04-09 23:54:37: Successfully started the EM Agents
INFO: 2016-04-09 23:54:47: Patching 12.1.0.2 Database Homes on the Node oda_base02


INFO: DB patching summary on node: oda_base01
SUCCESS: 2016-04-01 00:03:19: Successfully applied the patch on the Home /u01/app/oracle/product/11.2.0.4/dbhome_1
SUCCESS: 2016-04-01 00:03:19: Successfully applied the patch on the Home /u01/app/oracle/product/12.1.0.2/dbhome_1,/u01/app/oracle/product/12.1.0.2/dbhome_2

INFO: DB patching summary on node: oda_base02
SUCCESS: 2016-04-01 00:03:20: Successfully applied the patch on the Home /u01/app/oracle/product/11.2.0.4/dbhome_1
SUCCESS: 2016-04-01 00:03:20: Successfully applied the patch on the Home /u01/app/oracle/product/12.1.0.2/dbhome_1,/u01/app/oracle/product/12.1.0.2/dbhome_2

Post patching validation:

[root@oda_base01 ~]# /opt/oracle/oak/bin/oakcli validate -d
INFO: oak system information and Validations
RESULT: System Software inventory details
 Reading the metadata. It takes a while...
 System Version Component Name Installed Version Supported Version
 -------------- --------------- ------------------ -----------------
 12.1.2.6.0
                  Controller_INT   4.230.40-3739     Up-to-date
                  Controller_EXT   06.00.02.00       Up-to-date
                  Expander         0018              Up-to-date
 SSD_SHARED {
 [ c1d20,c1d21,c1d22,              A29A               Up-to-date
 c1d23 ]
 [ c1d16,c1d17,c1d18,              A29A               Up-to-date
 c1d19 ]
 }
 HDD_LOCAL                         A720               Up-to-date
 HDD_SHARED                        P554               Up-to-date
 ILOM                              3.2.4.42 r99377    Up-to-date
 BIOS                              30040200           Up-to-date
 IPMI                              1.8.12.4           Up-to-date
 HMP                               2.3.4.0.1          Up-to-date
 OAK                               12.1.2.6.0         Up-to-date
 OL                                6.7                Up-to-date
 OVM                               3.2.9              Up-to-date
 GI_HOME                         12.1.0.2.160119(2194 Up-to-date
                                 8354,21948344)
 DB_HOME {
 [ OraDb11204_home1 ]            11.2.0.4.160119(2194 Up-to-date
                                 8347,21948348)
 [ OraDb12102_home2,O            12.1.0.2.160119(2194 Up-to-date
 raDb12102_home1 ]               8354,21948344)
 }
RESULT: System Information:-
 Manufacturer:Oracle Corporation
 Product Name:ORACLE SERVER X5-2
 Serial Number:1548NM102F
RESULT: BIOS Information:-
 Vendor:American Megatrends Inc.
 Version:30040200
 Release Date:04/29/2015
 BIOS Revision:4.2
 Firmware Revision:3.2
SUCCESS: Controller p1 has the IR Bypass mode set correctly
SUCCESS: Controller p2 has the IR Bypass mode set correctly
INFO: Reading ilom data, may take short while..
INFO: Read the ilom data. Doing Validations
RESULT: System ILOM Version: 3.2.4.42 r99377
RESULT: System BMC firmware version 3.02
RESULT: Powersupply PS0 V_IN=230 Volts IN_POWER=180 Watts OUT_POWER=170 Watts
RESULT: Powersupply PS1 V_IN=230 Volts IN_POWER=190 Watts OUT_POWER=160 Watts
SUCCESS: Both the powersupply are ok and functioning
RESULT: Cooling Unit FM0 fan speed F0=5000 RPM F1=4500 RPM
RESULT: Cooling Unit FM1 fan speed F0=9100 RPM F1=8000 RPM
SUCCESS: Both the cooling unit are present
RESULT: Processor P0 present Details:-
 Version:Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
 Current Speed:2300 MHz Core Enabled:18 Thread Count:36
SUCCESS: All 4 memory modules of CPU P0 ok, each module is of Size:32767 MB Type:Other Speed:2133 MHz manufacturer:Samsung
RESULT: Processor P1 present Details:-
 Version:Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
 Current Speed:2300 MHz Core Enabled:18 Thread Count:36
SUCCESS: All 4 memory modules of CPU P1 ok, each module is of Size:32767 MB Type:Other Speed:2133 MHz manufacturer:Samsung
RESULT: Total Physical System Memory is 132037124 kB
SUCCESS: All OS Disks are present and in ok state
RESULT: Power Supply=24 degrees C
INFO: Checking Operating System Storage
SUCCESS: The OS disks have the boot stamp
RESULT: Device /dev/xvda2 is mounted on / of type ext3 in (rw)
RESULT: Device /dev/xvda1 is mounted on /boot of type ext3 in (rw)
RESULT: Device /dev/xvdb1 is mounted on /u01 of type ext3 in (rw)
RESULT: / has 19218 MB free out of total 55852 MB
RESULT: /boot has 384 MB free out of total 460 MB
RESULT: /u01 has 34501 MB free out of total 93868 MB
INFO: Checking Shared Storage
RESULT: Disk HDD_E0_S00_993971920 path1 status active device sdy with status active path2 status active device sda with status active
SUCCESS: HDD_E0_S00_993971920 has both the paths up and active
RESULT: Disk HDD_E0_S01_993379760 path1 status active device sdz with status active path2 status active device sdb with status active
SUCCESS: HDD_E0_S01_993379760 has both the paths up and active
RESULT: Disk HDD_E0_S02_993993052 path1 status active device sdaa with status active path2 status active device sdc with status active
SUCCESS: HDD_E0_S02_993993052 has both the paths up and active
RESULT: Disk HDD_E0_S03_993310956 path1 status active device sdab with status active path2 status active device sdd with status active
SUCCESS: HDD_E0_S03_993310956 has both the paths up and active
RESULT: Disk HDD_E0_S04_993385276 path1 status active device sdac with status active path2 status active device sde with status active
SUCCESS: HDD_E0_S04_993385276 has both the paths up and active
RESULT: Disk HDD_E0_S05_993388928 path1 status active device sdf with status active path2 status active device sdad with status active
SUCCESS: HDD_E0_S05_993388928 has both the paths up and active
RESULT: Disk HDD_E0_S06_993310572 path1 status active device sdae with status active path2 status active device sdg with status active
SUCCESS: HDD_E0_S06_993310572 has both the paths up and active
RESULT: Disk HDD_E0_S07_991849548 path1 status active device sdh with status active path2 status active device sdaf with status active
SUCCESS: HDD_E0_S07_991849548 has both the paths up and active
RESULT: Disk HDD_E0_S08_992415004 path1 status active device sdag with status active path2 status active device sdi with status active
SUCCESS: HDD_E0_S08_992415004 has both the paths up and active
RESULT: Disk HDD_E0_S09_992392444 path1 status active device sdj with status active path2 status active device sdah with status active
SUCCESS: HDD_E0_S09_992392444 has both the paths up and active
RESULT: Disk HDD_E0_S10_992233592 path1 status active device sdai with status active path2 status active device sdk with status active
SUCCESS: HDD_E0_S10_992233592 has both the paths up and active
RESULT: Disk HDD_E0_S11_992337644 path1 status active device sdl with status active path2 status active device sdaj with status active
SUCCESS: HDD_E0_S11_992337644 has both the paths up and active
RESULT: Disk HDD_E0_S12_993363524 path1 status active device sdm with status active path2 status active device sdak with status active
SUCCESS: HDD_E0_S12_993363524 has both the paths up and active
RESULT: Disk HDD_E0_S13_992394252 path1 status active device sdn with status active path2 status active device sdal with status active
SUCCESS: HDD_E0_S13_992394252 has both the paths up and active
RESULT: Disk HDD_E0_S14_993366344 path1 status active device sdam with status active path2 status active device sdo with status active
SUCCESS: HDD_E0_S14_993366344 has both the paths up and active
RESULT: Disk HDD_E0_S15_993407552 path1 status active device sdp with status active path2 status active device sdan with status active
SUCCESS: HDD_E0_S15_993407552 has both the paths up and active
RESULT: Disk SSD_E0_S16_1313537708 path1 status active device sdq with status active path2 status active device sdao with status active
SUCCESS: SSD_E0_S16_1313537708 has both the paths up and active
RESULT: Disk SSD_E0_S17_1313522352 path1 status active device sdr with status active path2 status active device sdap with status active
SUCCESS: SSD_E0_S17_1313522352 has both the paths up and active
RESULT: Disk SSD_E0_S18_1313531936 path1 status active device sds with status active path2 status active device sdaq with status active
SUCCESS: SSD_E0_S18_1313531936 has both the paths up and active
RESULT: Disk SSD_E0_S19_1313534520 path1 status active device sdt with status active path2 status active device sdar with status active
SUCCESS: SSD_E0_S19_1313534520 has both the paths up and active
RESULT: Disk SSD_E0_S20_1313568492 path1 status active device sdu with status active path2 status active device sdas with status active
SUCCESS: SSD_E0_S20_1313568492 has both the paths up and active
RESULT: Disk SSD_E0_S21_1313571440 path1 status active device sdv with status active path2 status active device sdat with status active
SUCCESS: SSD_E0_S21_1313571440 has both the paths up and active
RESULT: Disk SSD_E0_S22_1313568380 path1 status active device sdw with status active path2 status active device sdau with status active
SUCCESS: SSD_E0_S22_1313568380 has both the paths up and active
RESULT: Disk SSD_E0_S23_1313568480 path1 status active device sdx with status active path2 status active device sdav with status active
SUCCESS: SSD_E0_S23_1313568480 has both the paths up and active
INFO: Doing oak network checks
RESULT: Detected active link for interface eth0 with link speed 10000Mb/s and cable type as TwistedPair
RESULT: Detected active link for interface eth1 with link speed 10000Mb/s and cable type as TwistedPair
WARNING: No Link detected for interface eth2 with cable type as TwistedPair
WARNING: No Link detected for interface eth3 with cable type as TwistedPair
INFO: Checking bonding interface status
RESULT: No Bond Interface Found
SUCCESS: ibbond0 is running 192.168.16.27
 It may take a while. Please wait...
 INFO : ODA Topology Verification
 INFO : Running on Node0
 INFO : Check hardware type
 SUCCESS : Type of hardware found : X5-2
 INFO : Check for Environment(Bare Metal or Virtual Machine)
 SUCCESS : Type of environment found : Virtual Machine(ODA BASE)
 SUCCESS : Number of External SCSI controllers found : 2
 INFO : Check for Controllers correct PCIe slot address
 SUCCESS : External LSI SAS controller 0 : 00:04.0
 SUCCESS : External LSI SAS controller 1 : 00:05.0
 INFO : Check if JBOD powered on
 SUCCESS : 1JBOD : Powered-on
 INFO : Check for correct number of EBODS(2 or 4)
 SUCCESS : EBOD found : 2
 INFO : Check for External Controller 0
 SUCCESS : Controller connected to correct EBOD number
 SUCCESS : Controller port connected to correct EBOD port
 SUCCESS : Overall Cable check for controller 0
 INFO : Check for External Controller 1
 SUCCESS : Controller connected to correct EBOD number
 SUCCESS : Controller port connected to correct EBOD port
 SUCCESS : Overall Cable check for Controller 1
 INFO : Check for overall status of cable validation on Node0
 SUCCESS : Overall Cable Validation on Node0
 INFO : Check Node Identification status
 SUCCESS : Node Identification
 SUCCESS : Node name based on cable configuration found : NODE0
 INFO : Check JBOD Nickname
 SUCCESS : JBOD Nickname set correctly : Oracle Database Appliance - E0
 INFO : The details for Storage Topology Validation can also be found in the log file=/opt/oracle/oak/log/oda_base01/storagetopology/StorageTopology-2016-04-01-00:06:34_28446_1789.log

One takeaway

Despite the fact that patching an Oracle Engineered system should be a straight forward task, it is recommended to carefully read the instructions (README), and the MOS notes continuously updated with bug, known issues and other related information.

Patching Exadata Machine

################################################################
## EXADATA MACHINE INFRASTRUCTURE PATCHING of 1/8 RACK ##
################################################################

This post describe step-by-step how to patch the infrastructure components of an Exadata Machine

———————————————————–
— Cell Storage Pre-requisites
———————————————————–

--Stop CRS using dcli

[root@ch01db01 oracle]# dcli -g /home/oracle/dbhosts -l root '/u01/app/12.1.0.2/grid/bin/crsctl stop crs'
 [root@ch01db01 oracle]# dcli -g /home/oracle/dbhosts -l root '/u01/app/12.1.0.2/grid/bin/crsctl stat res -t -init'

ch01db01: CRS-4639: Could not contact Oracle High Availability Services

ch01db01: CRS-4000: Command Status failed, or completed with errors.

ch01db02: CRS-4639: Could not contact Oracle High Availability Services

ch01db02: CRS-4000: Command Status failed, or completed with errors.

--Stop All Cell Storage Services
 [root@ch01db01 oracle]# dcli -g /home/oracle/cellhosts_ALL -l root "cellcli -e alter cell shutdown services all"

ch01celadm01:

ch01celadm01: Stopping the RS, CELLSRV, and MS services...
 ch01celadm01: The SHUTDOWN of services was successful.
 ch01celadm02:
 ch01celadm02: Stopping the RS, CELLSRV, and MS services...
 ch01celadm02: The SHUTDOWN of services was successful.
 ch01celadm03:
 ch01celadm03: Stopping the RS, CELLSRV, and MS services...
 ch01celadm03: The SHUTDOWN of services was successful.

[root@ch01db01 oracle]#

———————————————————–
–Cell Storage patching
———————————————————–

[root@ch01db01 patch_12.1.2.1.0.141206.1]# ./patchmgr -cells /home/oracle/cellhosts -reset_force

2016-02-05 11:17:07 +0100 :DONE: reset_force

[root@ch01db01 patch_12.1.2.1.0.141206.1]# ./patchmgr -cells /home/oracle/cellhosts -cleanup

2016-02-05 11:19:19 +0100        :Working: DO: Cleanup ...

2016-02-05 11:19:20 +0100        :SUCCESS: DONE: Cleanup

[root@ch01db01 patch_12.1.2.1.0.141206.1]# ./patchmgr -cells /home/oracle/cellhosts -patch_check_prereq

2016-02-05 11:20:56 +0100        :Working: DO: Check cells have ssh equivalence for root user. Up to 10 seconds per cell ...
 2016-02-05 11:20:57 +0100        :SUCCESS: DONE: Check cells have ssh equivalence for root user.
 2016-02-05 11:20:59 +0100        :Working: DO: Initialize files, check space and state of cell services. Up to 1 minute ...
 2016-02-05 11:21:01 +0100        :SUCCESS: DONE: Initialize files, check space and state of cell services.
 2016-02-05 11:22:19 +0100        :SUCCESS: DONE: Initialize files, check space and state of cell services.
 2016-02-05 11:22:19 +0100        :Working: DO: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction. Up to 40 minutes ...
 2016-02-05 11:22:33 +0100 Wait correction of degraded md11 due to md partner size mismatch. Up to 30 minutes.

2016-02-05 11:22:34 +0100        :SUCCESS: DONE: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction.
 2016-02-05 11:22:34 +0100        :Working: DO: Check prerequisites on all cells. Up to 2 minutes ...
 2016-02-05 11:23:38 +0100        :SUCCESS: DONE: Check prerequisites on all cells.
 2016-02-05 11:23:38 +0100        :Working: DO: Execute plugin check for Patch Check Prereq ...
 2016-02-05 11:23:38 +0100        :INFO: Patchmgr plugin start: Prereq check for exposure to bug 17854520 v1.3. Details in logfile /u02/p17885582_121210_Linux-x86-64/patch_12.1.2.1.0.141206.1/patchmgr.stdout.
 2016-02-05 11:23:38 +0100        :SUCCESS: No exposure to bug 17854520 with non-rolling patching
 2016-02-05 11:23:39 +0100        :SUCCESS: DONE: Execute plugin check for Patch Check Prereq.

[root@ch01db01 patch_12.1.2.1.0.141206.1]#
 [root@ch01db01 patch_12.1.2.1.0.141206.1]# ./patchmgr -cells /home/oracle/cellhosts -patch

********************************************************************************
 NOTE Cells will reboot during the patch or rollback process.
 NOTE For non-rolling patch or rollback, ensure all ASM instances using
 NOTE the cells are shut down for the duration of the patch or rollback.
 NOTE For rolling patch or rollback, ensure all ASM instances using
 NOTE the cells are up for the duration of the patch or rollback.

WARNING Do not start more than one instance of patchmgr.
 WARNING Do not interrupt the patchmgr session.
 WARNING Do not alter state of ASM instances during patch or rollback.
 WARNING Do not resize the screen. It may disturb the screen layout.
 WARNING Do not reboot cells or alter cell services during patch or rollback.
 WARNING Do not open log files in editor in write mode or try to alter them.

NOTE All time estimates are approximate.
 NOTE You may interrupt this patchmgr run in next 60 seconds with CONTROL-c.

********************************************************************************

2016-02-05 11:27:08 +0100        :Working: DO: Check cells have ssh equivalence for root user. Up to 10 seconds per cell ...
 2016-02-05 11:27:09 +0100        :SUCCESS: DONE: Check cells have ssh equivalence for root user.
 2016-02-05 11:27:12 +0100        :Working: DO: Initialize files, check space and state of cell services. Up to 1 minute ...
 2016-02-05 11:27:32 +0100        :SUCCESS: DONE: Initialize files, check space and state of cell services.
 2016-02-05 11:27:32 +0100        :Working: DO: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction. Up to 40 minutes ...
 2016-02-05 11:27:45 +0100 Wait correction of degraded md11 due to md partner size mismatch. Up to 30 minutes.

2016-02-05 11:27:46 +0100        :SUCCESS: DONE: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction.
 2016-02-05 11:27:46 +0100        :Working: DO: Check prerequisites on all cells. Up to 2 minutes ...
 2016-02-05 11:28:50 +0100        :SUCCESS: DONE: Check prerequisites on all cells.
 2016-02-05 11:28:50 +0100        :Working: DO: Copy the patch to all cells. Up to 3 minutes ...
 2016-02-05 11:29:22 +0100        :SUCCESS: DONE: Copy the patch to all cells.
 2016-02-05 11:29:24 +0100        :Working: DO: Execute plugin check for Patch Check Prereq ...
 2016-02-05 11:29:24 +0100        :INFO: Patchmgr plugin start: Prereq check for exposure to bug 17854520 v1.3. Details in logfile /u02/p17885582_121210_Linux-x86-64/patch_12.1.2.1.0.141206.1/patchmgr.stdout.
 2016-02-05 11:29:24 +0100        :SUCCESS: No exposure to bug 17854520 with non-rolling patching
 2016-02-05 11:29:25 +0100        :SUCCESS: DONE: Execute plugin check for Patch Check Prereq.
 2016-02-05 11:29:25 +0100 1 of 5 :Working: DO: Initiate patch on cells. Cells will remain up. Up to 5 minutes ...
 2016-02-05 11:29:37 +0100 1 of 5 :SUCCESS: DONE: Initiate patch on cells.
 2016-02-05 11:29:37 +0100 2 of 5 :Working: DO: Waiting to finish pre-reboot patch actions. Cells will remain up. Up to 45 minutes ...
 2016-02-05 11:30:37 +0100 Wait for patch pre-reboot procedures

2016-02-05 11:44:56 +0100 2 of 5 :SUCCESS: DONE: Waiting to finish pre-reboot patch actions.
 2016-02-05 11:44:56 +0100        :Working: DO: Execute plugin check for Patching ...
 2016-02-05 11:44:56 +0100        :SUCCESS: DONE: Execute plugin check for Patching.
 2016-02-05 11:44:56 +0100 3 of 5 :Working: DO: Finalize patch on cells. Cells will reboot. Up to 5 minutes ...
 2016-02-05 11:45:17 +0100 3 of 5 :SUCCESS: DONE: Finalize patch on cells.
 2016-02-05 11:45:17 +0100 4 of 5 :Working: DO: Wait for cells to reboot and come online. Up to 120 minutes ...
 2016-02-05 11:46:17 +0100 Wait for patch finalization and reboot

2016-02-05 13:09:24 +0100 4 of 5 :SUCCESS: DONE: Wait for cells to reboot and come online.
 2016-02-05 13:09:24 +0100 5 of 5 :Working: DO: Check the state of patch on cells. Up to 5 minutes ...
 2016-02-05 13:10:09 +0100 5 of 5 :SUCCESS: DONE: Check the state of patch on cells.
 2016-02-05 13:10:09 +0100        :Working: DO: Execute plugin check for Post Patch ...
 2016-02-05 13:10:10 +0100        :SUCCESS: DONE: Execute plugin check for Post Patch.

[root@ch01db01 patch_12.1.2.1.0.141206.1]#

[root@ch01db01 patch_12.1.2.1.0.141206.1]# dcli -c ch01celadm01 -l root 'imageinfo'
 ch01celadm01:
 ch01celadm01: Kernel version: 2.6.39-400.243.1.el6uek.x86_64 #1 SMP Wed Nov 26 09:15:35 PST 2014 x86_64
 ch01celadm01: Cell version: OSS_12.1.2.1.0_LINUX.X64_141206.1
 ch01celadm01: Cell rpm version: cell-12.1.2.1.0_LINUX.X64_141206.1-1.x86_64
 ch01celadm01:
 ch01celadm01: Active image version: 12.1.2.1.0.141206.1
 ch01celadm01: Active image activated: 2016-02-05 20:14:52 +0100
 ch01celadm01: Active image status: success
 ch01celadm01: Active system partition on device: /dev/md5
 ch01celadm01: Active software partition on device: /dev/md7
 ch01celadm01:
 ch01celadm01: Cell boot usb partition: /dev/sdac1
 ch01celadm01: Cell boot usb version: 12.1.2.1.0.141206.1
 ch01celadm01:
 ch01celadm01: Inactive image version: 12.1.1.1.1.140712
 ch01celadm01: Inactive image activated: 2014-08-06 11:50:09 +0200
 ch01celadm01: Inactive image status: success
 ch01celadm01: Inactive system partition on device: /dev/md6
 ch01celadm01: Inactive software partition on device: /dev/md8
 ch01celadm01:
 ch01celadm01: Inactive marker for the rollback: /boot/I_am_hd_boot.inactive
 ch01celadm01: Inactive grub config for the rollback: /boot/grub/grub.conf.inactive
 ch01celadm01: Inactive kernel version for the rollback: 2.6.39-400.128.17.el5uek
 ch01celadm01: Rollback to the inactive partitions: Possible
 [root@ch01db01 patch_12.1.2.1.0.141206.1]#

-----------------------------------------------------------
-- DB Server Patching
-----------------------------------------------------------

[root@ch01db02 dbnodeupdate]# ./dbnodeupdate.sh -h

Usage: dbnodeupdate.sh [ -u | -r | -c ] -l <baseurl|zip file> [-p] <phase> [-n] [-s] [-q] [-v] [-t] [-a] <alert.sh> [-b] [-m] | [-V] | [-h]

-u                       Upgrade
 -r                       Rollback
 -c                       Complete post actions (verify image status, cleanup, apply fixes, relink all homes, enable GI to start/start all domU's)
 -l <baseurl|zip file>    Baseurl (http or zipped iso file for the repository)
 -s                       Shutdown stack (domU's for VM) before upgrading/rolling back
 -p                       Bootstrap phase (1 or 2) only to be used when instructed by dbnodeupdate.sh
 -q                       Quiet mode (no prompting) only be used in combination with -t
 -n                       No backup will be created (Option disabled for systems being updated from Oracle Linux 5 to Oracle Linux 6)
 -t                       'to release' - used when in quiet mode or used when updating to one-offs/releases via 'latest' channel (requires 11.2.3.2.1)
 -v                       Verify prereqs only. Only to be used with -u and -l option
 -b                       Perform backup only
 -a <alert.sh>            Full path to shell script used for alert trapping
 -m                       Install / update-to exadata-sun/hp-computenode-minimum only (11.2.3.3.0 and later)
 -i                       Ignore /etc/oratab - relinking will be disabled. Only possible in combination with -c.
 -V                       Print version
 -h                       Print usage

For upgrading from releases 11.2.2.4.2 and later:
 Example using iso  : ./dbnodeupdate.sh -u -l /u01/p16432033_112321_Linux-x86-64.zip
 Example using http : ./dbnodeupdate.sh -u -l http://my-yum-repo.my-domain.com/yum/unknown/EXADATA/dbserver/11.2.3.3.0/base/x86_64/
 Example: ./dbnodeupdate.sh -u -l http://my-yum-repo.my-domain.com/yum/unknown/EXADATA/dbserver/11.2.3.2.1/base/x86_64/
 Example: ./dbnodeupdate.sh -u -l http://my-yum-repo.my-domain.com/yum/unknown/EXADATA/dbserver/11.2.3.3.0/base/x86_64/

For upgrading from releases 11.2.2.4.2 and later in quiet mode:
 Example: ./dbnodeupdate.sh -u -l /u01/p16432033_112321_Linux-x86-64.zip -q -t 11.2.3.2.1.130302

For completion steps:
 Example: ./dbnodeupdate.sh -c

For rollback:
 Example: ./dbnodeupdate.sh -r

For pre-req checks only:
 Example using iso  : ./dbnodeupdate.sh -u -l /u01/p16432033_112321_Linux-x86-64.zip -v
 Example using http : ./dbnodeupdate.sh -u -l http://my-yum-repo.my-domain.com/yum/unknown/EXADATA/dbserver/11.2.3.3.0/base/x86_64/ -v

For backup only:
 Example: ./dbnodeupdate.sh -u -l /u01/p16432033_112321_Linux-x86-64.zip -b

See MOS 1553103.1 for more examples

[root@ch01db02 dbnodeupdate]#

———————————– –DB Server patching Verification ———————————–

[root@ch01db02 dbnodeupdate]# ./dbnodeupdate.sh -u -l /u01/exapatch/p20170913_121210_Linux-x86-64/p20170913_121210_Linux-x86-64.zip -v

##########################################################################################################################
 #                                                                                                                        #
 # Guidelines for using dbnodeupdate.sh (rel. 4.18):                                                                      #
 #                                                                                                                        #
 # - Prerequisites for usage:                                                                                             #
 #         1. Refer to dbnodeupdate.sh options. See MOS 1553103.1                                                         #
 #         2. Use the latest release of dbnodeupdate.sh. See patch 16486998                                               #
 #         3. Run the prereq check with the '-v' option.                                                                  #
 #                                                                                                                        #
 #   I.e.:  ./dbnodeupdate.sh -u -l /u01/my-iso-repo.zip -v                                                               #
 #          ./dbnodeupdate.sh -u -l http://my-yum-repo -v                                                                 #
 #                                                                                                                        #
 # - Prerequisite dependency check failures can happen due to customization:                                              #
 #     - The prereq check detects dependency issues that need to be addressed prior to running a successful update.       #
 #     - Customized rpm packages may fail the built-in dependency check and system updates cannot proceed until resolved. #
 #                                                                                                                        #
 #   When upgrading from releases later than 11.2.2.4.2 to releases before 11.2.3.3.0:                                    #
 #      - Conflicting packages should be removed before proceeding the update.                                            #
 #                                                                                                                        #
 #   When upgrading to releases 11.2.3.3.0 or later:                                                                      #
 #      - When the 'exact' package dependency check fails 'minimum' package dependency check will be tried.               #
 #      - When the 'minimum' package dependency check also fails,                                                         #
 #        the conflicting packages should be removed before proceeding.                                                   #
 #                                                                                                                        #
 # - As part of the prereq checks and as part of the update, a number of rpms will be removed.                            #
 #   This removal is required to preserve Exadata functioning. This should not be confused with obsolete packages.        #
 #      - See /var/log/cellos/packages_to_be_removed.txt for details on what packages will be removed.                    #
 #                                                                                                                        #
 # - In case of any problem when filing an SR, upload the following:                                                      #
 #      - /var/log/cellos/dbnodeupdate.log                                                                                #
 #      - /var/log/cellos/dbnodeupdate.<runid>.diag                                                                       #
 #      - where <runid> is the unique number of the failing run.                                                          #
 #                                                                                                                        #
 ##########################################################################################################################

Continue ? [y/n]
 y

(*) 2016-02-05 17:06:43: Unzipping helpers (/u01/exapatch/dbnodeupdate/dbupdate-helpers.zip) to /opt/oracle.SupportTools/dbnodeupdate_helpers
 (*) 2016-02-05 17:06:43: Initializing logfile /var/log/cellos/dbnodeupdate.log
 (*) 2016-02-05 17:06:44: Collecting system configuration settings. This may take a while...
 (*) 2016-02-05 17:07:10: Validating system settings for known issues and best practices. This may take a while...
 (*) 2016-02-05 17:07:10: Checking free space in /u01/exapatch/p20170913_121210_Linux-x86-64/iso.stage.050215170641
 (*) 2016-02-05 17:07:10: Unzipping /u01/exapatch/p20170913_121210_Linux-x86-64/p20170913_121210_Linux-x86-64.zip to /u01/exapatch/p20170913_121210_Linux-x86-64/iso.stage.050215170641, this may take a while
 (*) 2016-02-05 17:07:23: Original /etc/yum.conf moved to /etc/yum.conf.050215170641, generating new yum.conf
 (*) 2016-02-05 17:07:23: Generating Exadata repository file /etc/yum.repos.d/Exadata-computenode.repo
 (*) 2016-02-05 17:07:56: Validating the specified source location.
 (*) 2016-02-05 17:07:57: Cleaning up the yum cache.

—————————————————————————————————————————–
Running in prereq check mode
—————————————————————————————————————————–

Active Image version   : 12.1.1.1.1.140712
 Active Kernel version  : 2.6.39-400.128.17.el5uek
 Active LVM Name        : /dev/mapper/VGExaDb-LVDbSys1
 Inactive Image version : 12.1.1.1.0.131219
 Inactive LVM Name      : /dev/mapper/VGExaDb-LVDbSys2
 Current user id        : root
 Action                 : upgrade
 Upgrading to           : 12.1.2.1.0.141206.1 - Oracle Linux 5->6 upgrade
 Baseurl                : file:///var/www/html/yum/unknown/EXADATA/dbserver/050215170641/x86_64/ (iso)
 Iso file               : /u01/exapatch/p20170913_121210_Linux-x86-64/iso.stage.050215170641/exadata_ol6_base_repo_141206.1.iso
 Create a backup        : Yes (backup at update mandatory when updating from OL5 to OL6)
 Shutdown stack         : No (Currently stack is down)
 RPM exclusion list     : Function not available for OL5->OL6 upgrades
 RPM obsolete list      : Function not available for OL5->OL6 upgrades
 Exact dependencies     : Function not available for OL5->OL6 upgrades
 Minimum dependencies   : Function not available for OL5->OL6 upgrades
 Logfile                : /var/log/cellos/dbnodeupdate.log (runid: 050215170641)
 Diagfile               : /var/log/cellos/dbnodeupdate.050215170641.diag
 Server model           : SUN FIRE X4170 M3
 dbnodeupdate.sh rel.   : 4.18 (always check MOS 1553103.1 for the latest release of dbnodeupdate.sh)
 Note                   : After upgrading and rebooting run './dbnodeupdate.sh -c' to finish post steps.

The following known issues will be checked for and automatically corrected by dbnodeupdate.sh:
 (*) - Issue - Fix for CVE-2014-9295 AND ELSA-2014-1974

The following known issues will be checked for but require manual follow-up:
 (*) - Issue - Yum rolling update requires fix for 11768055 when Grid Infrastructure is below 11.2.0.2 BP12
 (*) - Issue - Exafusion silently enabled for database 12.1.0.2.0 with kernel 2.6.39-400.200 and later. See MOS 1947476.1 for more details.

---------------------------------------------------------------------------------------------------------------------
 NOTE:
 When upgrading to Oracle Linux 6 a backup is required for systems configured with logical volume manager (lvm).
 It appears no backup of the current image exist on the inactive lvm.
 This means a mandatory backup will be made using dbnodeupdate.sh before the actual update starts.
 ---------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------

-------------------------------------------
 Prereq check finished successfully, check the above report for next steps.
 -----------------------------------------------------------------------------------------------------------------------------

(*) 2016-02-05 17:08:01: Cleaning up iso and temp mount points

[root@ch01db02 dbnodeupdate]#

———————————–

–DB Server patching Execution

———————————–

[root@ch01db02 dbnodeupdate]# ./dbnodeupdate.sh -u -l /u01/exapatch/p20170913_121210_Linux-x86-64/p20170913_121210_Linux-x86-64.zip

##########################################################################################################################
 #                                                                                                                        #
 # Guidelines for using dbnodeupdate.sh (rel. 4.18):                                                                      #
 #                                                                                                                        #
 # - Prerequisites for usage:                                                                                             #
 #         1. Refer to dbnodeupdate.sh options. See MOS 1553103.1                                                         #
 #         2. Use the latest release of dbnodeupdate.sh. See patch 16486998                                               #
 #         3. Run the prereq check with the '-v' option.                                                                  #
 #                                                                                                                        #
 #   I.e.:  ./dbnodeupdate.sh -u -l /u01/my-iso-repo.zip -v                                                               #
 #          ./dbnodeupdate.sh -u -l http://my-yum-repo -v                                                                 #
 #                                                                                                                        #
 # - Prerequisite dependency check failures can happen due to customization:                                              #
 #     - The prereq check detects dependency issues that need to be addressed prior to running a successful update.       #
 #     - Customized rpm packages may fail the built-in dependency check and system updates cannot proceed until resolved. #
 #                                                                                                                        #
 #   When upgrading from releases later than 11.2.2.4.2 to releases before 11.2.3.3.0:                                    #
 #      - Conflicting packages should be removed before proceeding the update.                                            #
 #                                                                                                                        #
 #   When upgrading to releases 11.2.3.3.0 or later:                                                                      #
 #      - When the 'exact' package dependency check fails 'minimum' package dependency check will be tried.               #
 #      - When the 'minimum' package dependency check also fails,                                                         #
 #        the conflicting packages should be removed before proceeding.                                                   #
 #                                                                                                                        #
 # - As part of the prereq checks and as part of the update, a number of rpms will be removed.                            #
 #   This removal is required to preserve Exadata functioning. This should not be confused with obsolete packages.        #
 #      - See /var/log/cellos/packages_to_be_removed.txt for details on what packages will be removed.                    #
 #                                                                                                                        #
 # - In case of any problem when filing an SR, upload the following:                                                      #
 #      - /var/log/cellos/dbnodeupdate.log                                                                                #
 #      - /var/log/cellos/dbnodeupdate.<runid>.diag                                                                       #
 #      - where <runid> is the unique number of the failing run.                                                          #
 #                                                                                                                        #
 ##########################################################################################################################

Continue ? [y/n]
 y

(*) 2016-02-05 17:09:38: Unzipping helpers (/u01/exapatch/dbnodeupdate/dbupdate-helpers.zip) to /opt/oracle.SupportTools/dbnodeupdate_helpers
 (*) 2016-02-05 17:09:38: Initializing logfile /var/log/cellos/dbnodeupdate.log
 (*) 2016-02-05 17:09:39: Collecting system configuration settings. This may take a while...
 (*) 2016-02-05 17:10:07: Validating system settings for known issues and best practices. This may take a while...
 (*) 2016-02-05 17:10:07: Checking free space in /u01/exapatch/p20170913_121210_Linux-x86-64/iso.stage.050215170936
 (*) 2016-02-05 17:10:07: Unzipping /u01/exapatch/p20170913_121210_Linux-x86-64/p20170913_121210_Linux-x86-64.zip to /u01/exapatch/p20170913_121210_Linux-x86-64/iso.stage.050215170936, this may take a while
 (*) 2016-02-05 17:10:19: Original /etc/yum.conf moved to /etc/yum.conf.050215170936, generating new yum.conf
 (*) 2016-02-05 17:10:19: Generating Exadata repository file /etc/yum.repos.d/Exadata-computenode.repo
 (*) 2016-02-05 17:10:42: Validating the specified source location.
 (*) 2016-02-05 17:10:43: Cleaning up the yum cache.

Active Image version   : 12.1.1.1.1.140712
 Active Kernel version  : 2.6.39-400.128.17.el5uek
 Active LVM Name        : /dev/mapper/VGExaDb-LVDbSys1
 Inactive Image version : 12.1.1.1.0.131219
 Inactive LVM Name      : /dev/mapper/VGExaDb-LVDbSys2
 Current user id        : root
 Action                 : upgrade
 Upgrading to           : 12.1.2.1.0.141206.1 - Oracle Linux 5->6 upgrade
 Baseurl                : file:///var/www/html/yum/unknown/EXADATA/dbserver/050215170936/x86_64/ (iso)
 Iso file               : /u01/exapatch/p20170913_121210_Linux-x86-64/iso.stage.050215170936/exadata_ol6_base_repo_141206.1.iso
 Create a backup        : Yes (backup at update mandatory when updating from OL5 to OL6)
 Shutdown stack         : No (Currently stack is down)
 RPM exclusion list     : Function not available for OL5->OL6 upgrades
 RPM obsolete list      : Function not available for OL5->OL6 upgrades
 Exact dependencies     : Function not available for OL5->OL6 upgrades
 Minimum dependencies   : Function not available for OL5->OL6 upgrade
 Logfile                : /var/log/cellos/dbnodeupdate.log (runid: 050215170936)
 Diagfile               : /var/log/cellos/dbnodeupdate.050215170936.diag
 Server model           : SUN FIRE X4170 M3
 dbnodeupdate.sh rel.   : 4.18 (always check MOS 1553103.1 for the latest release of dbnodeupdate.sh)
 Note                   : After upgrading and rebooting run './dbnodeupdate.sh -c' to finish post steps.

The following known issues will be checked for and automatically corrected by dbnodeupdate.sh:
 (*) - Issue - Fix for CVE-2014-9295 AND ELSA-2014-1974

The following known issues will be checked for but require manual follow-up:
 (*) - Issue - Yum rolling update requires fix for 11768055 when Grid Infrastructure is below 11.2.0.2 BP12
 (*) - Issue - Exafusion silently enabled for database 12.1.0.2.0 with kernel 2.6.39-400.200 and later. See MOS 1947476.1 for more details.

Continue ? [y/n]
 y

(*) 2016-02-05 17:11:59: Verifying GI and DB''s are shutdown
 (*) 2016-02-05 17:12:00: Collecting console history for diag purposes
 (*) 2016-02-05 17:12:32: Unmount of /boot successful
 (*) 2016-02-05 17:12:32: Check for /dev/sda1 successful
 (*) 2016-02-05 17:12:32: Mount of /boot successful
 (*) 2016-02-05 17:12:32: Disabling stack from starting
 (*) 2016-02-05 17:12:33: Performing filesystem backup to /dev/mapper/VGExaDb-LVDbSys2. Avg. 30 minutes (maximum 120) depends per environment.......
 (*) 2016-02-05 17:18:44: Backup successful
 (*) 2016-02-05 17:18:47: ExaWatcher stopped successful
 (*) 2016-02-05 17:19:07: EM Agent (in /u01/app/oracle/product/agent12c/core/12.1.0.4.0) stopped successfully
 (*) 2016-02-05 17:19:07: Capturing service status and file attributes. This may take a while...
 (*) 2016-02-05 17:19:12: Service status and file attribute report in: /etc/exadata/reports
 (*) 2016-02-05 17:19:12: Validating the specified source location.
 (*) 2016-02-05 17:19:13: Cleaning up the yum cache.
 (*) 2016-02-05 17:19:14: Executing OL5->OL6 upgrade steps, system is expected to reboot multiple times.
 (*) 2016-02-05 17:21:37: Initialize of Oracle Linux 6 Upgrade successful. Rebooting now...

Broadcast message from root (pts/0) (Thu Feb  5 17:21:37 2015):

The system is going down for reboot NOW!

[root@ch01db02 dbnodeupdate]#

[root@ch01db02 dbnodeupdate]# ./dbnodeupdate.sh -c

———————————–
–Output new Image Version
———————————–

[root@ch01db01 ibdiagtools]# imageinfo

Kernel version: 2.6.39-400.243.1.el6uek.x86_64 #1 SMP Wed Nov 26 09:15:35 PST 2014 x86_64
 Image version: 12.1.2.1.0.141206.1
 Image activated: 2016-02-05 18:24:46 +0100
 Image status: success
 System partition on device: /dev/mapper/VGExaDb-LVDbSys1

My Oracle Notes

Tag: OVM