Oracle VM Server 3.4.5 – Kernel Memory Leak

Oracle VM Server instability on version 3.4.5 due to a Oracle Unbreakable Enterprise Kernel (UEK) bug.

The kernel version 4.1.12-124.14.5.el6uek.x86_64 has introduced a memory leak of the network module i40e.

Here is reported the backtraces collected after the problem from the /var/log/messages:

Dec 12 07:06:30 efuovs02 kernel: [1508192.885203] ntpd invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
Dec 12 07:06:30 efuovs02 kernel: [1508192.885208] ntpd cpuset=/ mems_allowed=0
Dec 12 07:06:30 efuovs02 kernel: [1508192.885217] CPU: 3 PID: 4751 Comm: ntpd Not tainted 4.1.12-124.14.5.el6uek.x86_64 #2
Dec 12 07:06:30 efuovs02 kernel: [1508192.885221] Hardware name: HPE ProLiant DL360 Gen10/ProLiant DL360 Gen10, BIOS U32 02/14/2018
Dec 12 07:06:30 efuovs02 kernel: [1508192.885224]  0000000000000000 ffff8804484cf678 ffffffff816e4bdb ffff88044d44aa00
Dec 12 07:06:30 efuovs02 kernel: [1508192.885230]  0000000000000000 ffff8804484cf708 ffffffff816e32d1 01ff8804484cf688
Dec 12 07:06:30 efuovs02 kernel: [1508192.885235]  ffff8804484cf718 ffff8804484cf6c8 ffffffff811fc561 ffff8804484cf800
Dec 12 07:06:30 efuovs02 kernel: [1508192.885241] Call Trace:
Dec 12 07:06:30 efuovs02 kernel: [1508192.885251]  [<ffffffff816e4bdb>] dump_stack+0x63/0x81
Dec 12 07:06:30 efuovs02 kernel: [1508192.885256]  [<ffffffff816e32d1>] dump_header+0x7f/0x1f3
Dec 12 07:06:30 efuovs02 kernel: [1508192.885264]  [<ffffffff811fc561>] ? vmpressure+0x21/0x90
Dec 12 07:06:30 efuovs02 kernel: [1508192.885272]  [<ffffffff8118e53c>] oom_kill_process+0x1cc/0x3c0
Dec 12 07:06:30 efuovs02 kernel: [1508192.885283]  [<ffffffff8108de0e>] ? has_capability_noaudit+0x1e/0x30
Dec 12 07:06:31 efuovs02 kernel: [1508192.885288]  [<ffffffff8118eaab>] __out_of_memory+0x31b/0x530
Dec 12 07:06:31 efuovs02 kernel: [1508192.885294]  [<ffffffff8118ee5b>] out_of_memory+0x5b/0x80
Dec 12 07:06:31 efuovs02 kernel: [1508192.885300]  [<ffffffff81194d42>] __alloc_pages_nodemask+0x952/0xab0
Dec 12 07:06:31 efuovs02 kernel: [1508192.885307]  [<ffffffff811de28d>] alloc_pages_vma+0xbd/0x260
Dec 12 07:06:31 efuovs02 kernel: [1508192.885311]  [<ffffffff8118a59e>] ? find_get_entry+0x1e/0xc0
Dec 12 07:06:31 efuovs02 kernel: [1508192.885317]  [<ffffffff811ce6bd>] read_swap_cache_async+0xed/0x170
Dec 12 07:06:31 efuovs02 kernel: [1508192.885322]  [<ffffffff811ce82d>] swapin_readahead+0xed/0x190
Dec 12 07:06:31 efuovs02 kernel: [1508192.885328]  [<ffffffff811bbfe0>] handle_mm_fault+0x12d0/0x1770
Dec 12 07:06:31 efuovs02 kernel: [1508192.885335]  [<ffffffff8121d910>] ? poll_select_copy_remaining+0x130/0x130
Dec 12 07:06:31 efuovs02 kernel: [1508192.885340]  [<ffffffff8106d57f>] __do_page_fault+0x1af/0x480
Dec 12 07:06:31 efuovs02 kernel: [1508192.885346]  [<ffffffff816f2c1c>] ? page_fault+0xcc/0x120
Dec 12 07:06:31 efuovs02 kernel: [1508192.885350]  [<ffffffff8106d87f>] do_page_fault+0x2f/0x80
Dec 12 07:06:31 efuovs02 kernel: [1508192.885354]  [<ffffffff816f2be4>] ? page_fault+0x94/0x120
Dec 12 07:06:31 efuovs02 kernel: [1508192.885359]  [<ffffffff816f2bdd>] ? page_fault+0x8d/0x120
Dec 12 07:06:31 efuovs02 kernel: [1508192.885363]  [<ffffffff816f2bd6>] ? page_fault+0x86/0x120
Dec 12 07:06:31 efuovs02 kernel: [1508192.885367]  [<ffffffff816f2c5f>] page_fault+0x10f/0x120
Dec 12 07:06:31 efuovs02 kernel: [1508192.885375]  [<ffffffff813316c5>] ? copy_user_enhanced_fast_string+0x5/0x10
Dec 12 07:06:31 efuovs02 kernel: [1508192.885379]  [<ffffffff8121d7d1>] ? set_fd_set+0x21/0x30
Dec 12 07:06:31 efuovs02 kernel: [1508192.885384]  [<ffffffff8121e5aa>] core_sys_select+0x1fa/0x2f0
Dec 12 07:06:31 efuovs02 kernel: [1508192.885392]  [<ffffffff810f8fc3>] ? ntp_notify_cmos_timer+0x23/0x30
Dec 12 07:06:31 efuovs02 kernel: [1508192.885396]  [<ffffffff810f8a1d>] ? do_adjtimex+0xed/0x100
Dec 12 07:06:31 efuovs02 kernel: [1508192.885402]  [<ffffffff810ed3ac>] ? SYSC_adjtimex+0x4c/0x80
Dec 12 07:06:31 efuovs02 kernel: [1508192.885410]  [<ffffffff810209e9>] ? read_tsc+0x9/0x10
Dec 12 07:06:31 efuovs02 kernel: [1508192.885414]  [<ffffffff810f68cb>] ? ktime_get_ts64+0x4b/0x110
Dec 12 07:06:31 efuovs02 kernel: [1508192.885419]  [<ffffffff8121e74b>] SyS_select+0xab/0x100
Dec 12 07:06:31 efuovs02 kernel: [1508192.885424]  [<ffffffff816ed451>] ? system_call_after_swapgs+0xdb/0x18c
Dec 12 07:06:31 efuovs02 kernel: [1508192.885428]  [<ffffffff816ed51a>] system_call_fastpath+0x18/0xd4
Dec 12 07:06:31 efuovs02 kernel: [1508192.885457] Mem-Info:
Dec 12 07:06:31 efuovs02 kernel: [1508192.885469] active_anon:1452 inactive_anon:1426 isolated_anon:65
Dec 12 07:06:31 efuovs02 kernel: [1508192.885469]  active_file:4559 inactive_file:873 isolated_file:0
Dec 12 07:06:31 efuovs02 kernel: [1508192.885469]  unevictable:1547 dirty:20 writeback:31 unstable:0
Dec 12 07:06:31 efuovs02 kernel: [1508192.885469]  slab_reclaimable:6776 slab_unreclaimable:8649
Dec 12 07:06:31 efuovs02 kernel: [1508192.885469]  mapped:3007 shmem:0 pagetables:1705 bounce:0
Dec 12 07:06:31 efuovs02 kernel: [1508192.885469]  free:33536 free_pcp:918 free_cma:0
Dec 12 07:06:31 efuovs02 kernel: [1508192.885483] Node 0 DMA free:15740kB min:60kB low:72kB high:84kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15988kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Dec 12 07:06:31 efuovs02 kernel: [1508192.885499] lowmem_reserve[]: 0 2661 15921 15921
Dec 12 07:06:31 efuovs02 kernel: [1508192.885508] Node 0 DMA32 free:64064kB min:11076kB low:13844kB high:16612kB active_anon:5912kB inactive_anon:5876kB active_file:112kB inactive_file:52kB unevictable:836kB isolated(anon):256kB isolated(file):0kB present:2781336kB managed:2751088kB mlocked:836kB dirty:0kB writeback:0kB mapped:668kB shmem:0kB slab_reclaimable:4792kB slab_unreclaimable:6452kB kernel_stack:912kB pagetables:1692kB unstable:0kB bounce:0kB free_pcp:1156kB local_pcp:248kB free_cma:0kB writeback_tmp:0kB pages_scanned:619296 all_unreclaimable? yes
Dec 12 07:06:31 efuovs02 kernel: [1508192.885524] lowmem_reserve[]: 0 0 13260 13260
Dec 12 07:06:31 efuovs02 kernel: [1508192.885532] Node 0 Normal free:54340kB min:54392kB low:67988kB high:81584kB active_anon:0kB inactive_anon:0kB active_file:18124kB inactive_file:3440kB unevictable:5352kB isolated(anon):4kB isolated(file):0kB present:13979888kB managed:13534768kB mlocked:5352kB dirty:80kB writeback:124kB mapped:11360kB shmem:0kB slab_reclaimable:22312kB slab_unreclaimable:28144kB kernel_stack:2880kB pagetables:5128kB unstable:0kB bounce:0kB free_pcp:2516kB local_pcp:572kB free_cma:0kB writeback_tmp:0kB pages_scanned:129384 all_unreclaimable? yes
Dec 12 07:06:31 efuovs02 kernel: [1508192.885546] lowmem_reserve[]: 0 0 0 0
Dec 12 07:06:31 efuovs02 kernel: [1508192.885551] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 1*32kB (U) 1*64kB (U) 0*128kB 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15740kB
Dec 12 07:06:31 efuovs02 kernel: [1508192.885573] Node 0 DMA32: 143*4kB (UE) 111*8kB (UEM) 209*16kB (UE) 140*32kB (UE) 94*64kB (UEM) 57*128kB (UEM) 28*256kB (UEM) 7*512kB (UEM) 4*1024kB (EM) 9*2048kB (MR) 2*4096kB (MR) = 64068kB
Dec 12 07:06:31 efuovs02 kernel: [1508192.885596] Node 0 Normal: 8736*4kB (UEM) 1360*8kB (UEM) 208*16kB (UEM) 32*32kB (UE) 2*64kB (UE) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 54400kB
Dec 12 07:06:31 efuovs02 kernel: [1508192.885615] 8002 total pagecache pages
Dec 12 07:06:31 efuovs02 kernel: [1508192.885618] 1494 pages in swap cache
Dec 12 07:06:31 efuovs02 kernel: [1508192.885621] Swap cache stats: add 3717250313, delete 3717248819, find 2895172777/5168362256
Dec 12 07:06:31 efuovs02 kernel: [1508192.885624] Free swap  = 4129656kB
Dec 12 07:06:31 efuovs02 kernel: [1508192.885626] Total swap = 4194300kB
Dec 12 07:06:31 efuovs02 kernel: [1508192.885628] 4194303 pages RAM
Dec 12 07:06:31 efuovs02 kernel: [1508192.885630] 0 pages HighMem/MovableOnly
Dec 12 07:06:31 efuovs02 kernel: [1508192.885632] 118864 pages reserved
Dec 12 07:06:31 efuovs02 kernel: [1508192.885634] 0 pages cma reserved
Dec 12 07:06:31 efuovs02 kernel: [1508192.885636] 0 pages hwpoisoned
Dec 12 07:06:31 efuovs02 kernel: [1508192.885638] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Dec 12 07:06:31 efuovs02 kernel: [1508192.885650] [  983]     0   983     2677      266      11       3      111         -1000 udevd
Dec 12 07:06:31 efuovs02 kernel: [1508192.885657] [ 3783]     0  3783   125771     1414      48       5        0         -1000 multipathd
Dec 12 07:06:31 efuovs02 kernel: [1508192.885662] [ 4334]     0  4334     6944      399      15       3      108         -1000 auditd
Dec 12 07:06:31 efuovs02 kernel: [1508192.885667] [ 4368]     0  4368    61281      438      23       3      377             0 rsyslogd
Dec 12 07:06:31 efuovs02 kernel: [1508192.885671] [ 4383]     0  4383     2832      352      11       3      164             0 irqbalance
Dec 12 07:06:31 efuovs02 kernel: [1508192.885675] [ 4412]    32  4412     4760      397      16       3       74             0 rpcbind
Dec 12 07:06:31 efuovs02 kernel: [1508192.885680] [ 4436]    29  4436     5853      354      17       3      112             0 rpc.statd
Dec 12 07:06:31 efuovs02 kernel: [1508192.885684] [ 4481]     0  4481     5790        0      15       3       50             0 rpc.idmapd
Dec 12 07:06:31 efuovs02 kernel: [1508192.885689] [ 4522]     0  4522     2106      268      12       5       28             0 fcoemon
Dec 12 07:06:31 efuovs02 kernel: [1508192.885694] [ 4537]    81  4537     5373        0      15       3       62             0 dbus-daemon
Dec 12 07:06:31 efuovs02 kernel: [1508192.885698] [ 4612]     0  4612     1030      310       9       5       41             0 o2hbmonitor
Dec 12 07:06:31 efuovs02 kernel: [1508192.885702] [ 4632]     0  4632    47286      410      51       3      221             0 cupsd
Dec 12 07:06:31 efuovs02 kernel: [1508192.885706] [ 4692]     0  4692     1039      323       9       5       30             0 acpid
Dec 12 07:06:31 efuovs02 kernel: [1508192.885711] [ 4718]     0  4718     1580      211       8       5       27             0 mcelog
Dec 12 07:06:31 efuovs02 kernel: [1508192.885715] [ 4738]     0  4738    16579      304      34       3      185         -1000 sshd
Dec 12 07:06:32 efuovs02 kernel: [1508192.885720] [ 4751]    38  4751     6644      567      18       3      162             0 ntpd
Dec 12 07:06:32 efuovs02 kernel: [1508192.885724] [ 4796]     0  4796     3235      331      15       6      143             0 xenstored
Dec 12 07:06:32 efuovs02 kernel: [1508192.885729] [ 4803]     0  4803    21126      307      21       6       69             0 xenconsoled
Dec 12 07:06:32 efuovs02 kernel: [1508192.885733] [ 4807]     0  4807    53362      393      65       3      525             0 qemu-system-i38
Dec 12 07:06:32 efuovs02 kernel: [1508192.885737] [ 4910]     0  4910    20252      478      45       3      239             0 master
Dec 12 07:06:32 efuovs02 kernel: [1508192.885742] [ 4922]    89  4922    20315      486      46       3      238             0 qmgr
Dec 12 07:06:32 efuovs02 kernel: [1508192.885746] [ 4930]     0  4930    29223      395      16       3      171             0 crond
Dec 12 07:06:32 efuovs02 kernel: [1508192.885750] [ 5036]     0  5036     5291      283      15       3       67             0 atd
Dec 12 07:06:32 efuovs02 kernel: [1508192.885755] [ 5345]     0  5345    38468      249      14       5       30             0 osmdaemon
Dec 12 07:06:32 efuovs02 kernel: [1508192.885759] [ 5366]     0  5366    85597      650      61       7     1514             0 python
Dec 12 07:06:32 efuovs02 kernel: [1508192.885764] [ 5378]     0  5378    24079      467      27       6      113             0 ovmport
Dec 12 07:06:32 efuovs02 kernel: [1508192.885768] [ 5390]     0  5390    60521      410      65       6      920             0 ovmwatch
Dec 12 07:06:32 efuovs02 kernel: [1508192.885772] [ 5405]     0  5405   208969      656      87       7     1558             0 python
Dec 12 07:06:32 efuovs02 kernel: [1508192.885777] [ 5772]     0  5772   177327     1015      89       6     1775             0 python
Dec 12 07:06:32 efuovs02 kernel: [1508192.885782] [ 5789]     0  5789    49154      741      71       7     1366             0 python
Dec 12 07:06:32 efuovs02 kernel: [1508192.885786] [ 5831]     0  5831    82559      555      70       6     1491             0 devmon
Dec 12 07:06:32 efuovs02 kernel: [1508192.885790] [ 5901]     0  5901     1031      292       9       5       18             0 mingetty
Dec 12 07:06:32 efuovs02 kernel: [1508192.885794] [ 5903]     0  5903     1031      292       8       5       19             0 mingetty
Dec 12 07:06:32 efuovs02 kernel: [1508192.885798] [ 5905]     0  5905     1031      292       9       5       19             0 mingetty
Dec 12 07:06:32 efuovs02 kernel: [1508192.885802] [ 5907]     0  5907     1031      292       9       5       19             0 mingetty
Dec 12 07:06:32 efuovs02 kernel: [1508192.885806] [ 5909]     0  5909     1031      292       9       5       19             0 mingetty
Dec 12 07:06:32 efuovs02 kernel: [1508192.885812] [26455]     0 26455    11091      458      44       5      108             0 socat
Dec 12 07:06:32 efuovs02 kernel: [1508192.885816] [27087]     0 27087    11091      458      45       5      108             0 socat
Dec 12 07:06:32 efuovs02 kernel: [1508192.885820] [27845]     0 27845    11091      458      44       5      109             0 socat
Dec 12 07:06:32 efuovs02 kernel: [1508192.885825] [27996]     0 27996    11091      458      44       5      107             0 socat
Dec 12 07:06:32 efuovs02 kernel: [1508192.885829] [14189]     0 14189    11091      458      44       5      109             0 socat
Dec 12 07:06:32 efuovs02 kernel: [1508192.885833] [16371]     0 16371    11091      458      44       5      109             0 socat
Dec 12 07:06:32 efuovs02 kernel: [1508192.885838] [14238]     0 14238     2676      256      11       3      129         -1000 udevd
Dec 12 07:06:32 efuovs02 kernel: [1508192.885842] [14374]     0 14374     2676      240      11       3      119         -1000 udevd
Dec 12 07:06:32 efuovs02 kernel: [1508192.885846] [15869]     0 15869    22957      931      62       6     1730             0 python
Dec 12 07:06:32 efuovs02 kernel: [1508192.885851] [16935]     0 16935    28695     2029      16       5       64             0 OSWatcher
Dec 12 07:06:32 efuovs02 kernel: [1508192.885855] [ 5867]    89  5867    20272     1250      45       3      229             0 pickup
Dec 12 07:06:32 efuovs02 kernel: [1508192.885860] [ 8948]     0  8948    27070      682      17       5       71             0 vmsub
Dec 12 07:06:32 efuovs02 kernel: [1508192.885864] [ 8951]     0  8951    27070      675      17       5       77             0 mpsub
Dec 12 07:06:32 efuovs02 kernel: [1508192.885868] [ 8953]     0  8953     1581      328      11       5       42             0 vmstat
Dec 12 07:06:32 efuovs02 kernel: [1508192.885872] [ 8958]     0  8958    27070      659      17       6       41             0 iosub
Dec 12 07:06:32 efuovs02 kernel: [1508192.885876] [ 8959]     0  8959    25258      441      13       5       46             0 mpstat
Dec 12 07:06:32 efuovs02 kernel: [1508192.885880] [ 8966]     0  8966    25261      420      11       5       19             0 iostat
Dec 12 07:06:32 efuovs02 kernel: [1508192.885884] [ 8971]     0  8971    27070      679      17       5        0             0 xtop
Dec 12 07:06:32 efuovs02 kernel: [1508192.885888] [ 8976]     0  8976    27070      695      17       5        0             0 psmemsub
Dec 12 07:06:32 efuovs02 kernel: [1508192.885892] [ 8977]     0  8977     3771      483      20       5        3             0 top
Dec 12 07:06:32 efuovs02 kernel: [1508192.885896] [ 8980]     0  8980    27070      680      17       5        0             0 oswsub
Dec 12 07:06:32 efuovs02 kernel: [1508192.885901] [ 8985]     0  8985    28695     1794      15       5      131             0 OSWatcher
Dec 12 07:06:32 efuovs02 kernel: [1508192.885905] [ 8986]     0  8986    27564      523      19       5        8             0 ps
Dec 12 07:06:32 efuovs02 kernel: [1508192.885909] [ 8987]     0  8987    27070       54      12       5        0             0 psmemsub
Dec 12 07:06:32 efuovs02 kernel: [1508192.885913] [ 8988]     0  8988    27070       52      11       5        0             0 oswsub
Dec 12 07:06:32 efuovs02 kernel: [1508192.885917] Out of memory: Kill process 5772 (python) score 0 or sacrifice child
Dec 12 07:06:32 efuovs02 kernel: [1508192.886216] Killed process 5772 (python) total-vm:709308kB, anon-rss:0kB, file-rss:4060kB

How to fix the OVS Kernel Memory Leak

Download the following kernel version which includes the memoy leak fix for the i40e module: link to Oracle RPM repository

kernel-uek-4.1.12-124.21.1.el6uek.x86_64.rpm
kernel-uek-firmware-4.1.12-124.21.1.el6uek.noarch.rpm


[root@efuovs02 new_Kernel]# rpm -qp --changelog kernel-uek-4.1.12-124.21.1.el6uek.x86_64.rpm | grep -B 3 28228724
warning: kernel-uek-4.1.12-124.21.1.el6uek.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID ec551f03: NOKEY
* Tue Oct 30 2018 Brian Maly <brian.maly@oracle.com> [4.1.12-124.20.8.el6uek] 
- scsi: lpfc: devloss timeout race condition caused null pointer reference (James Smart) [Orabug: 27994179] 
- scsi: qla2xxx: Fix race condition between iocb timeout and initialisation (Ben Hutchings) [Orabug: 28013813] 
- i40e: Add programming descriptors to cleaned_count (Alexander Duyck) [Orabug: 28228724] 
- i40e: Fix memory leak related filter programming status (Alexander Duyck) [Orabug: 28228724]

Install the new OVS Kernel

Using the steps reported below, the new kernel has been installed on all OVS servers of the farm.

[root@efuovs02 new_Kernel]# rpm -ivh kernel*
warning: kernel-uek-4.1.12-124.21.1.el6uek.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID ec551f03: NOKEY
Preparing... ########################################### [100%]
1:kernel-uek-firmware ########################################### [ 50%]
2:kernel-uek ########################################### [100%]
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.1.12-124.21.1.el6uek.x86_64
Found linux image: /boot/vmlinuz-4.1.12-124.14.5.el6uek.x86_64
Found initrd image: /boot/initramfs-4.1.12-124.14.5.el6uek.x86_64.img
done
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.1.12-124.21.1.el6uek.x86_64
Found initrd image: /boot/initramfs-4.1.12-124.21.1.el6uek.x86_64.img
Found linux image: /boot/vmlinuz-4.1.12-124.14.5.el6uek.x86_64
Found initrd image: /boot/initramfs-4.1.12-124.14.5.el6uek.x86_64.img
done
[root@efuovs02 new_Kernel]#

....

[root@efuovs02 new_Kernel]# reboot

[root@efuovs02 ~]# uname -a
Linux efuovs02 4.1.12-124.21.1.el6uek.x86_64 #2 SMP Tue Nov 6 13:31:13 PST 2018 x86_64 x86_64 x86_64 GNU/Linux

Grid Management DB filling up ASM disk space

Recently I discovered on Oracle Grid Infrastructure 12cR2 that the ASM disk group hosting the Management DB (-MGMTDB) was filling up the disk space very quickly.

This is due to a bug on the oclumon data purge procedure.

To fix the problem, two possibilities are available:

Recreate the Management DB
Manually truncate the tables not purged and shrinking the Tablespace Size

Below are described the two options.

Option 1 – Recreate the Management DB

As root user on each cluster node:

# /u01/app/12.2.0.1/grid/bin/crsctl stop res ora.crf -init
# /u01/app/12.2.0.1/grid/bin/crsctl modify res ora.crf -attr ENABLED=0 -init

As Grid from the local node hosting the Management Database Instance run the commands:

$ /u01/app/12.2.0.1/grid/bin/srvctl status mgmtdb
$ /u01/app/12.2.0.1/grid/bin/dbca -silent -deleteDatabase -sourceDB -MGMTDB
Connecting to database
4% complete
9% complete
14% complete
19% complete
23% complete
28% complete
47% complete
Updating network configuration files
48% complete
52% complete
Deleting instance and datafiles
76% complete
100% complete

How to recreate the MGMTDB:

$ /u01/app/12.2.0.1/grid/bin/dbca -silent -createDatabase -createAsContainerDatabase true -templateName MGMTSeed_Database.dbc
-sid -MGMTDB
-gdbName _mgmtdb
-storageType ASM
-diskGroupName GIMR
-datafileJarLocation <GI HOME>/assistants/dbca/templates
-characterset AL32UTF8
-autoGeneratePassword           
-skipUserTemplateCheck

Create the pluggable GIMR database

$ /u01/app/12.2.0.1/grid/bin/mgmtca -local

Option 2 – Manually truncate the tables

As root user stop and disable ora.crf resource on each cluster node:

# /u01/app/12.2.0.1/grid/bin/crsctl stop res ora.crf -init
# /u01/app/12.2.0.1/grid/bin/crsctl modify res ora.crf -attr ENABLED=0 -init

Connect to MGMTDB and identify the segments to truncate:

export ORACLE_SID=-MGMTDB
$ORACLE_HOME/bin/sqlplus / as sysdba
SQL> select pdb_name from dba_pdbs where pdb_name!='PDB$SEED';

SQL> alter session set container=GIMR_DSCREP_10;

Session altered.

SQL> col obj format a50
SQL> select owner||'.'||SEGMENT_NAME obj, BYTES from dba_segments where owner='CHM' order by 2 asc;

Likely those two tables are much bigger than the rest :

CHM.CHMOS_PROCESS_INT_TBL
CHM.CHMOS_DEVICE_INT_TBL

Truncate the tables:

SQL> truncate table CHM.CHMOS_PROCESS_INT_TBL;
SQL> truncate table CHM.CHMOS_DEVICE_INT_TBL;

Then if needed shrink the tablespace and job done!

RHEL 7.4 fails to mount ACFS File System due to KMOD package

After a fresh OS installation or an upgrade to RHEL 7.4, any attempt to install ACFS drivers will fail with the following message: “ACFS-9459 ADVM/ACFS is not supported on this OS version”

The error persists even if the Oracle Grid Infrastructure software includes the Patch 26247490: 12.2 ACFS MODULE ERRORS & CRASH DURING MODULE LOAD & UNLOAD WITH OL7U4 RHCK.

This problem has been identified by Oracle with BUG 26320387 – 7.4 kmod weak-modules not checking kABI compatibility correctly

And by Red Hat Bugzilla bug: 1477073 – 7.4 kmod weak-modules –dry-run changed output format missing ‘is compatible’ messages.

root@oel7node06:/u01/app/12.2.0.1/grid/crs/install# /u01/app/12.2.0.1/grid/bin/acfsroot install
ACFS-9459: ADVM/ACFS is not supported on this OS version: '3.10.0-514.6.1.el7.x86_64'

root@oel7node06:~# /sbin/lsmod | grep oracle
oracleadvm 776830 7
oracleoks 654476 1 oracleadvm
oracleafd 205543 1

The current Workaround consists in downgrade the version of the kmod RPM to kmod-20-9.el7.x86_64.

root@oel7node06:~# yum downgrade kmod-20-9.el7

After the package downgrade the ACFS drivers are correcly loaded:

root@oel7node06:~# /sbin/lsmod | grep oracle
oracleacfs 4597925 2
oracleadvm 776830 8
oracleoks 654476 2 oracleacfs,oracleadvm
oracleafd 205543 1

New Oracle version (12.2.0.1) old BUG!

In June 2016 I posted the following BUG: Bug on Oracle 12c Multitenant & PDB Clone as Snapshot Copy, promising to post an update once the version 12cR2 is available, because in the service request, originally opened with the version 12.1.0.2 Oracle stated that the bug would be fixed in 12cR2.

I was so impatient, that just few hours after the general availability of the Oracle Database 12c Release 2 I created a new cluster and tested the resolution.

For the record, it states that the resolution of this bug is important for one of my clients, where we have implemented the snapshot PDB on the application development lifecycle.

So let’s see if the bug has been fixed!

SQL*Plus: Release 12.2.0.1.0 Production on Wed Mar 1 21:06:54 2017

Copyright (c) 1982, 2016, Oracle. All rights reserved.

Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production


SQL> CREATE PLUGGABLE DATABASE PDBACFS1_SNAP1 from PDBACFS1 SNAPSHOT COPY;

Pluggable database created.

SQL> ALTER PLUGGABLE DATABASE PDBACFS1_SNAP1 OPEN instances=all;

Pluggable database altered.

SQL> select CON_ID, NAME, OPEN_MODE, SNAPSHOT_PARENT_CON_ID from v$pdbs where NAME in ('PDBACFS1','PDBACFS1_SNAP1');

 CON_ID     NAME               OPEN_MODE  SNAPSHOT_PARENT_CON_ID
---------- ------------------- ---------- ----------------------
 5          PDBACFS1            READ WRITE
 6          PDBACFS1_SNAP1      READ WRITE               <-- This should be 5 but is NULL

2 rows selected.

To a certain point of view progress has been made, in version 12.1.0.2 the column SNAPSHOT_PARENT_CON_ID was always zero (0) now is null!

I’m sorry for my customer, I’ll keep ~~testing~~ hoping …

Bug on Oracle 12c Multitenant & PDB Clone as Snapshot Copy

While automating the refresh of the test databases on Oracle 12c Multitenant environment with ACFS and PDB snapshot copy, I encountered the following BUG:

The column SNAPSHOT_PARENT_CON_ID of the view V$PDBS shows 0 (zero) in case of PDBs created as Snapshot Copy.

This bug prevents to identify the parent-child relationship between a PDB and its own Snapshots Copies.

The test case below explains the problem:

SQL> CREATE PLUGGABLE DATABASE LARTE3SEFU from LARTE3 SNAPSHOT COPY; 
 
 Pluggable database created. 
 
 SQL> select CON_ID, NAME, OPEN_MODE, SNAPSHOT_PARENT_CON_ID from v$pdbs where NAME in ('LARTE3SEFU','LARTE3'); 
 
 CON_ID      NAME          OPEN_MODE  SNAPSHOT_PARENT_CON_ID 
 ---------- -------------- ---------- ---------------------- 
 5          LARTE3         READ ONLY  0 
 16         LARTE3SEFU     MOUNTED    0  <-- This should be 5
 
 2 rows selected.

A Service Request to Oracle has been opened, I’ll update this post once I have the official answer.

Update from the Service Request: BUG Fixed on version 12.2

My Oracle Notes

Category: BUG