Oracle Multitenant supports database DevOps standards

As a consultant I constantly speak with my customers, and among a big number of them I noticed that the speed and flexibility of all database provisioning activities generate huge concern.

Hence I decide to describe on this post few Oracle Multitenant options to resolve those problems.

If production is the most critical environment to maintain, it is definetly not the one generating the greatest efforts in term of provisioning. The applications are more and more complex, and require continuous delivery;  to satify those needs the infrastructure has few provisioning challengers to overcome.

Now with the Oracle version 12.2 and the Mutitenet option, the DBaaS model becomes simpler than ever.


Clone PDB

The Clone PDB operation has been enhanced from Cold to Hot Clone. This improvement requires the usage of  PDB Local Undo. The Hot Clone is now the default method and can be devided in three major steps:

  1. PDB source datafile copy, because the PDB remains open in read/write at this stage the cloned datafiles are physically inconsistent (fuzzy data files).
  2. The Redo Log entries generated on the source PDB during the copy are applied to the targed PDB. This step makes the source and target PDBs two exact physical copies.
  3. Because the Redo Log entries coming from the source PDB contain committed and uncommitted transactions, to make the target PDB transactionally consistent, the undo entries of all uncommitted transations must be applied.


The command below shows how to clone a PDB open in read/write:

Create Pluggable Database ERP_Hot_Clone from ERP;


Refreshable PDB

Refreshable PDB leverages the Hot Clone PDB capability, creating an initial copy of the source PDB refreshed over the time at scheduled interval or on-demand.

To better understand the possible use cases, the graphical example below covers the development’s request to have every morning a copy of production data.




How to create a Refreshable PDB

Syntax to create an automatic refreshable PDB:

Create Pluggable Database CRM_Test from CRM_Prod@db_link refresh mode every 720; -- (12H)


Syntax to create a manual PDB refresh:

Create Pluggable Database CRM_Test from CRM_Prod@db_link refresh mode manual;


After the clone the refreshable PDB should then be opened in read-only:

Alter Pluggable Database CRM_Test read only;


How to invoke a manual PDB refresh:

Alter Pluggable Database CRM_Test refresh;


Creation of the snapshot databases:

Create Pluggable Database CRM_TEST_Snap01 FROM CRM_Test
FILE_NAME_CONVERT = ('/u03/oradata/CDB122/CRM_Test/','/u03/oradata/CDB122/CRM_Test_Snap01/')





New to Oracle Multitenant?

Multitenant is the biggest architectural change of Oracle 12c and the enabler of many new database options in the years to come. Therefore I have decided to write over the time, few blog posts with basic examples of what should be done and not in a multitenant database environment.


Rule #1   – What should not be done

If you are a CDB DBA, always pay attention to which container you are connected to and remember that application data should be stored on Application PDB only!

Unfortunately this golden rule is not-enforced by the RDBMS, but it is left in our responsibility as shown on the example below:

oracle@lxoel7n01:~/ [CDB_TEST] sqlplus / as sysdba

SQL*Plus: Release Production on Wed Sep 21 18:28:23 2016

Copyright (c) 1982, 2014, Oracle. All rights reserved.

Connected to:
Oracle Database 12c Enterprise Edition Release - 64bit Production
With the Partitioning, Automatic Storage Management, OLAP, Advanced Analytics
and Real Application Testing options

CDB$ROOT SQL> show user
CDB$ROOT SQL> show con_name



Once connected to the ROOT container let see if I can mistakenly create an application table:

(emp_id NUMBER,
emp_name VARCHAR2(25),
start_date DATE,
emp_status VARCHAR2(10) DEFAULT 'ACTIVE',
resume CLOB); 2 3 4 5 6

Table created.

CDB$ROOT SQL> desc emp_1
 Name                                Null?    Type
 ----------------------------------- -------- ----------------------------
 EMP_ID                                        NUMBER
 EMP_NAME                                      VARCHAR2(25)
 START_DATE                                    DATE
 EMP_STATUS                                    VARCHAR2(10)
 RESUME                                        CLOB

CDB$ROOT SQL> insert into emp_1 values (1, 'Emiliano', sysdate, 'active', ' ');

1 row created.

CDB$ROOT SQL> commit;

Commit complete.

CDB$ROOT SQL> select * from emp_1;

---------- ------------------------- --------- ---------- ----------------
 1          Emiliano                  21-SEP-16 active

CDB$ROOT SQL> show con_name


The answer is “YES” and the consequences can be devastating…


Rule #2   – Overview of Local and Common Entities

Non-schema entities can be created as local or common.  Local entities exist only in one PDB similar to a non-CDB architecture, while Common entities exist in every current and future container.

List of possible Local / Common entities in a Multitenant database:

  • Users
  • Roles
  • Profiles
  • Audit Policies

All Local entities are created from the local PDB and all Common entities are created from the CDB$ROOT container.

Common-user-defined Users, Roles and Profiles require a standard database prefix, defined by the spfile parameter COMMON_USER_PREFIX:

SQL> show parameter common_user_prefix

NAME                              TYPE        VALUE
--------------------------------- ----------- -----------------
common_user_prefix                string      C##


Example of Common User creation:


User created.

SQL> SELECT con_id, username, user_id, common

  2  FROM cdb_users where username='C##CDB_DBA1'  ORDER BY con_id;

---------- -------------------- ---------- ------
         1 C##CDB_DBA1               102    YES
         2 C##CDB_DBA1               101    YES
         3 C##CDB_DBA1               107    YES
         4 C##CDB_DBA1               105    YES
         5 C##CDB_DBA1               109    YES


Example of Local user creation:

SQL> show con_name



User created.

If we try to create a Local User from the CDB$ROOT container the following error occurs: ORA-65049: creation of local user or role is not allowed in CDB$ROOT

SQL> show con_name





ERROR at line 1:
ORA-65049: creation of local user or role is not allowed in CDB$ROOT



Rule #3  – Application should connect through user-defined database services only

We have been avoiding to create user-defined database services for many years, sometimes even for RAC databases. But in Multitenet or Singletenant architecture the importance of user-defined database service is even greater. For each CDB and PDB Oracle is still automatically creating a default service, but as in the past the default services should never be exposed to the applications.


To create user-defined database service in stand-alone environment use the package DBMS_SERVICE while connected to the corresponding PDB:

     SERVICE_NAME     => '',
     NETWORK_NAME     => '',

The database services will not start automatically after opening a PDB!  Create a database trigger for this purpose.


To create user-defined database service in clustered environment use the srvctl utility from the corresponding RDBMS ORACLE_HOME:

oracle@oel7n01:~/ [EFU1] srvctl add service -db EFU \
> -pdb MYPDB -service \
> -failovertype SELECT -failovermethod BASIC \
> -failoverdelay 2 -failoverretry 90


List all CDB database services order by Container ID:

SQL> SELECT con_id, name, pdb FROM v$services ORDER BY con_id;

    CON_ID NAME                                     PDB
---------- --------------------------------------- -----------------

         1 EFUXDB                                   CDB$ROOT   <-- CDB Default Service 
         1 SYS$BACKGROUND                           CDB$ROOT   <-- CDB Default Service 
         1 SYS$USERS                                CDB$ROOT   <-- CDB Default Service 
         1                 CDB$ROOT   <-- CDB Default Service 
         1           CDB$ROOT   <-- CDB User-defined Service  
         3               MYPDB      <-- PDB Default Service 
         3           MYPDB      <-- PDB User-defined Service  

7 rows selected.


EZCONNECT to a PDB using the user-defined service:

sqlplus <username>/<password>@<host_name>:<local-listener-port>/<service-name>
sqlplus application/pwd@oel7c-scan:1522/



Rule #4  –  Backup/Recovery strategy in Multitenant

As database administrator one of the first responsibility to fulfil is the “Backup/Recovery” strategy. The migration to multitenant database, due to the high level of consolidation density requires to review existing procedures. Few infrastructure operations, like creating a Data Guard or executing a backup, have been shifted from per-database to per-container consolidating the number of tasks.

RMAN in 12c covers all CDB, PDB backup/restore combinations, even though the best practice suggests to run the daily backup at CDB level, and in case of restore needed, the granularity goes down to the single block of one PDB.  Below are reported few basic backup/restore operations in Multitenant environment.


Backup a full CDB:

RMAN> connect target /;
RMAN> backup database plus archivelog;


Backup a list of PDBs:

RMAN> connect target /;
RMAN> backup pluggable database mypdb, hrpdb plus archivelog;


Backup one PDB directly connecting to it:

RMAN> connect target sys/;
RMAN> backup incremental level 0 database;


Backup a PDB tablespace:

RMAN> connect target /;
RMAN> backup tablespace mypdb:system;


Generate RMAN report:

RMAN> report need backup pluggable database mypdb;


Complete PDB Restore

RMAN> connect target /;
RMAN> alter pluggable database mypdb close;
RMAN> restore pluggable database mypdb;
RMAN> recover pluggable database mypdb;
RMAN> alter pluggable database mypdb open;



Rule #5  –  Before moving to Multitenant

Oracle Multitenant has introduced many architectural changes that force the DBA to evolve how databases are administered. My last golden rule suggests to thoroughly study the multitenant/singletenant architecture before starting any implementation.

During my experiences implementing multitenant/singletenant architectures, I found great dependencies with the following database areas:

  • Provisioning/Decommissioning
  • Patching and Upgrade
  • Backup/recovery
  • Capacity Planning and Management
  • Tuning
  • Separation of duties between CDB and PDB



Oracle Cloud Computing

What is a database Cloud Computing?

This looks like the million dollar question; what we know for sure is that it is a quite recent technology and different people identify the Cloud Architecture by different key features (On-Demand, Broad Network Access, Resource Pooling, Rapid Elasticity, Measured Service). There are two main categories: Private and Public Cloud, which identifies respectively in house and outsourced Cloud installation. Focusing on Oracle Database technology the Private Cloud is a clustered infrastructure hosted on the company?s data center, therefore the IT department is responsible of the installation, maintenance and life cycle of all hardware and software components. In case of Public Cloud the company demands the management of the databases to a third party, which owns the infrastructure used to manage the databases of different customers.

Beyond the different marketing definitions of database cloud computing, Oracle provides a reach set of features to realize this kind of setup. The main component of this architecture is the Grid Infrastructure, which provides the cluster and storage foundation of Oracle Cloud Computing. On top of the Grid Infrastructure we have the RDBMS which enables RAC, RAC One Node and stand-alone database setups.

At this point, anyone can say that with the exception of the name, there is almost nothing new compared to the earlier version of Oracle Real Application Cluster (RAC). But Oracle Cloud Computing is much more than a simple multi-node RAC which hosts several databases; the introduction of features like Quality of Service Management (QoS), Server Pool, Instance Caging (extension of Resource Manager) and the enhancement of the existing ones, allow to consolidate all the environments guaranteeing to each application: the performance expected, the scalability for future needs, the availability to respect the Service Level Agreement (SLA), the best time to market, the governance of the entire platform and last but not least cost saving.

Obviously Oracle provides all the instruments to reach such great result, but it is up to the single organization to define and implement the most appropriate modus operandi in terms of OM, Life Cycle, Capacity planning and management, to obtain the result promised by this great technology.