Category Archives: Linux

Starting Oracle Database on Linux 7 using systemd and making OEM start only after the database is up

Recently I created a new virtual machine for the Oracle Database Admin, Install and Upgrade class that I teach at ACC. Previously I’d used Oracle Virtual Box on my local machine and then uploaded the image to Oracle Cloud and used Ravello to give each of my students their own server.

It was actually pretty straight forward:

  1. Upload the latest Oracle Linux (7.6) ISO that I got from
  2. Create a new blank machine with the following:
    1. 4 CPUs
    2. 24 GB of RAM
    3. 200 GB of disk
    4. Mout the uploaded ISO as a CD-ROM
    5. An elastic IP
    6. Services as follows:
      1. SSH (port 22)
      2. HTTPS (port 443)
      3. VNC (port 5901)
      4. HTTPS (port 7803 for Cloud Control)
  3. Start the image, configuring Oracle Linux 7 with the following:
    1. Server with GUI for software
    2. A static IP address (I used, but you could use anything).
    3. IP filtering so my ACC students could access the servers while they are in the labs at ACC and I could access the machines from home
    4. Partition the disk into a 16 GB swap partition, a 10 GB /home partition and then the rest of the disk as the root partition.
    5. When it came time to reboot the server, remove the CD-ROM image and update the configuration before rebooting so the image boots up using the disk.
  4. Install Oracle 12c and create an emrep repository database for OEM 13.3.
  5. Install OEM 13.3.
  6. Install Oracle 11g and create a database that will be upgraded during the course.

At this point everything was great, but since I teach 3-hour classes on Mondays and Wednesdays and shut the servers down between classes, my databases and OEM need to come up cleanly. Oracle has documentation on creating services to automatically start up databases on Linux/Unix, but it uses the old System V method for starting services (which, to be fair does still work on Linux 7). Since this was a Linux 7 server, I wanted to use the new systemd method. Tim’s rather fantastic site had the basic framework, but where he used scripts that he called from the service, I wanted to use dbstart and dbshut so that we could maintain startup and shutdown from a single file (/etc/oratab) rather than modifying a script.

I created the following file:

[root@dba ~]# vim /usr/lib/systemd/system/oracle-database.service
Description=The Oracle Database Service

# systemd ignores PAM limits, so set any necessary limits in the service.
# Not really a bug, but a feature.


ExecStart=/usr/bin/echo 'Starting Oracle Databases with Y in /etc/oratab'
ExecStart=/u01/app/oracle/product/12.2.0/dbhome_1/bin/dbstart /u01/app/oracle/product/12.2.0/dbhome_1
ExecStart=/usr/bin/echo 'dbstart has completed'

ExecStop=/usr/bin/echo 'Stopping Oracle Databases'
ExecStop=/u01/app/oracle/product/12.2.0/dbhome_1/bin/dbshut /u01/app/oracle/product/12.2.0/dbhome_1
ExecStop=/usr/bin/echo 'dbshut has completed'


I then enabled the service using the following:

[root@dba ~]# systemctl daemon-reload
[root@dba ~]# systemctl enable oracle-database

While the above worked great to start the database (tested with a reboot of the server), it didn’t address another issue. Unlike the database, Oracle Enterprise Manager comes with ‘out of the box’ scripts to start and stop OEM. They are the old style System V scripts that run out of /etc/init.d, and it didn’t really seem worth going through the trouble of converting them to the new systemd format. Unfortunately, the OEM scripts always assume that the database is already up and running. If your repository database is running on the same server as your OMS (which isn’t really that big of a deal if your hardware can handle it) that can be fixed by modifying the OEM startup script and adding in a ‘check to make sure your database is up and running before you start OEM’ section. The content in bold below was added to the out of the box OEM script after the initial comments in the file.

[root@dba ~]# vim /etc/init.d/gcstartup
# 2019-03-05 Rich Soule
# OEM should only startup if the emrep database is already up and running
# on the local machine so the below was added to make sure that happens.
#################### Begin Rich Soule Added Lines #######################
if [ "$1" = "start" ]
  while [ $counter -le 24 ]
    if ! /usr/bin/ps -e | /usr/bin/grep -q ora_pmon_emrep
      echo 'OEM is waiting on Oracle database to start'
      sleep 10
   if [ $counter -ge 24 ]
     echo 'Oracle database did not start in time, exiting OEM startup'
     exit 1
   echo 'Oracle database started, waiting 20 more seconds for database to open'
   sleep 20
  echo 'OMS will now attempt to start as per remainer of the /etc/init.d/gcstartup script'
####################  End Rich Soule Added Lines ########################

The above first checks to make sure that gcstartup was called with the start argument. If so then we’ll check if there is an ora_pmon_emrep process running (emrep is the name of my OEM repository database). If it isn’t running, we’ll wait 10 seconds and check again, but only for 240 seconds total. Once that process is found, we break out of our while do loop. If we did hit our 240-second limit, then we exit out of the gcstartup script totally, otherwise, we wait 20 seconds for the database to open and then continue along with the rest of the gcstartup script.

So far this has been working like a charm.

Note that you could do the same type of thing with OEM and instead have the script make a connection to a remote server every 10 seconds to make sure that the remote OEM repository was up before attempting to start OEM.

Happy Linux-ing and Oracle-ing!


SQL Developer SSH Connections In Depth


Available now: A 29-page white paper on SQL Developer SSH Connections that explains in detail how SSH connections should be configured between your desktop/laptop and your organization’s database!

The vast majority of explanations out on the internet of how to configure SSH connections between Oracle SQL Developer and an Oracle Database use incredibly simplistic explanations, often using the oracle os account as the ssh user (absolutely a worst practice!), and not including important things like how firewalls will impact connections, how your ssh server should be configured, etc.

I wrote a 29-page white paper which has been published on the website. It covers the above and more in exacting detail. If you click the link above you’ll be taken to a description of the white paper and then to a page where you’ll be able to download it after supplying a name and email (a fair trade for the amount of time and effort I put into the white paper…).

Note: The image above is from the white paper and is used for explanation purposes only… None of those things actually exist except for the WordPress external firewall that protects just this blog, or, actually, all the blogs at

Oracle Virtualized ODA Install

I’ve been working with the brand new ODA update on an X5-2 and Oracle has made a number of changes that impact how an install and configuration is done. is the first release that supports a ‘reflash’ of an ODA with Oracle Linux 6. Previously update had Oracle Linux 5 and updated had Oracle Linux 6, but the only way to move to was to first already be at and then patch to is the first release that will support (re-)deployment of an ODA in a virtualized configuration where you load the Oracle VM ISO into each ILOM and then deploy the ODA_BASE virtual machine. Note that if your ODA was at or earlier you will still have to download the patch to ODA_BASE after you deploy ODA_BASE to patch some of the server and storage components. Oracle has changed from the old patch –infra, –grid, –database options and now has a –server (that does some hardware and the grid infrastructure) and –storage (which patches the shared storage hardware).

One very major change which is especially important for those of us who manage virtualized ODA installs remotely is that /etc/xen/xend-config.sxp on ODA_BASE now has the following section:

# The interface for VNC servers to listen on. Defaults
# to  To restore old 'listen everywhere' behaviour
# set this to
#(vnc-listen '')

This means that you when you attempt to create a VM and connect to the console the same way you did in past ODA versions you’re probably going to get something like this:

[root@oda-base-node0 ~]# oakcli show vmconsole OL7U2 
 OAKERR:8006 Error in VNC display configurations : DISPLAY=localhost:10.0

You can either modify the /etc/xen/xend-config.sxp file, or you can update the vm.cfg file for a particular VM that you’ve imported. Since the ODA will be using ACFS it might be a bit hard to find the vm.cfg file since it ends up in the hidden .ACFS directory on the shared repository ACFS file system.

I had created my VMs on a shared repository named sharedrepo so my vm.cfg file ended up here:

[root@oda-base-node0 OL7U2]# pwd 
[root@oda-base-node0 OL7U2]# vim vm.cfg

Update the vm.cfg file and add the following:

vnclisten = ''

Note that vnclisten ‘’ didn’t work and neither did vnclisten = It had to be vnclisten = ‘’ for the VM to start.

Now that you’ve got a ‘listen anywhere’ vnc console running, you can just use a VNC client on your workstation (I used Tiger VNC) and connect to your node0-dom0:590X where X is the number of VMs you’ve started past ODA_BASE.  ODA_BASE is at :5900 and if this is the first additional VM you’ve created you’ll be able to reach the console at :5901. Note that once you’ve done the initial network configuration of your VM you’ll probably want to take the vnclisten parameter out of the vm.cfg file.

I’m also currently working with Oracle support on an issue. We’ve got a RAC One Node multitenant database and won’t run scripts in the ‘root container and all pluggables’. For example:

[oracle@oda-base-node0 ~]$ $ORACLE_HOME/perl/bin/perl $ORACLE_HOME/rdbms/admin/ -u SYS -d $ORACLE_HOME/rdbms/admin -b utlrp_output utlrp.sql 
 catcon: ALL catcon-related output will be written to utlrp_output_catcon_77673.lst 
 catcon: See utlrp_output*.log files for output generated by scripts 
 catcon: See utlrp_output_*.lst files for spool files, if any 
 Enter Password: 
 catconInit: database is not open on the default instance 
 Unexpected error encountered in catconInit; exiting

This is important because apxremov_con.sql is just a wrapper for so you can’t remove APEX from the root container (CDB$ROOT) and PDB$SEED so that new pluggable databases won’t have APEX in them. We do this so that we’ll be able to potentially have different versions of APEX in different pluggable databases in the same container database.

Oracle has opened a P1 bug on this and has actively been working this for a few days.

Happy ODAing!


yum install tree -y

You need to do this right now!

As root on your Oracle Linux server, type:

[root@sqlfundamentals ~]# yum install tree -y

Then you can do this:

[oracle@sqlfundamentals ~]$ cd labs/
[oracle@sqlfundamentals labs]$ tree
├── sql1
│   ├── code_ex
│   │   ├── cleanup_scripts
│   │   │   ├── cleanup_10.sql
│   │   │   └── cleanup_11.sql
│   │   ├── code_09_retired_emp_tab.sql
│   │   ├── code_10_cretest.sql
│   │   ├── code_ex_02.sql
│   │   ├── code_ex_03.sql
│   │   ├── code_ex_04.sql
│   │   ├── code_ex_05.sql
│   │   ├── code_ex_06.sql
│   │   ├── code_ex_07.sql
│   │   ├── code_ex_08.sql
│   │   ├── code_ex_09.sql
│   │   ├── code_ex_10.sql
│   │   └── code_ex_11.sql
│   ├── demo
│   │   ├── demo_02_alias.sql
│   │   ├── demo_02_null.sql
│   │   ├── demo_03_and.sql
│   │   ├── demo_03_betw.sql
│   │   ├── demo_03_expr.sql
│   │   ├── demo_03_in.sql
│   │   ├── demo_03_ordernull.sql
│   │   ├── demo_03_or.sql
│   │   ├── demo_03_sal1.sql
│   │   ├── demo_03_sal2.sql
│   │   ├── demo_03_varno.sql
│   │   ├── demo_03_varyes.sql
│   │   ├── demo_04_case.sql
│   │   ├── demo_04_nest.sql
│   │   ├── demo_05_day1.sql
│   │   ├── demo_05_day2.sql
│   │   ├── demo_05_hire.sql
│   │   ├── demo_05_mask.sql
│   │   ├── demo_06_count1.sql
│   │   ├── demo_06_count2.sql
│   │   ├── demo_06_error.sql
│   │   ├── demo_06_job1.sql
│   │   ├── demo_06_job2.sql
│   │   ├── demo_06_listagg.sql
│   │   ├── demo_06_order1.sql
│   │   ├── demo_06_order2.sql
│   │   ├── demo_07_cart.sql
│   │   ├── demo_07_loc.sql
│   │   ├── demo_09_inters.sql
│   │   ├── demo_09_minus.sql
│   │   ├── demo_09_orderby.sql
│   │   ├── demo_09_union1.sql
│   │   ├── demo_09_union3.sql
│   │   ├── demo_10_cretabs.sql
│   │   ├── demo_10_grant.sql
│   │   ├── demo_10_revoke.sql
│   │   ├── demo_10_select.sql
│   │   ├── demo_10_update.sql
│   │   ├── demo_11_identity_col.sql
│   │   ├── demo_11_invisible_col.sql
│   │   ├── demo_11_sol_col_default.sql
│   │   └── demo_d_loc.sql
│   ├── labs
│   │   ├── lab_10_01.sql
│   │   ├── Online_Book_Store_Create_Table.sql
│   │   ├── Online_Book_Store_Drop_Tables.sql
│   │   └── Online_Book_Store_Populate.sql
│   └── soln
│       ├── sol_02.sql
│       ├── sol_03.sql
│       ├── sol_04.sql
│       ├── sol_05.sql
│       ├── sol_06.sql
│       ├── sol_07.sql
│       ├── sol_08.sql
│       ├── sol_09.sql
│       ├── sol_10.sql
│       ├── sol_11.sql
│       └── sol_ap.sql
└── sql2
├── code_ex
│   ├── cleanup_scripts
│   │   ├── cleanup_03.sql
│   │   ├── cleanup_05.sql
│   │   ├── cleanup_07.sql
│   │   ├── cleanup_09.sql
│   │   └── cleanup_10.sql
│   ├── code_cre_emp_history.sql
│   ├── code_cre_emp_sales.sql
│   ├── code_cre_mgr_history.sql
│   ├── code_cre_sales_info.sql
│   ├── code_cre_sales_source_data.sql
│   ├── code_cre_sal_high.sql
│   ├── code_cre_sal_history.sql
│   ├── code_cre_sal_low.sql
│   ├── code_cre_sal_mid.sql
│   ├── code_ex_02.sql
│   ├── code_ex_03.sql
│   ├── code_ex_04.sql
│   ├── code_ex_05.sql
│   ├── code_ex_06.sql
│   ├── code_ex_07.sql
│   ├── code_ex_08.sql
│   ├── code_ex_09.sql
│   ├── code_ex_10.sql
│   └── code_ins_sales_source_data.sql
├── demo
│   ├── demo_04_easyvu.sql
│   ├── demo_04_emp.sql
│   ├── demo_06_ex_1a.sql
│   ├── demo_06_ex_1b.sql
│   ├── demo_06_ex_2.sql
│   ├── demo_06_pairwise_a.sql
│   ├── demo_06_pairwise_b.sql
│   ├── demo_06_pairwise.sql
│   ├── demo_06_With.sql
│   ├── demo_07_delete_rows.sql
│   ├── demo_07_update_rows.sql
│   ├── demo_07_update_subquery1.sql
│   ├── demo_07_update_subquery.sql
│   ├── demo_08_revoke.sql
│   ├── demo_09_default_tab.sql
│   ├── demo_10_tz.sql
│   ├── demo_d_loc.sql
│   ├── demo_I_regexp_replace_a.sql
│   └── demo_I_regexp_replace_b.sql
├── emp_dir
│   ├── emp.dat
│   ├── library_items.dat
│   ├── LIBRARY_ITEMS_EXT_23121.log
│   ├── LIBRARY_ITEMS_EXT_5040.log
│   ├── OLDEMP_14719.log
│   └── OLDEMP_26721.log
├── labs
│   ├── confidence.sql
│   ├── lab_02_06_tab.sql
│   ├── lab_04_07.sql
│   ├── lab_05_09.sql
│   ├── lab_05_10.sql
│   ├── lab_05_11_a.sql
│   ├── lab_05_11_b.sql
│   ├── lab_05_11_g.sql
│   ├── lab_06_insert_empdata.sql
│   ├── lab_08_05.sql
│   ├── lab_09_01.sql
│   ├── lab_09_03.sql
│   ├── lab_09_05.sql
│   ├── lab_09_08_a.sql
│   ├── lab_09_08_b.sql
│   ├── lab_09_08_e.sql
│   ├── lab_09_09.sql
│   ├── lab_09_10.sql
│   ├── lab_09_11.sql
│   ├── lab_10_06.sql
│   ├── lab_ap_cre_mgr_history.sql
│   ├── lab_ap_cre_sal_history.sql
│   ├── lab_ap_cre_special_sal.sql
│   ├── Online_Book_Store_Drop_Tables.sql
│   └── Online_Book_Store_Populate.sql
└── soln
├── sol_02.sql
├── sol_03.sql
├── sol_04.sql
├── sol_05.sql
├── sol_06.sql
├── sol_08.sql
├── sol_09.sql
├── sol_10.sql
└── sol_ap.sql

13 directories, 154 files
[oracle@sqlfundamentals labs]$

And this is awesome!


Command line editing with bash

I use Oracle Linux just about every day. I teach Oracle classes using Oracle Linux and work with a whole bunch of Oracle servers which, more often than not, use Oracle Linux (or sometimes Redhat Linux which is basically very similar to Oracle Linux). The vast majority of the time these systems are set up to use bash (From wikipedia: The name itself is an acronym, a pun, and a description. As an acronym, it stands for Bourne-again shell, referring to its objective as a free replacement for the Bourne shell. As a pun, it expressed that objective in a phrase that sounds similar to born again, a term for spiritual rebirth. The name is also descriptive of what it did, bashing together the features of sh, csh, and ksh.) as the default shell for users.

While teaching my classes, I frequently type commands while my students watch and it’s not unusual for me to make typing mistakes. I often bang the BACKSPACE key a bunch of times while removing a single word, or worse yet, I hold it down to erase and entire line and start all over again. Every time I did this I would think “I used to know how to do this better.”

Today I looked it up a book that I have sitting on my bookshelf, (I last read it in 1995), which I would highly recommend if you’re a Unix System Admin: Using csh & tcsh Type Less, Accomplish More. It’s a really great book if you are going to be using a Unix shell all day every day interactively. I read it way back when I was learning Unix and it made a huge difference in my productivity with interactive ‘shelling’. These days I’m mostly only working with Linux to get Oracle systems set up and configured correctly, so I’m usually sticking with the default shell… which is bash.

I tested out some of the things in the book and they work a bit differently in tcsh than they do in bash. Since I’ve become (largely by default) a bash user, here goes:

Remove the entire command line in bash with CTRL-u or CTLR-w

[oracle@multitenant ~]$ This is a command line that I want to erase

Type CTRL-u or CTRL-w (as an aside, CTRL-w in tcsh erases a single word) and voila:

[oracle@multitenant ~]$

Sometime I only mess up the last word…

Remove the last word typed in bash with ESC BACKSPACE

[oracle@multitenant ~]$ env | grep ORALCE

Oops! Messed up that ORACLE. Type ESC BACKSPACE and voila:

[oracle@multitenant ~]$ env | grep

Now I can correctly add ORACLE to the above without having to use backspace six times.

Reuse part of the previous commands arguments with !$, !^, !*

Often we’ll look at the contents of a file and then decide we want to edit it.

[oracle@multitenant ~]# cat /etc/oratab

Hmmm…. I see something I want to change, so rather than typing vi /etc/oratab I can just do the following:

[oracle@multitenant ~]# vi !$

!$ uses the last argument from the previous command, !^ uses the first argument from the previous command, and !* uses all arguments from the last command, so actually any of those would have worked in the example above. Remember that above I have two parts in the line I’m typing; The first is the actual command (cat) and then the second is the argument (/etc/oratab).

Hopefully this (along with the CTRL-r post from earlier) will help a few folks become just a bit more efficient in their day to day operations!

CTRL-r is your friend!

You already knew that the up and down arrows in a Linux terminal window allow you to navigate through previously typed commands, but did you know that you can also search through your command history?

CTRL-r is great for finding those commands that are a bit tedious to type.

When you type CTRL-R in a terminal window your whole prompt turns into this strange looking text:


As you type characters the first command in your history, starting from the most recent and working back towards the top will be displayed.

Let’s say you’ve typed the following list of commands:

sqlplus / as sysdba
rman target "'/ as sysbackup'"
ls -als

At a command prompt you type CTRL-r and then type s. Your prompt will now look like this:

(reverse-i-search)`s': ls -als

And the s in the als will be highlighted. What you’ve done here is found the first s in your command history.

If you type Enter right now, you’ll rerun that command.

Things get more interesting when you start typing more letters. Instead of typing just s, type sq and you’ll get the following:

(reverse-i-search)`sq': sqlplus / as sysdba

Hitting Enter will rerun that sqlplus command.

What if you want to find something that you typed earlier than the last command that matched? Just type CTRL-r again. Each time you press CTRL-r you’ll find older and older strings that match. If you have a short search string (just s for example), then it’s not unusual to find that string in a single command multiple times. Once you’ve exhausted all the matches in one command, you’ll reach back into your history to find another that matches.

What if you want to find the last thing that you found with CTRL-r?

Start a new search (or just use backspace to remove any characters you’ve started typing) and press CTRL-r again. Let’s say that you used rm as your last successful search and retrieved the rman command from the above list of commands. You then ran a few more commands and decided to run rman again… Just type CTRL-r twice. The first CRTL-r says “I want to search through my history”, the second CTRL-r says “Remember the last successful search I used? I want that one again.”

What if you don’t want to run the command you found, but instead you just put the command at your prompt so you can edit it? Instead of hitting Enter after you find a command, just press Esc. This dumps the text of the command into your terminal at a command prompt.

Here’s where things get really cool… If you use rlwrap for various Oracle commands like sqlplus, rman, etc. (See this post to install rlwrap, and this post to configure the aliases to use with rlwrap) then CTRL-r will also work in those commands too!

Want to find the last alter user command that you ran? In sqlplus just type your good friend CTLR-r and start typing alter. You probably won’t have to actually type all the letters in alter and soon you’ll have the following:

(reverse-i-search)`al': alter user dbsnmp identified by oracle_4U account unlock;

Technically CTRL-r finds the spot in your history where the command you found lives, so once you’ve found a command, you can use the up and down arrows to cycle through the commands that are near that command. What was that command that I typed right after I did that alter user command? Just press the down arrow key and you’ve found it!

There are a few idiosyncrasies with CTRL-r. First it is case sensitive so you’ll have to remember if you used SELECT or select. Second it can be kind of easy to get it into a strange place with the characters you are searching for if you start typing characters that are not actually in your command history CTLR-r will beep at you a bunch and you may or may not be able to use the backspace key to get back to where you want to be… Just use Esc to get out of the CTRL-r search and try again.

Also you may want to give CTRL-r more history to work with. If you’d like to do this you’ll need export two different environment variables, HISTSIZE and HISTFILESIZE.

I added the following to my .bash_profile file in my home directory:

# Let's give the history command and CTRL-r more to work with!
export HISTSIZE=10000
export HISTFILESIZE=10000

CTRL-r has saved me much typing over the years, I hope it helps you too!

Thoughts on the Oracle users bashrc and bash_profile

For many, many years we’ve been clouding up the oracle user’s .bashrc and .bash_profile scripts with ‘lots of things’. Some of these things will be set automatically when we run the . oraenv command (that’s dot space oraenv). Our good old friend LD_LIBRARY_PATH gets set automatically, ORACLE_HOME gets set automatically, and the PATH is extended to search in your ORACLE_HOME/bin directory.

The below screenshot comes from a ‘new’ Oracle Linux 6 system where I’ve done a software only install of Oracle Database and have yet to create any databases. Because I don’t have a database yet, I have to type in the path to the ORACLE_HOME when I give some random SID. Once I have a database created, that database will get added to the /etc/oratab file and the SID will be recognized and automatically set the ORACLE_HOME for me.

bashrc and bash_profile

Click the image above to get a full size picture of what’s happening.

Based on the above, I might change things a bit in this post from my 12c Install Series.