Category Archives: Linux

Solved (?) SQL Developer 19.2: No more authentication methods available when using SSH Connections

When Oracle upgraded SQL Developer from 19.1 to 19.2 they updated some libraries. One of those updates meant that SSH keypairs that had been generated by PuttyGen didn’t work anymore and you should “Generate OpenSSH format ppk” (see Oracle Support Doc ID: 2575854.1)

However, when attempting to do so using PuttyGen it didn’t seem to work. Taking a step back, here was the situation in 19.1 or earlier:

  1. You had SQL Developer 19.1 running on your desktop and had used the SSH Hosts dialog to create an SSH connection pointing SQL Developer at the private key on your machine (typically, this is an id_rsa file).
  2. The server you are connecting to has the associated public key (typically, id_rsa.pub) in the authorized_keys file associated with the account you were connecting to on the remote server.
  3. Everything worked as expected and you could connect to databases on the remote database server using the SSH connection.

After an upgrade to 19.2 (during which SQL Developer will ask you to import settings from a previous release), there is a chance that the private key that you previously had pointed to will no longer work.

My ‘used to work in 19.1 and earlier’ private key file looked like this:

-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-128-CBC,abcdefghijklmnopqrstuvwxyz123456
                      
abcdefghijklmnopqrstuvwxyz1234567890abcdefghijklmnopqrstuvwxyz12
[SNIP]
abcdefghijklmnopqrstuvwxyz1234567890abcdefghijklmnopqrstuvwxyz12
-----END RSA PRIVATE KEY-----

If I opened that file in Putty Key Generator on a Windows 10 box, it said that it was already an OpenSSH SSH-2 Private key.Puttygen Notice

Since Oracle Support and others said that the key needed to be in OpenSSH format, one would assume that maybe you’d use the Conversion feature of Puttygen to convert the key…

Puttygen Conversion

Nope. No go.

After playing around for a long time I discovered the OpenSSH commands that are now built into Windows 10. After testing a million things, this is what actually worked:

C:\sshkeys> ssh-keygen -e -f id_rsa > id_rsa_for_sql_developer_19.2
Enter passphrase:

Once pointed at the id_rsa_for_sql_developer_19.2 file, SQL Developer happily prompted me for my (very long) passphrase and established a connection to my database.

Great! Problem solved.

However, and this is the strange thing that I don’t really understand… If you look at the contents of the generated file, you’ll see something like this:

---- BEGIN SSH2 PUBLIC KEY ----
Comment: "2048-bit RSA, converted by rich@ROTOR from OpenSSH"
abcdefghijklmnopqrstuvwxyz1234567890abcdefghijklmnopqrstuvwxyz12
abcdefghijklmnopqrstuvwxyz1234567890abcdefghijklmnopqrstuvwxyz12
abcdefghijklmnopqrstuvwxyz1234567890abcdefghijklmnopqrstuvwxyz12
abcdefghijklmnopqrstuvwxyz1234567890abcdefghijklmnopqrstuvwxyz12
abcdefghijklmnopqrstuvwxyz1234567890abcdefghijklmnopqrstuvwxyz12
abcdefghijklmnopqrstuv
---- END SSH2 PUBLIC KEY ----

Yep, that is what it looks like, a public key in SSH2 format (although I obviously obfuscated the above, for what reason, I do not know)…

Why an SSH2 Public Key works when you should have a private key (if I understand SSH connections correctly, and this sort of makes me feel like I don’t), I have no idea. But work it did, so I happily moved on.

As an aside, I did have an SR open with Oracle on this which I ended up closing myself, but I did ask them to update Doc ID: 2575854.1 with the above information before I closed it.

Happy SQL Developer-ing!


orabasetab is wrong if you install as grid

Traditionally when you install the Oracle Grid Infrastructure for RAC, the owner of the GI software is a user named grid and the owner of the database software is a user named oracle.

With the 19c version of the GI, the software is shipped as a zip file that you extract. Unfortunately, there is a bug (if you are doing a traditional install) in this extract where the orabasetab file (located here if you are using a traditional OFA compliant install:   /u01/app/19.0.0.0/grid/install/orabasetab) has the following contents:

#orabasetab file is used to track Oracle Home associated with Oracle Base
/u01/app/19.0.0/grid:/u01/app/oracle:OraGI19Home1:N:

Which is fine if you are going to use only the oracle user as the owner of both the GI software and the database software, but if you want to install the GI software as the grid user and the database software as the oracle user then you’ll need to modify the file as so:

#orabasetab file is used to track Oracle Home associated with Oracle Base
# 2019-09-29 Rich Soule updated the original value below to the new value
# so the GI could be installed as the grid user
#/u01/app/19.0.0/grid:/u01/app/oracle:OraGI19Home1:N:
/u01/app/19.0.0/grid:/u01/app/grid:OraGI19Home1:N:

If you don’t do this then there is a real good chance you’ll see the following when you attempt your GI install: Error 49802 initializing ADR

Good luck with your RAC installs…

Rich


X11 Forwarding with MIT Magic Cookies to Oracle Cloud as multiple users

Every time I do this, I always forget the steps, especially for allowing connections after you su to another user… So, here we go:

I’m going to connect two machines: rotor (it’s a palindrome), my local Windows machine, and cloudbox, my Oracle Cloud server.

The first thing I’m going to do is connect into cloudbox as the opc user, become root and then update the sshd_config file with an X11UseLocalhost no entry. I also commented the previous value (which was already commented out) and added a comment about who changed what and when. Then I’ll restart the ssh daemon.

[opc@cloudbox ~]$ sudo -s
[root@cloudbox opc]# vim /etc/ssh/sshd_config

# 2019-08-12 Rich Soule changed below to allow remote X11 Connections
#X11UseLocalhost yes ## Original value commented out as in this line
X11UseLocalhost no

[root@cloudbox ~]# systemctl restart sshd

Next, we’ll make sure that enough of the X11 tools are on the server.

[root@cloudbox ~]# yum install xclock -y

I’m going to use MobaXterm as my client on my Windows box and open up a local terminal on rotor. MobaXterm automatically gives me an X11 Server so I don’t have to use something like Xming to give me a local X11 Server.

When I created cloudbox, my Oracle Cloud server, a private key file was created. This private key file was saved to rotor, my Windows box, in a folder on my local computer, so I’m going to cd to that folder and start an ssh connection:

[Rich.Rotor] ➤ ssh -X -i id_rsa opc@cloudbox

At this point, I have to provide the passphrase for my private key file that is in this directory. After supplying the passphrase, I’ll be connected to cloudbox.

Last login: Mon Aug 12 22:23:10 2019 from somewhere on the internet
/usr/bin/xauth:  file /home/opc/.Xauthority does not exist
[opc@cloudbox ~]$

The message above should really be something like “.Xauthority does not exist, so I’m creating it.” because that is what just happened. The DISPLAY environment variable was set to the IP address of the cloud server with a :10.0 appended to the end, and we can see what ended up in .Xauthority file by using the xauth list command:

[opc@cloudbox ~]$ echo $DISPLAY
10.10.0.2:10.0
[opc@cloudbox ~]$ xauth list
cloudbox.myreg.myvcn.oraclevcn.com:10  MIT-MAGIC-COOKIE-1  6ab3d32cf1c543ecaf83c79297ee3fbc

At this point, X11 based commands will now work, but only for the opc user.

[opc@cloudbox ~]$ xeyes&
[1] 13177

xeyes

If I become another user, then X11 commands won’t work.

[opc@cloudbox ~]$ sudo su - oracle
Last login: Mon Aug 12 22:35:15 GMT 2019 on pts/0
[oracle@cloudbox ~]$ xeyes&
[1] 13595
[oracle@cloudbox ~]$ Error: Can't open display:

[1]+ Exit 1 xeyes
[oracle@cloudbox ~]$

Looking above, it appears that the DISPLAY environment variable for the oracle user has yet to be set. The blank line after the message essentially tells us it has a NULL value. However, even if we set it, it still doesn’t work yet.

[oracle@cloudbox ~]$ export DISPLAY=10.10.0.2:10.0
[oracle@cloudbox ~]$ xeyes&
[1] 14213
[oracle@cloudbox ~]$ X11 connection rejected because of wrong authentication
Error: Can't open display: 10.10.0.2:10.0

[1]+  Exit 1                  xeyes
[oracle@cloudbox ~]$

The trick at this point is to pass along the MIT Magic Cookie that got generated for the opc user to the oracle user. The easiest way to do this is to just copy and paste the full output from the xauth list command as the opc user into an xauth add command as the oracle user:

[oracle@cloudbox ~]$ xauth add cloudbox.myreg.myvcn.oraclevcn.com:10 MIT-MAGIC-COOKIE-1 6ab3d32cf1c543ecaf83c79297ee3fbc
xauth: file /home/oracle/.Xauthority does not exist
[oracle@cloudbox ~]$ xeyes&
[1] 14512
[oracle@cloudbox ~]$

At this point, everything works and X11 commands will now display on my local Windows box from Oracle Cloud as the oracle user.

Happy Linuxing!

 


Starting Oracle Database on Linux 7 using systemd and making OEM start only after the database is up

Recently I created a new virtual machine for the Oracle Database Admin, Install and Upgrade class that I teach at ACC. Previously I’d used Oracle Virtual Box on my local machine and then uploaded the image to Oracle Cloud and used Ravello to give each of my students their own server.

It was actually pretty straight forward:

  1. Upload the latest Oracle Linux (7.6) ISO that I got from eDelivery.oracle.com.
  2. Create a new blank machine with the following:
    1. 4 CPUs
    2. 24 GB of RAM
    3. 200 GB of disk
    4. Mout the uploaded ISO as a CD-ROM
    5. An elastic IP
    6. Services as follows:
      1. SSH (port 22)
      2. HTTPS (port 443)
      3. VNC (port 5901)
      4. HTTPS (port 7803 for Cloud Control)
  3. Start the image, configuring Oracle Linux 7 with the following:
    1. Server with GUI for software
    2. A static IP address (I used 10.0.0.15, but you could use anything).
    3. IP filtering so my ACC students could access the servers while they are in the labs at ACC and I could access the machines from home
    4. Partition the disk into a 16 GB swap partition, a 10 GB /home partition and then the rest of the disk as the root partition.
    5. When it came time to reboot the server, remove the CD-ROM image and update the configuration before rebooting so the image boots up using the disk.
  4. Install Oracle 12c and create an emrep repository database for OEM 13.3.
  5. Install OEM 13.3.
  6. Install Oracle 11g and create a database that will be upgraded during the course.

At this point everything was great, but since I teach 3-hour classes on Mondays and Wednesdays and shut the servers down between classes, my databases and OEM need to come up cleanly. Oracle has documentation on creating services to automatically start up databases on Linux/Unix, but it uses the old System V method for starting services (which, to be fair does still work on Linux 7). Since this was a Linux 7 server, I wanted to use the new systemd method. Tim’s rather fantastic site had the basic framework, but where he used scripts that he called from the service, I wanted to use dbstart and dbshut so that we could maintain startup and shutdown from a single file (/etc/oratab) rather than modifying a script.

I created the following file:

[root@dba ~]# vim /usr/lib/systemd/system/oracle-database.service
[Unit]
Description=The Oracle Database Service
After=syslog.target network.target

[Service]
# systemd ignores PAM limits, so set any necessary limits in the service.
# Not really a bug, but a feature.
# https://bugzilla.redhat.com/show_bug.cgi?id=754285
LimitMEMLOCK=infinity
LimitNOFILE=65535

Type=oneshot
RemainAfterExit=yes
User=oracle
Group=oinstall
Restart=no

ExecStart=/usr/bin/echo 'Starting Oracle Databases with Y in /etc/oratab'
ExecStart=/u01/app/oracle/product/12.2.0/dbhome_1/bin/dbstart /u01/app/oracle/product/12.2.0/dbhome_1
ExecStart=/usr/bin/echo 'dbstart has completed'

ExecStop=/usr/bin/echo 'Stopping Oracle Databases'
ExecStop=/u01/app/oracle/product/12.2.0/dbhome_1/bin/dbshut /u01/app/oracle/product/12.2.0/dbhome_1
ExecStop=/usr/bin/echo 'dbshut has completed'

[Install]
WantedBy=multi-user.target

I then enabled the service using the following:

[root@dba ~]# systemctl daemon-reload
[root@dba ~]# systemctl enable oracle-database

While the above worked great to start the database (tested with a reboot of the server), it didn’t address another issue. Unlike the database, Oracle Enterprise Manager comes with ‘out of the box’ scripts to start and stop OEM. They are the old style System V scripts that run out of /etc/init.d, and it didn’t really seem worth going through the trouble of converting them to the new systemd format. Unfortunately, the OEM scripts always assume that the database is already up and running. If your repository database is running on the same server as your OMS (which isn’t really that big of a deal if your hardware can handle it) that can be fixed by modifying the OEM startup script and adding in a ‘check to make sure your database is up and running before you start OEM’ section. The content in bold below was added to the out of the box OEM script after the initial comments in the file.

[root@dba ~]# vim /etc/init.d/gcstartup
# 2019-03-05 Rich Soule
# OEM should only startup if the emrep database is already up and running
# on the local machine so the below was added to make sure that happens.
#################### Begin Rich Soule Added Lines #######################
if [ "$1" = "start" ]
then
  counter=0
  while [ $counter -le 24 ]
  do
    ((counter++))
    if ! /usr/bin/ps -e | /usr/bin/grep -q ora_pmon_emrep
    then
      echo 'OEM is waiting on Oracle database to start'
      sleep 10
    else
      break
    fi
   done
   if [ $counter -ge 24 ]
   then
     echo 'Oracle database did not start in time, exiting OEM startup'
     exit 1
   fi
   echo 'Oracle database started, waiting 20 more seconds for database to open'
   sleep 20
  echo 'OMS will now attempt to start as per remainer of the /etc/init.d/gcstartup script'
fi
####################  End Rich Soule Added Lines ########################

The above first checks to make sure that gcstartup was called with the start argument. If so then we’ll check if there is an ora_pmon_emrep process running (emrep is the name of my OEM repository database). If it isn’t running, we’ll wait 10 seconds and check again, but only for 240 seconds total. Once that process is found, we break out of our while do loop. If we did hit our 240-second limit, then we exit out of the gcstartup script totally, otherwise, we wait 20 seconds for the database to open and then continue along with the rest of the gcstartup script.

So far this has been working like a charm.

Note that you could do the same type of thing with OEM and instead have the script make a connection to a remote server every 10 seconds to make sure that the remote OEM repository was up before attempting to start OEM.

Happy Linux-ing and Oracle-ing!


SQL Developer SSH Connections In Depth

Capture

Available now: A 29-page white paper on SQL Developer SSH Connections that explains in detail how SSH connections should be configured between your desktop/laptop and your organization’s database!

The vast majority of explanations out on the internet of how to configure SSH connections between Oracle SQL Developer and an Oracle Database use incredibly simplistic explanations, often using the oracle os account as the ssh user (absolutely a worst practice!), and not including important things like how firewalls will impact connections, how your ssh server should be configured, etc.

I wrote a 29-page white paper which has been published on the Insum.ca website. It covers the above and more in exacting detail. If you click the link above you’ll be taken to a description of the white paper and then to a page where you’ll be able to download it after supplying a name and email (a fair trade for the amount of time and effort I put into the white paper…).

Note: The image above is from the white paper and is used for explanation purposes only… None of those things actually exist except for the WordPress external firewall that protects just this blog, or, actually, all the blogs at WordPress.com.


Oracle 12.1.2.7 Virtualized ODA Install

I’ve been working with the brand new 12.1.2.7 ODA update on an X5-2 and Oracle has made a number of changes that impact how an install and configuration is done.

12.1.2.7 is the first release that supports a ‘reflash’ of an ODA with Oracle Linux 6. Previously update 12.1.2.5 had Oracle Linux 5 and updated 12.1.2.6 had Oracle Linux 6, but the only way to move to 12.1.2.6 was to first already be at 12.1.2.5 and then patch to 12.1.2.6.

12.1.2.7 is the first release that will support (re-)deployment of an ODA in a virtualized configuration where you load the Oracle VM ISO into each ILOM and then deploy the ODA_BASE virtual machine. Note that if your ODA was at 12.1.2.6 or earlier you will still have to download the 12.1.2.7 patch to ODA_BASE after you deploy ODA_BASE to patch some of the server and storage components. Oracle has changed from the old patch –infra, –grid, –database options and now has a –server (that does some hardware and the grid infrastructure) and –storage (which patches the shared storage hardware).

One very major change which is especially important for those of us who manage virtualized ODA installs remotely is that /etc/xen/xend-config.sxp on ODA_BASE now has the following section:

# The interface for VNC servers to listen on. Defaults
# to 127.0.0.1  To restore old 'listen everywhere' behaviour
# set this to 0.0.0.0
#(vnc-listen '127.0.0.1')

This means that you when you attempt to create a VM and connect to the console the same way you did in past ODA versions you’re probably going to get something like this:

[root@oda-base-node0 ~]# oakcli show vmconsole OL7U2 
 OAKERR:8006 Error in VNC display configurations : DISPLAY=localhost:10.0

You can either modify the /etc/xen/xend-config.sxp file, or you can update the vm.cfg file for a particular VM that you’ve imported. Since the ODA will be using ACFS it might be a bit hard to find the vm.cfg file since it ends up in the hidden .ACFS directory on the shared repository ACFS file system.

I had created my VMs on a shared repository named sharedrepo so my vm.cfg file ended up here:

[root@oda-base-node0 OL7U2]# pwd 
 /u01/app/sharedrepo/sharedrepo/.ACFS/snaps/OL7U2/VirtualMachines/OL7U2 
[root@oda-base-node0 OL7U2]# vim vm.cfg

Update the vm.cfg file and add the following:

vnclisten = '0.0.0.0'

Note that vnclisten ‘0.0.0.0’ didn’t work and neither did vnclisten = 0.0.0.0. It had to be vnclisten = ‘0.0.0.0’ for the VM to start.

Now that you’ve got a ‘listen anywhere’ vnc console running, you can just use a VNC client on your workstation (I used Tiger VNC) and connect to your node0-dom0:590X where X is the number of VMs you’ve started past ODA_BASE.  ODA_BASE is at :5900 and if this is the first additional VM you’ve created you’ll be able to reach the console at :5901. Note that once you’ve done the initial network configuration of your VM you’ll probably want to take the vnclisten parameter out of the vm.cfg file.

I’m also currently working with Oracle support on an issue. We’ve got a RAC One Node multitenant database and catcon.pl won’t run scripts in the ‘root container and all pluggables’. For example:

[oracle@oda-base-node0 ~]$ $ORACLE_HOME/perl/bin/perl $ORACLE_HOME/rdbms/admin/catcon.pl -u SYS -d $ORACLE_HOME/rdbms/admin -b utlrp_output utlrp.sql 
 catcon: ALL catcon-related output will be written to utlrp_output_catcon_77673.lst 
 catcon: See utlrp_output*.log files for output generated by scripts 
 catcon: See utlrp_output_*.lst files for spool files, if any 
 Enter Password: 
 catconInit: database is not open on the default instance 
 Unexpected error encountered in catconInit; exiting

This is important because apxremov_con.sql is just a wrapper for catcon.pl so you can’t remove APEX from the root container (CDB$ROOT) and PDB$SEED so that new pluggable databases won’t have APEX in them. We do this so that we’ll be able to potentially have different versions of APEX in different pluggable databases in the same container database.

Oracle has opened a P1 bug on this and has actively been working this for a few days.

Happy ODAing!

Rich


yum install tree -y

You need to do this right now!

As root on your Oracle Linux server, type:

[root@sqlfundamentals ~]# yum install tree -y

Then you can do this:

[oracle@sqlfundamentals ~]$ cd labs/
[oracle@sqlfundamentals labs]$ tree
.
├── sql1
│   ├── code_ex
│   │   ├── cleanup_scripts
│   │   │   ├── cleanup_10.sql
│   │   │   └── cleanup_11.sql
│   │   ├── code_09_retired_emp_tab.sql
│   │   ├── code_10_cretest.sql
│   │   ├── code_ex_02.sql
│   │   ├── code_ex_03.sql
│   │   ├── code_ex_04.sql
│   │   ├── code_ex_05.sql
│   │   ├── code_ex_06.sql
│   │   ├── code_ex_07.sql
│   │   ├── code_ex_08.sql
│   │   ├── code_ex_09.sql
│   │   ├── code_ex_10.sql
│   │   └── code_ex_11.sql
│   ├── demo
│   │   ├── demo_02_alias.sql
│   │   ├── demo_02_null.sql
│   │   ├── demo_03_and.sql
│   │   ├── demo_03_betw.sql
│   │   ├── demo_03_expr.sql
│   │   ├── demo_03_in.sql
│   │   ├── demo_03_ordernull.sql
│   │   ├── demo_03_or.sql
│   │   ├── demo_03_sal1.sql
│   │   ├── demo_03_sal2.sql
│   │   ├── demo_03_varno.sql
│   │   ├── demo_03_varyes.sql
│   │   ├── demo_04_case.sql
│   │   ├── demo_04_nest.sql
│   │   ├── demo_05_day1.sql
│   │   ├── demo_05_day2.sql
│   │   ├── demo_05_hire.sql
│   │   ├── demo_05_mask.sql
│   │   ├── demo_06_count1.sql
│   │   ├── demo_06_count2.sql
│   │   ├── demo_06_error.sql
│   │   ├── demo_06_job1.sql
│   │   ├── demo_06_job2.sql
│   │   ├── demo_06_listagg.sql
│   │   ├── demo_06_order1.sql
│   │   ├── demo_06_order2.sql
│   │   ├── demo_07_cart.sql
│   │   ├── demo_07_loc.sql
│   │   ├── demo_09_inters.sql
│   │   ├── demo_09_minus.sql
│   │   ├── demo_09_orderby.sql
│   │   ├── demo_09_union1.sql
│   │   ├── demo_09_union3.sql
│   │   ├── demo_10_cretabs.sql
│   │   ├── demo_10_grant.sql
│   │   ├── demo_10_revoke.sql
│   │   ├── demo_10_select.sql
│   │   ├── demo_10_update.sql
│   │   ├── demo_11_identity_col.sql
│   │   ├── demo_11_invisible_col.sql
│   │   ├── demo_11_sol_col_default.sql
│   │   └── demo_d_loc.sql
│   ├── labs
│   │   ├── lab_10_01.sql
│   │   ├── Online_Book_Store_Create_Table.sql
│   │   ├── Online_Book_Store_Drop_Tables.sql
│   │   └── Online_Book_Store_Populate.sql
│   └── soln
│       ├── sol_02.sql
│       ├── sol_03.sql
│       ├── sol_04.sql
│       ├── sol_05.sql
│       ├── sol_06.sql
│       ├── sol_07.sql
│       ├── sol_08.sql
│       ├── sol_09.sql
│       ├── sol_10.sql
│       ├── sol_11.sql
│       └── sol_ap.sql
└── sql2
├── code_ex
│   ├── cleanup_scripts
│   │   ├── cleanup_03.sql
│   │   ├── cleanup_05.sql
│   │   ├── cleanup_07.sql
│   │   ├── cleanup_09.sql
│   │   └── cleanup_10.sql
│   ├── code_cre_emp_history.sql
│   ├── code_cre_emp_sales.sql
│   ├── code_cre_mgr_history.sql
│   ├── code_cre_sales_info.sql
│   ├── code_cre_sales_source_data.sql
│   ├── code_cre_sal_high.sql
│   ├── code_cre_sal_history.sql
│   ├── code_cre_sal_low.sql
│   ├── code_cre_sal_mid.sql
│   ├── code_ex_02.sql
│   ├── code_ex_03.sql
│   ├── code_ex_04.sql
│   ├── code_ex_05.sql
│   ├── code_ex_06.sql
│   ├── code_ex_07.sql
│   ├── code_ex_08.sql
│   ├── code_ex_09.sql
│   ├── code_ex_10.sql
│   └── code_ins_sales_source_data.sql
├── demo
│   ├── demo_04_easyvu.sql
│   ├── demo_04_emp.sql
│   ├── demo_06_ex_1a.sql
│   ├── demo_06_ex_1b.sql
│   ├── demo_06_ex_2.sql
│   ├── demo_06_pairwise_a.sql
│   ├── demo_06_pairwise_b.sql
│   ├── demo_06_pairwise.sql
│   ├── demo_06_With.sql
│   ├── demo_07_delete_rows.sql
│   ├── demo_07_update_rows.sql
│   ├── demo_07_update_subquery1.sql
│   ├── demo_07_update_subquery.sql
│   ├── demo_08_revoke.sql
│   ├── demo_09_default_tab.sql
│   ├── demo_10_tz.sql
│   ├── demo_d_loc.sql
│   ├── demo_I_regexp_replace_a.sql
│   └── demo_I_regexp_replace_b.sql
├── emp_dir
│   ├── emp.dat
│   ├── library_items.dat
│   ├── LIBRARY_ITEMS_EXT_23121.log
│   ├── LIBRARY_ITEMS_EXT_5040.log
│   ├── OLDEMP_14719.log
│   └── OLDEMP_26721.log
├── labs
│   ├── confidence.sql
│   ├── lab_02_06_tab.sql
│   ├── lab_04_07.sql
│   ├── lab_05_09.sql
│   ├── lab_05_10.sql
│   ├── lab_05_11_a.sql
│   ├── lab_05_11_b.sql
│   ├── lab_05_11_g.sql
│   ├── lab_06_insert_empdata.sql
│   ├── lab_08_05.sql
│   ├── lab_09_01.sql
│   ├── lab_09_03.sql
│   ├── lab_09_05.sql
│   ├── lab_09_08_a.sql
│   ├── lab_09_08_b.sql
│   ├── lab_09_08_e.sql
│   ├── lab_09_09.sql
│   ├── lab_09_10.sql
│   ├── lab_09_11.sql
│   ├── lab_10_06.sql
│   ├── lab_ap_cre_mgr_history.sql
│   ├── lab_ap_cre_sal_history.sql
│   ├── lab_ap_cre_special_sal.sql
│   ├── Online_Book_Store_Drop_Tables.sql
│   └── Online_Book_Store_Populate.sql
└── soln
├── sol_02.sql
├── sol_03.sql
├── sol_04.sql
├── sol_05.sql
├── sol_06.sql
├── sol_08.sql
├── sol_09.sql
├── sol_10.sql
└── sol_ap.sql

13 directories, 154 files
[oracle@sqlfundamentals labs]$

And this is awesome!