DNS configuration for the SCAN used with Oracle Database 11gR2

This post provides the basic DNS configuration steps necessary to use the Single Client Access Name (SCAN) introduced in Oracle 11g Release 2 RAC.

I’m not an network guy and don’t claim to know much about DNS.

This post represents the minimum you can do to make a SCAN work. The DNS configuration in the post has been done on a DNS server installed on a separate VM called romford.mpoojari.co.uk where romford is the hostname (dns server name) and mpoojari.co.uk is the domain name. The IP address of this server will be 192.168.0.10 (static IP).

DNS installation

Install and configure RHEL 5.4 VM

[root@romford etc]# uname -a
Linux romford.mpoojari.co.uk 2.6.18-194.el5 #1 SMP Mon Mar 29 22:10:29 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

Ensure you have choose DNS Name server and Server configuration tools(select bind packages) during the installation phase.

Check if the following DNS Server Packages are installed

[root@romford ~]# rpm -qa | grep bind
bind-libs-9.3.6-4.P1.el5_4.2
ypbind-1.19-12.el5
system-config-bind-4.0.3-4.0.1.el5
bind-utils-9.3.6-4.P1.el5_4.2
bind-chroot-9.3.6-4.P1.el5_4.2
bind-9.3.6-4.P1.el5_4.2

Three packages must be installed on Linux for DNS Server:
•bind (includes DNS server, named)
•bind-utils (utilities for querying DNS servers about host information)
•bind-libs (libraries used by the bind server and utils package)

You should be able to locate /etc/named.conf
If not copy  all the files under /usr/share/doc/bind-9.3.4/sample/etc  to /etc
Also copy all the files under /usr/share/doc/bind-9.3.4/sample/var  to /var

[root@romford etc]# ls -ltr /usr/share/doc/bind-9.3.6/sample/etc
total 32
-rw-r--r-- 1 root root  524 Jan 20  2010 named.root.hints
-rw-r--r-- 1 root root  775 Jan 20  2010 named.rfc1912.zones
-rw-r--r-- 1 root root 4298 Jan 20  2010 named.conf
-rw-r--r-- 1 root root    0 Jan 20  2010 rndc.conf

[root@romford etc]# ls -ltr /usr/share/doc/bind-9.3.6/sample/var/
total 8
drwxr-xr-x 4 root root 4096 Nov 24 14:01 named
[root@romford etc]# ls -ltr /usr/share/doc/bind-9.3.6/sample/var/named/
total 88
-rw-r--r-- 1 root root  427 Jan 20  2010 named.zero
-rw-r--r-- 1 root root 1892 Jan 20  2010 named.root
-rw-r--r-- 1 root root  426 Jan 20  2010 named.local
-rw-r--r-- 1 root root  424 Jan 20  2010 named.ip6.local
-rw-r--r-- 1 root root  427 Jan 20  2010 named.broadcast
-rw-r--r-- 1 root root  195 Jan 20  2010 localhost.zone
-rw-r--r-- 1 root root  198 Jan 20  2010 localdomain.zone
-rw-r--r-- 1 root root   56 Jan 20  2010 my.internal.zone.db
-rw-r--r-- 1 root root   56 Jan 20  2010 my.external.zone.db
drwxr-xr-x 2 root root 4096 Jan 20  2010 data
drwxr-xr-x 2 root root 4096 Nov 24 14:01 slaves

1. Generate the DNS Key

[root@romford etc]# which dns-keygen
/usr/sbin/dns-keygen

[root@romford etc]# dns-keygen
j30zdeKhACTGFWSdxGgHVSC3jFm0ejIAn5p4mJkYmA0cyHp4jDaT30SYQLMR

2. Configure the named.conf file

Add the DNS key to the /etc/named.conf under the below section and should look like

replace

key ddns_key
{
        algorithm hmac-md5;
        secret "use /usr/sbin/dns-keygen to generate TSIG keys";
};

to

key ddns_key
{
         algorithm hmac-md5;
         secret "j30zdeKhACTGFWSdxGgHVSC3jFm0ejIAn5p4mJkYmA0cyHp4jDaT30SYQLMR";
 };

Add the below line in the /etc/named.conf .

zone "mpoojari.co.uk" IN {
 type master;
 file "mpoojari.co.uk.zone";
 allow-update { none; };
};

3. Create the zone file under /var/named

[root@romford etc]# ls -ltr /var/named
total 44
drwxr-xr-x 2 root root 4096 Nov 24 16:27 slaves
-rw-r--r-- 1 root root  427 Nov 24 16:27 named.zero
-rw-r--r-- 1 root root 1892 Nov 24 16:27 named.root
-rw-r--r-- 1 root root  426 Nov 24 16:27 named.local
-rw-r--r-- 1 root root  424 Nov 24 16:27 named.ip6.local
-rw-r--r-- 1 root root  427 Nov 24 16:27 named.broadcast
-rw-r--r-- 1 root root   56 Nov 24 16:27 my.internal.zone.db
-rw-r--r-- 1 root root   56 Nov 24 16:27 my.external.zone.db
-rw-r--r-- 1 root root  195 Nov 24 16:27 localhost.zone
-rw-r--r-- 1 root root  198 Nov 24 16:27 localdomain.zone
drwxr-xr-x 2 root root 4096 Nov 24 16:27 data

[root@romford etc]# cd /var/named
[root@romford named]# cp localdomain.zone mpoojari.co.uk.zone
[root@romford named]# ls -ltr
total 48
drwxr-xr-x 2 root root 4096 Nov 24 16:27 slaves
-rw-r--r-- 1 root root  427 Nov 24 16:27 named.zero
-rw-r--r-- 1 root root 1892 Nov 24 16:27 named.root
-rw-r--r-- 1 root root  426 Nov 24 16:27 named.local
-rw-r--r-- 1 root root  424 Nov 24 16:27 named.ip6.local
-rw-r--r-- 1 root root  427 Nov 24 16:27 named.broadcast
-rw-r--r-- 1 root root   56 Nov 24 16:27 my.internal.zone.db
-rw-r--r-- 1 root root   56 Nov 24 16:27 my.external.zone.db
-rw-r--r-- 1 root root  195 Nov 24 16:27 localhost.zone
-rw-r--r-- 1 root root  198 Nov 24 16:27 localdomain.zone
drwxr-xr-x 2 root root 4096 Nov 24 16:27 data
-rw-r--r-- 1 root root  198 Nov 24 16:33 mpoojari.co.uk.zone

Add the below entries to the zone file /var/named/mpoojari.co.uk.zone (and /var/named/localdomain.zone file not required as my domain is mpoojari.co.uk)

 
rac1            IN A            192.168.0.101
rac2            IN A            192.168.0.102
rac1-vip        IN A            192.168.0.221
rac2-vip        IN A            192.168.0.222
rac1-priv       IN A            192.168.1.101
rac2-priv       IN A            192.168.1.102
rac-scan        IN A            192.168.0.51
rac-scan        IN A            192.168.0.52
rac-scan        IN A            192.168.0.53

4. Configure a reverse proxy
To identify the SCAN name from any of the IPS’s we need to configure a reverse proxy .
Add the below entry to /etc/named.conf

zone "0.168.192.in-addr.arpa." IN {
 type master;
 file "0.168.192.in-addr.arpa";
 allow-update { none; };
 };

5. Create the reverse proxy file under /var/named

 
[root@ast1 named]# cat 0.168.192.in-addr.arpa
$ORIGIN 0.168.192.in-addr.arpa.
$TTL 1H
@       IN      SOA     mpoojari.co.uk.     root.mpoojari.co.uk. (      2
                                                3H
                                                1H
                                                1W
                                                1H )
0.168.192.in-addr.arpa.         IN NS      mpoojari.co.uk.

101     IN PTR  rac1.mpoojari.co.uk.
102     IN PTR  rac2.mpoojari.co.uk.
221     IN PTR  rac1-vip.mpoojari.co.uk.
222     IN PTR  rac2-vip.mpoojari.co.uk.
51      IN PTR  rac-scan.mpoojari.co.uk.
52      IN PTR  rac-scan.mpoojari.co.uk.
53      IN PTR  rac-scan.mpoojari.co.uk.

The named.conf file finally looks like as shown below

[root@romford etc]# cat /etc/named.conf
options
{
        // Those options should be used carefully because they disable port
        // randomization
        // query-source    port 53;
        // query-source-v6 port 53;

        // Set IP address correctly.
        listen-on port 53 { 127.0.0.1; 192.168.0.10; };

        // My Additions
        // Forwarder: Anything this DNS can't resolve gets forwarded to my ISPs DNS.
        // forwarders { 194.168.4.100; 194.168.8.100; };
        // End My Additions

        // Put files that named is allowed to write in the data/ directory:
        directory "/var/named"; // the default
        dump-file               "data/cache_dump.db";
        statistics-file         "data/named_stats.txt";
        memstatistics-file      "data/named_mem_stats.txt";

};
zone "mpoojari.co.uk" IN {
        type master;
        file "mpoojari.co.uk.zone";
        allow-update { none; };
};
zone "0.168.192.in-addr.arpa." IN {
        type master;
        file "0.168.192.in-addr.arpa";
        allow-update { none; };
};
key ddns_key
{
        algorithm hmac-md5;
        secret "j30zdeKhACTGFWSdxGgHVSC3jFm0ejIAn5p4mJkYmA0cyHp4jDaT30SYQLMR";
};

6. Restart the named service as below

[root@romford named]# service named stop
Stopping named:                                            [  OK  ]

[root@romford named]# service named start
Starting named:                                            [  OK  ]

7. Edit the /etc/resolv.conf and put your domain name and the DNS server name on all nodes

[root@romford named]# cat /etc/resolv.conf
search localdomain mpoojari.co.uk
nameserver 192.168.0.10

8. On the DNS server

[root@romford named]# cat /etc/nsswitch.conf |grep dns
#       dns                     Use DNS (Domain Name Service)
#hosts:     db files nisplus nis dns
hosts:      dns files nis

9. Do a nslookup on SCAN and observe the IP addresses round-robin resolution

[root@romford etc]# nslookup rac-scan.mpoojari.co.uk
Server:         192.168.0.10
Address:        192.168.0.10#53

Name:   rac-scan.mpoojari.co.uk
Address: 192.168.0.52
Name:   rac-scan.mpoojari.co.uk
Address: 192.168.0.53
Name:   rac-scan.mpoojari.co.uk
Address: 192.168.0.51

[root@romford etc]# nslookup rac-scan.mpoojari.co.uk
Server:         192.168.0.10
Address:        192.168.0.10#53

Name:   rac-scan.mpoojari.co.uk
Address: 192.168.0.51
Name:   rac-scan.mpoojari.co.uk
Address: 192.168.0.52
Name:   rac-scan.mpoojari.co.uk
Address: 192.168.0.53

[root@romford etc]# nslookup rac-scan.mpoojari.co.uk
Server:         192.168.0.10
Address:        192.168.0.10#53

Name:   rac-scan.mpoojari.co.uk
Address: 192.168.0.53
Name:   rac-scan.mpoojari.co.uk
Address: 192.168.0.51
Name:   rac-scan.mpoojari.co.uk
Address: 192.168.0.52

If try to ping rac-scan.testenv.com at this moment, you will find it displays one of SCAN ip addresses but it will not be able to be reached. This is correct behavior. Also at this stage, I have not installed rac1 and rac2 nodes hence they will not be able to be reached.

Once the GI software has been installed and is running it will bring these IP addresses online and at that point the SCAN IP should be pingable.

10. nslookup on VIP names

[root@romford etc]# nslookup rac1-vip
Server:         192.168.0.10
Address:        192.168.0.10#53

Name:   rac1-vip.mpoojari.co.uk
Address: 192.168.0.221

[root@romford etc]# nslookup rac2-vip
Server:         192.168.0.10
Address:        192.168.0.10#53

Name:   rac2-vip.mpoojari.co.uk
Address: 192.168.0.222

Troubleshooting

Error while restarting named service,

[root@romford etc]# service named restart
Stopping named:                                            [  OK  ]
Locating /var/named/chroot//etc/named.conf failed:
                                                           [FAILED]

I’ve commented, “#ROOTDIR=/var/named/chroot” in /etc/sysconfig/named.
And now service named will start!

[root@romford etc]# service named restart
Stopping named:                                            [  OK  ]
Starting named:                                            [  OK  ]

Network Configuration of rac nodes

Node1

  • Set DNS server ip address to 192.168.0.10
  • Set gateway ip address to 192.168.0.1
  • Set hostname as rac1.mpoojari.co.uk
  • set static IP to 192.168.0.101 (eth0)
  • set static IP to 192.168.1.101 (eth1)
  • Set subnet mask 255.255.255.0

Node2

  • Set DNS server ip address to 192.168.0.10
  • Set gateway ip address to 192.168.0.1
  • Set hostname as rac2.mpoojari.co.uk
  • set static IP to 192.168.0.102 (eth0)
  • set static IP to 192.168.1.102(eth1)
  • Set subnet mask 255.255.255.0

At this stage,

rac1 is reachable from DNS server, romford and from rac2
rac2 is reachable from DNS server, romford and from rac1

SCAN name is also pingable on rac1 and rac2.

Note that hosts files  do not contain any IP addresses as it is DNS resolved.

Advertisements

OUI does not recognize ASM disks in 11.2

Oracle Universal Installer (OUI) does not recognize ASM disks during the grid infrastructure installation.

Cause: Oracleasm (ASMLIB) was configured for user root instead of OS user that owns the Grid Infrastructure installation.
Configure oracleasm for OS user that owns the Grid Infrastructure installation.

E.g. if that user was oracle in dba group, as root run

/etc/init.d/oracleasm configure

and
answer oracle to ‘Default user to own the driver interface’.
answer dba to ‘Default group to own the driver interface’
answer y to Start ‘Oracle ASM library driver on boot’
answer y to ‘Fix permissions of Oracle ASM disks on boot’

[root@myrac1 logs]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets (‘[]’).  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface [grid]:
Default group to own the driver interface [asmadmin]:
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]

Resolved.

Also, make sure the following on all rac nodes:

[root@myrac1 ~]# id grid
uid=1100(grid) gid=1000(oinstall) groups=1000(oinstall),1200(asmadmin),1201(asmdba),1202(asmoper)

[root@myrac11 ~]# id oracle
uid=1101(oracle) gid=1000(oinstall) groups=1000(oinstall),1201(asmdba),1300(dba),1301(oper)

User equivalence check failed

I could not get the user equivalence check to work on my Solaris 10 server when trying to install 11gR2 Grid.

No issues were encountered during the install.

<< Message: Result: User equivalence check failed for user “grid”. >>

Cluvfy and the OUI tries to find SSH on Solaris at /usr/local/bin.

Workaround is to create a softlink from /usr/bin/ssh to /usr/local/bin.

Note: User equivalence is required for installations (IE using OUI) and patching. DBCA, NETCA, and DBControl also require user equivalence.

IPMP and 11gR2 Grid Infrastructure on Sun Solaris 10

11.2.0.1 clusterware fails to start or evict if IPMP is used for public or private network. This happened when I ran the root script. // <![CDATA[// Node eviction can also happen when IPMP fails over private IP from active NIC to other NIC in same group.

bug 9260196 is fixed in 11.2.0.2; for 11.2.0.1, fix is available in patch 9729439.  Applied this patch and re ran the root scripts.

Oracle Grid Infrastructure 11.2.0.1 does not support native multiple redundant network support. Therefore, an external redundancy mechanism must be used. Oracle Solaris provides two solutions: Trunking (Link Aggregation based) and IPMP (IP Multipathing).

Refer metalink note 1069584.1, 1067353.1

root.sh fails during 11gR2 Grid infrastructure installation

Whenever the root.sh script fails for the first time, do not run it again. Do the following steps and clusterware will be healthy up and running.

It happened that root.sh failed, when I did 11gR2 Rac Cluster set up on Sun Solaris and Enterprise Linux boxes. I could find the errors recorded in the logs.

Deconfigure and Reconfigure of Grid Infrastructure Cluster:

Identify cause of root.sh failure by reviewing logs in $GRID_HOME/cfgtoollogs/crsconfig and $GRID_HOME/log, once cause is identified, deconfigure and reconfigure with steps below – please keep in mind that you will need wait till each step finishes successfully before move to next one:

For Step1 and 2, you can skip node(s) on which you didn’t execute root.sh yet.

Step 1: As root, run “$GRID_HOME/crs/install/rootcrs.pl -verbose -deconfig -force” on all nodes, except the last one.

Step 2: As root, run “$GRID_HOME/crs/install/rootcrs.pl -verbose -deconfig -force -lastnode” on last node. This command will zero out OCR and VD disk also.

Step 3: As root, run $GRID_HOME/root.sh on first node

Step 4: As root, run $GRID_HOME/root.sh on all other node(s), except last one.

Step 5: As root, run $GRID_HOME/root.sh on last node.

Eevrything is working now.

Reference : ID 942166.1

  1. Deinstall of Grid Infrastructure Cluster
  2. Deinstall of Grid Infrastructure Standalone Server
  3. Deconfigure and Reconfigure of Standalone Server (Oracle Restart)

If there is a firewall between the nodes , then disable it between nodes, and refer to note 942166.1 or note 969254.1
to proceed further.