Diagnose Oracle Net and RAC Interconnect Routing with netstat -r

Diagnose Oracle Net and RAC Interconnect Routing with netstat -r

Purpose

Oracle Net connections and Oracle RAC private interconnect traffic both depend on correct IP routing. When an Oracle client cannot connect to the listener, when RAC Cache Fusion or heartbeat traffic is routing over the wrong interface, or when sqlnet.ora logs show unexpected source addresses, the OS routing table is the first place to look. netstat -r prints the kernel routing table — the same table that determines where every TCP packet on the Oracle server goes.

This command, from the shutdownabort.com DBA Quick Guides (Andrew Barry, 2007–2013, preserved via the Wayback Machine), is a quick orientation tool that works across every UNIX variant Oracle has ever run on: AIX, HP-UX, Solaris, and all Linux distributions. On modern Linux systems ip route is preferred, but netstat -r is ubiquitous — making it the one routing command a DBA can use regardless of which platform the Oracle server runs on.

Code

 1# Print the kernel routing table
 2netstat -r
 3
 4# Suppress DNS lookups — faster when DNS is broken or slow
 5netstat -rn
 6
 7# Linux only: routing table with per-interface statistics
 8netstat -rn -i
 9
10# Modern Linux equivalent (iproute2; preferred on RHEL 7+)
11ip route show

Code Breakdown

netstat -r

Prints the kernel IP routing table. Each row is a routing rule the kernel applies to outgoing packets.

-r — Show the routing table. Without -r, netstat shows network connection sockets, not routes.

netstat -rn

Adding -n suppresses DNS reverse lookups on the Destination and Gateway columns. Essential on Oracle database servers where DNS resolution can be slow or broken — netstat -r without -n can hang for 30 seconds per entry on servers with unreachable DNS. Always use -n for diagnostic work.

ip route show

The modern Linux replacement for netstat -r provided by the iproute2 package. Output format differs but information is equivalent. Oracle Linux 7+ and RHEL 7+ include it by default. Prefer netstat -rn for cross-platform UNIX scripts; use ip route for Linux-only automation or when net-tools is not installed.

Key Points

• RAC private interconnect must NOT route over the public interface. In a RAC cluster, the private interconnect (a dedicated interface, typically eth1, bond1, or ib0) carries Cache Fusion block transfers and cluster heartbeat. If the routing table sends interconnect traffic over the public interface (eth0), RAC performance degrades severely and the cluster can become unstable. netstat -rn immediately shows which interface the private subnet routes through.
• LOCAL_LISTENER registration depends on routing. The LOCAL_LISTENER init parameter tells PMON which address to register with the listener. If that VIP address's subnet routes via the wrong interface, PMON registration fails and clients get ORA-12514 "no service" errors. The routing table confirms whether the VIP subnet is reachable on the correct interface.
• netstat is deprecated on RHEL 8/9 and Oracle Linux 8/9. The net-tools package is no longer installed by default. Install with yum install net-tools or use ip route show instead. On AIX, HP-UX, and Solaris, netstat -rn remains the primary routing tool — no replacement needed.
• A default route pointing at the wrong gateway causes asymmetric routing. Oracle clients connect via the IP registered in tnsnames.ora. If the server's default route sends response packets out a different interface than where the connection arrived, the TCP session drops. netstat -rn reveals asymmetric routing immediately from the Iface column.
• SCAN listener routing (Oracle 11.2+). Single Client Access Name (SCAN) uses three VIPs that float across RAC nodes. Each SCAN VIP must be routable from every client subnet. If a client subnet is missing a route to the SCAN VIP range, connections fail at the SCAN listener before reaching node VIPs — a routing table problem, not an Oracle configuration problem.

Insights and Best Practices

RAC interconnect validation sequence

On a new RAC installation or after any network change, validate routing with this sequence:

 1# 1. Print routing table — confirm private subnet routes via eth1 / ib0, not eth0
 2netstat -rn
 3
 4# 2. Confirm the private IP is bound to the correct interface
 5ip addr show eth1
 6
 7# 3. Ping the other RAC node's private IP explicitly via the private interface
 8ping -I eth1 192.168.10.2    # Must succeed; if routing via eth0, routing is wrong
 9
10# 4. Check Oracle Clusterware's interface assignment
11$GRID_HOME/bin/oifcfg getif

oifcfg getif shows which interface Oracle Clusterware has assigned to each cluster role (public, cluster_interconnect, storage). The interface shown by netstat -rn for the private subnet must match what oifcfg expects.

Diagnosing ORA-12541 (no listener) with netstat

When a client cannot reach the listener on port 1521:

1# On the database server: confirm the listener is bound and the port is open
2netstat -tlnp | grep 1521
3
4# Confirm the listener's VIP address is routable on the correct interface
5netstat -rn | grep "<listener_vip_subnet>"
6
7# From the client: trace the route to the listener IP
8traceroute <listener_ip>

If netstat -rn on the server shows no route to the listener's VIP subnet, the VIP or SCAN address is not reachable on the server's network stack — likely a missing route or a VIP that failed to start on clusterware initialization.

Useful awk extractions from netstat output

1# Extract the default gateway (the router Oracle uses for external traffic)
2netstat -rn | awk '/^0\.0\.0\.0|^default/ {print "Default gateway:", $2; exit}'
3
4# Show only routes relevant to Oracle interfaces
5netstat -rn | awk 'NR<=2 || /eth0|eth1|bond0|ib0/'
6
7# Check whether a specific SCAN IP subnet is routable
8SCAN_SUBNET="10.10.20.0"
9netstat -rn | awk -v s="$SCAN_SUBNET" '$1 == s {print "SCAN subnet routable via:", $NF; found=1} END {if(!found) print "SCAN subnet NOT in routing table"}'

When to Run This

• After a RAC node reboot: confirm the private interconnect route is back on the correct interface
• When ORA-12541 (no listener) appears from clients that connected successfully before
• When Cache Fusion latency spikes unexpectedly: check whether private interconnect has rerouted to the public interface
• After a network configuration change (NIC bonding, VLAN change, IP address reassignment)
• When diagnosing asymmetric routing causing dropped TCP connections to the Oracle listener
• Before provisioning a new Oracle server: verify public, private, and storage network routes are all present before running the Oracle installer

Troubleshooting Common Issues

If netstat -r hangs for 30+ seconds, add -n to suppress DNS lookups. DNS issues are common on Oracle database servers — the server's /etc/resolv.conf may point at an internal DNS that is unreachable from the database network segment.

If netstat is not found on Oracle Linux 8 or RHEL 8, install the net-tools package: sudo yum install net-tools. Alternatively, use ip route show — the output format is slightly different but the routing information is identical.

If the RAC private interconnect route is missing from the routing table after a node reboot, the network interface may not have come up correctly. Check ip link show eth1 for interface state (UP/DOWN) and systemctl status network for network service failures. On Oracle Linux 8+, check nmcli device status for NetworkManager-managed interfaces.

References

• Oracle Database Net Services Administrator's Guide 19c — covers Oracle Net listener configuration, LOCAL_LISTENER parameter, SCAN listener architecture, and the network prerequisites the routing table must satisfy for listener registration and client connectivity
• Oracle Real Application Clusters Administration and Deployment Guide 19c — RAC private interconnect requirements, oifcfg interface assignment, and the network validation steps including routing table verification referenced in the RAC validation sequence
• shutdownabort.com — Miscellaneous Useful UNIX (Wayback, 2013-01-15) — original source of the netstat -r command from Andrew Barry's DBA Quick Guides, used for Oracle Net and RAC interconnect routing diagnostics
• Linux man page — netstat(8) — full flag reference including -r, -n, -t, -l, -p options used in this post; includes deprecation notice in favor of ss and ip on modern Linux
• Oracle Grid Infrastructure Administration and Deployment Guide 19c — oifcfg — oifcfg getif command used in the RAC validation sequence; maps cluster roles to network interfaces