It is Oracle’s standing recommendation to separate the various types of communication in an Oracle RAC cluster as much as possible. This general recommendation is the basis for the following separation of communication:
- Each node in an Oracle RAC cluster must have at least one public network.
- Each node in an Oracle RAC cluster must have at least one private network, also referred to as “interconnect”.
- Each node in an Oracle RAC cluster must have at least an additional network interface, if the shared storage is accessed using a network based connection.
In addition Oracle RAC and Oracle Clusterware deployment best practices recommend that the interconnect be deployed on a stand-alone, physically seperate, dedicated switch, since it represents the easiest to configure and most secure as well as stable configuration. Many customers, however, have consolidated or prefer to consolidate these stand-alone switches into larger managed switches.
Depending on the level of consolidation that is performed on the switch level, the switch thereby may become a single point of failure. Hardware redundancy within an enterprise switch may mitigate some of the risks, but there are limitations as far as maintenance operations are concerned. Mainaining switch redundancy is therefore highly recommended. Another consequence of this consolidation is a merging of IP networks on a single shared switch, segmented by VLANs in various levels, which include, but are not limited to:
- Sharing the same switch (and network channel) for private and public communication
- Sharing the same switch (and network channel) for the private communication of more than one cluster.
- Sharing the same switch (and network channel) for private communication and shared storage access.
While an increasingly powerful network infrastructure makes it more and more interesting for customers to consolidate network communication on fewer physical networks, it needs to be remembered that the latency and bandwidth requirements as well as availability requirements of the Oracle RAC / Oracle Clusterware interconnect IP network are more in-line with high performance computing. In a more abstract way, one should not look at the interconnect as a network, but rather as a backplane to connect the memory of the cluster nodes.
While observing the bandwidth requirements, Oracle generally recommends maintaining a 1:1 relation when VLANs are used in any possible way and if the usage of VLANs cannot be avoided. In this context, it needs to be noted that bandwidth and latency are not the only concerns. Security, ease of management, and unintended but possible side-effects of using a shared resource such as multicast flooding or spanning tree re-convergence also need to be considered. In detail:
Sharing the same switch (and network channel) for private and public communication
- and deploying the interconnect on a VLAN in this environment, there should be a 1:1 mapping of the VLAN to a non-routable subnet and the VLAN should not span multiple VLANs (tagged) or multiple switches.
Sharing the same switch (and network channel) for the private communication of more than one cluster,
- one VLAN per cluster is recommended for the purpose of a “cleaner” management and security (see above).
- Further consolidation, such as using only one VLAN for all clusters, is supported, but not recommended.
- It is supported to use the same, consolidated network infrastructure (within the same security domain) for various clusters without the use of VLANs, while separated channels are recommended.
Sharing the same switch (and network channel) for private communication and shared storage access
- is supported, if the underlying network infrastructure recognizes and prioritizes network based communication to the storage.