Hey there! Let’s walk through your setup question clearly— I’ve helped with similar cluster network upgrades before, so let’s break this down step by step.

一、链路连通性可行性 #

Short answer: Yes, this setup can work—as long as your hardware supports the necessary compatibility and you configure the switches correctly. Here’s why:

• Mellanox SN2100’s QSFP28 ports are designed to support backward compatibility with QSFP+ modules/cables, including 40G QSFP+ to 4x10G SFP+ breakout cables.
• Your existing 10GBe switch (Netgear XS728T) has SFP+ ports that should pair perfectly with the 10G SFP+ ends of the breakout cable, assuming the cable’s transceivers match your switch’s supported 10G standards (e.g., 10GBASE-SR/LR or copper SFP+).

Once the physical link is up, your compute nodes will be able to route traffic through the 100G switch → breakout cable → 10G switch → WAN to access repos, webpages, and other external resources.

二、必要的交换机配置 #

You’ll need to make a few key tweaks to one or both switches to get this running smoothly:

1. Mellanox SN2100: Enable Breakout Mode #

By default, QSFP28 ports on the SN2100 run in 100G mode. You need to split the target QSFP28 port into 4 independent 10G logical ports to use the breakout cable:

• If using the CLI, the command will look something like this (exact syntax depends on your firmware version):

  interface ethernet 1/1 breakout 4x10G
  

• You can also do this via the web UI: Navigate to the port settings, find the breakout option, and select "4x10G" for the port connected to your breakout cable.

2. VLAN & Broadcast Domain Alignment #

• If your existing 10G switch uses VLANs for WAN/internet traffic, add the breakout-enabled 10G ports on the SN2100 to the same VLAN as the WAN-facing port on your 10G switch.
• If you’re using a flat network (no VLANs), just ensure all involved ports (compute node ports on SN2100, breakout ports on SN2100, WAN port on 10G switch) are in the same default broadcast domain.

3. Link Negotiation Settings #

• Make sure both the breakout ports on the SN2100 and the connected SFP+ ports on your Netgear XS728T use the same link negotiation mode. The safest bet is enabling auto-negotiation on both ends, or manually setting them to 10G full-duplex if auto-negotiation causes issues.

4. Routing Configuration #

• If your SN2100 is a layer 3 switch: Configure a default route pointing to the gateway IP of your 10G switch (the one connected to the WAN).
• If your SN2100 is a layer 2 switch: Ensure your compute nodes have their default gateway set to the WAN gateway IP configured on the 10G switch, and that no ACLs are blocking traffic between the two switches.

三、Final Checks #

After applying these configurations, verify everything works by:

• Checking port status on both switches (look for "up/up" or "connected" status)
• Pinging a public IP (e.g., 8.8.8.8) from a compute node
• Trying to pull a package from a repo to confirm full internet access

备注：内容来源于stack exchange，提问作者Kyungrae Kang