Originating Author: User: Andersot

Background The following details out the process for creating and maintaining the FCIP Tunnels over Brocade Silkworm 7500’s crossing an OC-12. This assumes you have prior knowledge of the Brocade Fabric OS.

When using the Silkworm 7500, you have 32 ports available, 16 physical FC ports, 2 GE ports, and 16 Virtual Ports

Definitions

• Backbone Fabric – Silkworm 7500’s connected together via VE-Ports or VEX-Ports
• Edge Fabric – Normal switched fabric connected to the backbone via E-Ports or EX-Ports
• VE-Port – FCIP interconnected E-Port/ISL, i.e. fabrics will merge
• VEX-Port – FCIP interconnected EX-Port, routing needed via lsan zoning to connect initiator to a target.
• E-Port – Normal expansion connection between one switch to another.
• EX-Port – A connection that connects 2 fabrics together without merging services
• LSAN – A special zone that allows 2 fabrics interconnected via EX-Ports to communicate.

Link Generation

1. On the Backbone/Edge ensure that interopmode is disabled
   • Check to ensure you aren’t running in Mcdata Compatibility Mode

   >>interopmode

2. On the Backbone/Edge ensure that secmode is disabled
   • Check to ensure Secure-Fabrics isn’t enabled

   >>secmodeshow

3. On the Backbone ensure that msplatform DB is disabled
   • Check to ensure there is no external management to Brocade Fabrics

   >>msplatshow

4. Enable the FC Routing, by default its disabled on the backbone
   • Ensure its disabled first

   >>fosConfig –disable fcr

   • Set the Backbone Fabric ID

   >>fcrconfigure

   • Ensure you use the same backbone id for all the routers in that backbone.
   • Enable the FC routing

   >>fosconfig –enable fcr

5. Perform these steps on all Routers in the Backbone Fabric

Creating FCIP Connections

1. Step to enabling FCIP is to turn on your EX-Port from the backbone fabric (Routers) into the edge fabric (e.g. Tape SAN). On both sides of your WAN link.
   • Pick the same port for each backbone fabric i.e. port-4 on Source Router and Destination Router, to keep things easy to manage, on both sides
   • If plugging into the same switch types plug the router FC port into the same port on the other side of the EX-Port/Edge Fabric
   • By default all ports on the routers are persistently disabled, ensuring no-one can just plug in and go.
   • Here are the steps to configure the EX-Ports

     >>portcfgexport 4 -a 1 -f 30 (fcrap14c001)

     >>portcfgexport 4 –a 1 –f 31 (fcrap14c101)

     • This will configure port-4 to be an ex-port once its been enabled, with (-a 1, enabling), (-f 30, defines the fabric id)
   • On the other side of the backbone, you simply flip the EX-port fabric id’s, other wise the backbone fabric router will not merge properly

     >>portcfgexport 4 –a 1 –f 31 (fcrrp21001)

     >>portcfgexport 4 –a 1 –f 30 (fcrrp21101)

2. Next you need to enable the port
   • >>portcfgpersistentenable 4 on all routers, you have configured EX-ports on.
   • This will enable the port, and assuming its plugged into the edge fabric it will synch up, and you have a routable environment from the FC-Side
   • On the backbone you will see:

     EX-Port 10:00:00:05:1e:35:2c:9d "fcsap14c501" (fabric id = 30 )(Trunk master)

   • On the edge fabrics you will see:

     E-Port 50:00:51:e3:81:2e:4e:1e "fcr_fd_160" (downstream)(Trunk master)

   • One thing to remember the main reason for configuration of an EX-Port on the backbone, is so the fabrics will not merge themselves, thus ensuring WAN link outages won’t cause multi-fabric reconfigurations.
3. IPSEC is supported on these routers, however this configuration uses a PVC, not a internet based VPN.
   • You can configure only 1 tunnel per Ge port if IPSec is enabled, this configuration takes advantage of multiple tunnels over the same ports.
   • As well Tape Pipelining and Fast-Write aren’t supported over a secure-tunnel, and we plan to use both of these items.
4. Next you need to ensure you enable the virtual ports on both routers
   • Ge0 - 16 – 23 correspond to Physical 0-8
   • Ge1 - 24 – 31 correspond to Physical 9-15
5. Go ahead and enable the virtual ports you need based on the number of tunnels you are running across the ge-ports.

   • <code> >>portcfgpersistentenable 16 enables a tunnel for ge0

     >>portcfgpersistentenable 24 enables a tunnel for ge1

     • Ensure this is done on both sides of the WAN, and both set of routers
6. Create the IP Interfaces for the ge-ports on each router
   • LC-Router >>portcfg ipif ge0 create 10.255.2.3 255.255.255.0 2250
     • 2250 is the MTU size (2148, is FC Frame size)
   • FC-Router >>portcfg ipif ge0 create 10.255.2.4 255.255.255.0 2250
     • In a flat subnet there is no need to configure a ip route, once the tunnel is up the route will be generated for you
     • If you wish to enable tunnels on ge1 then you need to perform the same procedure on those ports on each router.
7. Next are is to create the tunnels on both sides
   • LC-Router >>portcfg fciptunnel ge0 create 0 10.255.2.4 10.255.2.3 512000 -c
     • Tunnel ID = 0
     • Dest IP = 10.255.2.4
     • Src IP = 10.255.2.3
     • comm_rate = 512000 (500 mbps) (What Networks has given us)
     • -c (compression enable)
     • -t (enables tapepiplining)
     • -f (enables fastwrite)

     **Note** HP Currently does not support the Fast Write option as it pertains to Continuous Access Journaling, if this is enabled you will not be able to create the needed pairs

   • FC-Router >>portcfg fciptunnel ge0 create 0 10.255.2.3 10.255.2.4 512000 -c
     • You will need to ensure that you create tunnels on the other pais of routers as well.
8. Once this is complete the tunnels should come online, provided you have done everything correctly

   Online VE-Port 10:00:00:05:1e:37:f2:fc "fcrrp21001" (upstream)

Creating LSAN zones for FC Routing Communications

At this point you are ready to create some LSAN zones and start moving some data

• On the Edge Fabrics
  1. Create an alias in the LC Fabric for the Target-Port on the FC Storage Platform

     aliCreate "Storage_1B_Alias","50:06:00:00:01:02:03:04"

  2. Create a zone with the remote alias and the local one

     zoneCreate "lsan_CAXP_Jrnl_Zone"," Storage_1B_Alias; XPAP14C001_1B_Alias"

  3. Add the new zone into the Defined zone schema

     cfgAdd "CFG_3"," lsan_CAXP_Jrnl_Zone"

  4. Add the new zone into the Effective zone schema

     cfgEnable "CFG_3"

• FC- Edge Fabric

  1. aliCreate "XPAP14C001_1B_Alias"," 50:06:0e:80:04:7e:69:01"

     zoneCreate "lsan_CAXP_Jrnl_Zone"," XPAP14C001_1B_Alias; XPRP21001_1B_Alias"

  2. Add the new zone into the Defined zone schema

     cfgAdd "FC_CFG4"," lsan_CAXP_Jrnl_Zone"

  3. Add the new zone into the Effective zone schema

     cfgEnable "Fc_CFG4"

  4. You will need to do the same process above for the other sides of the XP
     • XPAP14C001_2B_Alias – XPRP21001_2B_Alias (Initiator – RCU Target)
     • Since we are going both ways you will need to configure another lsan zone coming back the other way for the XPAP14C001_1D_Alias – XPRP21001_1D_Alias (RCU Target – Initiator)
  5. Must use LSAN or lsan in the beginning of any zone that will be routed.

Useful Commands to monitoring and status

   > portcmd --ping ge0 -s 10.255.2.3 -d 10.255.2.4
   Pinging 10.255.2.4 from ip interface 10.255.2.3 on 0/ge0 with 64 bytes of data
   Reply from 10.255.2.4: bytes=64 rtt=12ms ttl=64
   Reply from 10.255.2.4: bytes=64 rtt=12ms ttl=64
   Reply from 10.255.2.4: bytes=64 rtt=12ms ttl=64
   Reply from 10.255.2.4: bytes=64 rtt=12ms ttl=64
   Ping Statistics for 10.255.2.4:
   Packets: Sent = 4, Received = 4, Loss = 0 ( 0 percent loss)
   Min RTT = 12ms, Max RTT = 12ms Average = 12ms

   > portshow ipif ge0
   Port: ge0
   Interface IP Address NetMask MTU
   0 10.255.2.3 255.255.255.0 2250

   > portshow ipif ge1
   Port: ge1
   Interface IP Address NetMask MTU
   0 10.255.2.5 255.255.255.0 2250

   > portshow iproute ge0
   Port: ge0
   IP Address Mask Gateway Metric Flags
   10.255.2.0 255.255.255.0 10.255.2.3 0 Interface

   > portshow iproute ge1
   Port: ge1
   IP Address Mask Gateway Metric Flags
   10.255.2.0 255.255.255.0 10.255.2.5 0 Interface

   > portshow fciptunnel ge0 all
   Port: ge0
   Tunnel ID 0
   Remote IP Addr 10.255.2.4
   Local IP Addr 10.255.2.3
   Remote WWN Not Configured
   Local WWN 10:00:00:05:1e:38:12:e4
   Compression on
   Fastwrite on
   Tape Pipelining on
   Committed Rate 636928 Kbps (0.636928 Gbps)
   SACK on
   Min Retransmit Time 100
   Keepalive Timeout 10
   Max Retransmissions 8
   Status : Active
   Uptime 19 hours, 11 minutes, 2 seconds

   > portshow fciptunnel ge1 all
   Port: ge1
   Tunnel ID 0
   Remote IP Addr 10.255.2.6
   Local IP Addr 10.255.2.5
   Remote WWN Not Configured
   Local WWN 10:00:00:05:1e:38:12:e4
       Compression on
   Fastwrite on
   Tape Pipelining on
   Committed Rate 636928 Kbps (0.636928 Gbps)
   SACK on
   Min Retransmit Time 100
   Keepalive Timeout 80
   Max Retransmissions 9
   Status : Active
   Uptime 19 hours, 11 minutes, 13 seconds

   > portshow arp ge0
   Port: ge0
   IP Address Mac Address Flags
   10.255.2.4 00:05:1e:37:f2:e0 Resolved

   > portshow arp ge1
   Port: ge1
   IP Address Mac Address Flags
   10.255.2.6 00:05:1e:37:f2:e1 Resolved

   > portshow ge0
   Eth Mac Address: 00.05.1e.38.12.c8
   Port State: 1 Online
   Port Phys: 6 In_Sync
   Port Flags: 0x3 PRESENT ACTIVE
   Port Speed: 1G

   > portshow ge1
   Eth Mac Address: 00.05.1e.38.12.c9
   Port State: 1 Online
   Port Phys: 6 In_Sync
   Port Flags: 0x3 PRESENT ACTIVE
   Port Speed: 1G

   > portshow mode ge0
   Port: ge0
   Mode: FCIP

   > portshow mode ge1
   Port: ge1
   Mode: FCIP

   > fcrfabricshow
   FC Router WWN: 10:00:00:05:1e:38:12:e4, Dom ID: 50, Info: 10.236.190.51, "fcrap14c001"
     EX_Port FID Neighbor Switch Info (WWN, enet IP, name)
     4 30 10:00:00:05:1e:35:2c:9d 10.236.189.121 "fcsap14c301"
   FC Router WWN: 10:00:00:05:1e:37:f2:fc, Dom ID: 51, Info: 10.12.23.194, "fcrrp21001"
     EX_Port FID Neighbor Switch Info (WWN, enet IP, name)
     4 31 10:00:00:05:1e:03:11:e1 10.12.23.188 "fcsrp21001"

   > fcrresourceshow
   Daemon Limits:
   Max Allowed Currently Used
   LSAN Zones: 3000 0
   LSAN Devices: 10000 0
   Proxy Device Slots: 10000 0
   WWN Pool Size Allocated
   Phantom Node WWN: 8192 0
   Phantom Port WWN: 32768 0
   Port Limits:
   Max proxy devices: 2000
   Max NR_Ports: 1000

   > fcrrouteshow
   Destination NR_Port FCRP Cost WWN of Principal
   30 320400 1000 10:00:00:05:1e:35:2b:c7
   31 330400 1000 10:00:00:05:1e:03:11:e1