Vic/SAN
From Summerseas
Contents |
Overview
Storage Area Networks, SAN's, are becoming more and more prevelant in both large and small IT shops today. There are a number of reasons but probably the primary reason is that a SAN can greatly simplify the management of constantly growing storage requirements faced by many organizations today. You can imagine that managing 50 TB of storage all in one place is much simpler than managing the same amount of storage in the form of 100 500 GB arrays attached to a myriad of computers from different vendors running different OSes.
A typical modern SAN consists of 2 or more fibre channel switches and a high performance, high bandwidth array. The array will usually consist of 2 array controllers attached to dual ported fibre channel disks on several fibre channel loops. This provides the controllers with multiple paths to the disks for redundancy. Usually the disks and loops that they are attached to are referred to as the back-end connections. The controllers also provide multiple host connections. The connections are usually connected to fibre channel switches and are usually referred to as front-end connections.
Hosts are connected to the SAN via host bus adapters, HBAs. The HBA looks like a scsi controller to the OS and accepts scsi commands just like a scsi controller. The HBA along with the host's FCP software layer will then encapsulate the scsi commands and data into fibre channel packets and transport them via fibre channel cable. The HBA's in the hosts are attached to the fibre channel switches which are in turn attached to the array controllers. Typically the hosts are dual attached to the switches via 2 HBAs to provide redundant paths to the storage.
Modern fibre channel switches form what is referred to as a fabric. A fabric is made up of 1 or more switches and functions similar to multiple IP switches connected together in that devices on one switch can be configured to see devices on another switch. Typical SAN design is to have dual everything; dual fabrics each connected to both array controllers and dual HBAs in the hosts each connected to different fabrics. This way the host storage is very fault tolerant in that storage operations could continue through HBA, switch, cable, disk and array controller failures.
Most SAN array controllers provide the ability to create logical devices from the physical disks and present the logical disks to the hosts. When creating the logical devices it is possible to enhance performance and fault tolerance by using various combinations of striping with or without parity and mirroring. Usually in modern arrays RAID-5 is used. In the past RAID-5 was not considered appropriate for write intensive applications because every write required a parity calculation. Modern arrays have addressed this potential performance hit by putting several GB of ram cache in front of the disks and developing specialized hardware for parity calculations. The cache allows the write operation to complete immediately meanwhile the controller is calculating parity and flushing cache to disk.
Resources
- Brocade Switches
- McDATA Fibre Channel Switches
- >HP Enterprise Storage
- EMC
- Hitachi Data Systems
- IBM Storage
- LSI Logic
- StorageTek
Multipathing Notes
One of the benefits of modern SAN design is the ability to create very reliable and fault tolerant storage. It is now possible to have storage with no single point of failure. In the past this was not really possible. One of the things that makes SAN storage so fault tolerant is the possibility of having multiple paths from the host to the storage. This is possible because modern arrays allow access to storage on multiple ports and SAN design usually includes multiple independent fabrics each providing a path from the host to the storage. Even the individual components are fault tolerant. The fibre channel switches have redundant power supplies. The arrays and servers have redundant power supplies. The hosts have multiple HBAs and the storage itself is protected by mirroring or raid-5 with hot standby spares. All this adds up to very reliable storage.
Veritas DMP
On the host side additional functionality is required to manage the multiple paths to the storage for load balancing and failover. Veritas can provide some of this capability and most of the array vendors provide some sort of value added software to provide load balancing and failover functionality.
If you are going to rely on Veritas for DMP you will likely need an Array Support Library, ASL. Beginning with Veritas Volume Manager v3.2 it has been possible for array vendors to provide DMP through VxVM by providing the array user with an ASL. In Solaris, simply use pkgadd to add the package then run "vxdctl enable". After vxdctl completes you can verify that veritas is managing the paths by running "vxdmpadm getsubpaths ctrl=c2" if controller c2 were one of the controllers you're interested in.
Before you installed the ASL you probably noticed that in the output of "vxdisk list" you could see all the paths to the disks. After installing the ASL vxdisk list will show only one path. To learn more about other paths just run "vxdisk list cxtxdx" on the disk you're interested in. It will provide additional information, including the paths to the disk and whether they are enabled and which paths are active and passive.
Midrange Arrays
Midrange arrays usually lack active/active capability so load balancing across controllers doesn't work. In other words, if you have a LUN on both array controllers with controller 0 being primary and your host has 2 HBAs, one connected to each controller then the HBA connected to array controller 0 will be the active path and the other will be the passive path. The passive path can be used for failover only, not load balancing.
One way to effect load balancing with a midrange array is to simply divide the paimary assignment of LUNs accross the 2 controllers, taking usage into account.
Secure Path
Secure Path is a load balancing, failover software package that HP optionally supplies with their array. If you're not a Veritas customer you may want to consider Secure Path if you purchase an HP array. Secure Path manages the paths via device manipulation at a point before the OS creates disk devices for the user. Because of this, the OS and applications see only one path. On a system running Secure Path, if you did a "ls -l /dev/dsk/cxtxdxs2" where the cxtxdxs2 disk was a HP array LUN you would see that the device path was an swsp device. That is a storage works secure path.
A convenient utility comes with secure path called spmgr. This utility will allow you to configure notifications, logging and the addition and deletion of luns. You can also use spmgr to get status of the various paths and to configure an appropriate load balancing policy.
ALUA, Asymmetric logical unit access
ALUA is a relatively new multipathing technology for asymmetrical arrays. If the array is ALUA compliant and the host multipathing layer is ALUA aware then virtually no additional configuration is required for proper path management by the host. An Asymmetrical array is one which provides different levels of access per port. For example on a typical Asymmetrical array with 2 controllers it may be that a particular LUN's paths to controller-0 port-0 are active and optimized while that LUN's paths to controller-1 port-0 are active non-optimized. The multipathing layer should then use paths to controller-0 port-0 as the primary paths and paths to controller-1 port-0 as the secondary (failover) paths. Sun Solaris 10 MPxIO is an example of an ALUA aware multipathing solution. A Netapp FAS3020 with Data ONTAP 7.2.x is an exaample of an ALUA compliant array.
SAN Zoning
In a simple SAN configuration where only 1 or 2 hosts are connected zoning is not terribly important however in more complex SANs where many hosts of different types and OSes along with tape libraries are connected zoning becomes crucial. Essentially the goal of zoning is to isolate the connected hosts from each other for security purposes. For example say you have both unix and windows hosts connected to the SAN along with a fibre channel tape library. You may want your Unix Netbackup or Legato server to control the tape devices but you discover that when the windows box rebooted following a blue screen that it saw the tape drives and automatically loaded drivers for them and configured windows to use them. Next the clueless windows admin notices this and starts doing windows backups on top of your catelog tapes. If you're the SAN administrator you'll probably start reading up on zoning between job interviews.
Zone Types
There are 3 types of zones:
- Port Zones - These are zones where physical ports on the switch are placed into a zone.
- WWN Zones - These are zones where World Wide Numbers are zoned together. A typical example would be the worldwide numbers of an array controller port being zoned with whe world wide number of an HBA in a host.
- Mixed Zones - These zones contain at least 1 port number and 1 WWN.
Best Practices
For best security the zones should be hardware enforced and not software enforced. This is often referred to as "Hard Zoning" and "Soft Zoning". The distinction is that with software enforced zoning security is based on the assumption that hosts and devices logging into the switch fabric will only rely on the fabric name server for finding devices on the SAN. This is possibly a weak assumption. With hardware enforced zoning a host that is not zoned to see another device on the SAN will be prevented at a hardware level from seeing the device.
For newer Brocade silkworm switches, zones containing only port numbers and zones containing only WWNs are hardware enforced. Mixed zones are software enforced.
Zones should contain only 1 initiator; multiple targets in the zone is ok but the zone should have only 1 initiator. This is because the switch will send a RSCN to all initiators in the zone whenever certain changes occur for other devices in the same zone. A RSCN is a Registered State Change Notification and would be sent to all initiators in a zone if any device in the zone logs out of the fabric for example. In the past receiving an RSCN was in some cases disruptive for an initiator and in any case generates event logging on the affected host.
Aliases
Often zones are created from alias names of WWNs or port numbers for readability. For example, say you want your backup server whose HBA's WWN is 20:00:00:00:c9:67:23:11 to see an LTO-2 fibre channel drive connect to the SAN whose WWN is 50:01:04:f0:00:6f:7c:c1. First create an alias named backup1_hba0 then add 20:00:00:00:c9:67:23:11 to the alias. Next create an alias named lto2_Drive1 and add 50:01:04:f0:00:6f:7c:c1 to the alias. Finally, create a zone named backup1_hba0_lto2_Drive1 and add the aliases backup1_hba0 and lto2_Drive1 as zone members. This kind of naming convention will make things easier for admins left to support the SAN.
