Client added an HBA to a TSM server, but this causes problems that needed to be anticipated with Windows servers.  TSM Device names for tapes and the library itself will change as Windows does not keep names consistent as hardware changes.   The TSM server thus lost communication with the tape library and tape drives, as the device names changed to the Operating System.

This was diagnosed by remote tech support as a faulty Library controller board, and IBM was dispatched to change it.

In all the up/down with the situation somehow now TSM and DB2 is not coming up correctly.

So your plan of attack should be:

1) Get TSM and DB2 resolved.  This does NOT include library/tape communication.  This comes later, after TSM will function.

2) Fix the tape paths.

Here is a document that goes over the steps in good detail:

http://www-01.ibm.com/support/docview.wss?uid=swg21190124

Title
Potential DS8100/DS8300/DS8700/DS8800 loss of access issue when running VMWare 4.1 or higher with vStorage API for Array Integration ( VAAI ) enabled

Abstract
The extended features of VAAI are not yet supported by DS8000, but are enabled by default on ESX 4.1 or higher.  

Mitigation
To avoid any issues, use the following to disable them.  All OTHER non VAAI features of VMWare ESX 4.1 or higher are certified with DS8000 (all models) and function normally.

Here’s a KB for disabling: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1033665 
Here’s a description on VAAI: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1021976 

VAAI uses these fundamental operations:

* Atomic Test & Set (ATS), which is used during creation of files on the VMFS volume
* Clone Blocks/Full Copy/XCOPY, which is used to copy data
* Zero Blocks/Write Same, which is used to zero-out disk regions
* Thin Provisioning in ESXi 5.x and later hosts, which allows the ESXi host to tell the array when the space previously occupied by a virtual machine (whether it be deleted or migrated to another datastore) can be reclaimed.

Due to a known Linux kernel issue, Storwize V7000, Storwize V7000 Unified block node canisters and SAN Volume Controller nodes will reboot after running for 208 continuous days since their last power on or software upgrade.

A widely documented Linux kernel issue will result in a kernel panic occurring after 208 days of continuous uptime, due to an internal counter overflow. For Storwize V7000, Storwize V7000 Unified block or SAN Volume Controller nodes or node canisters running V6.2.0.x, V6.3.0.x or V1.3.0.x, this will trigger a self-recovering reboot event once this amount of time has elapsed.

Although each reboot event is only expected to last between five to ten minutes, for clusters in which both nodes in an I/O group were last booted in close proximity to one another there is a risk that a loss of host access may occur after 208 days. A typical scenario would be if both nodes in a new installation were powered on simultaneously.

The Software Upgrade Test Utility can be used to determine the uptime of all nodes in a cluster which is exposed to this issue. The following URL contains download and usage instructions for this utility:

http://www-01.ibm.com/support/docview.wss?uid=ssg1S4000585

This issue was first introduced in the V6.2.0.1 release, which shipped on 10 June 2011. For convenience, the following table lists the published date for all code levels that are exposed to this issue, and the date at which 208 days will have elapsed.

Read the full IBM alert

IBM System x3650 M4 sets new record for 2-processor server performance on TPC-E

March 6, 2012 … IBM has published a benchmark result that sets a new record for 2-processor

performance on the TPC-E benchmark, which is designed to enable clients to more objectively

measure and compare the performance and price of OLTP systems.

The IBM System x3650 M4 server achieved 1,863.23 tpsE (transactions per second E) at $207.85

USD / tpsE. (1) This result is faster than all the other currently published TPC-E results for 2-

processor servers, and represents a significant performance benefit compared to systems using

previous-generation processors. For example, the x3650 M4’s result is more than 45% faster than the

HP ProLiant DL380 G7 server’s result. (2)

The x3650 M4 achieved this result using Microsoft® SQL Server 2012 Enterprise Edition and

Microsoft Windows® Server 2008 R2 Enterprise Edition SP1. The x3650 M4 was configured with two

Intel® Xeon® E5-2690 processors at 2.9GHz with 20MB L3 cache per processor (2 processors/16

cores/32 threads), 512GB of memory (3) and solid state drive (SSD) storage, which can enable faster

database access.

The IBM System x3650 M4 is the IBM flagship, 2-socket, 2U rack server optimized for performance

and uptime for business-critical applications and cloud deployments. The flexible x3650 M4 features

the following:

• Intel Xeon processors with up to 8 cores and up to 768GB of memory capacity for

virtualization and other memory-intensive solutions.

• Optional slotless 10GbE NIC with IBM Virtual Fabric that delivers high performance for

bandwidth-heavy workloads.

• Available with the IBM exclusive high-performance solid state drive technology.

Results referenced are current as of March 6, 2012. To view all TPC results, visit www.tpc.org.

See the details for this result: http://www.tpc.org/tpce/results/tpce_last_ten_results.asp

(1) The total solution availability for this TPC-E benchmark result is May 31, 2012.

(2) HP ProLiant DL380 G7 Server with two Intel Xeon X5690 processors at 3.46GHz (2

processors/12 cores/24 threads), Microsoft SQL Server 2008 R2 Enterprise Edition, Microsoft

Windows Server 2008 R2 Enterprise Edition; 1,284.14 tpsE at $250.00 USD / tpsE, total solution

availability of May 4, 2011.

(3) The IBM memory option used in the benchmark is Samsung’s Green DDR3 32GB LRDIMM.

IBM and System x are registered trademarks of IBM Corporation.

Intel and Xeon are trademarks or registered trademarks of Intel Corporation.

Microsoft and Windows are registered trademarks of Microsoft Corporation in the United States

and/or other countries.

TPC, TPC Benchmark, TPC-E and tpsE are trademarks of the Transaction Processing Performance

Council.

All other company/product names and service marks may be trademarks or registered trademarks of

their respective companies.

Level Release Date

740_077 03/08/2012

730_066 12/08/2011

720_108 01/23/2012

710_119 12/07/2011

Regarding the newest firmware level, which was released for Power7 systems at 740_xxx, it is known as 740_077. It is a minor tweak to the prior-released 740_075, which was released two weeks ago.

Some important fixes are included for those coming from 740_045 or earlier.

These include:

A problem was fixed that caused multiple service processor dumps to be unnecessarily taken during a concurrent firmware update. SRC B181EF9A, which indicates that the dump space on the service processor is full, was logged as a result.

A problem was fixed that caused SRCs B181843C and B181EF88 to be logged erroneously, and a service processor dump to be generated unnecessarily.

The firmware was enhanced to increase the threshold for recoverable SRC B113E504 so that the processor core reporting the SRC is not guarded out. This prevents unnecessary performance loss and the unnecessary replacement of processor modules.

A problem was fixed that caused SRCs B7006790 and B7006A21 to be erroneously logged.

A problem was fixed that caused SRCs 11001512 and 11001522 to be erroneously logged during a field replaceable unit (FRU) replacement or service processor reset.

A problem was fixed that caused SRC B18138B4 to be erroneously logged when the system is rebooted.

The firmware was enhanced to provide a more complete list of field replaceable units (FRUs) for SRCs B1xxC803, B1xxC804, and B1xxC829.

A problem was fixed that prevented a node from being deconfigured manually using the Advanced System Management Interface (ASMI).

A problem was fixed that caused the system to fail to boot with SRC B1xxB507.

A problem was fixed the caused system fans to be erroneously called out as failing.

A problem was fixed that caused SRC B7000602 to be erroneously logged at power on.

I realize very few of us actually enjoy updating firmware, but please take the time to review your firmware levels, and consider the fixes and enhancements contained with each progressive step.

As always, if you have any questions, or would like to engage us for assistance, please contact us.

I understand (please correct me if I’m wrong) that the DCS3700 has a max cap of 20 SSD drives per storage subsystem. Is this same limitation true for the DS3524 as they both use the same controllers?

If you can verify that the 192 drive value in the charts are also applicable to the SSD performance #’s that would be great. Again, I’m not sure if Snowmass supports 192 SSDs at once?

Answer to our customer

There is a limit of 20 SSDs in the storage subsystem (this limit is the same for DS5000, DS3500, and DCS3700). I just called a colleague in engineering to verify if the limit was lifted yet or not. The answer was “not”.

I believe the thought is that 20 SSDs is more than enough to max out the controller, so going to a larger SSD count would not be advantageous. Given the current pricing of the SSDs, the priority to get the drive count very high isn’t there. The relative cost of additional controllers is small compared to the drives, and the additional drives don’t provide value except for “capacity”.

If you want to have a rack of SSDs, then put in a rack of controllers also, and you’ll get a much larger aggregate performance number. The only thing that doesn’t make sense is why 20? So, with the next firmware release (mid-year), I am told the number will increase to 24 (for a fully loaded DS3524).

Per the previous logic, I am told that 60 would be a waste, so even with the DCS3700, the number is only going up to 24.

That’s the bad news.

The good news is that a rack of 20 DS3524s, each with 20 SSDs should yield about 20 X 70K IOPS reads – that’s 1,400,000 IOPS! And 20 X 18K writes IOPS – or 360,000 IOPS writes – (depending on IO size, and assuming you can fit 20 systems in a rack).

In setting up a new VMware installation with our DS3524 with 10Gbps iSCSI HICs, we configured MRU and so no real issues when connected to 1Gb ports. When we moved to 10GbE ports we started to see SCSI disconnects whenever we placed a high IO load such as a storage VMotion or VM clone. We changed the path policy to round robin (RR) and all the errors and disconnects stopped. We have only done 4 isolated servers so far but the results have been good. My problem is the VMware HCL lists MRU as the path policy for the DS3500, and our customer may validate that we’re configuring his systems “correctly”.

There is a note on the HCL that talks about RR as the path selection policy, but it’s more of a generic note that says “contact the storage array manufacturer for recommendations and instructions”. This note appears on almost every storage array, so I don’t believe it carries much weight.

http://www.vmware.com/resources/compatibility/detail.php?deviceCategory=san&productid=14700&releaseid=148&deviceCategory=san&partner=43&releases=148&arrayTypes=1&isSVA=1&page=1&display_interval=10&sortColumn=Partner&sortOrder=Asc

Answer to our customer

Please note that currently IBM supports MRU, and I’m not sure when that will be modified (further explained below). Because of that, this Tip may be “stepping slightly out of bounds” a little with my response.

I’m currently on a crusade to get the VMware HCL updated to reflect both MRU and RR as valid path policies. I spoke to some NetApp/Engenio engineers, and the introduction of higher latencies with storage connected over IP seemed reasonable, especially if all traffic is going down a single path as would be the case with MRU. They also said that their storage (NOT A STATEMENT ON IBM STORAGE, just splitting hairs) works with either MRU or RR as path selection policies (with newer versions of VMware).

I believe the issue is that VMware has, over the years, improved their failover driver to be much more sophisticated. In the old days, it might not have recognized the difference between paths to the controller that owned the lun, and the one that didn’t. That all seems to have been cleared up in the 4.x timeframe.

As always, I hate putting myself in front of a speeding bus, and would not ask anyone else to do so either. So, for supporting documentation (and the reasoning behind my crusade), note the following:

From the HCL at the URL above, the note alludes to the improvements made to ESX 4.0 and later.

“Attention: Storage partners using ESX 4.0 or later may recommend VMW_PSP_RR for path failover policy for certain storage array models. If desired, contact the storage array manufacturer for recommendation and instruction to set VMW_PSP_RR appropriately.”

From a Vmware Knowledge Base article describing multipath policies in ESX/ESXi 4.x and ESXi 5.x, that seems to also support the point that it’s “safe” (listed as a Note): http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1011340

“Switching to Round Robin from MRU or Fixed is safe and supported for all arrays, Please check with your vendor for supported Multipathing policies for your storage array. Switching to a unsupported pathing policy can cause an outage.”

From the 4.1 iSCSI SAN Configuration Guide (Vmware http://www.vmware.com/pdf/vsphere4/r41/vsp_41_iscsi_san_cfg.pdf)

“If your array does not support the ALUA protocol(the DS products currently do not support ALUA) and you want your host to do automatic load balancing, configure your devices to use the Round Robin PSP.”

From the ESXi 5.0/vCenter Server 5.0 Storage Guide (http://pubs.vmware.com/vsphere-50/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenter-server-50-storage-guide.pdf) in describing storage path policies:

VMW_PSP_RR The host uses an automatic path selection algorithm rotating through all active paths when connecting to active-passive arrays, or through all available paths when connecting to active-active arrays. RR is the default for a number of arrays and can be used with both active-active and active-passive arrays to implement load balancing across paths for different LUNs.

Another blog supporting the argument: http://www.boche.net/blog/index.php/2011/09/28/changing-the-default-vsphere-5-0-psp-to-round-robin/

Of course none of this is as nice as having the HCL updated, so we’ll keep working on that.

The LSI SAS switches are very economical and may be daisy chained to connect up to 1,000 end devices with up to 25 meters between SAS switches. Coupled with the IBM DS3500 line of disk storage which all have (2) SAS connectors for each controller in the base product and can be double with a daughter card. We see this as a perfect solution for small to large ESX clusters for either departmental needs or development.

A small ESX server cluster with (2) IBM 3650 M3s with memory up to 288GB and a DS3512 with up to 24TB of shared storage allows for this configuration to be a robust high available versatile solution. Add to this a LTO 5 SAS tape drive or site to site replication built into the DS product line and you can have local or remote backup.

Next time you come by our innovation center please ask to see a demo. I think you will be surprised.

• Eliminates the costs of SAS-based “storage islands” in direct-attached storage (DAS) environments and enables independent growth for servers and storage in traditional SANs
• Allows for multiple servers to connect to one or more independent external storage systems enabling more efficient “scaling out” of both storage and servers in data centers and other large storage installations

inspection of your ServeRAID battery. We have seen a number of installations where the ServeRAID

adapter has failed due to either a depleted or defective battery. This has resulted in unscheduled outages

and extreme high levels of stress.

IBM Retain Tip H001648 states: “ Users should install and monitor battery conditions using the IBM

ServeRAID Manager application for the following IBM ServeRAID controller models: 4M, 4H, 4Mx, 5i, 6i,

6i+, 6M, 7k, 8i, 8k, and 8s. All have cache backup batteries of which the battery status can be checked on

the controller properties Status tab.

Users should install and monitor battery conditions using the IBM MegaRAID Storage Manager (MSM)

application for the following controller models: MegaRAID 8480, IBM ServeRAID MR10i, MR10is,

MR10ie, MR10M, MR10k, and M5015. If installed, the cache backup battery status can be checked on

the battery backup unit (BBU) properties tab.”

We emphasize the need to visually inspect the battery. We have seen a number of instances where the

batteries had swelled prior to failing. IBM Retain Tip H191476 addresses this and provides guidance on

replacing the defective part.