Question: What’s worse than spending six or seven figures on an expensive SAN that doesn’t meet your needs or perhaps you didn’t even need in the first place?
Answer: Being the person in your company blamed for making that six or seven figure recommendation.
I’ve been doing database work for 19 years and have been a performance tuning jock for most of that time. I’ve been tuning databases since before RAID was commercially viable, never mind SANs. IO has always been the most common hardware-related bottleneck. In general, people have typically had more memory and CPU than they need in a server and are trying to get by with undersized IO components. This disparity has become even more acute in recent times. Moore’s law applies to silicon, not spinning plates.
If sizing a SAN or DAS system was easy or cheap, then most database servers wouldn’t have undersized IO components. Reading this week’s commentary won’t be the silver bullet to slay your IO monsters. Instead, I want to throw a few ideas out to you for further consideration and research on your part. Considering these ideas will help you more effectively address IO needs within your data ecosystem.
First, you don’t always need more IO bandwidth, even if you think that’s the problem. What if all your IO counters are in the red? Is that bad? Maybe. What if they’re in the red because you’re consistently doing a table scan on a billion-row table because you’re missing an index? What if adding an index turns your scan to a seek and your redlined IO counters drop to green flat lines? Do you need to spend money on more IO? Probably not. Spending more money on IO probably doesn’t solve that problem.
Second, who should you trust to recommend and spec out your SAN? Most people have their SAN’s configured and optimized by their SAN vendor. Newsflash: The people selling you six and seven figure hardware make a lot of money on commission. I’m not saying they sell you things you don’t need, but they make more money selling you more expensive hardware.
Third, who do you have provision your SAN and layout what the disks will look like? Typically, it’s the vendor. I’m on a bit more politically riskier ground here. I don’t want to suggest that typical SAN vendor system engineers don’t know their products and technology. Typically, they’re highly trained and skilled professionals. Alas, they often don’t have a database background. Letting your SAN vendor provision what your SAN looks like will often lead to a layout that isn’t ideal for the database workload you’re running.
Fourth, let’s assume you really do need more IO capacity. Is a SAN always the right answer? Nope. SANs have incredibly useful management features making them super flexible, but all things considered, you can almost always get substantially better IO throughput on a DAS solution and spend a heck of a lot less money. By the way, I’ve been consulting for 19 years and have learned to avoid the use of “never” and “always.” When I say “almost always” above, what I really mean to say is SANs will never, ever, ever in a gazillion years out perform properly spec’d DAS solutions. But I’m not saying that directly because I almost always like to avoid using the words “never” and “always” when making technical recommendations.