Over the past few months, I've been doing a lot of traveling and I've neglected this blog. (If for some reason, anybody is interested in how I spent my summer, see the end of this post for my quick summary. It feels odd to be writing about myself, but I guess that's what people do in blogs.) Anyway, I'm back now and have a million things to write about. Microsoft has been incredibly busy! I'll start by sharing a letter from a reader, Jeff Vandervoort, who asked me to get a response from Microsoft. At the end of Jeff's letter, you'll see the response Microsoft gave me. I'll be eager to hear what you think.
Jeff's Letter
It's frustrating to my clients (and me) that Microsoft is pushing virtualization for all kinds of uses -- as long as it doesn't involve Microsoft's own products. Virtualization, particularly with Windows Server 2003 Enterprise Edition's [Server 2003 EE] licensing, potentially adds a lot of value to Microsoft products for SMBs, but Microsoft's archaic, VM-hostile support policies make it risky to make use of it. I've been told by Microsoft [Customer Support Services] CSS that even in the absence of a published support policy for running a product in a VM [virtual machine], Microsoft may not support it unless we reproduce the problem on physical hardware. When asked for specifics of what can go wrong in a VM, I get only vague answers and guesses. Does Microsoft support Virtual Server for production use, or not? Is the real reason Virtual Server is free that, had we paid for it, we'd expect its use to be supported? In this specific case, I'm charged with setting up 4 small branch offices: 2 with ~5 users, 1 with ~10 and another with ~25. The 2 smallest are project-specific and will exist for less than 2 years. Connectivity to the main office is critical, but so is economy. They have determined that Terminal Services does not meet their needs. They'll be using an RRAS VPN endpoint in a VM at a small site with Web proxy clients to Microsoft's ISA, and an ISA VPN and edge firewall in a VM at the 3 other sites. The host machine in each case will be a DC/file and print server. Using Virtual Server with Server 2003 EE on the host means we buy only one server and one Server 2003 license, which puts the project in budget. I've advised my client of Microsoft's support position in writing, and they're prepared to move forward at their risk. Their alternative is either no connectivity to the main office at all, or reducing security of the system as a whole by using SOHO firewall/VPN endpoints in lieu of ISA. Neither is acceptable to the client. Microsoft CSS has confirmed there will be no Internet connectivity to the host machine in our config. But our configurations are still either "not recommended" or "not supported." In the case of ISA, the config is "not recommended" in the release notes and the ISA BPA [Best Practices Analyzer], and "not supported" on TechNet! So, while neither answer is acceptable: Which is it? Neither CSS nor I could find any documentation about RRAS in a VM. Microsoft does not appear to have given virtualization very much thought. Unfirewalled honeypots are often run in VMs. The honeypots are attacked, but the host is unaffected, and survives to allow use of the undo disk to put the honeypot back online quickly. If VS can host honeypots safely, without compromising the host, why not ISA or RRAS? Empirically, we have tested the ISA and RRAS VM configs and they work well, but it sure would be nice to have Microsoft's blessing while going into production. Beyond ISA and RRAS, if Microsoft is going to encourage virtualization, they need to step up and support virtualizing their products, except where specific reasons can be furnished and documented that shows why they should not be.
Microsoft's Response to Jeff's Letter
We're sorry to hear your reader had a frustrating experience when deploying and maintaining solutions built on Microsoft software. For over 30 years, our design goal has always been to offer quality products, an excellent customer experience at a reasonable price. But when we don't meet these design goals, we listen to our customers and we make changes as needed. Microsoft takes virtualization seriously. We're making investments across our business, to include computing infrastructure, applications, systems management, licensing, support and interoperability so that customers can deploy critical workloads and applications in a virtual environment. One way we've helped meet customers' needs is the Common Engineering Criteria, which allows customers and partners to see the design goals for Microsoft server products as it relates to other Microsoft server software, including server virtualization. Virtual Server 2005 was added to the 2005 Common Engineering Criteria and Windows Server virtualization, which is a feature of Windows Server 2008, has been added to the 2008 Common Engineering Criteria. Exemptions are only granted due to OS or hardware dependencies. Specific to your reader, Microsoft does support Virtual Server 2005 in production environments and intends to keep on doing so with Windows Server virtualization. For instance, ISA Server 2006 is fully supported within a Virtual Server 2005 R2 guest; whereas previous versions of ISA Server were not. ISA Server 2006 can run as a virtual guest, but because of performance considerations and potential security risks due to misconfiguration, this configuration isn't recommended by Microsoft, especially in network firewall deployment scenarios. The ISA Server product team is committed to supporting virtualization in the future versions of ISA Server, and is committed to security and providing sound deployment and configuration guidance to customers. Microsoft has published two KB articles that state our support policy for software running in a virtualized environment: - Microsoft Virtual Server support policy: http://support.microsoft.com/kb/897613 - Support policy for Microsoft software running in non-Microsoft hardware virtualization software: http://support.microsoft.com/kb/897615/
Consistent with software industry practice, Microsoft doesn't provide general product support for any third-party software. However, as virtualization software matures and the industry adoption goes beyond today's 4% penetration, we recognize that new support models are needed. Customers have told us that they want a consistent support experience across their physical and virtual computing systems. Microsoft offers a progressive technical support policy covering the Microsoft virtualization software, the Windows OS and most Microsoft applications. And Microsoft is working with the industry to define such a model so that customers receive a consistent technical support experience for their computing systems, be it physical or virtual.
How I Spent My Summer
And now for something completely different: As I said, while I was spending the summer traveling for pleasure and for work, Microsoft was very busy with all sorts of new announcements that I need to write about. But since this is a blog, I feel compelled to talk about myself first. My husband (whose name is Ossi, short for Oswald) and I drove down the coast from our home near Seattle. Our first night was spent in a creepy little town called Ocean Shores, WA. The coast is gorgeous there, but the town consists of nothing but big "resort" chain hotels and looks like it was built to become a big tourist center, but nobody came. But the drive down the coast was stunningly beautiful. We only got lost once when our navigation system sent us down a dirt road to nowhere as a "shortcut." But it was a pretty drive. We spent a night in Ft. Bragg, CA (yep, it's a West Coast Ft. Bragg), and found an outstanding restaurant, Mendo Bistro, in that little town. Then we cut across California to Lake Tahoe and spent July 4, at a wonderful oasis of a B&B called the Black Bear Inn. (The only evidence of Tahoe's big forest fire that we could see were signs all over town thanking the heroic firefighters.) From Tahoe, we drove over to Utah to see the stunning Canyonlands and Arches National Parks. We decided against hiking in the 105-degree weather, though. Along the way, I bought a great handmade turquoise necklace at a scenic overlook where all sorts of trinkets were spread on the sidewalk under signs forbidding any selling. (Fortunately, the signs didn't say anything about buying.) Then we drove through the beautiful (though suffering from pine-bark beetle damage) Colorado Rockies, over I-70, past Vail, to Denver for Microsoft's Worldwide Partner Conference. (See, I told you there was also work involved.) After spending a week at the lovely Windows IT Pro headquarters in Loveland, CO, and getting some quality time with my team, Ossi and I drove home via Yellowstone National Park -- gorgeous, too many people, lots of geysers. I hadn't been to Yellowstone since a family vacation when I was a kid. All I remember from that childhood trip was the smell of sulfur, but I didn't notice that smell much at all this time. Strange how memory plays tricks on you. We were home for a week and then flew to Germany to visit Ossi's parents in Neumarkt/Opf., near Nuremburg. Between watching goofy German TV with the in-laws, we made it our mission to check out as many Biergarten as we could. I lived in Germany from 1975 to 1984 and we visit every year, but I had somehow failed to consciously think about how wonderful German cafes and beer gardens are. You can find beautiful scenery and sit as long as you like after a hike or bike ride and enjoy the beer and the people. (sigh) Ossi and I spent the entire day of our 30th wedding anniversary in Regensburg, where we had lived and attended the university back in the 70s. Regensburg is a remarkable town. It's on the northernmost part of the Danube (which is anything but blue, BTW) and is one of only two medieval cities in Germany that weren't destroyed in WWII bombings. So you can walk down the narrowest little streets you've ever seen and feel what it must have been like living in those 13th-century buildings. The town was actually founded by Markus Aurelius in 179 AD, and you can still see Roman ruins downtown. My favorite juxtaposition is that you can stand inside McDonalds on the main street (Maximillianstrasse) and look out a big picture window to see remains of the Roman wall. Anyway... there was lots more, but I this is long enough to bore anybody. Suffice it to say that I'm back now and will be posting regularly -- about Microsoft, not me. It's hard to think where to even begin with all the Microsoft stuff that's going on....
End of Article
Embedded in all the marketing-speak in Microsoft's response is this key sentence:
“ISA Server 2006 can run as a virtual guest, but because of performance considerations and potential security risks DUE TO MISCONFIGURATION, this configuration isn't recommended by Microsoft, especially in network firewall deployment scenarios.” (EMPHASIS mine.)
This is as vague as the CSS responses I’ve gotten.
There is apparently nothing *inherently* wrong with this design from a security standpoint. It has only to be *configured* properly. Well, that goes for nearly everything we do in IT, doesn’t it?
Let's see....I could create an Access Rule in ISA that allows all protocols, both directions,for all users, from all networks & Local Host, to all networks and Local Host. If I do, I have, in one step, negated the value of ISA. Would that not also be a "potential security risk due to misconfiguration"?
If that's their criterion, Microsoft should "not recommend" using ISA! But Microsoft gives me documentation about creating Access Rules that is neither vague nor arbitrary, and that persuades me not to create such an Access Rule. And as long as I abide by that documentation, they support my use of ISA.
As for performance, the client is satisfied with performance of this design in our tests, so that’s a non-issue to us.
Microsoft posts many gigabytes of configuration documentation. Why balk at documenting this one?
Is there a real vulnerability or not? If there is, document it and I'll move on. If not, why aren’t they “fully supporting” this design, provided that I use their recommended configuration?
JRV September 25, 2007 (Article Rating: )
JRV, I totally agree with you. I was just speaking with my boss the other day about virtualization vs. using blade servers and he asked if Microsoft supports virtualization of their software in a production environment. Needless to say, I didn't have an answer for him yet - this response however cements to me that virtualization of any of microsoft's products in a production environment is not supported - therefore I cannot reccommend its use internally in production or to our customer base (over a thousand clients).
Man, what a bummer. :(
CFConner September 26, 2007 (Article Rating: )
On my home network I've been using an almost identical setup that you have describe. A host server which is a DC / File / Print / WSUS server that is also a Virtual host server running ISA 2006, exchange front-end in a virtual DMZ and a back-end exchange 2003. It works great after I ironed the kinks out of it and often times i have it running for over a month without any issues. Lately the only time I ever need to reboot any of the servers is for windows updates. That being said would I run a similar setup in a critical business environment? I would have to say no. There are just too many points of failures and too many things going on on a single box for me comfortable running any sort of real business on it. At a minimum I would keep the domain controller on separate physical hardware from the Virtual host server. Infact that's an improvement I'm planning to make on my network at home also. I would also definetly make sure that your virtual guests are running on separate physical disks than what the host server uses for its OS and data. It's great to see people pushing the potential of the virtual server environment.
collide.six September 26, 2007 (Article Rating: )
One routinely runs much more than collide.six does on a fully-utilized SBS 2003 Premium server. ISA is then running on the same copy of Windows as the DC/File/Print/Exchange/SQL/WSUS/IIS/SharePoint/Whatever server. The protection afforded by banishing ISA to a VM is not an option, there.
While MS's claims that the SBS component products have been tweaked to make them secure, I don't think the argument can be made that SBS with ISA running on the same box with ANY number of tweaks could be more secure than ISA in its own, dedicated, guest VM. Yet MS supports SBS Premium for systems up to 75 users.
Put in perspective, 4 of this company's 5 sites COMBINED are well under SBS' limit of 75 users. In a sense, this is not only an example of Microsoft failing to support virtualization, but also an example of how Microsoft doesn't "get" mid-sized businesses. I'm hardly the first to observe there's a big gap between SBS on one extreme and dedicated, stanadlone servers running Microsoft [Insert Product Name Here] Enterprise Edition on 100s of clustered servers in a gigantic data center at the other extreme.
I wouldn't do this for a 10,000 seat system, just like I wouldn't use SBS for a 10,000 user company (even if it was possible!). And I doubt I'd be asked to. And I am not doing it for this SMB without advising them of MS's support posture, and the as-yet still-unspecified "potential security risks".
I'd love to do this on physical hardware...it would require no disclaimers, no research--and would increase my billable hours as their consultant. But it's out of the question for them, economically. That question has been asked and answered. Even physically out of the question; their smallest site has a tiny server closet that won't house 2 servers.
I also fully expect this discussion to seem very quaint 5 years from now, when virtualization is the rule rather than the exception.
JRV September 26, 2007 (Article Rating: )
I want to clarify the situation as the MS response above lacked details. MS does support ISA 2006 when running inside VM in production, with the sole limitation that it can’t protect the VM host. CSS personnel know that ISA 2006 is supported as a VM guest in production as long as ISA is not deployed as the edge protection service.
For scenarios where ISA 2006 is used as a proxy server or/and as a remote access server will be supported. ISA 2006 is supported running in production on a virtual machine as long as it is not used for layer 3 security (given the host OS will not be protected). It also supports evaluation of layer 3 protection scenarios on virtual machines as long as they do not go into production (internet).
porourke38 September 26, 2007 (Article Rating: )
porourke38, thanks for your post.
MS CSS has already acknowledged that if the only thing bound to the host's external NIC is the VS Server Service (and, specifically, TCP/IP is NOT bound to it), then there is NO Internet connectivity to the VM host. That's the best Internet protection you can get--isn't it?
That's the question I've been trying to get answered, and unanswered it remains: Is there an actual attack vector? If so, what is it? No one seems to know. I sure don't.
If there isn't one, why is the config "not recommended"?
JRV September 27, 2007 (Article Rating: )
in MS VS2k5 site there is a list of supported products I checked about march 2007 - it said ISA 2K6 WAS supported. I didn't understand Jeff lacking of information (but I recommend MS site search, not talking with MS CSS :-)
Free Online Event! Virtualization:Get the Facts! Register now and attend this free, live in-depth online conference on November 13 and 20, 2008, produced by Windows IT Pro. All registrants are eligible to receive a complimentary one-year digital subscription to Windows IT Pro (a $49.95 value)!
Ease Your Scripting Pains with the Flexibility of PowerShell! Join MVP Paul Robichaux on December 11, 2008 at 11:00 AM EDT as he equips you with PowerShell basics in 3 introductory lessons, each followed by a live Q&A session—all on your own computer!
Latest Advancements in SSL Technology There are a variety of different kinds of SSL to explore to ensure customer data is kept confidential and secure. In this paper, we will discuss some of these SSL advances to help you decide which would be best for your organization.
Order Your SQL Fundamentals CD Today! Learn how to use SQL Server, understand Office integration techniques and dive into the essentials of SQL Express and Visual Basic with this free SQL Fundamentals CD.
Maximize Your SharePoint Investment: Get Your Data Moving Watch this web seminar now to learn how to maximize your SharePoint investment! Join us as we take a look at the complex business of securing, accessing and managing vast amounts of information in a global network and various ways to get your data moving.
List Your Products in Our Technology Resource Directory Don't miss the chance to post your free listing in this comprehensive directory for IT and developer professionals, powered by Windows IT Pro. But hurry! Deadline ends Oct. 9.
“ISA Server 2006 can run as a virtual guest, but because of performance considerations and potential security risks DUE TO MISCONFIGURATION, this configuration isn't recommended by Microsoft, especially in network firewall deployment scenarios.” (EMPHASIS mine.)
This is as vague as the CSS responses I’ve gotten.
There is apparently nothing *inherently* wrong with this design from a security standpoint. It has only to be *configured* properly. Well, that goes for nearly everything we do in IT, doesn’t it?
Let's see....I could create an Access Rule in ISA that allows all protocols, both directions,for all users, from all networks & Local Host, to all networks and Local Host. If I do, I have, in one step, negated the value of ISA. Would that not also be a "potential security risk due to misconfiguration"?
If that's their criterion, Microsoft should "not recommend" using ISA! But Microsoft gives me documentation about creating Access Rules that is neither vague nor arbitrary, and that persuades me not to create such an Access Rule. And as long as I abide by that documentation, they support my use of ISA.
As for performance, the client is satisfied with performance of this design in our tests, so that’s a non-issue to us.
Microsoft posts many gigabytes of configuration documentation. Why balk at documenting this one?
Is there a real vulnerability or not? If there is, document it and I'll move on. If not, why aren’t they “fully supporting” this design, provided that I use their recommended configuration?
JRV September 25, 2007 (Article Rating: