XenServer Tips – HBAs, HA, and HOSTDEVSCAN

In this installment of the ManageOps Video Series, Steve Parlee, our Director of Engineering, talks about:

  • Why we always use iSCSI HBAs in our Citrix XenServer deployments.
  • The possible risks of using HA in a two-server pool. (NOTE: Initial testing indicates that XenServer v5.6 may not present the same problems in a two-server pool as earlier versions. When we have completed our testing, we will post an update here.)
  • A useful utility for XenServer called “hostdevscan.”

Desktop Virtualization for the SMB

One of the criticisms that’s been leveled at XenDesktop by its competitors is that it is too complex – too many components that have to be configured to get everything to work. And while that’s partially true, it’s not the whole story. As we’ve discussed in previous posts, XenDesktop is extremely flexible in that it allows you to mix and match different kinds of virtual desktops in your environment to best meet the needs of various groups of users. As you bring more kinds of virtual desktops into the mix, you add more infrastructure components to manage them. More infrastructure components = more complexity but also more flexibility.

If you don’t need all that flexibility – if, for example, you just want to deploy “classic” VDI, by which I mean a bunch of virtual PCs running on the hypervisor of your choice – then you don’t need all that complexity, either.

In this video, Dan Feller of Citrix presents a reference architecture for a straightforward VDI deployment of up to 500 users. The video takes about 50 minutes to watch, but it’s worth your time. You’ll learn some interesting things.

For example, you’ll note that Dan is recommending that the XenServers in the XenServer pool that supports the virtual Windows 7 machines should have local disk drives, in a RAID 10 configuration, that will be used for the local host cache for the provisioned Windows 7 systems, for two reasons: First, it’s less expensive than using SAN storage. Second, the limiting factor for how many virtual PCs you will be able to run on a XenServer host is not processing power, and it’s not RAM – it’s IOPS. And he walks you through the calculation of how many functional IOPS the local storage on the XenServer can support, and how many virtual desktops you can therefore reasonably expect to support.

In fact, my only reservation about this video is that, like just about every other discussion I’ve seen regarding Windows 7 virtualization, it doesn’t mention the Microsoft license activation issue that’s inherent in provisioning Vista and Windows 7 desktops, the need for the Microsoft Key Management Service, and the nuances of getting KMS to work properly. But we’ve pummeled that issue elsewhere in this blog.

So, with that in mind, heeeerrrrrreeee’s Dan (P.S.: the audio doesn’t start until about 15 seconds into the video):

A Better Way to Backup Your Data

ManageOps has been building and supporting networks for a long time. And during most of that time we’ve had a real love-hate relationship with most of the backup technologies we’ve implemented and/or recommended.

Tape backups – although they are arguably the best technology for long-term archival storage – are a pain to manage. Tapes wear out. Tape drives get dirty. People just don’t do test restores as often as they should. As a result, all too often, the first time you realize that you’ve got a problem with your backups is when you have a data loss, try to restore from your backups, and find out that they’re no good.

Add to that the astronomical growth in storage capacity, meaning that all the data you need to back up often won’t fit on one tape any more. So, unless you have someone working the night shift who can swap out the tape when it gets full, you’re faced with…

  • Buying multiple tape drives, which typically means you’re going to spend more on your backup software. And if your servers are virtualized, where are you going to install those tape drives?
  • Buying a tape library (a.k.a. autoloader), which can also get expensive.
  • Changing the tape when you come in the next morning, which means that your network performance suffers because you’re trying to finish the backup job(s) while people are trying to get work done.

Then there’s the issue of getting a copy of your data out of the building. Typically, that’s done by having multiple sets of tapes, and a designated employee who takes one set home every Friday and brings the other set in. If s/he remembers. Or isn’t sick or on vacation.

Backing up to external hard drives is a reasonable alternative for some. It solves the capacity issue in most cases. But over the years, we’ve seen reliability issues with some manufacturers’ units. We’ve uncovered nagging little issues like some units that don’t automatically come back on line after a power interruption. And they’re not necessarily the best for long-term archival storage, unless you keep them powered on – or at least power them on once in a while – because hard disks that just sit for long periods of time may develop issues with the lubrication in their bearings and not want to spin back up.

But we’ve finally found an approach that we really, really like. One that, as one of our engineers said in an internal email thread, we actually enjoy managing. In fact, we like it so much we built a backup appliance around it. It’s Microsoft’s System Center Data Protection Manager (SCDPM).

In this installment of the ManageOps Video Series, our own Scott Gorcester gives you a quick overview of SCDPM 2010:



For more detail on how it works, check out the description of our MooseSentryTM backup appliance.

Urgent Security Alert – “VBMania” Email Worm

Watchguard LiveSecurity has released an urgent security alert for an email worm. It generally arrives with one of the following subject lines:

  • “Here you have”
  • “Just for you”
  • “This is the Free Dowload Sex Movies, you can find it Here”

The email contains a link to what appears to be a PDF document or WMV video, but is actually a link to a malicious Windows screen saver (.SCR) file. If you run the malicious .SCR file, it…

  • Copies itself to your Windows directory as CSRSS.EXE (the legitimate CSRSS.EXE is actually in your WindowsSystem directory), and modifies the Windows registry so it can restart after a system reboot
  • Sends itself to your email and IM contacts
  • Copies itself to mapped drives and removable USB media
  • Tries to disable popular security applications
  • Downloads and installs various other pieces of malware
  • Steals sensitive information (including passwords cached in your Web browser)

This worm does not appear to use any new techniques, and should be detected by most major antivirus vendors, so it is not cause for panic. You should, however, make sure you have the latest AV signature updates installed on your systems. Also, remind your users never to open unexpected attachments or click on unexpected Web links, even if they appear to come from friends, co-workers, or other trusted parties. The bad guys appear to be spamming this very aggressively, and it only takes one user to cause you a lot of headaches.

The Cost of a Windows 7 Migration

According to an August 26 Gartner press release, your Windows 7 migration may have a painful impact on your budget. The heart of the problem is summed up in this quote from Gartner managing vice president Charles Smulders:

Corporate IT departments typically prefer to migrate PC operating systems (OSs) via hardware attrition, which means bringing in the new OS as they replace hardware through a normal refresh cycle. Microsoft will support Windows XP for four more years. With most migrations not starting until the fourth quarter of 2010 at the earliest, and PC hardware replacement cycles typically running at four to five years, most organizations will not be able to migrate to Windows 7 through usual planned hardware refresh before support for Windows XP ends.

Because of this time crunch, Gartner says that you really have only one of three options:

  1. Accelerate your PC replacement schedule. This obviously will impact your capital budget.
  2. Upgrade some of your existing PCs. Unfortunately, not all of your PCs are likely to support Windows 7 without some upgrades. In fact, Gartner estimates that 25% of the installed base of PCs will require some kind of hardware upgrade to run Windows 7. Also, unless you’re prepared to stretch out the life of these upgraded PCs beyond your usual upgrade cycle, those users are going to end up being migrated twice, not once, during the next four years. Gartner’s estimate of the migration cost per PC, assuming a large enterprise with 10,000 PCs where all PCs are upgraded: between $1,274 and $2,069, depending on how well-managed the environment is to begin with, which, by the way, is not a heck of a lot less than their estimated migration cost if you do just replace them.
  3. Migrate some users to a “hosted virtual desktop” instead of a new PC.

If you’ve been following this blog for any length of time, you know were we stand on the “hosted virtual desktop” issue. To most people, the term “hosted virtual desktop” refers to a virtual instance of a PC OS (e.g., Windows 7) running on a virtualized infrastructure such as VMware, Hyper-V, or XenServer. However, this is only one way to deliver a virtual desktop to a user. Other ways include:

  • Delivering a shared desktop from a server using Remote Desktop Services and XenApp (we’ve been doing this for years).
  • Streaming the PC OS from a common, shared image to a physical PC across the local area network. (Note that this would still require that the hardware in the physical PC be able to support the new OS.)
  • Streaming the PC OS to a client-side hypervisor (XenClient) so the client device can be disconnected from the network and continue to operate.

We’re also of the opinion that no single one of these approaches will fit all use cases. But the nice thing about Citrix XenDesktop is that you can mix and match any and all of these use cases to the needs of your users, all under a single license model.

It still isn’t going to be inexpensive. As Gartner points out, you have to build the virtual infrastructure to deliver those desktops, which will involve both capital costs and labor costs. Anyone who tells you that VDI will save you money in immediate capital costs compared with buying new PCs is not being straight with you. But you can, according to other studies, save up to 40% in your “Total Cost of Ownership” (“TCO”).

And your other alternatives aren’t inexpensive either. So why not take advantage of this opportunity to change the way you deploy and manage PCs? Take a look at what you can do with XenDesktop today, think about how much easier and less costly your Windows 7 roll out would be if you already had XenDesktop in place, and then think about how much easier and less costly your next major PC upgrade project will be if you deploy XenDesktop now.

Windows 7 is going to impact your budget one way or another. Gartner estimates that if you just decide to accelerate your upgrade cycle, the percentage of your IT budget that you spend on PCs will need to increase somewhere between 20% and 60% in 2011 and 2012. If, as in many organizations, your PC spending accounts for 15% of your overall IT budget, that means that in 2011 and 2012 you’re going to be spending between 18% and 25% of your budget on PCs instead of 15%. And that will impact other projects.

As if that wasn’t bad enough, Gartner also predicts that the demand for “highly qualified Windows 7 migration IT personnel” will exceed supply in 2011 and 2012. Remember those discussions about supply & demand back in Economics 101? Yep, that means that IT labor costs are going to go up. In fact, Gartner predicts that the labor shortage, and higher costs, will persist into 2013 as organizations realize that they’re behind in their planned migration schedule and try to figure out what to do about it.

Mr. Smulders had a recommendation on that as well: “Begin talks with suppliers now about putting in place contracts that can deliver flexible levels of resources at a fixed rate over the migration period.”

If you want to purchase a copy of the full report from Gartner, you can order one through their Web site. Or, if you just want to take Mr. Smulders’ advice, you can reach us at (206) 774-0619, or by email at sales@manage-ops.com, or by using our handy information request form. We’re here to help.

What Is Storage Virtualization (Part 2 of 2)

This is the second of two videos addressing virtual storage and its benefits. In Part 1, we addressed thin provisioning and virtual volumes. In this video, Steve talks about multipathing, and how it contributes to a high availability storage solution:

What is Storage Virtualization? (Part 1 of 2)

This is the first of two videos addressing virtual storage and its benefits. There are a number of storage solutions out there on the market but we have chosen to focus on DataCore of the purposes of this video. DataCore is an iSCSI SAN solution and you can learn more about their products here.

In part one, we address thin provisioning and virtual volumes. Watching this video will help you understand part 2 of “What is Storage Virtualization” where we talk about how multipathing relates to virtual volumes and contributes to a highly available SAN solution.

Citrix Formally Announces XenClient and XenVault

Yesterday (August 25), Citrix formally announced XenDesktop 4 Feature Pack 2. It’s expected to be available by the end of September, and, of course, will be available at no charge to existing XenDesktop customers whose Subscription Advantage is current. The big news in this Feature Pack is the incorporation of XenClient and XenVault.

We’ve talked a lot about XenClient here, but haven’t said much about XenVault. It’s high time we did, because it’s a pretty cool piece of technology in its own right.

If you’ve used Citrix products in the past, you know that we have administrative control over whether, for example, users who are running applications on a XenApp server are able to save data back to a disk drive on their client device. With the advent of Smart Access (enabled by Access Gateway Enterprise policies), we can get even more granular: we might allow a user to save data to a client drive if they’re connecting from within the protected network, or connecting from a corporate-owned laptop, but deny that same user the ability to do so if they’re connecting from a personal device or public location like a hotel business center.

Unfortunately, once the data is on a client device, you now have a security risk. It could potentially be copied to a USB drive. The corporate laptop could be lost or stolen. (For some of the more high-profile examples, check out the “laptop losers hall of shame.”) Nevertheless, it’s often viewed as a risk we have to take so that our mobile users can be productive.

XenVault, which was first previewed at the Synergy event last May, is designed to address this risk. XenVault is a new plug-in for the Citrix Receiver. As such, its deployment and configuration are controlled through the Citrix Merchandising Server. To quickly review, Merchandising Server is the preferred tool Citrix has provided for installing and configuring client software. The first time a user authenticates to the Merchandising Server (through a simple browser interface), the Citrix Receiver will be pushed down and installed on the client device, together with whatever plug-ins and configuration details the administrator has defined for that user. Subsequently, the Citrix Receiver will check back with the Merchandising Server behind the scenes, and receive any configuration updates that may be available.

The XenVault plug-in creates a secure, encrypted (256-bit AES) storage area on the client hard disk. Typically, any application that is running remotely on a XenApp server or XenDesktop virtual PC will only be able to store data in the secure, encrypted location, if it is allowed to store data on the client drive at all. Same for an application that has been streamed via XenApp for local execution on the client (regardless of whether it was packaged with the Citrix streaming tools or with App-V). While the user will be able to use Windows Explorer to look at the secure location and see what files are there, the user will not be able to copy files from the secure location to a non-secured area of the hard disk, nor open the files with applications other than those specified by the administrator. For a deeper explanation of how this works, see Joe Nord’s blog post on the subject.

If the laptop is lost or stolen, the administrator can issue a “kill pill” that will cause the secure, encrypted area to be locked or deleted the next time the Receiver checks in with the Merchandising Server. Pretty cool.

If you can’t wait until the end of September to try it out, and you have a mycitrix login, you can download the XenVault technology preview now. And keep watching this space, because I’ve got a feeling that this will be a good subject for a future video blog.

Interview With DataCore

Recently Steve Parlee, ManageOps’s Director of Engineering, sat down with Tim Warden, DataCore’s Western Region Director of Sales. ManageOps has installed a number of DataCore solutions over the last few years and highly recommends their software to anyone looking into storage virtualization. We’ve also mentioned DataCore a number of times in our blog and newsletters. If your still not sure what DataCore does, this is a great introduction to their storage solutions. In the interview, Tim Warden explains the benefits of the DataCore software and what their solution can bring to your data center.

More on Provisioning Services and KMS

Last fall, we posted about Citrix Provisioning Services and Microsoft KMS activation. To briefly recap, here’s the issue:

  • When you convert a Windows 7 OS image to a shared image for provisioning, it breaks the Microsoft license key.
  • The way you deal with that is to use Microsoft’s Key Management Services (KMS) to auto-activate systems as they boot.
  • A KMS server must have a minimum number of systems checking in for activation before it will activate anything (5 different server systems must check in before it will begin activating servers, and an aggregate of 25 servers and/or workstations must check in before it will begin activating workstations.)
  • If your KMS server is running on Windows Server 2008 R2, both physical and virtual systems will increment the counter. If it’s running on an earlier server version, only physical systems will increment the counter.

In the comment thread of that earlier post, “Chris” stated that he was trying to use Provisioning Server to provision Windows 7 systems, but that they were not incrementing the counter on the KMS server. It turns out that he was absolutely right, and I thought this was important enough to bump the issue by writing another post rather than just going back and commenting on the older one.

It turns out that, although Provisioning Server changes the host name as systems boot, it does not change the machine ID (“CMID”). And, unfortunately, the CMID is what a KMS server looks at to determine whether a machine that’s checking in is a new one that hasn’t previously checked in. Therefore, all of your provisioned Windows 7 systems will look to the KMS server like the same system checking in over and over again, and will not continue to increment the threshold counter.

According to a blog post by Thomas Koetzing a couple of weeks ago, Citrix has told him that this will be fixed in the next release of Provisioning Services, scheduled for sometime in Q4.

Frankly, I’m pretty disappointed by this whole issue. Windows 7 has been out now for almost a year. The big push by both Citrix and Microsoft is that XenDesktop is a great way to roll out Windows 7. Provisioning Services is a must for any significant VDI deployment, because otherwise you eat up far too much of your expensive SAN storage. But yet we’re still stuck in a situation where we can’t use Provisioning Services to provision Windows 7 unless we have at least 25 physical systems checking in with our KMS server for activation. In my opinion, there is no excuse for this issue not being addressed long ago…particularly when it’s been a known issue since the release of Windows Vista.

I did find a workaround described by Kirk Kosinski in a Citrix forum post:

What I did was create a VM with VL media, sysprep and power off, convert to a template, then deploy the template 25 times and boot each VM once (a few required a reboot before contacting the KMS for whatever reason). My KMS server could then activate clients successfully, at least for a while… the activation count will decrease over time if the machine doesn’t contact the KMS server, so you will periodically need to redo this process.

The VMs don’t have to join the domain to activate so you don’t need a complicated sysprep script, just make sure to not include any license key in the script…

This strikes me as a bit of a pain, particularly when you’ve got to do it every six months or so to keep your systems alive, but it should at least work until Citrix and Microsoft get this sorted out.