Do you run test environments on your production hardware? If you're like many IT professionals, you're offended that I even have the nerve to ask such a question. As Michael Moore implies in Understand Your Workload, we've accepted segregation of production from non-production hardware as a central tenet of data center management. However, it's not obvious to me that such segregation is a valid best practice for virtual servers. The main justifications for segregating production on its own hardware involve security, availability, and performance. Let's examine these justifications more closely.

Security

We don't want security vulnerabilities in development code to expose our production deployments. Virtualization vendors have gone to a great deal of trouble to isolate virtual machines from each other. In fact the Department of Defense considers two IBM Logical Partitions (LPARs) to be just as isolated as seperate physical machines. We also grant production server access to a more select group of people. For example, developers may be able to log onto development servers, but should not be able to log onto production servers. Again, hypervisor and OS virtualization solutions provide environments that act like completely seperate systems.

Availability

The fact that hypervisors isolate virtual servers from a security point of view also addresses half of the availability exposure. We can be pretty certain that unauthorized personnel won't accidently (or maliciously) shut down, misconfigure, or otherwise disrupt our production servers.

Performance

Of course, we are also concerned that performance could degrade enough to become an availability issue. An errant load test could cause a test server to consume all available resources, starving the production servers. Luckily, most hypervisor solutions provide robust ways to limit resource consumption by a specify virtual server. IBM's VM hypervisor even supports complex level of service rules to provide fine resource allocation control. We can even gain some resource control when stacking databases or Java Virtual Machines (JVMs) by tuning processes, threads, sockets, etc. That said, it's important to realize that we can't get the same level of isolation with these technologies as we can with hypervisors.

Mixing workloads is an important step toward smoothing out the overall load profile. Smooth load profiles with smaller peaks and valleys allows us to reduce our white space requirements -- that is, increase server utilization targets. So, let go of the old ways. Prod and non-prod living together is not a sign of the coming apocalypse.