Isn't IT automation inherently evil? (I mean, you saw 'The Terminator,' right?)

While the general take on CloudWorld in San Francisco last week may have been that it was merely a shadow of what industry attendees were expecting, at least one presentation seems to have registered on the "worthy-of-discussion" meter. Lew Tucker from Sun was written up by Larry Dignan of ZDNet, Reuven Cohen, The Register, and others, for Lew's commentary on self-provisioning applications and "future cloud apps that won't need humans."

A couple things strike me about this "humanless computing" (as Reuven put it): first, whether people really think it through or not, this kind of automation is absolutely required for cloud computing. The types of dynamic infrastructures that businesses are hoping to get from the cloud just can't have a human in the minute-by-minute IT operations loop. (See also: human telephone switch operators.)

OK, fine. But that brings me to point #2: people hate automation. It's assumed to be faceless, out of control, and most likely up to no good. There's a whole Terminator movie & TV franchise built on the premise that the "rise of the machines" is to be avoided at all costs.

But while the evil of IT automation is great fodder for summer blockbusters, it's bad IT policy. Some specifics to think about:

Automation is a big part of cloud computing

Automation similar to what Lew was explaining is at the core of what cloud computing really is (or more honestly, what it will be when it's fully realized). Contrary to what many folks assume, the core requirement for a cloud is not virtualization. Virtualization is a technology that can be helpful in creating a cloud-style model, but it's not an absolute necessity. On the other hand, automation is one of those core cloud requirements -- the ability for systems to scale up and down without manual efforts, adding and releasing resources, and in many respects managing themselves.

Lori MacVittie of F5 explained this well in her post about "Putting the Cloud Before the Horse." "It isn't really a cloud unless it's automated," said Lori. "Without that automation what do you have? A bunch of servers running applications. That those applications are virtualized is really irrelevant to the architecture because you haven't done anything but changed which physical server they're being deployed on. Without the ability for the infrastructure to make decisions based on actionable data that's shared between the components, you really don't have anything all that much different than you did before."

Automation remains scary to IT shops

Earlier this year, before CA's acquisition of the Cassatt assets and expertise, I wrote a bit about some of the reasons that IT remains leery of automation and boiled it down to three reasons: the IT operations folks are a conservative bunch (and rightly so), automation requires a great deal of trust, and vendors have made it hard on themselves by frequently overpromising and under-delivering.

The complexity of IT environments is part of the problem here, something that Forrester’s Glenn O’Donnell underscored in "IT Operations 2009: An Automation Odyssey" (Forrester client link) last month (Denise Dubie of Network World did a nice analysis of Glenn's paper here.) "A combination of forces, including skyrocketing complexity and severe economic pressure, are radically and irreversibly altering the IT landscape," said Glenn. "Evidence indicates an automation 'tipping point' is already under way this year."

2009 as the tipping point for automation?

So, if you agree that automation is an important piece for enterprises to begin to use public and private clouds, and you agree that IT avoids automation like the plague, are we at an impasse? Why would this year (of all years) see huge shifts?

As I speculated earlier and as Glenn notes, "the one-two punch of virtualization and economic pressure represents a tectonic shift for IT. For many IT services, complexity has surpassed human ability." (Once again, see also: human telephone switch operators.)

So does that mean it's time to jump in whole hog? As with anything in the enterprise, don't bet on it. Glenn has two good bits of advice here. First, find your threshold for automation. For every IT organization that will be different.

To steal (and probably mangle) an analogy from Cassatt's former CTO Rob Gingell, some IT folks are like new Prius owners that go through their daily commute entranced by the little dashboard graphics showing when & how different parts of their hybrid engine are charging up or firing away. Others are less interested in the moment-by-moment changes going on and trust that the engine will do what it's supposed to do. (Often these are the same people, just a few weeks later.) This second group are the ones who are more willing to try more automation (after they see it perform admirably on its test runs, of course). Those are also the ones where the costs of tools, people, and process improvements (what Glenn calls "automation pain") are going to be offset by their org's "automation gain," an important balance to strike.

Another point Rob Gingell often makes is that the history of computing has been all about the drive toward increasing automation. Cloud computing is just another step. Previously you'd have to tell your operating system to do things in order for computing to proceed -- a step that we've thankfully evolved beyond. It will be the same with applications self-provisioning to maintain their service levels. Eventually.

OK, but shouldn't we still be a little worried about the Terminator scenario?

Lew Tucker's CloudWorld talk referenced Skynet, the self-aware (and quite nasty) computing system from the Terminator movies. He gave a nod to the underlying worry we humans have (at least, IT operations humans have) that the moral equivalent of a T-1000 is going to show up and take out some critical part of your data center and bring down important applications. And, sure, there are some twisted ways that automation can go wrong. (Stuart Charlton of Elastra pointed me to some amusing ones, which also note where humans make things worse.)

But, it was with no small amount of irony that Cassatt used the Skynet name as the internal codename for our next-generation product architecture, in which more and more of the things your system needs to do to were handled autonomously, not for purposes of world domination or hunting down the last remaining humans, but instead to maintain application service levels.

Be on the correct side of history

So, scary as things could seem around cloud computing and automating IT operations, it's logical that it will follow the pattern that enterprise IT has followed since it began: there is a gradual move toward making the impossible or hard-to-imagine things possible, even if the rate of that movement is somewhat unpredictable. Not long ago, doing e-commerce and Internet banking were seen as science fiction. Little by little, the risks became manageable and/or understood. IT incorporated what was needed and moved on.

I expect much the same is in store for the automation that's needed to enable cloud computing. Why? Lori MacVittie put it well: "If you want to take it to the next level, you're going to have to automate processes because that's where real operational benefits will be realized."

One other suggestion for those whose jobs are potentially impacted by automation comes from Glenn's work at Forrester: "Be the automator not the automated." Especially as the economy is forcing drastic cuts or at least hard choices, being on the side of providing the solution is more advisable. So, think a bit about how automation could actually make a difference, and help advocate that change.

So, in the end, IT automation is both important and possible. It's the key to some of the new cloud-driven capabilities that are within reach for organizations. Some organizations are jumping in with both feet now, and if Glenn's right a lot more will be experimenting this calendar year.

Which by some estimates is even a little late. I mean, wasn't Skynet supposed to take over starting in 1997?

Cassatt cloud innovation added to CA muscle

For those who have been watching and wondering what's next for Cassatt, the wait is over. CA announced today that they are the new home for much of what Cassatt has to offer: technology assets, patents, and a very significant chunk of the employees.

After the dire situation that Quentin Hardy wrote about in Forbes back in April, this is certainly a win for Cassatt and those with vested interest in seeing its ideas -- often described as both innovative and too far ahead of the market for its own good -- find a backer.

Here are a couple relevant excerpts from the CA-Cassatt press release:

CA Acquires Cassatt Data Center Automation Innovation and Expertise:
Acquisition Enables Dramatic Expansion of CA’s Automation and Cloud Capabilities

ISLANDIA, N.Y. and SAN JOSE, CA, June 2, 2009 -- CA, Inc. (NASDAQ:CA) today announced the acquisition of certain data center automation and policy-based optimization expertise and assets from Cassatt Corporation, a provider of innovative cloud computing software that makes data centers more efficient. Terms of the deal were not disclosed.

Some comments from Ajei Gopal, executive vice president of the Products and Technology Group at CA:

"This acquisition strengthens CA's ability to deliver on the promise of Lean IT for our customers. With the addition of Cassatt's engineering team and advanced data center automation assets, CA will accelerate its development of software that helps customers make more intelligent, business policy-based decisions. We are ensuring that CA will continue to be at the forefront of managing our customers’ evolving IT environments."

And some comments from Cassatt's chairman and CEO, Bill Coleman:

"Cassatt has long been a champion for using a cloud-style architecture to manage data centers like a 'compute utility.' This is a great move for both organizations because of the vision we share -- delivering a new, dramatically more efficient way to run data centers. The acquisition of Cassatt's data center automation technology and expertise by CA, one of the world's largest and most successful software companies and an innovator in business-driven automation, will help make this vision into a reality for customers."

And, finally, from Donald Ferguson, CA's Chief Architect:

"Cassatt invented an elegant and innovative architecture and algorithms for data center performance optimization. Incorporating Cassatt's analysis and optimization capabilities into CA's world-class business-driven automation solution will enable cloud-style computing to reliably drive efficiencies in both on-premises, private data centers and off-premises, utility data centers. We believe the result will be a uniquely comprehensive infrastructure management approach, spanning monitoring, analysis, planning, optimization and execution."

As I look at CA's infrastructure management portfolio as a (now former) outsider, I see a lot of product breadth in something near and dear to the hearts of those of us from Cassatt: data center efficiency. This deal combines CA's already broad product set and its formidable execution capabilities with some new data center optimization, cloud computing, and automation expertise from Cassatt. That seems like good news to me, especially since one of the dings against Cassatt has always been that infrastructure management is pretty difficult to entrust to a small vendor.

Neither Cassatt nor CA are going to have much more comment on today's announcement beyond the official press release. Don't take that as negative. We're spending our time and effort (starting today) to bring it all together in a way that will make sense for everyone involved. We'll share those details when we have them.

One piece of bittersweet news is that while Cassatt CEO Bill Coleman has been the face and personality behind Cassatt since its beginnings in 2003 (including a starring role as a Forbes cover boy in September 2006), he won't be coming with us to CA. I'd like to personally thank Bill for his vision and passion, and for giving me the opportunity to be a part of the story at Cassatt. I like to think that even though we didn't achieve the massive customer adoption we were hoping for, we did have a significant impact on the industry discussions about how to run and manage a data center, about utility computing, and about what has come to be known as cloud computing.

Along the way, we at Cassatt have all tried hard to make sure we're part of a real, on-going conversation with the industry and with end users. The name of this blog is just one reflection of that. Many things will likely change for us Cassatt folks as a result of the CA deal, but I for one am confident that we'll keep the dialog going. Stay tuned.

You can continue to follow Jay Fry on Twitter as @jayfry3.

Recoverability: How an internal cloud makes it easier to get your apps back up and running after a failure

Ok, back to talking about the "-ilities" this week and how cloud computing can help you address one of the key issues you are concerned with in your data center. On deck is the recoverability of your IT environment when run on an internal cloud infrastructure (like Cassatt Active Response).

As discussed in my last post, there can be a fair amount of organizational and operational change required to adopt an internal cloud infrastructure, but there are many benefits from taking on the task. The next couple of posts will outline one of the major benefits (recoverability) that comes from separating the applications from their network, computing, and storage resources, and how this separation allows for both intra-datacenter and inter-datacenter recoverability of your applications.

Five Internal Cloud Recovery Scenarios
To walk you through the discussion, I'm going to take you through a number of levels of recoverability as a function of the complexity of the application and the failure being addressed. Taking this approach, I've come up with five different scenarios that start with the recovery of a single application and end with recovery of a primary datacenter into a backup datacenter. The first four scenarios (intra-datacenter recovery) are covered in this post and the last one (inter-datacenter recovery) will be covered in my next post. So, enough background: let’s get into the discussion.

Application Recovery
Let's begin with the simplest level of recoverability and start with a single application (think a single workgroup server for a department that might be running things like a wiki, mail, or a web server.) From talking to many admins over the years, the first step they do when they find that a given server has dropped offline is to perform a "therapeutic reboot" to see if that gets everything back to a running state. The reality of IT is that many of the frameworks/containers/applications that you run leak memory slowly over time and that a reboot is the easiest way to clean up the issue.

In the case of a traditional IT environment, a monitoring tool would be used to monitor the servers under management and if the monitors go offline then an alert/page is generated to tell the admin that something is wrong that needs their attention. With an internal cloud infrastructure in place, the creation of monitors for each managed server comes for free (you just select in the UI how you want to monitor a given service from a list of supported monitors). In addition, when the monitor(s) drop offline you can configure policy to tell the system how you want the failure handled.

In the case of an application that you'd like rebooted prior to considering it failed, you simply instruct the internal cloud infrastructure to power cycle the node and to send an alert only if it doesn't recover correctly. While this simple level of recoverability is interesting in making an admin more productive (less chasing unqualified failures), it really isn't that awe-inspiring, so let's move on to the next level of recovery.

Hardware Recovery
In this scenario, we'll pick up where the last example left off and add an additional twist. Assume that the issue in the previous example wasn't fixed by rebooting the node. With an internal cloud infrastructure, you can enable the policy for the service to not only reboot the service on failure (to see if that re-establishes the service) but also to swap out the current piece of hardware for a new piece (complete with reconfiguring the network and storage on the fly so that the new node meets the applications requirements).

Let's explore this a bit more. With current IT practices, if you are faced with needing to make a singleton application available (like a workgroup server) you probably start thinking about clustering solutions (Windows or U*nx) that allow you to have a single primary node running and a secondary backup node listening in case of failure. The problem with this approach is that it is costly (you have to buy twice the hardware), your utilization is poor (the back-up machine is sitting idle), and you have to budget for twice the cooling and power because the back-up machine sits on, but idle.

Now contrast that with an internal cloud approach, where you have a pool of hardware shared among your applications. In this situation, you get single application hardware availability for free (just configure the policy accordingly). You buy between 1/5th and 1/10th the backup hardware (depending the number of concurrent failures you want to be resilient to) as the back-up hardware can be shared across many applications. Additionally, the spare hardware you do purchase sits powered off while awaiting a failure so it consumes zero power and cooling.

Now, that level of recoverability is interesting, but IT shops typically have more complex n-tier applications where the application being hosted is spread across multiple application tiers (e.g. a database tier, a middleware/service tier and a web tier).

Multi-tier Application Recovery
In more complex application types (including those with four tiers, as is typical with JBoss, WebLogic, or WebSphere), an internal cloud infrastructure continues to help you out. For this example let's take JBoss as our working example as many people have experience with that toolset. JBoss' deployment model for a web application will typically consist of four tiers of services that work in concert to provide the desired application to the end user. There will be a database tier (where the user's data is stored), a service tier that provides the business logic for the application being hosted, and a web tier that will interact with the business logic and dynamically generate the desired HTML page. The fourth and final tier (which isn't in the IT environment) is the user's browser that actually renders the HTML to human-readable format. In this type of n-tier application stack there are implicit dependencies between the different tiers that are usually managed by the admins who know what the dependencies are for the various tiers and, as a result, the correct order for startup/shutdown/recovery (e.g. there is no point in starting up the business or web tiers if the DB is not running).

In the case of an n-tier application, an internal cloud computing infrastructure can help you with manageability, scalability, as well as recoverability (we're starting to pull out all the "-ilities" now…) We'll cover them in order and close with the recoverability item as that's this week’s theme. On the manageability front, an internal cloud infrastructure can capture the dependencies between the tiers and orchestrate the orderly startup/shutdown of the tiers (e.g. first verify that the DB is running, then start the business logic, and finally finish with the web tier). This means that the specifics of the application are no longer kept in the admin's head, but rather in the tool where any admin can benefit from the knowledge.

On the n-tier scalability front, usually a horizontal rather than vertical scaling approach is used for the business and web tiers. With an internal cloud infrastructure managing the tiers and using the monitors to determine the required computing capacity (to do this with Cassatt Active Response, we use what we call Demand-Based Policies), the infrastructure will automatically increase/decrease the capacity in each tier as a function of the demand being generated against the tier.

Finally, on the recoverability front, everything outlined in the last recovery scenario applies (restart on failure and swap to a new node if that doesn't work), but now you also get the added value of being able to restart services in multiple dimensions. As an example, in many cases connection pooling is used in the business tier to increase the performance of accessing the database. One downside (depending on the solution used for managing the connection pool) is that if the database goes away, then the business tier has to be restarted to re-establish the connections. In a typical IT shop this would mean that the admin would have to manage the recovery across the various tiers. However, in an internal cloud computing environment, the infrastructure has sufficient knowledge to know that if the DB went down, there is no point in trying to restart the failed business tier until the DB has been recovered. Likewise, there is no point in trying to recover the web tier when the business tier is offline. This means that even if the root failure can not be addressed by the infrastructure (which can happen if the issue is not transient or hardware related) the admin can focus on the recovery of the specific item that has failed and the system will take care of the busywork associated with restoring the complete service.

Intra-datacenter recovery
Ok, we're now into hosting non-trivial applications within a cloud infrastructure so let's take that same n-tier application example, but add in the extra complexity that there are now multiple n-tier applications being managed. What we'll do, though, is throw in a datacenter infrastructure failure. This example would be relevant for issues like a Computer Room Air Conditioning (CRAC) unit failure, loss of one of the consolidated UPS units, or a water leak (all of which would not cause a complete failure in a datacenter, but would typically knock out a block of computing capacity).

Before we jump into the failure, we need to explore one more typical practice for IT shops. Specifically, as more applications are added to an IT environment it is not uncommon for the IT staff to begin to stratify the applications into support levels that correspond to the importance the business places on the specific application in question (e.g. revenue systems and customer facing systems typically have a higher availability requirement than, say, a workgroup server or test and development servers.) For this example lets say that the IT department has three levels of support/availability that they use, with level one being the highest priority and level three being the lowest. With Cassatt Active Response, you can put this type of policy directly into the application and allow it to optimize the allocation of your computing resources to applications per your defined priorities. With that as background, let’s walk through the failure we outlined above and see what Cassatt Active Response will do for you in the face of a major failure in your datacenter (we're going to take the UPS example for this discussion).

We'll assume prior to the failure that the environment is in a steady state with all applications up and running at the desired levels of capacity. At this point, one of the shared UPS units goes offline, which affects all compute resources connected to that UPS unit. This appears to Cassatt Active Response as a number of node failures that go through the same recovery steps outlined above. However, as mentioned above, usually you will plan for a certain number of concurrent failures and you will keep that much spare capacity available for deployment. Unfortunately, when you loose something shared like a UPS, the number of failures quickly consumes the spare capacity available and you find yourself in an over-constrained situation.

This is where being on a cloud infrastructure really starts to shine. Since you have already identified the priority of the various applications you host in the tool, it can dynamically react to the loss in compute capacity and move resources as necessary to maintain your service levels on your higher priority applications. Specifically, in this example lets assume that 30% of the lost capacity was in your level 1 (most important) applications. The infrastructure will first try to shore up those applications from the excess capacity available, but when that is exhausted, it will start repurposing nodes from lower priority applications in support of re-establishing the service level of the higher priority applications. Now, because the cloud infrastructure can manage the power, network, and images of the applications, it can do all of this gracefully (existing lower priority applications get gracefully shut down prior to their hardware being repurposed) and without user interaction. Within a short period of time (think 10s of minutes), the higher priority applications have been re-established to their necessary operating levels.

The final part of this example is what occurs when the issue causing the failure is fixed (in our example, the UPS is repaired and power is re-applied to the effected computing resources.) With a cloud infrastructure managing your environment, your lower priority applications that were effected in support of shoring up the higher priority applications all just recover automatically. Specifically, once the power is reapplied, all you have to do is mark the hardware as available and the system will do the rest of the work to re-inventory and re-allocate the hardware back into those tiers that are running below the desired capacity levels.

Well, there you have it. We've walked through a variety of failure scenarios within a datacenter and discussed how an internal cloud infrastructure can offload much of the busy/mundane work of recovery. In the next post I'll take the example we just finished and broaden it to include recovery into a completely different datacenter. Until then…

Is it safe to discuss automating your data center yet?

I've been at Cassatt for over 3 years now and during that time one of words that I've wanted to use the most in describing what our software does is "automation." The only problem: it's a four letter word for IT. Until now. Maybe.

In the strictest sense of the word, Cassatt's software is all about automation. The goal of the software is to run your data center more efficiently by doing most of the work for you. It balances the resources you have (your hardware, virtual servers, applications, networks) with the application demand on those resources using policies you set -- and it does the balancing and re-balancing automatically. You end up with a big pool of compute power that gets called into service to support your apps only as needed, without someone having to manually adjust anything.

(By the way, you should probably be saying about now: "hey, that sounds like a cloud, but one that uses the stuff you already have inside your own data center" -- but that's another topic altogether.)

So why be timid about using the word "automation," then, if it's accurate? Because, in fact, it's something that IT folks have proven to be very gun-shy about. It's a concept that, frankly, doesn't have a lot of fans. As an example, the InfoWorld article by Eric Knorr debating the legitimacy of internal clouds also expressed some of this skepticism about automation.

Why is everyone afraid of automation?

From our experiences with customers, there are probably three reasons. First, IT and data center operations groups fear change. And with good reason. Their job is to make sure stuff works. Their motto usually is: if it's working, don't mess with it.

Second, automation itself requires a great deal of trust. You are replacing what a thinking person has been doing with a set of code. An IT ops person is going to want to see it working in action before he or she feels good about that. It's the way these guys are wired. (See previous paragraph for the reason.)

Third, vendors have shot themselves in the foot repeatedly by overpromising and underdelivering. (This was Eric Knorr's point noted above.) When things don't live up to the hype and fail to work as promised, successful IT ops folks write that down in their little black book of "see, I told you sos."

However, the bad economy may be changing all that. The 451 Group and others deeply involved in this space (including us here at Cassatt) have been noting that change may be afoot. Why? Automation is something that higher-up execs begin to think about in a downturn, especially when lay-offs are happening all around them. Wouldn't it be great to be able to do more stuff with fewer people? Or worse: now that we have fewer people, how do we keep the lights on in the data center?

In the Great Recession, automating how you run your data center may be something you need to consider in order to survive.

Of course, there are several angles on what "automation" actually is. There's the HP/Opsware and BMC/BladeLogic automation, which sets up configurations of your software and apps on your servers. And then there's automating the run-time of your data center -- the day-to-day running of your IT systems after you have them configured -- where a huge percentage of your operational expenses go, like Cassatt does.

Either way, it's smart to get ahead of this. It's always a good idea to be one of the folks helping to push forward change rather than clinging to the current business-as-usual plans. Especially when business-as-usual might actually mean going-out-of-business. Or at least an extended "vacation" for some of your IT staff.

The thing that got me thinking about all this was a podcast that Mike Vizard of eWeek did with Cassatt CEO Bill Coleman this week about how data centers are run, why it has been so hard for IT to adopt automation to help them out, and how we might get out of this "rut of complacency."

The main problem running data centers, said Bill, "is not scale; it's complexity. We've had a good run with virtualization and data center consolidation, but they've attacked the scale problem and in doing so have added to the complexity problem. The issue going forward is that if we don't attack this, we'll get beyond human scale. I don't think there's any way to do that without automating the operations part, doing for data centers what the telephone switching system did for the telephone system."

So how do company IT departments begin to make these changes? "The only way to do this is an evolutionary process, not a revolutionary process," said Bill. "And I think we are seeing the beginning of that with cloud computing, both internally and externally."

You can listen to the whole podcast here. Total running time is about 18 minutes.