testing devops

Testing in the DevOps World-Test Automation the Sequel

“The most powerful tool we have as developers is automation.” – Scott Hanselman

In a previous blog about some stats that told the developing software testing story, we had identified the impact on testing strategies due to the growing adoption of DevOps as a key trend. Our CEO has also written in greater detail on how testing is changing due to DevOps. Looking back though it’s clear this story is still developing and that something more remains to be said. In other words, it deserves a sequel and the central role in this continuing story is reserved for Test Automation.

You may ask, why does Test Automation deserve this star billing? Well, consider the DevOps way for a bit. This is a world with several, almost continuous iterative releases, each within days, even minutes, of each other, all being pushed out to the final production environment into the demanding hands of paying customers. So many releases, so little testing time and so much pressure to deliver quality – has there ever been a more, theoretically, perfect case for automated testing? Let’s hope that puts the “Why” question to bed – now let’s move on to the “How” and “What”.

First, a look at the “How”. As was already apparent in Agile, with so many iterative releases following so close on the heels of each other, it is absolutely impossible to build your automation in parallel with the product-under-test. Thus, with DevOps, it becomes critically important to involve the test automation team at an early enough stage of the product planning to be able to anticipate, as much as possible, the direction the product is likely to take and automate early. This is also the time to plan for the automation carefully. Factors to consider include what conditions are most likely to remain reasonably unchanged and which are likely to undergo frequent changes? How reusable can you make the components of the automation framework? This is also a good time to define the objectives you are looking to achieve with the automation – Faster deployment? Better code quality? Greater confidence based on better regression tests? Essentially, start with the end in mind and measure as you go along to know if you are on the right track.

  1. So, on to the “What”. There is both the opportunity for creating a comprehensive test automation framework and the threat that some of it could be rendered irrelevant by the pace of change in the product.
  2. That said, though, there is value to considering automating the Unit Tests as there is a reasonable chance that several specific components will remain relatively stable over the course of the many iterations.
  3. The greatest value could well be in automating the regression testing – the maximum bang for the buck considering the sheer number of releases. Many DevOps models allow code to be delivered late in the cycle and fixes can be applied right until the time that specific release goes live. Automating the regressions will allow you to test the entire code after each such addition and this makes it far more likely that a high-quality, bug-free product goes out to the end customer.
  4. Among the central value propositions of the DevOps, way is continuous deployment on the operational infrastructure. Continuous deployment means continuous integration of code and of the code into the operational infrastructure. This is where automation can play a key role. An interesting approach being followed by many is to run integration testing separately and parallel from unit testing and sometimes even before unit testing. This approach believes that the integration testing is not impacted by the business logic and is only concerned with whether the product works on the deployment infrastructure. Automation helps test this quickly and relatively comprehensively.
  5. There is also great value on automating the testing for the build – deployment stage starting at the test environment itself. The objective is to run the tests in the test or development environment and ensure smooth deployment in the production environment.

A quote we like about DevOps goes, “DevOps is not a goal, but a never-ending process of continual improvement”. While agreeing with Jez Humble, who said this, perhaps we would like to add that this continual improvement is driven by continual testing, which is turn is based on a solid test automation platform. What do you think?

devops qa

Role of DevOps in QA

DevOps, a compendium of development and operations is an organisational strategy, focusing on a close collaboration and communication of the software developer team with the other professionals belonging to the testing and releasing teams. The planning employs automated processes in a symbiotic environment which ultimately results in building, testing and releasing the software with clockwork and guaranteed reliability.

How does QA benefit from DevOps:

Some ten years back, QA was seen as a group disparate from the Developers’ teams, with different skill sets and responsibilities and management. Fast forward into the DevOps age, and things are quite different today. Here’s how we look at QA through the glasses of Devops…

1. Automated Deployment:

The conventional approach of a software “release” is now passé with devops facilitating the delivery of the product on a monthly, weekly and even hourly basis into the market through automated processes. This has been made possible through a continuous cycle of improvement where the developers, testers and operations people all working in sync and moving in the same direction.

2. Environment is now a part of the product:

Traditionally here’s how the flowchart used to be like….you create a software, get it verified in a testing environment of the QA team and when the litmus test is over, so to speak, unleash it into the big bad world of the user. If anything then went wrong, it was the problem of the operations teams. Not any more….As is evident from the success of Google’s Unbounce , the QA verifies the environment with their chief enabling infrastructure being the code itself. At the occurrence of any change/problem, the QA team initiates the requisite deploys, examines that the intended change, functions as expected and move over to the latest deployed code with the option of a roll back if needed.

3. Prevention is better than discovery:

In a devops environment, the priority for QA is prevention of faults and not just finding them. As opposed to let’s say ten-fifteen years ago, the QA teams of today have the luxury of pushing the code on when it’s fully functional and rolling it back when things go awry. This has positive ramifications in that, the QA team can continuously track the quality of the product. Thus the QA team has a profound influence on the development and operational phases of the software.

4. Less of Human error:

Devops enables more of automated testing in QA thus reducing the glitches due to fatigue associated with manual testing. This also ensures 100% code coverage and quick scripting of test cases.

5. Greater teamwork and rapport:

At the individual level, the testers and the operations team get a chance to be on the same page and level as the developers. This improves coordination ticking the boxes for higher market outreach and efficiency gains

automate testing for devops

The Role of Software Testing in a DevOps World

By Rajiv Jain (CEO, ThinkSys)

Mark Berry said, “Programmers don’t burn out on hard work, they burn out on change-with-the-wind directives and not ‘shipping’”. I don’t know about the first but the desire to “ship” seems to be a powerful motivation for the push towards the adoption of DevOps practices among software development teams and companies everywhere. That and the business benefits obviously. I remember a Puppet Labs survey from a couple of years ago that showed that of the organizations surveyed, those that adopted DevOps shipped code up to 30 times more frequently with lead times of as little as a few minutes. They also experienced only half as many failures and had the ability to recover from those failures up to 12 times faster. These are not just numbers, I recall reading about one of the early DevOps adopters, Etsy. Apparently they update their site every 20 minutes without any service disruption to over 20 million users – that’s a phenomenal competitive edge to possess. Small wonder then that the DevOps tide is rising.

Across most SaaS-based product development efforts, the approach seems to be to bring Development and Operations together in, a largely, automation-driven effort to keep pushing releases out, more or less continuously. Given the sheer number, it’s perhaps not even fair to term these as releases any longer! Coming from a company that has a significant Software Testing practice though an interesting facet of this partnership between Development and Operations occasionally occurs to me. It seems to me that a third, equally interested party, Testing and QA, may not be getting the attention it deserves in this discussion. In fact, the case has sometimes been made that the greater degree of automation and the souped-up release cycles imply a reduced role for software testing. I couldn’t disagree more.

Let’s go back to the Puppet Labs survey I quoted earlier – notice that one of the key benefits stated was the reduced rate of failures and the ability to recover from those failures quickly. Clearly the aim is not to ship code, it’s to ship market-ready product on a “Continuous” basis. To me it seems that would be impossible to achieve without an even greater emphasis on testing. Testing, while Agile had brought out the need to involve testing strategy at a very early stage of the product planning. How else to keep up with the short release cycles? DevOps with its Continuous Integration and Continuous Delivery is, in this context, like Agile on steroids and the need, thus, is even greater to get testing involved early in the product definition and planning stage. In many ways, the approach now has to be to plan to prevent defects rather than detect them.

So, in DevOps testing comes in early, plans for the way the product is going to pan out and prepares itself accordingly so that as code starts rolling out it gets tested. This is a significant change, at least of mindset. Earlier the development used to get done and the testing would start – today these seem to have to go, more or less, in parallel. Many have called this Continuous Testing – a perfectly valid term.
Another significant change that this “Continuous” model is engineering is the integration of functions. DevOps teams now tend to contain people from Development, Operations and, increasingly, Testing. Without that level of integration, getting quality code out in such short time frames would be impossible. This integration is also throwing up new roles for everyone, including testing. Carl Schmidt, CTO of Unbounce explains it well, “I’m of the mindset that any change at all (software or systems configuration) can flow through one unified pipeline that ends with QA verification. In a more traditional organization, QA is often seen as being gatekeepers between environments. However, in a DevOps-infused culture, QA can now verify the environments themselves, because now infrastructure is code.”

That statement points to a third significant change for testing in the DevOps way – what to test? Now that the Operations, i.e the nuts and bolts that get the SaaS product out to the millions of subscribers, is a part of the effort of getting the product out is it not necessary to test that too? There are also some subtle changes of emphasis that emerge here – functional testing is always important but now there is a premium placed on testing for load conditions and for performance. The environment is also, as much a part of the SaaS product now as the code. There is a role here that testing can excel in – clearly they know more than most about the difficulties of taking the application code, deploying it in test environments, testing it and then moving it out to production. The process may have become shorter and sharper but the skills are the same.

I like a quote, sadly unattributed, that I read somewhere, it goes, “Software testers do not make software, they only make them better.” In the context of DevOps it would seem that finally they are getting an opportunity to not only make software but also to make it better!

Categories