Friday, 30 November 2012

Automated Regression Testing Challenges in Agile Environment

Agile Projects present their own challenges to the Automation team; Unclear project scope, Multiple iterations, Minimal documentation, early and frequent Automation needs and active stakeholder involvement all demand lot of challenges from the Automation Team.Some of these challenges are:

Challenge 1: Requirement Phase
Test Automation developer captures requirements in the form of "user stories", which are brief descriptions of customer-relevant functionality.

Each requirement has to be prioritized as follows:

High: These are mission critical requirements that absolutely have to be done in the first release
Medium: These are requirements that are important but can be worked around until implemented.
Low: These are requirements that are nice-to-have but not critical to the operation of the software.

Once priories are established, the release "iterations" are planned. Normally, each Agile release iteration takes between 1 to 3 months to deliver. Customers/software folks take liberty to make too many changes to the requirements. Sometimes, these changes are so volatile that the iterations are bumped off. These changes are greater challenges in implementing Agile Automation testing process.

Challenge 2: Selecting the Right Tools

Traditional, test-last tools with record-and-playback-features force teams to wait until after the software is done. More over, traditional test automation tools don't work for an Agile context because they solve traditional problems, and those are different from the challenges facing Agile Automation teams. Automation in the early stages of an agile project is usually very tough, but as the system grows and evolves, some aspects settle and it becomes appropriate to deploy automation. So the choice of testing tools becomes critical for reaping the efficiency and quality benefits of agile.

Challenge 3: Script Development Phase

The Automation testers, developers, business analysts and project stakeholders all contribute to kick-off meetings where "user-stories" are selected to next sprint. Once the "user-stories" are selected for the sprint, they are used as the basis for a set of tests.

As functionality grows with each iteration, regression testing must be performed to ensure that existing functionality has not been impacted by the introduction of new functionality in each iteration cycle. The scale of the regression testing grows with each sprint and ensures that this remains a manageable task the test team use the test automation for the regression suite.

Challenge 4: Resource Management

The Agile approach requires a mixture of testing skills, that is, test resource will be required to define unclear scenarios and test cases, conduct manual testing alongside developers, write automated regression tests and execute the automated regression packages. As the project progresses, specialist skills will also be required to cover further test areas that might include integration and performance testing. There should be an appropriate mix of domain specialist who plan and gather requirements. The challenging part in the Resource management is to find out test resources with multiple skills and
allocate them.

Challenge 5: Communication

Good communication must exist among Automation testing team, developers, business analysts and stake holders. There must be highly collaborative interaction between client and the delivery teams. More client involvement implies more suggestions or changes from the client. It implies more bandwidth for communication. The key challenge is that the process should be able to capture and effectively implement all the changes and data integrity needs to be retained. In traditional testing, developers and testers are like oil and water, but in agile environment, the challenging task is that they both must work together to achieve the target.

Challenge 6: Daily Scrum Meeting

Daily Scrum Meeting is one of the key activities in Agile Process. Teams do meet for 15 minutes stand up sessions. What is the effectiveness of these meetings? How far these meetings help Automation practice Developers?

Challenge 7: Release Phase

The aim of Agile project is to deliver a basic working product as quickly as possible and then to go through a process of continual improvement. This means that there is no single release phase for a product. The challenging part lies in integration testing and acceptance testing of the product.

If we can meet these challenges in a well optimized manner, then Automated Regression Testing in Agile environment is an excellent opportunity for QA to take leadership of the agile processes. It is better placed to bridge the gap between users and developers, understand both what is required, how it can be achieved and how it can be assured prior to deployment. Automation practice should have a vested interest in both the how and the result, as well as continuing to assure that the whole evolving system meets business objectives and is fit for purpose.

Transitioning from Waterfall to Agile Testing – ‘Challenges & Approach’

Transitioning from Waterfall to Agile Testing – 'Challenges & Approach'

More and more software projects in the mainstream are embracing Agile, these days.  It's important to understand how this paradigm shift impacts testing teams.  And, what are the challenges testing teams encounter and how do they overcome them?

This blog post is an attempt to answer some of the questions above by taking inferences from a recent customer engagement of Aspire, where Client's team and Aspire's offshore team collaboratively plans the move to Agile.

Driving the transformation to Agile Testing

Our customer is a leader in enterprise fraud management solutions in US.  This customer has been engaged with us for more than 6 years now.  Aspire is involved in Manual Testing and Test Automation of their suite of products.  Because of a heightened need for shorter testing cycles and higher visibility, customer decided to move ahead with Agile.

Primary Differences in working style- Waterfall Model vs. Agile

In the waterfall model, requirements along with product documentation were entrusted to the team.  There was enough time to understand requirements, perform exploratory testing, write test cases, and automate the QA.  On the Test Automation front, the builds were stable, with the UI and functionality intact, making it easier for preparing good test scripts.

All these practises that were working earlier were not applicable for agile approach. So the team decided to pursue each problem one by one and analyze the strategy for agile testing and define new methodologies.

The typical activities that were carried out include:

Requirements Understanding and Scheduling:

Less Documentation/Regular Communication Meetings: Meetings with the Product Owners, Scrum Masters and development team to understand the requirements one by one and validated our understanding by questioning defined Use Cases.  The requirements were approached from the customer perspective.

Refining the requirements from the customer perspective: The requirements were refined to the smallest possible unit.  This enabled the team to commit the unit requirement into specific Sprint.  Then the team proceeded onto writing test scenario with test cases as soon as the sprint started.

Estimate the effort required to complete the estimation: Based on the refined requirements, team was able to precisely estimate the effort required to complete the design, development, QA and documentation.  This helped everyone in the team to know when the preceding task (say development) would be completed and the work product would be available for succeeding tasks (say QA).  This helps to track the schedule and negate any slippage.

Test Automation:

Transition to Test Automation was not as easy as it was in manual testing.   Earlier, the team had a complete set of test cases and cushion of stable builds for categories like smoke and regression. This helped to create module based scripts with good reusability.

Development & Test automation in Different Sprint: Initially Test Automation was followed up by development activities and manual testing with a gap of one sprint.  The team automated test cases of previous sprint, so that the team can minimize test script maintenance activities.  Since this was not in tune with agile, we followed below approach.

  • Developing feature, writing test cases and creating test scripts went in parallel for a particular requirement.
  • Developer/Product Owner reviews the test cases based on scoped requirements
  • Test automation engineer starts writing stub test scripts and common functions based on reviewed test scripts
  • Transparent & Collaborative Approach: Since each member in the team was aware of the team member's work and this helped to inform the changes better and avoid any rework in the sprints. The integrated work-approach within the team played a crucial role here.

Regression testing:

Delivering working products faster requires shorter end of the release activities. So we need to minimize the end of release activities (specifically regression testing).  To accommodate this, we empowered our QA Infrastructure to perform regression quickly, as and when needed, without pushing it to the end of the release.

  • At the end of each sprint, the test cases (which are mostly automated) were classified into categories like sanity, smoke, regression etc.
  • The Smoke test cases in particular were executed as nightly tests which helped to uncover any major issues instantly.
  • Importantly, these test suites had to be properly maintained to have confidence in the scripts. So the testing team worked with developers to maintain these tests where they will review the scripts and will also execute the tests wherever they feel something has changed/affected.
  • This helped to extensively bring down the regression testing effort required at the end of each release

Certifying alpha / beta kits

  • In the earlier waterfall model, resources spent dedicated time to certify alpha and beta kits.
  • With the current approach in regression testing and test automation, qualifications of kit were integrated and were implied part of sprint deliverables.
  • Delivery of fully working product in each sprint

Team Work

All these need consensus within team that "Quality of product is every ones responsibility".  The emphasis has been on review of test cases and test scripts, involving ourselves in requirement grooming, giving feedback on usability and most importantly approaching testing with quality in mind.

Conclusion:

Agile helped us to improve ourselves and scale up in new areas. In the team, people who were used to performing manual testing alone, started developing automation scripts and reviewing scripts developed by others.

The team aggressively interacted and worked closely with developers and product owners.

And thus, they were able to deliver product quickly (~50% less time than waterfall model approach) and with good quality. Most importantly they are learning each day in agile and "Transforming Product Development".

Tags:

agile testing life cycle

agile testing methodology

agile testing ppt

agile methodology

agile testing process

agile model

agile testing scrum

scrum

agile testing

agile testing life cycle

agile testing scrum

agile testing process activities

agile testing process ppt

agile testing interview questions

software testing process agile development

agile testing process wikipedia

agile testing process examples

agile testing scrum ppt

what is agile testing scrum and sprint

testing agile scrum methodology

agile scrum testing methodology

what is scrum process in testing

agile test plan

what is agile methodology

scrum test plan

janet gregory agile test plan template

agile test plan template

quality assurance agile test plan

agile test plan forums

agile test plan quality methodologies

kind agile test plan

agile test plan sample

test plan in agile methodology

agile scrum testing process

agile testing process diagram

agile testing process ppt

agile software testing process

agile testing challenges

agile testing basics

agile regression testing

what is agile testing in software testing

agile testing process ppt

testing challenges in software testing

testing in agile environment

what is agile testing in software testing

regression testing in agile

define agile testing

agile testing benefits

role of tester in agile

gile test strategy

agile testing methodology wiki

testing agile scrum methodology

agile testing methodology tutorial

agile testing methodology diagram

performance testing agile methodology

agile methodology in software testing

agile model in testing


Agile Automation

Agile Automation

According to the World Quality Report 2010/2011 by Capgemini and HP over 60% of companies have either already adopted agile methods or are planning to do so in the near future.  Agile Testing has been gaining popularity over the last few years due to several reasons such as the faster product development lifecycles and quicker Go to market cycles (GTM.

Some of the factors that affect progress and outcome of sprints are:

  • Reduced Coverage: Compressed time frames in each sprint, forces the Test Engineers to test only the areas that have changed in the sprint.  A complete regression cycle is not being conducted on the rest of the application leaving it susceptible to major flaws
  • Traditional tools that were designed to work in a 'test last' environment, struggle to work in agile environment which is designed for 'test first, test continuously' model. New agile specific tools from Version One or Rally Software, can be used
  • Agile Testing allows requirements to creep in any time during any phase of the project and it will be accommodated in the already allotted functionalities to be validated

To address these issues, a suite of automated testing solutions can be formulated that enable smarter sprints that help to meet deadlines and timely qualitative releases.

And some of the best practices that can yield great results for such short GTMs and provide quick ROI are:

  • Refactoring/Regression-Performing  Bug fixes, refactoring old code and running extensive regression testing, every few weeks to focus on stability, takes care of missed bugs during ongoing testing process
  • Testing from a Customer Point of View: Sharing good stories with customer relevant material, with the development team as well as the testing team
  • Test ideas can be compiled in a presentation shared with the developers and developers can take live feedback helping developers start thinking about whether their software will pass such tests
  • Frequently run tests or procedures, or those that take the most time, if automated, can generate a big ROI

Adopting these core points can hugely impact the overall testing process.


Tags:

agile testing life cycle

agile testing methodology

agile testing ppt

agile methodology

agile testing process

agile model

agile testing scrum

scrum

agile testing

agile testing life cycle

agile testing scrum

agile testing process activities

agile testing process ppt

agile testing interview questions

software testing process agile development

agile testing process wikipedia

agile testing process examples

agile testing scrum ppt

what is agile testing scrum and sprint

testing agile scrum methodology

agile scrum testing methodology

what is scrum process in testing

agile test plan

what is agile methodology

scrum test plan

janet gregory agile test plan template

agile test plan template

quality assurance agile test plan

agile test plan forums

agile test plan quality methodologies

kind agile test plan

agile test plan sample

test plan in agile methodology

agile scrum testing process

agile testing process diagram

agile testing process ppt

agile software testing process

agile testing challenges

agile testing basics

agile regression testing

what is agile testing in software testing

agile testing process ppt

testing challenges in software testing

testing in agile environment

what is agile testing in software testing

regression testing in agile

define agile testing

agile testing benefits

role of tester in agile

What to be automated and what to be not

What to be automated and what to be not

Automation of whole application, product or process is not possible or one can say it is near to impossible to achieve 100% automation. So it is important to decide which test cases should be automated on High priority.

The main idea of Automating the test is to save the time of manual resource and how to increase the test coverage. So one should determine the test cases which are repeated very often and taking large amount of data on the same set of actions like login of a screen. There could be many combination of testing Login screen with different set of data, which takes very much time of manual tester.
One should not automate the Test cases which take most of the Automation test engineer effort and time and in future that test script usage is very less.

One can get the most benefit out of your automated testing efforts by automating:

• Repetitive tests that run for multiple builds.
• Tests that tend to cause human error.
• Tests that require multiple data sets.
• Frequently used functionality that introduces high risk conditions.
• Tests that is impossible to perform manually.
• Tests that run on several different hardware or software platforms and configurations.
• Tests that take a lot of effort and time when manual testing
When trying to automate the test cases, always try to make the modules independent and focused on an objective. By using this strategy, one can reuse the independent modules to run the test repeatedly

When to be automated:

Testing of a product or application is not sufficient in a single round as developer may change their code due to some previous bug fix or due to new enhancements to application. In each and every build we need to test the same functionality number of times. Lot of manual effort is involved. To save the time and effort of manual effort, we will automate the test cases.

If we encounter the following question, then it's perfect time that we automated the test cases:

What if a programmer then changes the code and introduces a bug? What if I don't catch that bug because I didn't rerun the test after the change?

There are some scenarios in which one should think of automating the test:

• Both automation and manual testing are plausible. That's not always the case. For example, load testing often requires the creation of heavy user workloads. Even if it were possible to arrange for 300 testers to use the product simultaneously, it's surely not cost-effective. Load tests need to be automated.

What Technology should be used for Automation?
For some Automation tools they used their own Scripting language like Rational Robot use SQABasic. Now a days there are so many tools they used global Scripting languages like RFT use .NET and Java.
Selenium used Ruby, Python, java,.NET etc.

Choice of Automation tool?

Selecting an automated testing tool is essential for test automation. There are a lot of automated testing tools on the market, and it is important to choose the right automated testing tool that best suits your overall requirements.

Consider these key points when selecting an automated testing tool:

• Support for your platforms and technology. Are you testing .Net, C# or WPF applications and on what operating systems?
• Flexibility for testers of all skill levels. Can your QA department write automated test scripts or is there a need for keyword testing?
• Feature rich but also easy to create automated tests. Does the automated testing tool support record-and-playback test creation as well as manual creation of automated tests; does it include features for implementing checkpoints to verify values, databases, or key functionality of your application?
• Create automated tests that are reusable, maintainable and resistant to changes in the applications UI. Will my automated tests break if my UI changes?
• Support for Cross Platform or cross OS test support of tool
• Good Support with the AUT.
• Testing Multi-Language Applications
• Automated Testing Tool: Support for Team Work
• Command Line and OLE Automation Support
• Integration with Team Systems and Build Software
• Technical Support
• Pricing Policy
• Free Trial Versions

Now once the tool gets selected, the primary focus of Automation test Engineer is to setup the Automation framework. It should be Tool Independent. One can create Automation framework in ASP.Net or java. Framework should be user friendly and independent of AUT. One Automation framework could be used for multiple Applications.