Test Terminology

Software Testing Life Cycle

 

There is no fixed standard of STLC in the world and it basically varies as per the following:

 

 

Nevertheless, Software Testing Life Cycle, in general, comprises of the following phases:

 

Software Testing Life Cycle

 

 

Phase Activity Deliverables Necessity
Requirements/Design Review You review the software requirements/design (Well, if they exist.)
  • Review Defect Reports
Curiosity
Test Planning Once you have gathered a general idea of what needs to be tested, you ‘plan’ for the tests. Farsightedness
Test Designing You design/detail your tests on the basis of detailed requirements/design of the software (sometimes, on the basis of your imagination). Creativity
Test Environment Setup You setup the test environment (server/client/network, etc) with the goal of replicating the end-users’ environment.
  • Test Environment
Rich company
Test Execution You execute your Test Cases/Scripts in the Test Environment to see whether they pass. Patience
Test Reporting You prepare various reports for various stakeholders.
  • Test Results (Final)
  • Test/Defect Metrics
  • Test Closure Report
  • Who Worked Till Late & on Weekends Report
Diplomacy

 

Interestingly, no matter how well-defined a Software Testing Life Cycle you have in your project or organization, there are chances that you will invariably witness the following widely-popular cycle:

  • Testing

  • Cursing

 

In this type of STLC, you skip phases like requirement/design review, test planning, and test designing –in the high hope that the skipping will save you some time and/or cost.

Also, note that the Software Testing Life Cycle phases mentioned above do not necessarily have to be in the order listed; some phases can sometimes run in parallel (For instance, Test Designing and Test Execution). And, in extreme cases, the phases might also be reversed (For instance, when there is Cursing prior to Testing).

 

 

Software Testing Life Cycle STLC - Explained

 
 

Contrary to popular belief, Software Testing is not a just a single activity. It consists of series of activities carried out methodologically to help certify your software product. These activities (stages) constitute the Software Testing Life Cycle (STLC).

 

The different stages in Software Test Life Cycle -

software-test-life-cycle

 

 

Each of these stages have a definite Entry and Exit criteria  , Activities & Deliverables associated with it.

 

In an Ideal world you will not enter the next stage until the exit criteria for the previous stage is met. But practically this is not always possible. So for this tutorial , we will focus of activities and deliverables for the different stages in STLC. Lets look into them in detail.

 

Requirement Analysis

 

During this phase, test team studies the requirements from a testing point of view to identify the testable requirements. The QA team may interact with various stakeholders (Client, Business Analyst, Technical Leads, System Architects etc) to understand the requirements in detail. Requirements could be either Functional (defining what the software must do) or Non Functional (defining system performance /security availability ) .Automation feasibility for the given testing project is also done in this stage.

 

Activities

  • Identify types of tests to be performed.
  • Gather details about testing priorities and focus.
  • Prepare Requirement Traceability Matrix (RTM).
  • Identify test environment details where testing is supposed to be carried out.
  • Automation feasibility analysis (if required).

 

Deliverables

  • RTM
  • Automation feasibility report. (if applicable)

 

Test Planning

 

This phase is also called Test Strategy phase. Typically , in this stage, a Senior QA manager will determine effort and cost estimates for the project and would prepare and finalize the Test Plan.

 

Activities

  • Preparation of test plan/strategy document for various types of testing
  • Test tool selection
  • Test effort estimation
  • Resource planning and determining roles and responsibilities.
  • Training requirement

Deliverables

 

Test Case Development

 

This phase involves creation, verification and rework of test cases & test scripts. Test data , is identified/created and is reviewed and then reworked as well.

 

Activities

  • Create test cases, automation scripts (if applicable)
  • Review and baseline test cases and scripts
  • Create test data (If Test Environment is available)

Deliverables

  • Test cases/scripts
  • Test data

 

Test Environment Setup

 

Test environment decides the software and hardware conditions under which a work product is tested. Test environment set-up is one of the critical aspects of testing process and can be done in parallel with Test Case Development Stage. Test team may not be involved in this activity if the customer/development team provides the test environment in which case the test team is required to do a readiness check (smoke testing) of the given environment.

 

Activities

  • Understand the required architecture, environment set-up and prepare hardware and software requirement list for the Test Environment.
  • Setup test Environment and test data
  • Perform smoke test on the build

 

Deliverables

  • Environment ready with test data set up
  • Smoke Test Results.

 

Test Execution

 

During this phase test team will carry out the testing based on the test plans and the test cases prepared. Bugs will be reported back to the development team for correction and retesting will be performed.

 

Activities

  • Execute tests as per plan
  • Document test results, and log defects for failed cases
  • Map defects to test cases in RTM
  • Retest the defect fixes
  • Track the defects to closure

 

Deliverables

  • Completed RTM with execution status
  • Test cases updated with results
  • Defect reports

 

Test Cycle Closure

 

Testing team will meet , discuss and analyze testing artifacts to identify strategies that have to be implemented in future, taking lessons from the current test cycle. The idea is to remove the process bottlenecks for future test cycles and share best practices for any similar projects in future.

 

Activities

  • Evaluate cycle completion criteria based on Time,Test coverage,Cost,Software,Critical Business Objectives , Quality
  • Prepare test metrics based on the above parameters.
  • Document the learning out of the project
  • Prepare Test closure report
  • Qualitative and quantitative reporting of quality of the work product to the customer.
  • Test result analysis to find out the defect distribution by type and severity.

 

Deliverables

  • Test Closure report
  • Test metrics

 

Finally, summary of STLC along with Entry and Exit Criteria

STLC Stage

Entry Criteria

Activity

Exit Criteria

Deliverables

Requirement Analysis

Requirements Document available (both functional and non functional)

Acceptance criteria defined.

Application architectural document available.

Analyse business functionality to know the business modules and module specific functionalities.

Identify all transactions in the modules.
Identify all the user profiles.

Gather user interface/authentication, geographic spread requirements.

Identify types of tests to be performed.

Gather details about testing priorities and focus.

Prepare Requirement Traceability Matrix (RTM).

Identify test environment details where testing is supposed to be carried out.

Automation feasibility analysis (if required).

Signed off RTM

Test automation feasibility report signed off by the client

 

 

RTM

Automation feasibility report (if applicable)

 

 

Test Planning

Requirements Documents

Requirement Traceability matrix.

Test automation feasibility document.

Analyze various testing approaches available

Finalize on the best suited approach

Preparation of test plan/strategy document for various types of testing

Test tool selection

Test effort estimation

Resource planning and determining roles and responsibilities.

Approved test plan/strategy document.

Effort estimation document signed off.

 

Test plan/strategy document.

Effort estimation document.

 

Test case development

Requirements Documents

RTM and test plan

Automation analysis report

Create test cases, automation scripts (where applicable)

Review and baseline test cases and scripts

Create test data

Reviewed and signed test Cases/scripts

Reviewed and signed test data

 

Test cases/scripts

Test data

 

Test Environment setup

System Design and architecture documents are available

Environment set-up plan is available

Understand the required architecture, environment set-up

Prepare hardware and software requirement list

Finalize connectivity requirements

Prepare environment setup checklist

Setup test Environment and test data

Perform smoke test on the build

Accept/reject the build depending on smoke test result

Environment setup is working as per the plan and checklist

Test data setup is complete

Smoke test is successful

 

Environment ready with test data set up

Smoke Test Results.

 

Test Execution

Baselined RTM, Test Plan , Test case/scripts are available

Test environment is ready

Test data set up is done

Unit/Integration test report for the build to be tested is available

Execute tests as per plan

Document test results, and log defects for failed cases

Update test plans/test cases, if necessary

Map defects to test cases in RTM

Retest the defect fixes

Regression testing of application

Track the defects to closure

 

All tests planned are executed

Defects logged and tracked to closure

 

Completed RTM with execution status

Test cases updated with results

Defect reports

Test Cycle closure

Testing has been completed

Test results are available

Defect logs are available

Evaluate cycle completion criteria based on - Time, Test coverage , Cost , Software Quality , Critical Business Objectives

Prepare test metrics based on the above parameters.

Document the learning out of the project

Prepare Test closure report

Qualitative and quantitative reporting of quality of the work product to the customer.

Test result analysis to find out the defect distribution by type and severity

Test Closure report signed off by client

 

Test Closure report

Test metrics

 

 

 

Test Strategy and Test Plan

 

Test Strategy:


A Test Strategy document is a high level document and normally developed by project manager. This document defines “Software Testing Approach” to achieve testing objectives. The Test Strategy is normally derived from the Business Requirement Specification document.

 

The Test Strategy document is a static document meaning that it is not updated too often. It sets the standards for testing processes and activities and other documents such as the Test Plan draws its contents from those standards set in the Test Strategy Document.

Some companies include the “Test Approach” or “Strategy” inside the Test Plan, which is fine and it is usually the case for small projects. However, for larger projects, there is one Test Strategy document and different number of Test Plans for each phase or level of testing.

 

Components of the Test Strategy document

 

  • Scope and Objectives

  • Business issues

  • Roles and responsibilities

  • Communication and status reporting

  • Test deliverability

  • Industry standards to follow

  • Test automation and tools

  • Testing measurements and metrices

  • Risks and mitigation

  • Defect reporting and tracking

  • Change and configuration management

  • Training plan

 

Test Plan:


The Test Plan document on the other hand, is derived from the Product Description, Software Requirement Specification SRS, or Use Case Documents.
The Test Plan document is usually prepared by the Test Lead or Test Manager and the focus of the document is to describe what to test, how to test, when to test and who will do what test.

 

It is not uncommon to have one Master Test Plan which is a common document for the test phases and each test phase have their own Test Plan documents.

There is much debate, as to whether the Test Plan document should also be a static document like the Test Strategy document mentioned above or should it be updated every often to reflect changes according to the direction of the project and activities.
My own personal view is that when a testing phase starts and the Test Manager is “controlling” the activities, the test plan should be updated to reflect any deviation from the original plan. After all, Planning and Control are continuous activities in the formal test process.

  • Test Plan id

  • Introduction

  • Test items

  • Features to be tested

  • Features not to be tested

  • Test techniques

  • Testing tasks

  • Suspension criteria

  • Features pass or fail criteria

  • Test environment (Entry criteria, Exit criteria)

  • Test deliverables

  • Staff and training needs

  • Responsibilities

  • Schedule

This is a standard approach to prepare test plan and test strategy documents, but things can vary company-to-company

 

 

 

What is Test Strategy? 

 

A strategy plan for defining testing approach, what you want to accomplish and how you are going to achieve it.

 

This document removes all uncertainty or vague requirement statements with a clear plan of approach for achieving the test objectives. Test strategy is one of the most important documents for QA team. Writing it effectively is a skill every tester should achieve in their career.

 

It initiates your thought process which helps to discover many missing requirements. Thinking and test planning activities help team to define testing scope and test coverage.  It helps test managers to get clear state of the project at any point. The chances of missing any test activity are very low when there is a proper test strategy in place.

 

Test execution without any plan rarely works. I know teams who write strategy document but never refer it back while test execution. Testing strategy plan must be discussed with whole team so that the team will be consistent on approach and responsibilities. In tight deadlines, you can’t just waive any testing activity due to time pressure. At least it must go through a formal process before doing so.

 

Test Strategy vs. Test Plan:

 

Over the years, I see a lot of confusion between these two documents. So let’s start with basic definitions. Generally it doesn’t matter which comes first. Test planning document is a combination of strategy plugged with overall project plan. According to IEEEStandard 829-2008, strategy plan is a sub item of test plan.

 

Every organization has their own standards and processes to maintain these documents. Some organizations include strategy details in test plan itself (here is good example of this). Some organizations list strategy as a subsection in testing plan but details are separated out in different test strategy document.

Project scope and test focus is defined in test plan. Basically, it deals with test coverage, features to be tested, features not to be tested, estimation, scheduling and resource management.

 

Whereas test strategy defines guidelines for test approach to be followed in order to achieve the test objectives and execution of test types defined in testing plan. It deals with test objective, approach, test environment, automation strategy and tools, and risk analysis with contingency plan.

 

To summarize test plan is a vision of what you want to achieve and test strategy is an action plan designed to achieve this vision!

 

Hope this will clear all your doubts. James Bach has more discussion on this topic here.

 

Process to develop a good test strategy document:
Don’t just follow the templates without understanding what works best for your project. Every client has its own requirements and you must stick to the things that work perfectly for you. Do not copy any organization or any standard blindly. Always make sure if that is helping for you and your processes.

Below is a sample strategy template that will outline what should be covered in this plan along with some examples to illustrate what make sense to cover under each component.

 

Test Strategy in STLC:
 

STLC

 

Common sections of test strategy document:

 

Step #1 – Scope and Overview:

Project overview along with information on who should use this document. Also include details like who will review and approve this document. Define testing activities and phases to be carried out with timelines with respect to overall project timelines defined in test plan.

Step #2 – Test Approach:

Define testing process, level of testing, roles and responsibilities of every team member. For every test type defined in test plan (e.g.unit, integration, system, regression, installation/uninstallation, usability, load, performance, and security testing) describe why it should be conducted along with details like when to start, test owner, responsibilities, testing approach and details of automation strategy and tool if applicable.

In test execution there are various activities like adding new defects, defect triage, defect assignments, re-testing, regression testing and finally test sign-off. You must define exact steps to be followed for each activity. You can follow same process which worked for you in your previous test cycles. A visio presentation of all these activities including number of testers and who will work on what activity is very helpful to quickly understand roles and responsibilities in the team.

E.g. defect management cycle – mention the process to log new defect. Where to log it, how to log new defects, what should be the defect status, who should do defect triage, whom to assign defects after triage etc.

Also define change management process. This includes defining change request submission, template to be used, and process to handle the request.

Step #3 – Test Environment:

Test environment setup should outline information about number of environments and required setup for each environment. E.g. one test environment for functional test team and another for UAT team. Define number of users supported on each environment, access roles for each user, software and hardware requirements like operating system, memory, free disk space, number of systems etc.

Defining test data requirements is equally important. Provide clear instruction on how to create test data (either generate data or use production data by masking fields for privacy).

Define test data backup and restore strategy. Test environment database may run into problems due to unhandled conditions in the code. I remember the problems we faced on one of the projects when there was no database backup strategy defined and we lost whole data due to code issues. Backup and restore process should define who will take backups, when to take backup, what to include in backup, when to restore database, who will restore it and data masking steps to be followed if database is restored.

Step #4 – Testing Tools:

Define test management and automation tools required for test execution. For performance, load and security testing describe the test approach and tools required. Mention whether it is open source or commercial tool and how many users are supported on it and plan accordingly.

Step #5 – Release Control:

As mentioned in our last UAT article, unplanned release cycle could result into different software versions on test and UAT environments. Release management plan with proper version history will ensure test execution of all modifications in that release.

E.g. set build management process which will answer – where new build should made available, where it should be deployed, when to get new build, from where to get the production build, who will give go, no-go signal for production release etc.

Step #6 – Risk Analysis:

List all risks that you envision.  Provide a clear plan to mitigate these risks and also a contingency plan in case if you see these risks in reality.

Step #7 – Review and Approvals:

When all these activities are defined in test strategy plan it needs to be reviewed for sign-off by all entities involved like project management, business team, development team and system administration (or environment management) team. Summary of review changes should be tracked at the begging of the document along with approver name, date and comment. Also it’s a living document meaning this should be continuously reviewed and updated with the testing process enhancements.

Conclusion:

 

Test strategy is not a piece of paper. It’s the reflection of whole QA activities in software testing life cycle. Refer this document time to time in test execution process and follow plan till the software release. When project nears the release date it’s fairly easy cut on testing activities by ignoring what you have defined in test strategy document. But it is advisable to discuss with your team whether or not cutting down on any particular activity will help for release without any potential risk of major issues post release.

Most of the agile teams cut down on writing strategy document as team focus is on test execution rather than documentation. But having a basic test strategy plan always help to clearly plan and mitigate risks involved in the project. Agile teams can capture and document all high level activities to complete test execution on time without any issues.

I’m sure developing a good test strategy plan and committing to follow it will definitely improve testing process and quality of the software. It would be my pleasure if this article inspires you to write a test strategy plan for your project!

 

Test Plan

 

TEST PLAN DEFINITION:

 

A Software Test Plan is a document describing the testing scope and activities. It is the basis for formally testing any software/product in a project.

ISTQB Definition

  • test plan: A document describing the scope, approach, resources and schedule of intended test activities. It identifies amongst others test items, the features to be tested, the testing tasks, who will do each task, degree of tester independence, the test environment, the test design techniques and entry and exit criteria to be used, and the rationale for their choice,and any risks requiring contingency planning. It is a record of the test planning process.
  • master test plan: A test plan that typically addresses multiple test levels.
  • phase test plan: A test plan that typically addresses one test phase.

 

 

 

 

 

 

TEST PLAN TYPES:

 

One can have the following types of test plans:

  • Master Test Plan: A single high-level test plan for a project/product that unifies all other test plans.
  • Testing Level Specific Test Plans:Plans for each level of testing.
    • Unit Test Plan
    • Integration Test Plan
    • System Test Plan
    • Acceptance Test Plan
  • Testing Type Specific Test Plans: Plans for major types of testing like Performance Test Plan and Security Test Plan.

 

 

 

TEST PLAN TEMPLATE:

 

The format and content of a software test plan vary depending on the processes, standards, and test management tools being implemented. Nevertheless, the following format, which is based on IEEE standard for software test documentation, provides a summary of what a test plan can/should contain.

Test Plan Identifier:

  • Provide a unique identifier for the document. (Adhere to the Configuration Management System if you have one.)

Introduction:

  • Provide an overview of the test plan.
  • Specify the goals/objectives.
  • Specify any constraints.

References:

  • List the related documents, with links to them if available, including the following:
    • Project Plan
    • Configuration Management Plan

 

Test Items:

  • List the test items (software/products) and their versions.

Features to be Tested:

  • List the features of the software/product to be tested.
  • Provide references to the Requirements and/or Design specifications of the features to be tested

Features Not to Be Tested:

  • List the features of the software/product which will not be tested.
  • Specify the reasons these features won’t be tested.

Approach:

  • Mention the overall approach to testing.
  • Specify the testing levels [if it's a Master Test Plan], the testing types, and the testing methods [Manual/Automated; White Box/Black Box/Gray Box]

Item Pass/Fail Criteria:

  • Specify the criteria that will be used to determine whether each test item (software/product) has passed or failed testing.

Suspension Criteria and Resumption Requirements:

  • Specify criteria to be used to suspend the testing activity.
  • Specify testing activities which must be redone when testing is resumed.

Test Deliverables:

  • List test deliverables, and links to them if available, including the following:
    • Test Plan (this document itself)
    • Test Cases
    • Test Scripts
    • Defect/Enhancement Logs
    • Test Reports

 

Test Environment:

  • Specify the properties of test environment: hardware, software, network etc.
  • List any testing or related tools.

Estimate:

  • Provide a summary of test estimates (cost or effort) and/or provide a link to the detailed estimation.

Schedule:

  • Provide a summary of the schedule, specifying key test milestones, and/or provide a link to the detailed schedule.

Staffing and Training Needs:

  • Specify staffing needs by role and required skills.
  • Identify training that is necessary to provide those skills, if not already acquired.

Responsibilities:

  • List the responsibilities of each team/role/individual.

Risks:

  • List the risks that have been identified.
  • Specify the mitigation plan and the contingency plan for each risk.

Assumptions and Dependencies:

  • List the assumptions that have been made during the preparation of this plan.
  • List the dependencies.

Approvals:

  • Specify the names and roles of all persons who must approve the plan.
  • Provide space for signatures and dates. (If the document is to be printed.)

 

 

TEST PLAN GUIDELINES

 

  • Make the plan concise. Avoid redundancy and superfluousness. If you think you do not need a section that has been mentioned in the template above, go ahead and delete that section in your test plan.

  • Be specific. For example, when you specify an operating system as a property of a test environment, mention the OS Edition/Version as well, not just the OS Name.

  • Make use of lists and tables wherever possible. Avoid lengthy paragraphs.

  • Have the test plan reviewed a number of times prior to baselining it or sending it for approval. The quality of your test plan speaks volumes about the quality of the testing you or your team is going to perform.

  • Update the plan as and when necessary. An out-dated and unused document stinks and is worse than not having the document in the first place.

Test Strategy


A Test Strategy document is a high level document and normally developed by project manager. This document defines “Software Testing Approach” to achieve testing objectives. The Test Strategy is normally derived from the Business Requirement Specification document.

 

The Test Strategy document is a static document meaning that it is not updated too often. It sets the standards for testing processes and activities and other documents such as the Test Plan draws its contents from those standards set in the Test Strategy Document.

Some companies include the “Test Approach” or “Strategy” inside the Test Plan, which is fine and it is usually the case for small projects. However, for larger projects, there is one Test Strategy document and different number of Test Plans for each phase or level of testing.

 

Components of the Test Strategy document

  • Scope and Objectives
  • Business issues
  • Roles and responsibilities
  • Communication and status reporting
  • Test deliverability
  • Industry standards to follow
  • Test automation and tools
  • Testing measurements and metrices
  • Risks and mitigation
  • Defect reporting and tracking
  • Change and configuration management
  • Training plan

 

Test Plan


The Test Plan document on the other hand, is derived from the Product Description, Software Requirement Specification SRS, or Use Case Documents.
The Test Plan document is usually prepared by the Test Lead or Test Manager and the focus of the document is to describe what to test, how to test, when to test and who will do what test.

 

It is not uncommon to have one Master Test Plan which is a common document for the test phases and each test phase have their own Test Plan documents.

 

There is much debate, as to whether the Test Plan document should also be a static document like the Test Strategy document mentioned above or should it be updated every often to reflect changes according to the direction of the project and activities.


My own personal view is that when a testing phase starts and the Test Manager is “controlling” the activities, the test plan should be updated to reflect any deviation from the original plan. After all, Planning and Control are continuous activities in the formal test process.

  • Test Plan id
  • Introduction
  • Test items
  • Features to be tested
  • Features not to be tested
  • Test techniques
  • Testing tasks
  • Suspension criteria
  • Features pass or fail criteria
  • Test environment (Entry criteria, Exit criteria)
  • Test deliverables
  • Staff and training needs
  • Responsibilities
  • Schedule

This is a standard approach to prepare test plan and test strategy documents, but things can vary company-to-company

Print Print | Sitemap
© 2016 Automation Learn. All rights reserved.