What is Software Testing
Software Testing is the process of evaluating and verifying that a software application or system meets specified requirements and functions correctly. The primary goal of software testing is to gather information about the product and its users, analyze requirements from the user’s perspective, identify defects, ensure the quality of the software, and validate that it performs as expected in various scenarios. Software testing is a critical part of the software development lifecycle, helping to deliver a stable and effective product.
Testing is the process of evaluating a product by learning about it through exploration and experimentation, which includes to some degree: exploring, discovering, analyzing, learning, and reporting about the product to reveal new information. so that they can make more empirically informed decisions.
Tester can explore products deeply, imaginatively, and suspiciously to help find issues or problems that discourage the end user.
To test is to evaluate a product by learning about it through experiencing, exploring, and experimenting.
What is Software Testing Life Cycle (STLC)
The Software Testing Life Cycle (STLC) is an approach to test an application to make sure that it meets the requirements or specification. It is a process that follows a series of steps or phases, and each phase has specific objectives and deliverables.
STLC is part of the SDLC (Software Development Life Cycle). It includes both verification and validation. Therefore, STLC consists of only the testing phases. The main aim of STLC is to discover discrepancies in the application as early as possible so quality products can be delivered to the end users. STLC begins once requirements are shared by the Client with the team.
STLC consists of below 6 phases:
PHASE 1
Requirement Analysis
In this phase, the tester understands the requirements, identify testable aspects. If there is any gap, missing requirements, or an incomprehensible requirement then the test team can consult with the client to clarify and understand the E2E flow of the requirement. Also identify any risk may impact on testing process.
PHASE 2
Test Planning
In this phase, a detailed test plan document is being created to outline entire testing process which highlights testing scope, types of testing needs to be carried out, tools needed, Identify test environments required, roles & responsibilities of testing team, schedule, and cost associated with it.
PHASE 3
Test Case Development
In this phase, the test team creates detailed test scenarios and test cases. Also prepares the test data to test these scenarios. Review of these test scenarios carried out by the internal team and Client to get better test coverage of the application.
PHASE 4
Test Environment Setup
In this phase, testing environments are configured. Sanity test is performed to ensure that all configured environments are working as expected.
PHASE 5
Test Execution
The test team starts executing test cases once the feature deploys into the environment. Here, expected test results are compared to actual test results.
Defect Reporting and tracking – If any test case is not working as per specification then defect needs to be raised to track and fix it. All defects needs to be tracked until it marks to Close status.
PHASE 6
Test Closure
Test Closure summary report is created which gives brief information on entire testing process, test execution result/report and defect report.
Below diagram illustrate the Software Testing Life Cycle (STLC)
Types of Testing
There are many different types of testing. At a very high level, there are two testing commonly refers –
- Functional Testing
- Non-functional Testing
Functional Testing
| Unit Testing | A unit is the smallest testable module of an application. Testing individual methods and functions of the classes and modules used by software application OR test teams perform such tests to ensure that individual parts of an application work correctly |
| Integration Tests | Validates different modules used by the software application work together properly. The test team continues to add modules one by one until they have tested the entire application. |
| System Testing/ E2E Testing | Evaluates how the complete and fully integrated application works. Here test team just verifies workflow of the application to check whether application is working smoothly without any interruption. |
| Regression Testing | Verify new changes in application code, like new feature or defect fixes or any small change, do not impact existing functionality. Overall working of existing functionality does not break due to deployment of new changes. |
| Smoke Testing | Perform as a preliminary check to verify whether all the application’s main features work correctly. Test teams usually use smoke tests after deploying a new build to evaluate the system’s basic functionality quickly before running more thorough. |
| Sanity Testing | Sanity tests perform on a specific part of an application after a new build deployment. It verifies deeper into the areas impacted by recent/new changes, ensuring that specific functionality or feature is working as expected. |
| Acceptance Testing | Acceptance testing is performed by the customer. These tests help to ensure if an application meets all business requirements and ready to deliver.This is the final testing stage before an application is deployed to production. |
| Usability Testing | Usability testing is a method of testing the functionality of the application by observing real end users. Usability tests evaluate the overall user experience, user friendliness and how easily users can interact with the application. |
| Exploratory Testing | Testers actively explore the application to identify issues and assess user experience without following any specific test scenarios/test cases. This Testing focuses on tester’s experience, decision making and intuitions. |
Non-functional Testing
| Performance Testing | Performance testing is a testing method that evaluates how the system behaves and performs. It measures and examines responsiveness, stability, scalability, reliability, speed, and resource usage of your software and infrastructure. It evaluates the application’s overall efficiency. |
| Load Testing | Application’s loading capability is being tested during load testing. It checks how the application behave when number of concurrent users performs transactions over a certain period of time. |
| Stress Testing | This testing checks the upper limits of your system by testing it under extreme loads. |
| Security Testing | Security testing checks to find flaws and security vulnerabilities exist in the application. The main goal of this testing is to identify and protect against unauthorized access, data breaches, and other security-related issues. |
| Volume Testing | This testing refers to testing the data load capabilities of the application. This testing checks the system’s performance under huge data volume to get response time of the application. |
| Recovery Testing | This testing checks how well an application is able to recover from crash, Hardware failure, software failure etc. Key goal of this testing is to ensure that data backups are recoverable. |
Agile Methodology
Agile methodology is a project management framework that splits projects into several phases, commonly known as sprints. It is more focused on continuous improvement as it is an iterative process that involves planning, execution and evaluation.
Agile Testing
Agile testing is an inseparable part of agile methodology. In agile, testing is one key aspect of the software development lifecycle. It supports Continuous integration and continuous delivery of applications.
Just as the Agile development methodology breaks down larger components of feature development into smaller such as user epic, stories, tasks, and technical requirements, an Agile testing methodology breaks down larger testing components into smaller, more specific tests with a narrower focus on a specific user story, task, technical requirement, etc.
This allows testing to happen in a more timely and efficient manner.
Automation Testing
Introduction
Test automation is the practice of using software tools, libraries and scripts to automate human driven manual processes in software applications. Automation can assist, accelerate and enable all kinds of testing tasks that are not checks :
- Configuring/Generating data
- Collecting data
- Configuring test environment
- Creating & executing test suites or test scripts
- Installing the product/application
- Importing test cases
- Logging
- Visualizing output
- Automating repetitive tasks
- Generating Reports
Manual testing vs Automation Testing
Both manual testing vs automation testing are not equivalent. ‘Automation in tests’ always helps to complete testing tasks faster or view unexpected results quickly but actually does not perform validation. At the end of any tasks, it needs verification to make sure tasks are completed as per requirement or expectation and this needs manual verification. Automation in tests is always supporting hand for testing.
Automaton in tests always helps testing to improve efficiency, accuracy, and repeatability. Tools can aid testing powerfully.
As Cem Kaner so perfectly says, “When we’re testing, we’re not talking about something you’re doing with your hands or tools, we’re talking about something you do with your head.“
What kinds of tests should be automated?
Below are few test or application can be part of automation
- Repetitive Tests : Ideal for tests that need to be run frequently and repeatedly such as Sanity, test Smoke test , regression test.
- Business-Critical : Useful for business critical flow which can cause human error and time consuming.
- Stable requirement: Ensures that automated tests run smoothly and produce reliable results. Reduces time and effort of analysis and re-running failed tests
- High-Volume Applications: Suitable for applications requiring extensive testing due to their size and complexity
- Multiple Data-set: Same test needs to be executed with different and huge dataset
- Run Against Multiple Configurations: Tests that need to be run on different OS or browsers and configurations.
- Regression Testing: Essential for ensuring that new code changes do not break existing functionality.
What kinds of tests are best done manually?
Everything can not be automated. Below are tests or application can be done manually which does not give ROI in the automation
- Constantly changing requirements
- Exploratory test
- New Functional test
- Subjective validation (validity of words, statements, or initials)
- Ad-hoc tests
- User Experience (When there is matter of user experience then nothing can compete with the human eye)
- Low return on investment
Benefits of Automation Testing
Cost Savings
Improved Accuracy
Increased Speed
Improved Test Reusability
Enhanced Test Coverage
Automation Framework
A testing framework is used as a guide of rules and best practices to be followed and used for building test cases. It helps testers to be more efficient when designing and developing test cases. This type of test reduces time, costs, the possibility of errors and improves the precision of the tests.
These rules or guidelines include coding standards, test-data handling methods, object repositories, processes for storing test results, or appropriate ways on how to access external resources like csv, excel, PDF, etc., maintaining file structure.
Using a framework in test automation is important because it mainly improves the automation testing team’s efficiency, ensure maximum test coverage within minimum time period
Types of Automation Frameworks:
Below are common types of automation frameworks. Every framework has different architecture, benefits. One or more of the following automation frameworks can be utilized for developing and executing tests, generating reports and maintaining all test suites
1. Linear Automation Framework
This is the Record and playback framework. Testers just record each and every step and then play the script automatically to execute the test. Testers do not need to write code to test it. Just record it and play to execute it. This is the fastest way to generate and execute test scripts and However, developed scripts are not reusable as most of the user entered data is hard coded in the test script. Also maintenance is really tough and rework.
2. Modular Based Testing Framework
In this framework, the entire application is divided into different modules. Testers need to create separate scripts for every module and then combine to build E2E tests in sequential order. All modules would not be affected if any changes were made in the application. if any changes made in the application then only the module and it’s associated test script will be changed and rest of the application will remain untouched.
3. Data-Driven Framework
This framework is commonly used. When a single test case needs to be executed multiple times with different dataset. This framework separates the test data from the test script or logic layer. It means that testers can store test data externally or separate files so test data is not hard-coded in the script itself.
4. Keyword-Driven Framework
Every test case that needs to be run maintains a series of instructions in consecutive order in the table. It is called a Keyword. All keywords are stored in an external data table to keep the test data and script logic separate. These keywords represent the various actions being performed to test.
5. Hybrid Testing Framework
This framework usually combines the data-driven and keyword-driven frameworks to execute the most out of the frameworks above. Hybrid framework is a combination of any of the previously mentioned frameworks set up to utilize the benefits of some and mitigate the weaknesses of others.
Test Automation Tools/Library
Defect Management and Life Cycle
A defect is the unexpected behavior of the software. If the functionality or feature does not work as per the end user’s requirement then it is called a defect. It affects the end user experience and usability while using the application.
Defects can be observed due to requirement mismatch, poor design, incorrect data, unexpected errors, technical fault, human error while interacting with application.
Types of Defects
Following are some of the types of defects in application which tester can find while performing Static testing or functional testing
- Arithmetic defects
- Logical defects
- Syntax defects
- Interface/UI defects
- Design defects
- User Level defects
- Navigation defects
- Integration defects
- Performance defects
- Security defects
Defect Severity and Defect Priority
Both are commonly used when a tester finds and tracks defects. These are important factors used to prioritize and classify issues found while testing an application
Defect Severity :
Severity refers to the impact of a defect on the functionality or the end-user experience. It measures how severe the issue is and how critical it is to fix it. It is classified into 3 levels such as :
- Minor
- Major
- Critical
Defect Priority
Priority refers to the urgency which a defect needs to be fixed. It measures how quickly the issue should be resolved based on the business impact. It can be classified into different level
- Highest
- High
- Medium
- Low
- Lowest
Defect Management
Defect management is a process used in software development lifecycle to identify, raise, prioritize, track, fix and resolve. This is an iterative approach that continues throughout the development lifecycle to ensure that application meets the user’s requirement or specification.
It is impossible to say that application is complete bug free but testers can reduce the issues by fixing and resolving it Defect management Process :
This process helps to find, report, fix and resolve defects. The software defect management process follows few steps such as
- Defect identification – Defect can be identified and tracked while performing different testing stages such as unit testing, integration testing, Sanity Testing, System testing, Acceptance Testing etc.
- Defect logging (NEW/TODO) – Defect can be logged by providing details such as Description, Steps to produce, Environment, Logs, Screenshots, Videos etc into Defect Management tool
- Defect triage (OPEN) – Identified defects are discussed to determine whether a fix is required, evaluating defect priority & severity, and the resources required to fix it
- Defect Assignment (ASSIGNED) – Defects are assigned to developers to resolve it
- Defect resolution (FIXED) – The development team finds the root cause of the defect and resolves it by fixing the code.
- Defect verification (RETEST) – Once the defect is fixed, it is assigned to testers to retest and ensure that it has been fixed and working as expected. Also, make sure that no new defects are introduced.
- Defect closure (CLOSE) – Once the defect has been verified, it is closed and the status is updated in the Defect Management Tool
Defect Life Cycle
A defect/bug life cycle is the sequence of steps a defect goes through from its identification to its resolution in software development. These steps standardize the defect management process; teams can manage or track all defects easily, and coordinate and communicate the current status of the defect.
Defect status is the current state from which the defect is currently going through. The number of states the defect goes through varies from project to project. Below diagram represents different states of the defect which usually follows during defect lifecycle
Test Documentation
Test documentation refers to artifacts that may be produced before, during, and after the testing phase to enable successful testing.
It is a collection of documents which allows the test team to describe and document test planning, test design, test execution , test results , test closure. Test documentation makes all testing phases easy and verifiable because all information such as schedule, requirements, issues, risks, environments, resources, test coverage etc are described in a detailed manner. Therefore, testing activities consume 25-30% for documentation.
| Test Strategy | This document describes the high-level approach to testing for a project, guiding the overall testing process. |
| Test Plan | A test plan document is a detailed overview of all activities involved in the testing phase. It contains the scope, out of scope, approach, resources, schedule, environment, types of testing, Risk etc. of testing activities. |
| Requirements Traceability Matrix (RTM) | This document contains a mapping between requirements and test cases. |
| Test Scenario | Test Scenario is a key functionality which could be verified by referring to one or more Test cases. |
| Test Case | Test case is a detailed description of the feature, test data, exact steps needed to be performed while executing the specific feature. |
| Defect Report | This document contains all the defects or issues observed during testing. |
| Test Closure | This report is generated after completing testing. It is a high-level document which summarizes testing activities conducted as well as the test result defect list. |
Testing Challenges
Below are the common challenges faced by testing team –
- Dealing with Changes – Requirements can be continuously changing and testers need to keep up with these changes. Incorporating these changes into the existing test cases/scenarios needs more effort than before. It is really important to have a process in place to deal with the changes.
- Time Constraints – One of the biggest challenges in the testing is time constraints. Sometimes there is not enough time to test a feature before it needs to be deployed in production. Due to tight deadlines, testing does not get sufficient time.
- Communication Issue – Communication problems commonly happen between the development team and the testing team. There might be a misunderstanding about the requirements or about the features that need to be tested. Another type of communication issue can occur within the testing team itself. The members of the testing team might not be on the same page when it comes to the approach or the process of software testing. It is important to have clear communication between or within the team to avoid these issues.
- Lack of Resources – This can include both human resources and software resources. There might not be enough or skilled testers available to test the software. Or, there might not be enough software licenses available for the testing tools.
- Missing or Insufficient Documentation – Documentation plays a key role in testing. Unavailability of any documentation makes it difficult to understand the requirements or to know what needs to be tested.
Another challenge related to documentation is outdated documentation. This can make it difficult to know what is currently working and what needs to be updated in the documentation. It is important to keep the documentation up-to-date with the software changes. - Unstable Environment -The test teams depend on support teams to deal with hardware, software,or upgrade challenges.This often takes time and causes delays. Dedicated test environments and support teams are essential.
- Test Data Management – An application can be tested with a different set of data. Missing any data set leads to a bigger impact of an application. Need to analyze and create a data set before starting the test execution phase.
- Inadequate Testing – This can happen when the software is not tested thoroughly enough before it is deployed to the end users. This can lead to production defects that could have been avoided if more time was spent on testing. It is important to make sure that the software is effectively tested before it is released. This can help to avoid software defects and improve the quality of the software.
About the Authors
Shilpa Waghmare
Passionate Software Quality Assurance Engineer with over 10 years of expertise in diverse testing methodologies, including Automation, ETL, Functional, and Database Testing. Known for a solution-focused approach, strong problem-solving abilities, and exceptional analytical and communication skills. Shilpa holds a Bachelors and Masters in Computer Applications.
Chinkal Patel
Ex-Deloitte Consultant with 20+ years of cross-functional experience in diverse industries. Over 10+ years of experience implementing software systems and delivering large-scale digital transformations. Chinkal has led numerous discovery and specifications workshops across multiple industries spanning across Canada and USA. Chinkal brings big consulting experience in managing large scale implementations, with strong expertise in group decision facilitation, training, change management planning and strategic project management.
