Reliability testing
What is reliability testing?
‘Reliable’ means ‘something that can be trusted or believed because it is working in the way you expect’. Reliability testing is testing the software to check software reliability and to ensure that the software performs well in given environmental conditions for a specific period without any errors.
Reliability testing ensures that the software is reliable, it is performing in an expected manner and it satisfies the client’s requirements.
Some of the objectives of reliability testing are to find out the pattern of repetitive failures, the number of failures that occur in a specific amount of time, the main cause of the failure and to test the modules once defects have been eliminated.
Types of reliability testing
Following are the three types of reliability testing:
Feature testing
In feature testing, all the features of the software are checked, and it involves the following steps.
First, test each feature of the software documented in software requirements in isolation.
Test two individual features together and reduce the interaction between them as much as possible.
Make sure each feature is working in the desired manner.
Load testing
Load testing is done to ensure that the system is able to handle the required load and does not suffer a malfunction. In this type of testing, multiple users will be using the system and doing similar kinds of operations simultaneously to check its consequences on the system performance.
Regression testing
Regression testing is done whenever any change is made in the system i.e. new feature is added or any existing feature is changed. Regression testing checks that the changes do not affect other unchanged parts of the system.
Categories of Reliability Testing
Following are the three categories of reliability testing:
Modeling
In software modeling techniques we observe the failure data and analyze them. It is of two types-
Prediction modelling
Such models use old historical data and predict future failure behavior. Generally, these models are used before the beginning of the development phase and software reliability can be predicted even before the coding starts. Examples of prediction models are Musa’s Execution Time Model and Putnam’s Model.
Estimation modelling
Estimation models use data from ongoing software development and estimate the failure behavior of the system. Generally, these models are used once the development process has started and enough data has been collected. Examples of such models are the Weibull Distribution Model and Exponential Distribution Models.
It is important to note that no individual model is best in all situations. Based on the software requirement, the correct model will have to be chosen by the team.
Measurement
Measuring the reliability of software is difficult but we can identify the characteristics related to software reliability and based on them, reliability metrics can be measured. Let’s discuss such metrics in this section.
Following are the sub-categories of the software reliability measurement practices:
Product metrics
Product metrics are computed from data collected from source code, requirements, design models and test cases. These metrics are useful in assessing various software characteristics and gain insight into software quality.
Software size
To measure software size, an important approach is ‘Line of Code’ (LOC). In this approach, lines of source code are counted, the calculation does not include comments and non-executable statements. This is not a perfect method to calculate the software size as every software uses different languages and some parts of the code may have been reused.
Function point metric
Function Point (FP) metric is used to measure developed software’s functionality. Function points are calculated based on countable measures such as Number of external inputs (EIs), Number of external outputs (EOs), Number of external inquiries (EIs), Number of internal logical files (ILFs) and Number of external interface files (EIFs). This metric is independent of the programming language used and it is used to measure the functional complexity of the software.
Complexity
Complexity metrics are used to measure the complexity of the software control structure by simplifying the code into a graphical representation. One such metric is McCabe’s Complexity Metric. In this metric, any software module can be described by a Control Flow Graph (CFG) using nodes ad edges.
Test coverage
Such metrics are used to estimate faults in the system by performing various types of testing on the software.
Project management metrics
Project management metrics are used to evaluate the management process. Better management by developer results in a better development process that will reduce the cost and complete the project on time. Examples of such metrics are Schedule Variance, Effort Variance, Size variance, etc.
Process metrics
Process metrics focus on process quality by measuring attributes of the software development process. Examples of such metrics are Cost of Quality, Defect Density, Testing Efficiency, etc.
Fault and failure metrics
These metrics are used to achieved failure-free software execution. These metrics use faults found by the testing team during testing and failures found by end users; this data is then gathered and analysed. One of the important metrics used for this purpose is MTBF i.e. Mean Time between Failures.
MTBF is an addition of two other metrics MTTF i.e. Mean Time to Failure (The difference of Time between two consecutive failures) and MTTR i.e. Mean Time to Repair (The time required to fix the failure).
In other words, MTBF = MTTF + MTTR
Improvement
After development and before deployment, the software is tested to identify any defects or bugs in the system. There are some analysis tools that can also be used to reduce the possibility of defects such as Fault Tree Analysis (FTA) and Orthogonal Defect Classification (ODC).
Approaches used for reliability testing
Reliability testing is done to check whether the software works in the desired manner without any faults. It is difficult to calculate exact reliability, but it can be estimated using various approaches. A couple of them are discussed below:
Test-retest reliability
In this type of reliability, consistency of results is measured when the same test is performed by or on the same sample at a different time. Once the results are ready, the correlation between the two results is calculated. Usually, if the correlation is greater than 0.8, it is considered a reliable test.
Parallel forms reliability
It is used to measure reliability when two similar kinds of tests are performed by or on the same or two different samples. Response or results from samples are collected and their co-relation is calculated.
Reliability testing tools
Some of the tools or programs available in the market to measure software reliability are mentioned in brief below:
SOFTREL
SOFTRELL LLC is a US-based company founded in 1991. It provides multiple products and services to measure software reliability. Some of the products are ‘Software Reliability Toolkit’, ‘Frestimate Software’, etc.
SoREL
SoREL tool is used for software reliability analysis and prediction. This tool offers four types of reliability growth tests i.e. Arithmetical Mean, Laplace Test, Kendall Test, and Spearmann Test. They allow four types of reliability growth models. SoREL allows two types of failure data processing i.e. inter-failure data and failure intensity data.
SMERFS
SMERFS stands for Statistical Modelling ad Estimation of Reliability Functions for Software. It was created in 1982 and has two versions: SMERFS and SMERFS Cubed. SMERFS collects raw data and after examination, failure and fault detection rates are predicted.
Conclusion
Software reliability is an important area of software quality. Reliability testing is a costly process and it should be done with proper planning. Even with the cost factor, reliability testing gives confidence to the development team and clients that the software being developed is of high reliability and works in the desired manner.