Software Testing
A test case is a repeatable execution situation of a software system that produces recordable outcomes. A test is a particular attempt of a test case:
- the outcomes may be expected
- specified in advance, e.g., we expect passing 1+1 to a calculator to return 2
- generally boolean outcoms (pass or fail)
- the outcomes may be measurement results
- e.g., we want to find the time it takes to compute 1+1
- the outcomes should testify to some software quality
- e.g., correctness, but also efficiency, usability, etc
A Classification Of Tests
Based on a 5W+H approach by Ray Sinnema (archived)
- Who (Programmer vs. customer vs. manager vs…)
- What (Correctness vs. Performance vs. Useability vs…)
- When (Before writing code or after)
- Or even before architecting!
- Where (Unit vs. Component vs. Integration vs.System)
- Or lab vs. field
- Why (Verification vs. specification vs. design)
- How (Manual vs. automated)
- On demand vs. continuous
Who
Tests give confidence in the system. I.e., they are evidence of a quality.
Who is getting the evidence?
- Users? Tests focus on external qualities
- Can I accept this software?
- Programmers? Tests focus on internal qualities
- Can I check in this code?
- Managers? Both?
- Are we ready to release
What
Which qualities am I trying to show?
- Internal vs. external
- Functional vs. non-functional?
- Most developer testing is functional (i.e.,correctness)
- And at the unit level
- Does this class behave as designed
When
When is the test written?
- Before the code is written?
- After the code is written?
Where
Different types of testing
- Unit Testing
- Component testing
- Integration testing
- Regression testing
- System testing
Unit Testing
Unit testing is the execution of a complete class, routine, or small program that has been written by a single programmer or team of programmers, which is tested in isolation from the more complete system.
Component testing
Component testing is the execution of a class, package, small program, or other program element that involves the work of multiple programmers or programming teams, which is tested in isolation from the more complete system.
Integration testing
Integration testing is the combined execution of two or more classes, packages, components, or subsystems that have been created by multiple programmers or programming teams. This kind of testing typically starts as soon as there are two classes to test and continues until the entire system is complete.
Regression testing
Regression testing is the repetition of previously executed test cases for the purpose of finding defects in software that previously passed the same set of tests.
System testing
System testing is the execution of the software in its final configuration, including integration with other software and hardware systems. It tests for security, performance, resource loss, timing problems, and other issues that can’t be tested at lower levels of integration.
Why
- Verification (or validation)
- Does the system possess a quality to a certain degree?
- Design
- Impose constraints on the design space
- Both structure and function
- Comprehension
- How does the system work?
- How do I work with the system?
How
- Manual
- Typically interactive (Human intervation for more than initiation)
- Expectations flexible
- Automated
- The test executes and evaluates on initiation
- Automatically run (i.e., continuously)