Engineering quality in software

The basic objectives in verification and validation of software requirements and design specifications are to identify and resolve software problems and high-risk issues early in the software life cycle.

An IEEE Software paper in its very first issue in 1984 expanded on this.

Verification and Validation Sequence

The above figure shows the basic sequence of functions performed by the V&V agent, specification agent (the analyst or software system engineer), the project manager, and the customer.

The key portions are the iterative loops in which

• the V&V agent analyzes the specifications and issues problem reports to the specification agent;
• the specification agent isolates the source of the problem and develops a solution resulting in an iteration of the specification;
• the project manager and customer approve any proposed iterations that would perceptibly change the requirements baseline;
• the V&V agent analyzes the iterated specification and issues further problem reports if necessary.

Verification and Validation Criteria

The four basic V&V criteria for requirements and design specifications are completeness, consistency, feasibility, and testability.

Completeness

A software specification must exhibit several properties to assure its completeness.

• No TBDs: e.g. The system shall handle a peak load of (TBD) transactions per second
• No nonexistent references: i.e. references in the specification to functions, inputs, or outputs (including databases) not defined in the specification.
• No missing specifications: e.g. no interface specifications
• No missing functions: These are functions that should be part of the product but are not called for in the specification e.g. no backup/ recovery functions
• No missing products: These are products that should be part of the delivered software but are not called for in the specification e.g. test tools

Consistency

A specification is consistent to the extent that its provisions do not conflict with each other or with governing specifications and objectives. Specifications require consistency in several ways.

• Internal: Items within the specification should not conflict with each other e.g. saying at 1 place that interrupt X will have highest precendence and at another that interrupt Y will have highest precedence
• External: Items in the specification should not conflict with external specifications or entities e.g. mentioning that system X will handle 'all' payment functionalities and then mentioning in X's spec that 'some' payment functionalities will be handled by system Y.
• Traceability: Items in the specification have clear antecedents in earlier specifications or statements of system objectives. Particularly on large specifications, each item should indicate the item or items in earlier specifications from which it is derived to prevent misinterpretation or embellishments.

Feasibility

A specification is feasible to the extent that the life cycle benefits of the system specified exceeds its life cycle costs. Furthermore, feasibility implies identifying and resolving any high-risk issues before committing a large group of people to detailed development.

Testability

A specification is testable to the extent that one can identify an economically feasible technique for determining whether or not the developed software will satisfy the specification. To be testable, specifications must be specific, unambiguous, and quantitative wherever possible. For e.g. if the specification is that a query should respond in <= 2 seconds, then it still needs clarification whether response time should include terminal delay, communication delay or just the time in computer processing. Also does the 2 seconds refer to 90% confidence limits or never-to-exceed constraints.

Verification and Validation Techniques

From all the techniques mentioned above, I will touch upon few of the techniques, which I could wrap my head around and believe could be still relevant.

Reading

is having someone other than the originator read the specification to identify any blind spots or misconceptions that the specification developer may have. Reading can also potentially waste time by not looking for anything in particular or looking for the wrong things.

Manual cross-referencing

involves constructing cross-reference tables and various diagrams - e.g. state transition, data flow, control flow, and data structure diagrams - to clarify interaction among specified entities such as functions, databases, and interfacing hardware, software or personnel.

Interviews

Discussing a specification with its originator will identify potential blind spots, misunderstandings, and high-risk issues.

Checklists

are specialized lists, based on experience, of significant issues. One pitfall with checklists is that items may be considered absolute requirements rather than suggestions.

Simple scenarios

are scenarios describing how the system will work once it is in operation.

Prototypes

are working throwaway software, the process of building which will surface ambiguities, inconsistencies, blind spots, and misunderstandings incorporated in the specification.

The upshot

Verification and Validation provide with a good starting point for identifying and resolving software problems while they are still easy to handle.