Engineering quality in software

The idea of property based testing (PBT) is to express the properties that a program must satisfy in the form of input-output relations, and present the general structure of valid input messages, while letting the system handle the creation of progressively more complex test cases in an attempt to find a counterexample for the property.

A 2012 paper tried to explore the possibility of using PropEr in conjunction with their own tool (source code not available) to test web-services.

PropEr is an Erlang-based PBT tool that accepts property information (i.e. general structure of valid inputs, and input-output relations) and creates progressively more complex test cases, executes them and then monitors the response to make sure it conforms with the specified properties. In addition, should a failing test case occur, PropEr will try to locate the part of the test case that is actually responsible for the fault by shrinking (i.e. simplifying) the offending test case. Below is my attempt to understand the paper.

1. getURI / HTTP GET to fetch the WSDL

Given a URI, testing starts by obtaining the WSDL specification of the web service. The WSDL describes the web service: its operations, input messages, output messages, bindings, endpoints, and embedded or referenced XSD schema.

2. xmerl extracts type information

Once the WSDL is available, one part of the tool sends the schema/type part to xmerl (Erlang's XML parser) which parses the XSD schema inside the WSDL and extracts type information.

3. Yaws extracts operation information

In parallel, the tool uses Yaws (one of Erlang HTTP web server) to extract information about the supported SOAP operations.

4. Test file generator creates proper_ws_autogen.erl

A file with test case generators and properties is created which typically includes

• PropEr generators
• PropEr properties
• helper call functions
• imports/includes/header references
• answer extraction helpers

5. PropEr runs the generated property

The file created in 4. above is compiled by PropEr which then proceeds to generate random test cases, invoke the web service using Yaws as a SOAP wrapper and then analyzes the result.

6. SOAP request is sent to the web service

For each generated test case, PropEr does not directly call the web service. It uses the generated call function (from 4. above), which uses Yaws. Yaws wraps the generated Erlang data into a proper SOAP request and sends it to the real service

7. SOAP response comes back and PropEr evaluates it

The web service returns a SOAP response. Yaws decodes it and returns an Erlang tuple to PropEr. PropEr uses the tuple along with the property (from 4. above) to evaluate that no exceptions (SOAP faults) were generated and that the XML responses were well formed.

The upshot

The tool supposedly created by the authors can test whether the SOAP response is structurally correct but not whether it is functionally correctly i.e. whether a business logic has been satisfied.