The Isartor test suite is named after the location of the first meeting of the Technical Working Group, a medieval gate towards the river Isar in the center of Munich/Germany. This test suite comprises a set of files which can be used to check the conformance of software regarding the PDF/A-1 standard. More precisely, the Isartor test suite can be used to “validate the validators”: It deliberately violates the requirements of PDF/A-1 in a systematic way in order to check whether PDF/A-1 validation software actually finds the violations.
The Isartor test suite has been constructed according to the following basic principles:
- Atomic: A test file attempts to examine a particular aspect of the standard. Wherever possible test files avoid to touch more than one aspect of the standard.
- Complete: The Isartor test suite provides full coverage of all requirements in the PDF/A-1 standard.
- Self-documenting: The test files contain intrinsic information about the expected result of performing the test.
- Traceable: Each test is based on a specific clause in PDF/A-1 which can easily be identified.
- Reproducible: All test files (with only a few exceptions) have been created programmatically with a documented change history. This creation process support reproducibility.
Through a rigid quality assurance and testing process a lot of effort has been spent to make sure that all test files adhere to these basic principles.
What the Test Suite checks
The Isartor test suite for PDF/A-1b deliberately violates each requirement of the PDF/A-1b standard in order to check whether validation software detects all possible kinds of violation of the standard requirements, and whether it provides appropriate explanation for rejected nonconforming documents.
A PDF/A validator which completely passes the Isartor test suite is known to implement all required checks.
For the sake of simplicity, only a single aspect of the standard is violated at a time. However, in a few cases this is not possible because of interdependencies among multiple aspects of the standard.
The test requirements (assertions) are the inverse of requirements in the standard. For example, PDF/A-1 mandates that fonts be embedded, so the test suite contains several documents where the fonts are deliberately not embedded. PDF/A validators must report this standard violation in order to pass the test.
Who created the Test Suite?
The Isartor test suite is a project of the PDF/A Competence Center, and was developed by its Technical Working Group (TWG). Members of the TWG are volunteers from member companies of the PDF Association. All companies who send TWG delegates are software vendors. Individual TWG members are experts in the field of PDF and/or standardization. Most members are involved in the design, development, and testing of software for creating, processing, or validating PDF/A documents. Several TWG members are also members of TC 171 SC2 WG 5, the ISO committee which defined the PDF/A family of standards.
Design and structure of the Isartor test suite have been devised by the TWG. Test files have been created by a core group after thoroughly discussing the respective criteria in the standard. All test files have been formally reviewed by at least two other members of the TWG.