Evaluating evaluation instruments

In the technical world, check lists (CL) have since long been regarded as useful instruments for product/process assessment and evaluation. Recent growing interest for standard-guided inspection, evaluation and certification of software has led to the production and usage of a nice variety of check lists. Although in many technical fields verifying and validating the instruments used to obtain measures is common practice (and standards exist, defining classification and operating requirements for instruments, as in ISO 9001), defining requirements for a CL as an evaluation instrument is little addressed in literature: a CL just gets thought ofand written down. As a result, CLsbeing natural language expressions - are prone to be ambiguous, uncompleted and underspecified and their usage for software evaluation is not always satisfying. In this paper a technique to analyse CLs is presented, based on Natural Languages Understanding (NLU) methods This technique allows the analysis ofa given CL to be performed on the basis ofa quality model, defined as a list of essential requirementsfor CLs. and a grammar, used to parse the CL. Moreover, the definition ofa checker is given, to perform the evaluation ofa CL against the quality model.Examples are provided of the application of the presented techniques on some parts of real check lists used to evaluate quality characteristics ofsoftware products.