The purpose of rating test quality
The quality of psychodiagnostic instruments (referred to as 'tests') is rated by the COTAN. The purpose of these ratings is twofold. First, the ratings inform test users about the quality of available instruments. This information can be helpful in choosing an instrument. Second, the ratings supply feedback to test developers about the quality of their products. For them the rating system can serve as a guideline for the development of tests and the writing of manuals.

The rating system
The text of the rating system as used by the COTAN is public and can be purchased from NIP. The complete text is published in Dutch. Download an English translation of the rating system. This rating system provides grades for seven criteria:

• Theoretical basis and the soundness of the test development procedure 
• Quality of the testing materials 
• Comprehensiveness of the manual 
• Norms 
• Reliability 
• Construct validity 
• Criterion validity

 
Which instruments are reviewed?
A test is admitted to the rating procedure when the test is made available for actual use by the author or publisher. As a consequence the rating may take place when the documentation concerning the test is still incomplete, for example because a Technical Manual is lacking. However, the COTAN takes the position that test users should be informed about the actual quality of a test as soon as possible.

The tests come from different sources. Sometimes they are submitted by test publishers, alternatively they may be bought by COTAN, provided by individual test authors, or be obtained from published journal articles or dissertations that have been drawn to the attention of the literature researcher. However, not all Dutch tests are rated by the COTAN.

Criteria rating procedure
To be included in the rating procedure the following criteria must be met: 

• The test must be available to others than the test author alone 
• The test must be in the Dutch language 
• At least some Dutch research on norms, reliability or validity have to be reported 
• The administration procedure must be standardised

Rating tests in 7 steps

1. Sending your material
The material is sent to the Information and Documentation Centre of the COTAN. This includes all material that may be relevant for the rating, such as the User's Guide and the Technical Manual, the answer sheets and scoring keys, and all relevant articles, reports, dissertations, etc. For computer-based testing the relevant material also concerns diskettes and CD-ROM's. In exceptional cases (for instance, when the testing equipment cannot be moved, or when PCs with specific qualities are required) the test can be rated on location.

2. Two raters
The material is sent successively to two raters on the condition of strictest secrecy. The material is never lent out or given to third parties for inspection. It is only used for rating by the COTAN and for the description of the test in the COTAN Documentation. It is then filed in the COTAN archive.

3. Anonymous rating
Successively both raters rate the test independently. The selection of a particular rater depends on his or her expertise. However, a rater will never rate his or her own test, a test developed by a colleague, or a test of a competing company. Raters always remain anonymous, as is the case in review procedures for scientific journals, in order to prevent possible arguments between them and the test authors.

4. Discrepancies
The independent coordinator of the rating procedure discusses any discrepancies in the ratings with both raters. The coordinator will, if necessary, call in a third expert.

5. Responding to the rating
Prior to publication the grades for the seven criteria are sent to the test author for comments. The author is given one month to respond. Depending upon this response some criteria may be re-rated (by the same raters). For instance, grades may be adjusted if the results of research not previously known to the COTAN are provided.

6. Final rating
The final grades are established and published in COTAN Documentation (Dutch only).

7. Reassessment
As a result of the publication of new research data or the release of a new manual, a test may be reassessed. In this case, the rating process will be repeated, revisiting steps one through six. However, a reassessment procedure will only be started at least one year after the previous rating.


Costs and duration of the procedure
The rating procedure is free of charge for the test author and/or the test publisher. Because the interests of many persons and many institutions are involved, the COTAN works according to a careful and sound procedure. This may take some time, though the COTAN aims at completing a procedure within six months.