2.2.7. Principles governing the choice of psychodiagnostic instruments

2.2.7    Principles governing the choice of psychodiagnostic instruments

If the psychologist wants to use psychodiagnostic instruments, it is important that they sort out which instruments are available for the purpose intended. Usually, multiple instruments with the same measuring pretension can be considered. In that case it is especially important, in making a choice, to take into consideration the quality of these instruments. The information that a test publisher provides, such as in the manual, can be helpful here.

In addition, an objective quality assessment by the COTAN can also be made available and serve as a tool for the psychologist in choosing among tests. Detailed information on the working method of the COTAN is available on the NIP’s website. The COTAN test reviews can be consulted by subscribers in the online COTAN documentation. Individual test reviews can also be requested.

The COTAN assesses the quality of a psychodiagnostic instrument based on seven criteria: the principles of test construction, the quality of the test material, the quality of the manual, norms, reliability, construct validity and criterion validity. See BOX 8 for further details. The rating for each of these criteria can be “insufficient”, “sufficient” or “good”. It is thus a nuanced evaluation system: the COTAN does not provide a general quality seal “approved psychodiagnostic instrument”. Neither does it discourage the use of certain instruments, even if they are given a “insufficient” according to one or more of the seven criteria.

It is and remains the responsibility of the psychologist to choose instruments that are of the highest possible quality and are aligned with the question as it is phrased. Whereas according to Article 101 of the Code, “Use of effective and efficient methods”, the psychologist is professionally responsible for choosing methods that will allow them to fulfil the intended purpose of the test as efficiently and effectively as possible, the same actually applies, too, to the choice of tests.

What does it mean, then, for the use of a psychodiagnostic instrument if it earns a “sufficient” or a “good” on one of the COTAN’s seven assessment criteria? In addition to a brief explanation of the assessment criteria laid down in the COTAN test review system, BOX 8 also contains a number of examples in which a “insufficient” is called for.

Of course, it is preferable for the psychologist to use tests that, for as many of the COTAN criteria as possible, have earned at least a “sufficient”. But that does not mean that instruments that have not earned a “sufficient” could not be used safely and sensibly. If an instrument is given a “insufficient” under one or more of the criteria, or if there is no COTAN test review available at all or for the moment, the psychologist should be able to make a convincing argument for the use of this instrument. They should be aware of any imperfections in the instrument, if possible by removing these during use, and by taking account of them in interpreting results.

The psychologist might be tempted to look at the test that has the highest number of “sufficient” or “good”, while the context of the individual client should determine the choice of the set of instruments. The psychologist must support and substantiate their choice. There are, of course, limits: although the COTAN by no means recommends or discourages certain instruments, a large number of “insufficient” is indeed a signal. It can indicate a potentially good instrument whose imperfections might yet be resolved through further research and development. However, it may also be that the psychologist, after reading the COTAN comments on the test review, concludes that the instrument has clearly lagged behind the state of the science. In that case, taking into account, among things, Article 16, “Professional standards”, they should ask themselves whether they want to use the instrument, and if so whether they can.
In addition to considerations about the quality of an instrument, the psychologist, in selecting it, must wonder whether they themselves, on the basis of their education, training and experience are qualified enough to use a particular instrument (see Article 105, “Qualification”)
In the choice of psychodiagnostic instruments, it is increasingly important that the psychologist ask themselves whether the instrument can give a clear image of the client’s characteristics regardless of their age, sex, language and culture. The psychologist is required to do this under Article 58, “Respect for individuality and diversity”. Since mid-2015, the COTAN has explicitly characterised the search for the “fairness” or “impartiality” of tests as a complement to the test review. See also the addendum of the COTAN on fairness on the website of the NIP, and the publication by Huijding, Hemker, and Van den Berg (2012).

Precisely because the psychologist must have insight both into the possibilities an instrument offers and into its limitations, as well as its pros and cons – all of which derive from Article 101, “Use of effective and efficient methods” – it is not enough just to note the seven outcomes of the COTAN test review. The list of “insufficient”, “sufficient” and “good” speaks volumes, but does not tell the whole story. The psychologist should also take cognisance of the comments of the COTAN that accompany each test review and that form an integral part of it. In addition to descriptive information and research data given in summary, the comments contain, for each criterion, the test reviewers’ considerations and arguments in coming to the rating in question and, where possible, information regarding limitations on the use of the instrument.

Two examples can make it clear why it is important to take cognisance of the comments included in the COTAN test reviews. First, the Code requires that, with the use of recently developed methods about which not so much is known yet, extra caution be exercised in going about using them (see Article 17, “Care and caution with regard to new methods”). Recently, digital versions, mostly online, have been made of many psychodiagnostic instruments that previously had to be taken with a pencil and paper. In certain fields of application, digital versions are the rule rather than the exception these days. Usually that entails not only differences in terms of instructions and examples, but frequently, too, in the presentation of the items themselves. If the test authors or publishers have done no additional research, one cannot simply assume that the hard-copy and digital versions are equivalent. If there is both a paper-and-pencil and a digital version, the comments will offer details, so that the psychologist gets extra information that will allow them to come up with a good assessment in choosing an instrument. Moreover, developments in new types of digital diagnostics and test use are moving rather fast, such as serious games and situational judgment tests. Also with these new forms, care and caution are still advised.

A second example concerns the criterion “reliability”. The assessment itself consists of one summary rating, to which a footnote is sometimes added if, for instance, the judgment for subgroups is different. For especially with reliability, the coefficients that are found for various age groups can diverge considerably, and as often as not go from “insufficient” to “good”. Similar footnotes also arise in other criteria, such as norms and construct validity. In the comments on the test review, the difference of opinion is shown in a full and balanced way. This is why reading these comments is an important and necessary form of support in choosing an instrument wisely.

From the foregoing it will be clear that, however valuable COTAN reviews of psychodiagnostic instruments are, it is nothing more and nothing less than an aid for the psychologist. Choosing and working with psychodiagnostic instruments remain at all times the responsibility of the psychologist. It is expected that the psychologist actively follows developments in psychodiagnostics and psychometrics that are relevant to their field and, in accordance with Article 100, “Maintenance and development of professional expertise”, that they take follow-up training as needed. In the mental-health field, for example, Routine Outcome Monitoring (ROM) – taking repeated measurements of a client in order to follow up on treatment and evaluation – has secured an important place for itself. This means that the psychologist will also have to delve into test-technical questions related to repeated measurements, such as the question of when, with regard to the reliability of the instrument, there is actually a real difference between two scores. After all, with respect to norms, reliability and validity, psychometrics imposes requirements on tests that are used for repeated measurements that differ from those that are intended for a one-time measurement. The psychologist will need to be aware of this, partly because the test reviews of the COTAN often apply to the test being taken once, and because the psychometric requirements for a test are, or may be, different depending on whether the data is from a single or from multiple administrations of the test.