Mrs Nanette Tredoux, Dr Nicola Taylor, Prof David Maree & Prof Sumaya Laher
Test classification in South Africa is currently receiving much scrutiny from various sectors within and outside of the psychological profession. This is largely due to the promulgation of a fourth clause in the amended Employment Equity Act. The amendment as per the image below calls for tests to be subjected to a process of review and classification to receive certification by the Health Professions Council of South Africa (HPCSA).
Prof David Maree in his capacity as Chair of the Psychometrics Committee at the HPCSA presented a brief overview of the test classification process as it currently exists at the Professional Board for Psychology (PBP) as well as some envisaged changes. Mrs Nanette Tredoux (Psytech and ex PBP member) and Dr Nicola Taylor (JvR Psychometrics and Association of Test Publishers of South Africa) responded to Prof Maree’s presentation as part of a roundtable discussion at the 22nd South African Psychology Congress on the 23 September 2016 at Emperor’s Palace.
Where are we in terms of classification?
As it exists test classification falls under the auspices of the Psychometrics Committee of the Professional Board for Psychology at the HPCSA. Before the process of test classification begins the Psychometrics Committee uses a screening process to decide whether the assessment is evaluating any psychological construct and/or being used to perform a psychological act or an act limited to profession of psychology. Hence the Committee screens at the outset for tests that could be potentially be classified as psychological and those that would be classified as non-psychological.
For tests that could be classified, classification is a 3-phased process involving:
- Notification – inform the PBP of the test under development / to be adapted
- Progress Report – submit annual reports updating the PBP of progress made with the development/adaptation
- Final submission for classification – submit all manuals, test material and other evidence as to the psychometric functioning of the test in South Africa. This has an associated cost which presently stands at R10 000 per submission
The test then goes for review to two expert reviewers. Based on the reviewer comments, feedback is given to the applicant regarding whether the test (1) is ready for classification, (2) needs more information or further data collection and to be resubmitted demonstrating that the issues have been addressed, or (3) is rejected as inappropriate for use in the proposed populations.
After having successfully evaluated and classified a test, the Psychometrics Committee issues a certificate that contains the name of test, the classification, the date on which the classification decision was made and any points to be noted. Hence certification follows classification. It is mandatory for test developers to include the certificate on the first page of the test manual. The PBP argues for the value of inclusion of the certificate in the manual, indicating that this will emphasise to test developers and users the responsibility of developing or using tests that are psychometrically sound.
The test publisher’s response
Having submitted a large number of tests for classification, publishers are of the opinion that preparing the portfolios for submission is an extremely onerous process. The long delays between meetings of the Psychometrics Committee (only three a year) are very frustrating and demoralising for test publishers. The extremely long process makes it almost impossible for test publishers to comply with the newly instituted Clause 4, even if they have submitted tests for review. In one case, the review cycle exceeded eight years from first submission to classification, and exceeded ten years from first submission to receipt of the classification certificate. Thankfully the system seems to have been approved with better turnaround times but the benchmark of ten years to two to three years for classification is still not acceptable.
The quality of many of the review reports do not inspire confidence in the process. It is not clear whether this is a result of the integration that happens when the reports from different reviewers are combined for feedback to the submitter of the test, or whether it is a result of the reviewers not fully perusing or understanding the material that is submitted. The lack of objective criteria for review and the training of the test reviewers may be at fault.
There are benefits to early compliance though. Early compliance with the classification process, even with its frustrations and shortcomings, has been worthwhile in terms of credibility with clients, hence the amended Employment Equity act has been to the commercial advantage of test publishers who are in a position to meet the requirements of the review process. That said, the current PBP guidelines for test classification requirements are opaque. As a publisher it is difficult to determine what the acceptable standards are to ensure that a test will meet test classification criteria.
Concerns with regards to the current list of classified and certified tests
There is a gazetted list of classified and certified tests as per the Board Notice 93 of 2014 available. This list divides tests into three sections:
- Tests which have been classified and certified
- Tests which have been classified but not certified
- Tests that have been classified as tests under development/being adapted and which should not be used for financial gain.
This list of tests is problematic. There are assessments on the list that are old and outdated, and some that are obsolete or have been discontinued. These tests by virtue of still appearing on this list are legally superior to other assessments that have not yet been certified. The PBP does not seem to have a standard process for removing these from the list. As is evident, changes and updates to this list need to be gazette. Presently it is not clear from the existing PBP guidelines how regularly that happens in order to reflect newly classified and certified assessments. Currently there is no section on this list that provides for tests that were screened and classified as non-psychological. The matter of non-psychological measures submitted to the PBP needs to be addressed.
The matter of “related measures” that are not psychological in nature
The Employment Equity Amendment Act requires not only that psychological tests be “certified” but also extends this requirement to related measures that are not psychological in nature. It is not clear who will do the evaluation and classification of these measures. Currently there is no public record of tests that have failed classification or tests submitted and classified as non-psychological. It is not feasible to add the classification of non-psychological tests to the Psychometrics Committee’s responsibilities, because the system is already overburdened. A decision needs to be made at the Department of Labour, where the requirement originated, and a suitable clearinghouse for such tests needs to be created.
Reflections as a board member and ex-member of the psychometrics committee
The time and effort it takes to draft regulations has proven extraordinary. The regulations are a team effort, and have taken on board proposals from external stakeholders. Whenever there is a change to the composition of the Psychometrics Committee, there is a delay in progress because new members need to assimilate and understand the work that has already been done, and they may want to bring in other changes. It is a complex process, and it is difficult to communicate the professional issues to the legal team members. External review of the draft regulations by the HPCSA’s consulting lawyers also takes an extremely long time.
The fact that the test classification process has been run based on hard copy and physical meetings has contributed to the delays. The HPCSA’s information systems were not capable of handling the proposed electronic review platform, and there was resistance at high level to the proposal that such a system should be hosted independently of the HPCSA information technology systems.
The question of whether the HPCSA has the technical capacity and competence to handle the evaluation and classification process if it goes online needs to be given serious and objective consideration. Attempts were made to liaise with the Human Sciences Research Council in this regard, but these did not bear fruit.
Where are we going with test classification?
Prof Maree expressed very clearly that the concerns raised by test publishers were warranted. The Psychometrics Committee is well aware that the current system is problematic. Hence the PBP conducted a Stakeholder meeting in 2015 to discuss test review and classification. The European Federation of Psychologists Associations (EFPA) Guidelines for test classification were presented as an example of a detailed and more transparent test classification system. Prof Maree indicated that revised guidelines based on the EFPA and other such guidelines have already been developed by the PBP and will be released for comment in the very near future. The PBP has over this year introduced training sessions for test reviewers. There are plans to put the entire process online.
Respondents indicated that more needed to be done by the PBP. There are currently a number of unclassified tests being used. Furthermore the situation on the ground is such that psychology practitioners are compromised in having to use non-classified tests. Test publishers in particular requested a moratorium be placed on Clause 4 until such time as the test classification process has been revised and a more effective and efficient process has been put in place. The PBP was not in a position to grant this as this was within the ambit of the Department of Labour. This request forms the core of the court case between the Association of Test Publishers and the Department on Labour. This case is set to be heard on 21 November, 2016.
It was suggested that the PBP consider mandating a separate body to assist with test classification which can possibly work monthly on test classification and work on getting tests ready for the Psychometrics Committee meetings that happen every three months to ensure a quicker turnaround time. It was further suggested that a chief reviewer be assigned to collate reviewer feedback in a logical, sensible and timeous manner for applicants. An online submission process is strongly supported together with more transparent and rigorous guidelines for test review and classification. Reviewer training for classification of online tools is still inadequate and needs to be addressed. Furthermore there is a need for guidance on the use of open access tests and whether the submission processes for these remain the same.
While the test classification process has always drawn significant criticism and contention in South Africa, there is now a real spirit of collaboration between parties in trying to move the process forward. Test publishers, psychological membership bodies, the PBP and stakeholders in academia and industry are all committed to creating a process that works for everyone. We hope that this collaboration allows us to reach new heights as a community, and reflects those principles that we hold dear to our profession.