Recent research suggests that cognitive-behaviour therapy (CBT) can significantly improve outcomes for patients with severe mental health problems. However, there are no tools specifically designed to assess competence in delivering CBT to psychotic patients. This study investigates the psychometric properties of the Cognitive Therapy Scale for Psychosis (the CTS-Psy) for assessing the quality of CBT with psychotic patients. Inter-rater reliability of trained raters using the CTS-Psy was investigated using taped therapy of trainees engaged in a CBT oriented psychosis training course. Validity was investigated in relation to examining the degree to which the scale could be used to assess a range of therapist ability and patient severity and by assessing the degree to which the CTS-Psy could pick up changes in skill acquisition during the training course over a 9-month period. The CTS-Psy demonstrated excellent inter-rater reliability and good validity in relation to it being able to rate all standards of therapy and all types of patient sessions in the sample studied. In addition, the scale was sensitive to changes in clinical skills during a training course and could discriminate between those who had received training and those who had not.