Abstract ID: A14

Abstract Title: Video Assisted Validation of an Assessment System of Manual Skills in Epidural Analgesia

Authors: Friedman Z¹, Katznelson R², Sidiqui N³, Chan V⁴
Mount Sinai Hospital Toronto Ontario Canada¹, Toronto General Hospital Toronto Ontario Canada², Mount Sinai Hospital Toronto Ontario Canada³, Toronto Western hospital Toronto Ontario Canada⁴
Poster Type: Either

ABSTRACT BODY

Introduction:
There is a growing demand for objective assessment of manual skills and competencies in all aspects of invasive pain treatment procedures. Epidural analgesia is a relatively complex procedure comprising of many crucial technical components. The aim of this study was to validate an objective tool for assessing residents’ competency in performing epidural analgesia by using a global assessment scale and a 3-scale 27-stage checklist. We wish to demonstrate that this tool can differentiate operators with different levels of training.

Materials and Methods:
Six second year anesthesia residents were recruited after their previous experience with epidural analgesia was assessed by questionnaire. The residents were videotaped several times during epidural catheter insertion over a 6-month period. Videotaping was done in a blinded manner masking the identity and level of training of the resident. Three blinded independent examiners evaluated each session, using a specifically devised assessment comprised of a global rating scale and a 27 stage 3-scale checklist to judge the skill level, and grade the videotaped sessions.

Results:
21 sessions were videotaped over 6 months. The first session was taped after 12 epidurals were performed independently and the last ones after up to 220 insertions. Inter-rater reliability for the different checklist and global rating form items shows moderate to high degree of agreement for most stages. Total scores demonstrate almost perfect agreement (Kappa/ICC+/-SE= 0.90+/-0.03 and 0.83+/-0.13 respectively, p<0.0001) between examiners. To test whether higher total scores are associated with greater experience, a series of repeated measures ANCOVAs were performed. Differences between raters were not found to exist in either of the tests (checklist:F=2.44, df=2,10, p=0.1373; global rating:F=1.22, df=2,10, p=0.3365). In both the global rating form and the checklist, a significant relationship between total scores and epidurals done was found to exist (checklist:F=86.55, df=1,53, p<0.0001; global rating:F=36.49, df=1,54, p<0.0001).

Discussion:
The results of our study show that scores on a system comprised of a global rating form and a task specific checklist had a very good correlation with experience in performing epidural analgesia and a significant relationship between total scores and the number of epidurals insertions performed was found to exist. The inter-rater reliability of these assessment tools is very strong, suggesting we will be able to obtain meaningful data from these scales without being concerned about inter-examiner differences.
Evaluating technical skills by an objective tool under direct observation, as opposed to laboratory setting may create a “gold standard” of assessment. Furthermore, residency programs could use these evaluations for identifying deficiencies in teaching programs and trainees that require extra instruction.

ATTACHED FILES

A14_Global Rating Scale and checklist.doc

A14_ValidationofAssessment-graphsl.doc

A14_ValidationofAssessment-tables.doc

Reg Anesth Pain Med 2005; 30(3):A14