OSCE Boursicot

8/9/2019 OSCE Boursicot

1/55

OSCE

Kathy Boursicot

Train the Trainer Assessment Workshop

October 29, 2003

Hong Kong International Consortium


2/55

OSCE

Format

Purpose

Advantages

Writing principles

Training observers

Scoring considerations


3/55

What is an OSCE?

Objective

Structured

Clinical

ExaminationHarden RG and Gleeson FA Assessment of clinical competence using an

objective structured clinical examination (OSCE Medical Education,!"#"$

%ol !&' !)*


4/55

Obsere! Stations"c#inician e$aminers

OSCE test design


5/55

arieties o! OSCEs

Patient-based Clinical task Written task

Traditional OSCE ….SP-based test ….

Station couplets

….Integral consultations ….


6/55

OSCE

%ormat

Purpose

Advantages

Writing principles

Training observers



7/55

Simple model o! competence

"iller #E$ The assessment o! clinicals%ills&competence&per!ormance$ Academic

"edicine 'Supplement( )**+, -./ S-01S2$

Knows

Shows how

Knows how

Does

P r o f e s s i o n a l a u t h e n t i c i t y

P r o f e s s i o n a l a u t h

e n t i c i t y


8/55

Testing !ormats

Knows

Knows how

Shows how OSCEs

E+,s$ SE,s

-rofessional practice

+C,s

3ehaviour4attitude&s%ills

Cognition4%no5ledge

Does


9/55

OSCE & Objective

All the candidates are presented 5iththe same test

Speci6c s%ill modalities are tested ateach station

7istor8 ta%ing

Explanation Clinical examination

Procedures


10/55

OSCE 1 Structured

The mar%ing scheme !or each stationis structured

Structured interaction bet5eenexaminer and student


11/55

OSCE 9 Clinical Examination

Test o! per!ormance o! clinical s%ills

candidates have to demonstrate their

s%ills: not just describe the theor8


12/55

OSCE

Format

'urpose

Advantages

Writing principles

Training observers



13/55

Characteristics o! assessmentinstruments

;tilit8 <

=eliabilit8

alidit8

Educational impact Acceptabilit8

Feasibilit8

%an der %leuten$ C. /0e assessment of professional competence'

developments$ researc0 and practical implications$

Advances in Health Science Education, !""1$ %ol !' !)1#


14/55

Test characteristics

=eliabilit8 o! a test & measure

reproducibilit8 o! scores acrossraters: >uestions: cases:occasions

capabilit8 to dierentiate

consistentl8 bet5een good @poor students


15/55

Test Sample

Test Sample

Domain of Interest

Sampling


16/55

=eliabilit8

Competencies are highl8 domain1speci6c

3road sampling is re>uired to obtainade>uate reliabilit8

across content: i$e$: range o! cases&situations

across other potential !actors that cause

error variance: i$e$: testing time: number o! cases: examiners:

patients: settings: !acilities


17/55


alidit8 o! a test & measure The content is deemed appropriate b8

the relevant experts

The test measures the characteristic'e$g$ %no5ledge: s%ills( that it isintended to measure

The per!ormance o! a particular tas%predicts !uture per!ormance


18/55


alidit8 o! a test & measure


19/55

OSCE

Format

Purpose

A!anta(es

Writing principles

Training observers



20/55

Advantages o! using OSCEs in clinicalassessment

Care!ul speci6cation o! content


21/55

OSCE

Format

Purpose

Advantages

Writin( princip#es

Training observers



22/55

OSCE Station Writin(


23/55

)o* to start

ecide 5hat tas%s 8ou 5ant to

can

should

test in an OSCE !ormat

OSCEs test per!ormance: not%no5ledge


24/55

Constructive alignment

Beed to %no5 the learningobjectives o! 8our course &programme

"ap these across / Subject areas

no5ledge areas

S%ill areas


25/55

3lueprinting

Content o! the assessment shouldalign 5ith the learning objectives o!the course

3lueprinting allo5s mapping o! test items to

speci6c learning outcomes ensures ade>uate sampling across

subject area and s%ill domains


26/55

OSCE blueprint/ s8stems1based

Hx taking(incl. iag!

"h#s exam(incl. iag!

"roceures Counseling$Eucation

Oreringin%estigs

C&S

Enocrine

'astro

H )

Haem *)

+usculoskl

etc


27/55

OSCE blueprint/ discipline1based

Hx taking(incl. iag!

"h#s exam(incl. iag!

"roceures Counseling$Eucation

Oreringin%estigs

,naes CC

Clin "harm

Comm Health

Emergenc# me

-amil# me

+usculoskel

etc


28/55

e8 !eatures o! success in designingOSCEs

Feasibilit8

Congruence


29/55

Feasibilit8

Ds it a reasonable tas% to expect thecandidates to per!orm?

Can the tas% be examined at anOSCE station?

Can the tas% be per!ormed in thetime allo5ed?

ibili


30/55

Ds it a reasonable tas% to expect the

candidates to per!orm? Ds it authentic?

Can the tas% be examined at an OSCEstation?

"atch clinical situations as closel8 as possible

Some tas%s ma8 re>uire simulated patients

Some tas%s ma8 re>uire mani%ins

Some tas%s simpl8 cannot be examined in this!ormat

Feasibilit8


31/55

Can tas% be per!ormed in timeallo5ed?

Pilot the stations to see i! the8 are!easible

Chec% e>uipment &helpers&practicalities

Feasibilit8

C


32/55

Ds it testing 5hat 8ou 5ant it totest?

Station construct/ describe 5hatstation is testing

Congruence

C


33/55

Ensure that all parts o! station coordinate

Candidate instructions

"ar%ing schedule

Examiner instructions

Simulated patient instructions

E>uipment

Congruence

St ti t t


34/55

This station tests the candidatesabilit8

to

Station construct

C did t i t ti


35/55

Candidate instructions

State circumstances/ e$g$ outpatient clinic:5ard: A @ E: #P surger8

Speci!8 the tas% re>uired o! the candidate/

e$g$ ta%e a histor8: per!orm a neurologicalexamination o! the legs: explain a diagnosis

Speci!8 tas%s BOT re>uired

Dnstruct on summing up/ e$g$ tell thepatient: tell the examiner

E i i t ti


36/55

Cop8 o! candidate instructions

Speci6c instructions appropriate to the tas%/

e$g$: do not prompt: explicit prompts: managinge>uipment

Examiner instructions

Si l t d ti t i t ti


37/55

#ive as much detail as possible so the8can be consistent tr8 to leave as little as possible !or them to ad

lib

#ive enough in!ormation to enable them toans5er >uestions consistentl8

3e speci6c about aect in each role

Speci!8 patient demographics

i$e$: gender: age: ethnicit8: social class: etc$

Simulated patient instructions

" %i h d l


38/55

Ensure mar%s are allocated !or tas%s thecandidates are as%ed to per!orm

ecide relative importance o! diagnosis vsprocess 'histor8 ta%ing: examination(

Separate chec%list !or process s%ills

"ar%ing schedule

E i t


39/55

3e detailed Thin% o!

Chairs G table & couch & bench

"ani%ins 1 speci!8 "edical e>uipment

Stethoscope: ophthalmoscope: sph8g:suturing materials: etc

E>uipment

i i t ti


40/55

;se 8our blueprint

3e clear 5hat 8ou are testing/ de6nethe construct

Chec% !or congruence

Pilot !or !easibilit8

esigning stations

OSCE


41/55

OSCE

Format

Purpose

Advantages

Writing principles

Trainin( obserers


Training observers


42/55

Training observers

;nderstand the principles o! OSCEs

Enhance inter1rater consistenc8

/ 0 i


43/55

/ec0ni2ues

Examiners must train together ideos

HliveI stations

iscussion o! mar%ing

inconsistencies

Training observers


44/55

Training observers

#eneral training

Station1speci6c training

OSCE


45/55

OSCE

Format

Purpose

Advantages

Writing principles

Training observers

Scorin( consi!erations



46/55


#lobal vs chec%list scoring

Weighting

Standard setting

C0 3li i


47/55

C0ec3list scoring

A!anta(es 7elps examiner %no5 5hat the station setters are

loo%ing !or

7elps the examiner be objective

Facilities the use o! non1expert examiners

+isa!anta(es Can just re5ard process&thoroughness

"a8 not suJcientl8 re5ard the excellent candidate

Dgnores the examiners expertise

#lobal scoring


48/55

#lobal scoring

A!anta(es ;tilises the expertise o! the examiners

The8 are in a position to ma%e a 'global(

judgement about the per!ormance

+isa!anta(es

Examiners have to be expert examiners ietraine!

Examiners must be !amiliar 5ith expectedstandards !or the level o! the test

Weighting


49/55

Weighting

Dn a chec%list: some items ma8 be 5eightedmore than others

"ore complicated scoring s8stem

"a%es no dierence to ver8 good @ ver8 badcandidates

Can enhance discrimination at the cut score

Standard setting


50/55

Standard setting

Bo per!ect method

Should be criterion1re!erenced method e$g$ Ango: Ebel: etc$

3ut

are these suitable !or per!ormance basedtests?

Per!ormance based standard setting


51/55

Per!ormance1based standard settingmethods

3orderline group method

Contrasting group method

=egression based standard method

4ramer A$ +uijtjens A$ 5ansen 4$ 67sman H$ /an 8$ van der %leuten C

Comparison of a rational and an empirical standard setting procedure for an OSCE$

Medical Education, 9::& %ol ;ssue 9$ -age !&9

3orderline method


52/55

3orderline method

Chec%list)$ Ησ σηϕσ σϕνησσ σϕησ σϕσ σϕ

K$ Κσκσ σκσµσιθοπθλ θλθµθ θ θ θκλ

0$ Λαλκα κδµ δδκκ δλκλ δλλδ

L$ Κεψω δδ ε ρ ρρµτ τµκ

.$ ϑφϕφκ δδ

-$ Ησκλ;σ σκϕ σλσ σκα ακ ακλ αλδ

2$ Ηδηηδδη σησ αηηακκ ασ

TOTAM

Passing score

3orderline score distribution

Pass: Fail: 3orderline P&3&F

Test score distribution

Contrasting groups method


53/55

Contrasting groups method





.$ ϑφϕφκ δδ



TOTAM

Pass: Fail: 3orderline P&3&F

Test score distribution

Passing score

PassFail

=egression based standard


54/55

=egression based standard





.$ ϑφϕφκ δδ



TOTAM

Overall rating ) K 0 L .

) K 0 L .

Chec%list

Score

N

N < passing score

) < Clear !ailK < 3orderline0 < Clear passL < v good pass

. < excellent pass

Clear 3orderline Clear v good pass excellent pass !ail pass

3orderline&contrasting&regression based


55/55

3orderline&contrasting&regression basedmethods

Panel e>uals examiners =eliable/ cut1o score based on large sample

o! judgments 'no$ o! stations x no$ o! candidates(

Credible/ based on expert judgment in directobservation

Passing score not %no5n in advance 'as 5ith allexaminee centered methods(

Qudgments not independent o! chec%listscoring

Documents

OSCE Boursicot