65
Institut sur la gouvernance d’organisations privées et publiques Collection Gouvernance des institutions universitaires Les travaux de cette collection sont dirigés par Jean-Marie Toulouse, Ph.D., professeur titulaire, HEC Montréal Document #4 RAPPORT DE RECHERCHE SUR LES INDICATEURS Préparé par Roland Proulx Consultant en planification institutionnelle et veille stratégique Montréal, mai 2008 Roland Proulx est professeur honoraire de l’Université de Montréal où il a été Directeur de la planification de 1984 à 2003. Il a été membre du bureau des Directeurs de la Society for College and University Planning (19992004). Il est actuellement consultant en planification institutionnelle et veille stratégique, membre de l’International Ranking Expert Group (UnescoCepes) et partner de Strategic Initiative, management consultants (USA)

Institut sur la gouvernance d’organisations privées et …igopp.org/wp-content/uploads/2014/05/Rapport_final_-_Indicateurs... · Institut sur la gouvernance d’organisations privées

Embed Size (px)

Citation preview

Institut sur la gouvernance drsquoorganisations priveacutees et publiques

Collection Gouvernance des institutions universitaires

Les travaux de cette collection sont dirigeacutes par Jean-Marie Toulouse PhD professeur titulaire HEC Montreacuteal

Document 4

RAPPORT DE RECHERCHE SUR LES INDICATEURS

Preacutepareacute par Roland Proulx

Consultant en planification institutionnelle et veille strateacutegique

Montreacuteal mai 2008

Roland Proulx est professeur honoraire de lrsquoUniversiteacute de Montreacuteal ougrave il a eacuteteacute Directeur de la planification de 1984 agrave 2003 Il a eacuteteacute membre du bureau des Directeurs de la Society for College and University Planning (1999‐2004) Il est actuellement consultant en planification institutionnelle et veille strateacutegique membre de lrsquoInternational Ranking Expert Group (Unesco‐Cepes) et partner de Strategic Initiative management consultants (USA)

Institut sur la gouvernance drsquoorganisations priveacutees et publiques 2008 3000 chemin de la Cocircte‐Sainte‐Catherine Bureau 4311 Montreacuteal (Queacutebec) H3T 2A7 Teacuteleacutephone (514) 340‐6398 Site Web wwwigopporg Deacutepocirct leacutegal ndash Mai 2008 Bibliothegraveque nationale du Queacutebec Bibliothegraveque nationale du Canada ISBN 978‐2‐9810410‐5‐0 Ce document est disponible gratuitement sur le site wwwigopporg La reproduction drsquoextraits est autoriseacutee agrave des fins non commerciales avec mention de la source Toute reproduction partielle doit ecirctre fidegravele au texte utiliseacute

wwwigopporg 2

PREMIEgraveRE PARTIE

INVENTAIRE DES INDICATEURS COURANTS

Agrave DES FINS DE PLANIFICATION DrsquoEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

Preacutesentation

Inventaire des indicateurs

Annexes quelques exemples de listes drsquoindicateurs

On peut reacutesister agrave lrsquoinvasion drsquoune armeacutee mais pas agrave une ideacutee - Victor Hugo Histoire drsquoun crime

wwwigopporg 3

La production drsquoindicateurs et leur application aux universiteacutes remontent agrave la fin des anneacutees

soixante Les premiegraveres listes apparaissent en 1968 soit il y a deacutejagrave une cinquantaine

drsquoanneacutees Les premiers indicateurs se preacutesentent sous la forme de donneacutees statistiques et

de caracteacuteristiques institutionnelles portant sur le nombre drsquoeacutetudiants la qualiteacute des

programmes et les problegravemes financiers que rencontrent les universiteacutes Les anneacutees quatre‐

vingt teacutemoignent drsquoun changement majeur lrsquointroduction dans le domaine de

lrsquoenseignement supeacuterieur des concepts et des techniques de planification strateacutegique

emprunteacutes au monde des affaires incite au deacuteveloppement drsquoautres types drsquoindicateurs et agrave

leur application agrave lrsquoensemble des domaines drsquoactiviteacutes universitaires Le livre de George

Keller publieacute en 1983 sous le titre de Academic Strategy The Management Revolution in

Higher Educations est significatif du virage majeur qui srsquoopegravere dans les universiteacutes nord‐

ameacutericaines

Nous assistons au cours de la mecircme peacuteriode agrave la naissance drsquoune veacuteritable typologie des

indicateurs Aux indicateurs simples et descriptifs viennent srsquoajouter des indicateurs

composeacutes (ratios pourcentage) les deux cateacutegories drsquoindicateurs se reacutefeacuterant de plus en plus

agrave des normes agrave des standards agrave des objectifs et agrave des cibles Les critegraveres retenus eacutevoluent

en direction drsquoindicateurs de rendement de performance de suivi La notion de laquo key

performance indicators (KPIs) raquo se retrouve au cours des derniegraveres anneacutees dans le titre de

la plupart des publications et dans la production drsquoindicateurs Plusieurs facteurs rendent

compte de cette eacutevolution Il faut retenir agrave cet eacutegard lrsquointeacutegration des indicateurs tant

qualitatifs que quantitatifs au processus de planification strateacutegique qursquoadoptent de plus les

universiteacutes lrsquoeacutevaluation par les pairs des faculteacutes et des deacutepartements des universiteacutes les

demandes de reddition de compte formuleacutees entre autres par les gouvernements les

Conseils drsquoadministration les laquo stakeholders raquo et le positionnement strateacutegique tant

national qursquointernational des universiteacutes Il est devenu eacutevident pour tous que les indicateurs

repreacutesentent un certain modegravele de production qui srsquoarticule autour de quatre cateacutegories

inter‐relieacutes de mesures que lrsquoon retrouve dans la plupart des listes et drsquoindicateurs input ndash

processus ndash output ndash reacutesultat

wwwigopporg 4

Les indicateurs font partie deacutesormais des outils indispensables auxquels ont recours les

gouvernements la communauteacute universitaire les eacutetudiants et leurs parents Leur nombre

varie entre dix et 300 indicateurs Bon nombre drsquoauteurs (Barbara Taylor (1993) Gaither amp

alii (1994) drsquoorganismes (COU) de producteurs de classements nationaux et internationaux

favorisent des listes courtes de 10 agrave 20 indicateurs ces listes courtes sont suffisantes pour

caracteacuteriser les activiteacutes drsquoune universiteacute repreacutesentent des indicateurs courants dont les

sources sont accessibles et permettent le repeacuterage (benchmarking) des universiteacutes

comparables En produisant des listes longues de 100 agrave 300 indicateurs les producteurs

preacutesentent plutocirct un reacutepertoire ou une banque drsquoindicateurs parmi lesquels les requeacuterants

comme les utilisateurs peuvent piger selon les objectifs poursuivis et les inteacuterecircts privileacutegieacutes

On trouvera dans les pages qui suivent un inventaire de quelque 175 indicateurs regroupeacutes

autour de 13 chefs de division ou domaines drsquoactiviteacutes (parmi lesquels deux seacuteries

drsquoindicateurs ndash environnement et planification et gestion strateacutegique constituent une

innovation)

Eacutetudiants

Enseignants

Personnels

Activiteacutes de recherche

Internationalisation

Inteacutegration des technologies de linformation

Bibliothegraveques

Revenus et deacutepenses

Espaces

Environnement

Reacuteputation

Planification et gestion strateacutegiques

Eacuteducation permanente

wwwigopporg 5

wwwigopporg

Tous les indicateurs des listes longues et tous les domaines drsquoactiviteacutes nrsquoont pas eacuteteacute retenus

Il srsquoagit drsquoun inventaire des indicateurs courants tels que recenseacutes dans plusieurs listes

courtes et longues et utiliseacutes agrave des fins de planification drsquoeacutevaluation de la performance et

de reddition de comptes De plus dans un contexte de mondialisation et

drsquointernationalisation il est devenu important et pertinent drsquoun point de vue strateacutegique de

retenir les indicateurs utiliseacutes dans les principaux classements nationaux (9 sur 20 possibles)

et internationaux (4) 1 Lrsquoencadreacute qui suit reacutesume ce choix

classements nationaux classements internationaux

universiteacutes canadiennes

et MEQ

USNews and

Report ampThe

Center

Macleans CHEDAAD GUARDIAN ASIAWEEKDEST

AUSTRALIE amp

MELBOURNE REPUBLICA

QS-TIMES SHANHAI TAIWAN

WEB

1 On trouvera en fin de document lrsquoensemble des reacutefeacuterences consulteacutees ainsi qursquoun annexe preacutesentant quelques exemples de listes drsquoindicateurs

6

1INVENTAIRE DES INDICATEURS COURANTS

Agrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

EacutetudiantsUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIEampMel

bourneREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre deacutetudiantsNombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input MEQ G10 DETYA THESARWUproportion deacutetudiantes input DETYA des eacutetudiants agrave temps partiel input AOTQU CHEDAAD Republicainscriptions au 1er cycle agrave temps plein input MEQ CHEDAAD inscriptions au 2e cycle input MEQ inscriptions au 3e cycle input MEQ eacutetudiants aux 2e et 3e cycles input G10 DETYA eacutetudiants internationaux input TU THES des eacutetudiants internationaux input Macleans CHEDAAD THES des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input MEQ stagiaires postdoctoraux input MEQ reacutesidents et reacutesidentes en meacutedecine input MEQ

inputrecrutement et admission inputdemandes dadmission aux programmes reacuteguliers input OTQ CHEDAAD offres dadmissiondemandes dadmission i t CHEDAAD offres dadmissiondemandes dadmission input CHEDAAD inscriptionsdemandes dadmission input T CHEDAAD Asiaweek Republica

inputdossier scolaire des nouveaux eacutetudiants (bulletin Cote R) input US NEWS Macleans Asiaweek

provenance des eacutetudiants input Macleanspopulation cible inputreacutegion meacutetropolitaine inputQueacutebec inputCanada inputInternational input

inputdiversiteacute ethnique input US NEWS Guardian

Bourses accordeacutees aux eacutetudiants - Financement moyen input TMEQpremier cycle input Macleans Republicadeuxiegraveme cycle input G10troisiegraveme cycle input G10

cheminement des eacutetudiants (temps complet) perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output QT MEQ US NEWS Macleans Republica diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output G10 MEQ US NEWS Macleans Melbourne Republica dilplomationmaicirctrise apregraves 4 ans output G10 MEQ US NEWS Macleans Republica diplomationdoctorat apregraves 6 ans output G10 MEQ US NEWS Macleans Republica

Deacutepenses de fonctionnement par eacutetudiant input CAUBOG10 US NEWS Macleans Guardian Asia weekFrais de scolariteacute input G10frais affeacuterents inputDeacutepenses pour les Services aux eacutetudiants input Macleans

taux demployabiliteacute apregraves 1 an outcome Ontario CHEDAAD Guardian de retour aux eacutetudes outcome Melbourne

Valeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome Guardian

RP gouvernance - 2008-06-23

2INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

EnseignantsUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Nombre denseignants Nombre total denseignants inputNombre de professeurs reacuteguliers input G10 MEQStatCan USNEWS Melbourne de professeures input OQ MelbourneNombre de chercheurs inputNombre de professeurs avec diplocircme international THESNombre de professeurs de clinique eacutequivalents temps complet input G10StatCanNombre de chargeacutes de cours et autres personnels eacutequivalents temps complet input USNEWSNombre dauxiliaires en eacutequivalence temps complet input

Caracteacuteristiques des professeurs reacuteguliers input des professeurs et chercheurs avec PhD (ou grade terminal) input USNEWS Macleansstructure dage du corps professoral input Asiaweek Melbourne Republicaacircge moyen des professeurs reacuteguliers inputacircge moyen des professeurs adjoints inputacircge moyen des professeurs agreacutegeacutes inputacircge moyen des professeurs titulaires input

reacutemuneacuteration moyenne des professeurs reacuteguliers input Q T U U S NEWS Asiaweek Melbournereacutemuneacuteration moyenne des professeurs reacuteguliers input QTU USNEWS Asiaweek Melbournedes professeurs adjoints inputdes professeurs agreacutegeacutes inputdes professeurs titulaire input

Activiteacutes denseignement inputnombre deacutetudiants EETC Professeur reacutegulier input G10 MEQ USNEWS Guardian Asiaweek Melbourne Republica THESnombre deacutetudiants EETC Professeur reacutegulier en sciences sociales input Melbournenombre deacutetudiants EETC Professeur reacutegulier en sciences input Melbournenombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input

Nombre de cours dispenseacutes par lInstitution (ANNUEL) input Republica cours assumeacutes par les professeurs reacuteguliers input MEQ USNEWS Macleans Republica cours assumeacutes par les chargeacutes de cours eacutequivalents temps complet input MEQTaille des groupe cours input MEQ USNEWS Macleans Guardian

RP gouvernance - 2008-06-23

3INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

PersonnelsUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Le personnel de soutientotal de tous les personnels reacuteguliers acadeacutemiques et non acd agrave temps complet inputnombre des personnels de soutien (staff) input G10 MelbourneDeacutepenses des personnel non-enseignant input MEQnombre des personnels de soutien (staff)- secteur enseignement inputnombre des personnels de soutien (staff) - secteur soutien acadeacutemique inputnombre des personnels de soutien (staff) - secteur administration inputnombre des personnels de soutien (staff) - secteur service aux eacutetudiants inputnombre des personnels de soutien (staff) - entreprises auxiliaires input

rectoratpersonnel dencadrement non acadeacutemique et cadres acadeacutemiques inputratio sur lensemble des personnels acadeacutemiques et non acadeacutemiques inputeacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input G10 Guardianpersonnel de soutien professeur reacutegulier input G10

RP gouvernance - 2008-06-23

4INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Activiteacutes de recherche USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Revenus de recherche de professeurs et chercheurs ayant fait une demande de subvention processus de professeurs et chercheurs ayant obtenu une subvention inputRevenus totaux de recherche (subventions et contrats) input CAUBO G10 The Center Asiaweek Melbournepart de la recherche contractuelle input MEQSubventions publiques input Macleans Melbournerevenus de recherche obtenus des conseils de recherche feacutedeacuteraux input AOQTU The Center Macleans Melbournerevenus moyens de recherche par professeur et chercheur input InfosourceMEQ des deacutepenses de recherchebudget total des deacutepenses input TNombre de chaires et de centre dexcellence output AFondation canadienne pour linnovation (FCI) input G10 -nombre de projets output G10 -valeur $ output G10

nombre de programmes de maicirctrise de recherche input The Center Asiaweeknombre de programmes de doctorat input The Center Asiaweek

Publications et prix outputNombre de publications output Infosource Asiaweek Melbourne

Shanghai Leiden Taiwan

Nombre de publications par professeur et chercheur output Infosource CHEDAAD ShanghaiPublications dans Nature and Science output TaiwanNombre total de citations output InfosourceT Asiaweek MelbourneNombre de citations par professeur et chercheur output InfosourceT Melbourne THESNombre de citations par publication output T CHEDAAD LeidenHighly cited researchers dans 21 secteurs disciplinaires

output

ShanghaiTaiwan Leiden

Impact des publications selon les secteurs disciplinaires output Infosource TaiwanTHES Shanghai

Domaines disciplinaires dexcellence inputTaiwanShanghai THES

Prix et meacutedailles de niveau national et international output G10 The Center Macleans Melbourne Republica ShanghaiNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur output CHEDAADnombre de brevets et de licences output G10 CHEDAADnombre de spin-off companies G10

RP gouvernance - 2008-06-23

5INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

InternationalisationUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

deacutepenses pour le secteur internationalisation input MEQnombre deacutetudiants en stage agrave leacutetrangernombre deacutetudiants temps complet input tes un cannombre deacutetudiants internationauxnombre deacutetudiants temps complet input tes un can Melbourne THESnombre deacutetudiants en cotutellesnombre deacutetudiant des cycles supeacuterieurs input CHEDAADnombre de professeurs inviteacutes internationaux processus tes un can CHEDAADprofesseurs inviteacutes agrave leacutetranger processus CHEDAADnombre deacutetudiants posdoctoraux processus tes un can CHEDAADproportion de cours de langues eacutetrangegraveres processusnombre de programmes deacutechange processus tes un can CHEDAADEntentes de partenariat outputPrix internationaux output G10 The Center Macleans Melbourne Republica Shanghai

RP gouvernance - 2008-06-23

6INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Inteacutegration des technologies de linformation USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Deacutepenses totales consacreacutees aux TIC input MEQpersonnel techniqueenseignant reacutegulier inputpersonnel techniqueeacutetudiant en eacutequivalence temps complet inputExistence dun reacuteseau informatique (accessible agrave linterne et agrave lexterne) outputaccegraves au reacuteseau avec fils et sans fils processusaccegraves aux ressources documentaires des bibliothegraveques outputService de courriel interne et externe en mode WEB outputNombre dordinateurs par professeur et par eacutetudiant outputNombre de cours en ligne processusnombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) outputAccegraves des professeurs aux banque de donneacutees et aux logiciels processus

Site WEB (eacutevaluation du site WEB) WEBOMETRICSSize (nombre de pages recenseacutees) outcomevisibilitty (consultation observeacutee selon le nobre de lien externe unique) outcomeRich files (intensiteacute de lactiviteacute acadeacutemique) outcomeScholar (activiteacute de recherche publications et citations) outcome

RP gouvernance - 2008-06-23

7INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

BibliothegravequesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre total de volumes et de documents audiovisuels et eacutelectroniques input G10 ARLacquisition de volumes (au cours de lanneacutee) input ARL Macleansnombre total de volumes eacutetudiants en eacutequivalence temps complet input G10 ARL Macleans Deacutepenses des bibliothegraveques input MEQ Macleans AsiaweekDeacutepenses des bibliothegravequesbudget de fonctionnement input ARLDeacutepenses des bibliothegravequeseacutetudiants en eacutequivalence temps complet input G10 ARLPersonnels reacuteguliers agrave temps complet input G10 ARLClassement outcome ARL

RP gouvernance - 2008-06-23

8INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Revenus et deacutepenses USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Produits et revenusSubvention du ministegravere de lEacuteducation du Queacutebec input MEQ subvMEQrevenus de fonctionnement input G10 MacleansDroits de scolariteacute input G10 droits de scolariteacute revenus de fonctionnement input G10autres produits input G10Total des revenus de fonctionnement input G10 MEQ Guardian revenus de fonctionnement eacutetudiants en eacutequivalence temps complet input CAUBO USNews MacleansRevenus de recherche input G10encaissement par nature (dons immobilisations chaires bourses) input G10 fonds de dotation input G10 CenterContribution des alumni aux encaissements et fonds de dotation input USNews Macleans fonds de dotation eacutetudiants en eacutequivalence temps complet input G10Total des produits G10

Charges et deacutepenses G10salaire des professeurs input G10 MEQsalaire des chargeacutes de cours input G10salaires des auxiliaires input MEQsalaires des auxiliaires input MEQsalaire des personnels (staff) input G10autres deacutepenses input G10total des deacutepenses input G10 deacutepenses denseignement deacutepenses totales de fonctionnement input UBC MEQ

Chargesproduits processusEacutequilibre budgeacutetaire revenus et deacutepenses de fonctionnement processus MEQ

RP gouvernance - 2008-06-23

9INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

EspacesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Nombre de megravetres carreacutes bruts input G10MCB eacutetudiants en eacutequivalence temps complet input G10megravetres carreacutes nets alloueacutes input MEQNombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input Tbesoin despace en megravetres carreacutes nets inputExistence dun plan directeur des espaces inputBudget preacutevu au plan directeur des espaces input TBudget alloueacute aux espaces input MEQdeacutepenses de fonctionnement nombre de megravetres carreacutes bruts input G10

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Environnement USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

consommation de leacutenergie (exprimeacutee en BTU processus universiteacutes canadiennes et ameacutericaines BTUpopulation totale de luniversiteacute processus voir en particulier Penn State et Umich de leacutenergie produit par des sources renouvelable processusUtilisation de leau processuslitres deau par personne processusGaz agrave effet de serre (en tonnes meacutetriques CO2 eacutequivalents) processusMT CO2 par personne processusDeacutechets solides (en tonnes) processusTonnes de deacutechets solides per capita processus de deacutechets solides recycleacutes processus

RP gouvernance - 2008-06-23

10INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

ReacuteputationUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

enquecircte meneacutee aupregraves des eacutetudiants outcome NSSE Melbourneenquecircte meneacutee aupregraves des diplocircmeacutes outcomeenquecircte meneacutee aupregraves des pairs outcome USNews Asiaweek THESenquecircte meneacutee aupregraves des employeurs outcome Macleans Melbourneenquecircte meneacutee aupregraves des stakeholders outcome Macleans Melbourneenquecircte meneacutee aupregraves du monde des affaires outcome Macleans Melbournecouverture meacutediatique outcome AT

RP gouvernance - 2008-06-23

11INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Planification et gestion strateacutegiquesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Pouvoirs des corps universitaire processus un Canliens dynamiques entre Conseil Assembleacutee universitaire commission des eacutetudes processus un Canprocessus formel de planification strateacutegique processus un Canproduction de donneacutees et indicateurs processus un Canprocessus deacutevaluation des faculteacutes deacutepartements et services processus un Canrevue systeacutematique des responsabiliteacutes et efficaciteacute des comiteacutes processus un Candeacutepenses de ladministration centrale deacutepenses de fonctionnement processus un Can

RP gouvernance - 2008-06-23

12INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Eacuteducation permanenteUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre de programmes offerts input un Cannombre deacutetudiants inscrits input un CanRevenus et deacutepenses du service input un Can

RP gouvernance - 2008-06-23

13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes

REacuteFEacuteRENCES

Aperccedilu densemble des principaux indicateurs

GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost

Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes

MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf

Indicateurs utiliseacutes dans les classements internationaux

Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml

RP gouvernance - 2008-06-23

ANNEXES

Quelques exemples de listes drsquoindicateurs

1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)

Voir page suivante

Preacutepareacute par Roland Proulx

Consultant en planification institutionnelle et veille strateacutegique

Montreacuteal mai 2008

Performance Indicators

Discussion Paper

May 2004 (Prepared by the Office of Planning amp Analysis)

Table of Contents

Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25

2

Table 1 ndash Number of Indictors by Type of Data Source

Table 1 shows that surveys are the least frequently used source of data for indicators

Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a

plurality of indicators come from this source This measure somewhat underestimates the

importance of surveys however as it does not account for the weighting given to each

indicator in each study In the THES World Rankings for instance there is only a single

survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos

Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one

indicator is worth 20 of the final score

Raw

indIcator count

Surve

y data

Third parties

Universities

AsiaweekmdashAsias Best Universities 18 - - 18

Daily Telegraph (2003) 1 - 1 -

Education18com 9 3 4 2

Excelencia 2001 71 - 71 -

Financial Times (2003) 17 - 17 -

Guangdong Institute of Management Science 17 - 14 3

GuardianmdashUniversity Guide 2005 7 - 2 5

La Repubblica 23 2 21 -

Macleans University Rankings 24 1 5 18

Melbourne Institutemdash International Standing of Australian Universities

26 3 23 -

Netbig 2004 18 1 10 7

Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15

Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities

6 - 5 1

The TimesmdashGood University Guide 2005 9 - 9 -

Times Higher Education SupplementmdashWorld University Rankings

5 1 1 3

US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11

Washington MonthlymdashCollege Rankings 2005 8 - 1 7

Wuhan University Centre for Science Evaluation 45 2 22 21

3

A Indicators of Beginning Characteristics

ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of

students at the time they begin their studies Fourteen of the 18 rankings examined in this

study use one or more indicators of the beginning characteristics of students to arrive at

their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on

these factors with 33 of the total ranking coming from this class of indicators but the

Guardian Education18 Asiaweek and the two other North American surveys also place

considerable emphasis on this category

There are six main indicators used to determine which institutions have students with

positive ldquobeginning characteristicsrdquo

The most common measure of beginning characteristics is performance on national

standardized tests with nine surveys using this as a measure Education18 and the

Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used

by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)

Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this

data is collected and standardized by national bodies it has the benefit of being seen as a

relatively impartial method of determining the relative ldquostrengthrdquo of the students entering

each institution Institutionsrsquo results can be scored by showing either averages or the

percentage of entering students meeting a particular standard

Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses

secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This

is a second-best solution made necessary by the absence of any national standardized test

in Canada (or indeed of any provincial standardized tests at the end of secondary school

in provinces other than Alberta) The lack of national standardization makes this an

undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is

truly equivalent to an ldquoArdquo in another jurisdiction

Another measure of the strength of the student body is the percentage of incoming

students receiving (third-party) scholarships which is worth 11 of the score in the

Wuhan survey One can also approach the issue by measuring institutional selectivity In

effect this method infers the strength of the student body by the proportion of

4

applicants rejected the theory being that the higher the number of rejected applicants the

stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)

Normally this measure is expressed as a straight ratio of acceptances to applications but it

can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the

greatest weight on this measure) as a ratio of enrolments to applications Within the US

there is some dispute as to what constitutes an offer of admission and whether or not late

admissions are included as noted by Ganeshananthan (2003)

Student bodies are often considered to be strong if the school is able to attract a large

number of international or out-of-district students or if they contain people from diverse

ethnic backgrounds A number of league tables use the international student indicator

(which like the selectivity indicator is arguably as much an indicator of prestige and

reputation as it is of student characteristics) although in no case does this indicator

account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a

quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data

on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian

puts a somewhat larger emphasis on this indicator in comparison to other league tables

which use similar variables

A very different take on this idea is present in the Washington Monthly which released its

first set of College Rankings in September 2005 With the declared aim of using an

institutionrsquos commitment to social mobility as a measure of quality it uses the percentage

of students from low-income backgrounds as an indicator (with percentage of students

receiving need-based government (Pell) grants used as a proxy)

Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy

statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the

percentage of the student population who are graduate students (arguably this is a

research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the

number of graduate students auditing classes is used as an indicator the assumption is

presumably that if people are auditing then the classes must be very attractive The Italian

La Repubblica ranks an institution according to the number of part-time students it has

contrary to prevailing North American views on the undesirability of part-time study the

Italian rankings see higher numbers of part-time students in a positive light as it is

evidence that an institution is becoming less rigid in its timetabling and

5

permitting students to juggle both work and study something which was nearly

impossible in that country just a few years ago

The Washington Monthly also has a final category of indicators which reflect studentsrsquo

beginning characteristics namely their likelihood of performing community service as

measured by the percentage of students in the US Peace Corps and Reserve Officer

Training Corps (ROTC) and the percentage of students involved in work-study in the

community3 Together these three indicators account for 33 of an institutionrsquos total

ranking

B Indicators of Learning InputsmdashStaff

Generally both the quantity and quality of staff are positively correlated with institutional

quality The problem of course is finding useful metrics for each of these factors

especially if one excludes as we have done here measures of research performance and

research intensity putting them in a separate category4

The simplest measure is simply the number of faculty unadjusted for things like size of

student body Most national league tables however prefer to use variations on the

concept of facultystudent ratio Others try to measure teaching intensity with measures

such as courses per teacher or hours spent in class per student (both in La Repubblica)

These kinds of measures usually account for between 2-5 of the final rankings although

in some cases (ie the Guardian) this figure can be as high as 20

Another important way of measuring how faculty resources are deployed is the measure

of average class size which is used only by Macleanrsquos and the US News and World Report

Ostensibly the reason for measuring class size is to account in some form for the degree

3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category

6

of attention devoted to actually teaching students Despite the fact that important research

(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level

the use of this indicator appears to be a spillover from the North American debates on

class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein

2000) Regardless of why the indicators are used they are extraordinarily important to

these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and

World Reportrsquos rankings respectively

A number of ranking systems try to look at staff qualifications such as the number of

PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the

Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further

than other surveys and actually looks at the proportion of classes taught by tenure-track

staff Others (ie THES) look at the number of foreign faculty based on the assumption

that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still

others (ie La Repubblica) look at the age structure of the faculty Another proxy for

institutional quality is the pay rates for tenured staff on the assumption that institutions

with higher rates of pay on average attract better faculty this measure has been used both

by the US News and World Report and Asiaweek

Finally a number of league tables rank faculty inputs on the basis of standardized third-

party evaluations 6

Education18 the Financial Times the Times the Guardian and the US

News and World Report league tables all use some sort of ranking criterion based at least in

part on this indicator or variations thereof

C Indicators of Learning InputsmdashResources

Resource inputsmdashcrudely the amount of current dollars equipment and books available

to students at an institutionmdashare widely considered an important measure of quality Yet

despite the apparent simplicity of counting dollars and measuring assets

5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each

university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future

7

the means by which institutional wealth is measured varies considerably between ranking

systems

There are a number of revenue-based measures of resources Macleanrsquos uses public funding

of institutional budgets as a factor in its analysis conversely the Financial Times uses the

private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of

total score) and the US News and World Report (5 of total score) also measure alumni

financial support as a measure of quality

For reasons that are not entirely clear league tables tend to favour measures of

expenditures rather than revenues The Guardian looks at total institutional expenditures

as an indicator Institutional expenditure on student services is used as a measure of

institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total

institutional scores respectively) Rzezspospolita does not measure student services

expenditures directly but does measure student services outputs such as number of

student athletes and number of study clubs which amounts to more or less the same thing

Macleanrsquos also gives out 433 of its total score based on institutional expenditures on

scholarships and bursaries

Various aspects of physical infrastructure are also used as measures of institutional

resources most directly in the case of La Repubblica which bases 317 of its total rank on

the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding

assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type

of physical infrastructure measured is available Internet bandwidth which was used by

Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth

roughly 3 of the total score

By some distance the infrastructure indicators most favoured by the compilers of league

tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on

this with 12 of the total quality mark being taken from various types of library

infrastructure measurements (including acquisitions per year total volumes average

number of volumes per student and yearly library expenditure outside of acquisitions)

Netbig and Education18 also use library volume holdings while Asiaweek the Financial

Times and the Times also use measures of library expenditures outside of acquisitions or

computerization of library resources as measures of institutional quality

8

One important factor to note is that most ranking systems do not normalize their resource

and infrastructure measures That is to say it is raw spending power or simple size of

assets that is usually measured rather than spending per studentprofessor or assets per

studentprofessor As a result a number of these rankings systems have inbuilt biases

towards larger institutions

D Indicators of Learning Outputs

Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge

learned over the course of a baccalaureate degreemdashshould be a basic indicator of

institutional quality Unfortunately good means of measuring these outputsmdashlike the

National Survey of Student Engagement (NSSE) and the College Learning Assessment

(CLA)mdashhave only recently become available and for the most part institutions are still

keeping their scores secret Outside of these measures only a few very crude indicators are

available which likely explains why learning outputs do not feature especially prominently

in most ranking schemes

The simplest types of measures of learning outputs are those linked to graduation and

retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong

and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7

the latter three also use rates of graduation from Masterrsquos programs as indicators In some

cases the weights on these measures can be very highmdashin the Guangdong rankings

graduation rates account for over 50 of the rankingmdashbut in most cases the weights are

10 or less Retention rates commonly meaning the progression rate of first-year students

into second year are accorded less importance The US News and World Report Macleanrsquos

the Melbourne Institute and La Repubblica all employ retention measures as indicators but

none of them are worth more than 4 of total weighting Two publications make specific

indicators for retention and graduation of international students Macleanrsquos (graduation

rates of international students) and the Melbourne Institute (retention rates of international

students) The Washington Monthly looks specifically at institutional retention rates

adjusted for the participation of lower-income students and gives higher scores to

institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores

and number of Pell Grant recipients the US

7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18

9

News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos

ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)

E Indicators of Final Outcomes

Final outcomes are indications of generalized outcomes for students after graduation

Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness

good citizenship) but given the somewhat utilitarian justifications for education that are

currently in fashion (see Wolf 2000) employment outcomes are the most commonly used

measure of final outcomes These are given particular emphasis by the Guardian (where

employment outcomes are worth 17 of the total score) but are also used by the Financial

Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the

Times are interestingly not concerned with employment per se but with ldquoemployment in

an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher

Education Statistics Agency (HESA) uses Standard Occupational Classifications to

measure the proportion of graduates in professional or white-collar jobs anyone not in

such a job is considered not to be working in an area related to their studies (it is unclear

what methodology is used by the Financial Times and the Times although we suspect their

methods are broadly similar)

The only other measure of final outcomes in use is percentage of graduates returning for

additional education which is an indicator used by both the Melbourne Institute and the

Financial Times This is a particularly important indicator for the latter as it is worth 21

of the final ranking

The lack of indicators concerning final outcomes is interesting since most government-

sponsored performance-indicator regimes around the world are very much concerned

with such measures especially with respect to employment Possibly this indicates that

ranking systems simply do not view education outcomes as relevant measures of

educational quality Alternatively it may be the case that they simply have not found a

reliable indicator of outcomes or that there are reliable indicators but that there is so little

variation between institutions that it makes no sense to rank based on the data

10

F Indicators of Research

Many of the league tables covered in this survey include sections and weightings related

to universitiesrsquo research efforts It is in this field of measurement that we see the greatest

diversity of indicators Presumably this is because research inputs and outputs lend

themselves much more easily to measurement and manipulation than other areas of

institutional activity

Three studies include research staff as part of their ranking scheme La Repubblica at 952

the Melbourne Institute (4) and Wuhan (078)

Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used

method of looking at research quality but it is not universally admired because different

disciplines use different means to communicate major advances in knowledge (leading

scientists invariably produce large numbers of journal articles leading social scientists

may produce fewer journal articles but instead have one or two long important

monographsmdashsee Hicks 2004) There is also some concern among non-English speaking

countries that they are penalized in international rankings because so many of the major

journals (notably Science and Nature) are printed in English However the one set of

rankings that uses separate indicators to monitor articles published in English and articles

published in another language (the Wuhan rankings) shows that the two indicators are

positively correlated institutions that have more Chinese publications are also likely to

have more English publications and vice versa

Several sets of league tables measure bibliometric citations in various publication indices

The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a

weight of 20 of the final total Guangdong also monitors other Chinese universities

specifically for citations in engineering publications and weights this at 29 Moreover it

tacks on an additional 1049 for citations in science-oriented indices such as the Science

Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the

same category of scientific citations while the Melbourne Institute rates science citations at

68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social

Science Citation Index which does not include the humanities) are noted in only two

league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne

Institute (32) Another way of measuring research

11

impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a

weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the

Melbourne Institute

The complement of citations is of course publications Listing the number of publications

an individual group department or whole university releases can act as a weak substitute

for citationsmdashweak because simply publishing a paper or monograph is no guarantee that

the general public or other researchers will even glance at the work Guangdong gives an

1179 weighting to publications in science-oriented indices such as the Science Citation

Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same

indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong

even has a separate category just for measuring publications in Science and Nature

although it accords it an almost derisory weighting of 06 Under publications in social

science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their

final totals For publications in other indices (where the subject indices are

undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for

Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of

rankings to include research monographs weighted at 033

In countries where there are specific third-party evaluations of research output academic

quality of research is sometimes used as a research indicator The Times puts a very large

30 weight on this indicator while the Financial Times puts it at 119

Research awards are another handy third-party measurement of quality as the number of

international and national awards won by faculty andor graduates is often considered a

useful measure of institutional success International research awardsmdash specifically the

number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator

by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as

particularly suspect in some quarters given that the points are based on where the

recipient went to school rather than on where they are or were on the

8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the

most prominent publication indices

9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences

12

faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead

for nearly a century Wuhan uses similar measures but only accords them a weight of 14

National research awards are more common as a quality indicator used by La Repubblica

(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)

Financial indicators of research are also very common Research budgets as a factor in the

overall assessment of research in universities are covered by the Financial Times (9)

Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for

total amount of research expenditure unfortunately it is unclear precisely what this

research expenditure represents or how it is determined although it is clearly indicated

that it does not represent the total number of grants or projects at a university Total

number of research-based grants and projects is weighted by Education18 at 15 and

Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science

and engineering and another 55 to those for social sciences and humanities Similarly

the Melbourne Institute gives 6 of its overall weight to public-source grants making no

distinction between areas of study

In a slightly different vein Netbig (46) and Wuhan (278) both list the number of

research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica

(095) all weigh research-basedaffiliated research institutions or centres for studies

Finally one can also measure research not simply in terms of the amount of money it

generates but also in terms of the amount of future income it will generate Both

Guangdong (245) and Wuhan (193) measure the number of patents issued to

universities as a quality indicator

A final way of measuring an institutionrsquos research intensity is to look at the range of its

course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of

doctoral and Masterrsquos programs offered as a proxy for research intensity

As with physical and financial resources few if any of the research indicators are

normalized to account for institutional size (either by student or faculty numbers) In the

world of rankings bigger almost always means better an institution with 100 faculty with

ten citations apiece will always look worse than an institution with 1001 faculty

13

with one citation each To the extent that the raw production of knowledge matters this

form of measurement is acceptable To the extent that rankings are meant to show how

well institutions are doing on a like-to-like basis or to show the efficiency of universities it

is plainly inadequate This should be of particular concern to Chinese policy-makers

whose ranking systems are especially reliant on research-based indicators

G Indicators of Reputation

The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo

Those rankings systems which use the results of reputation surveys as an indicator do so as

an indirect measure of quality based on the assumption that the employers academics and

academic administrators surveyed have opinions of institutional quality that are informed

up-to-date and impartial While these assumptions are clearly open to debate they

nevertheless form an important basis for many ranking systems Another reason for using

reputation measures is the paucity of other data availablemdashsome countries have few

independent measures of teaching effectiveness university resources or output and

reputation can thus act as a useful surrogate Reputation rankings are often criticized as

simply quantifying the common ignorance of the people being surveyed However to the

extent that the people being surveyed hold positions which have the potential to affect

large numbers of young people and whose positions actually require some knowledge of

institutional quality (ie officials in charge of graduate admissions corporate recruiters

etc) then reputation rankings make sense because they provide useful information for

students about the perceived value of the degrees that they could obtain from various

universities

The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and

the Times which both accord reputation a weighting of 50 in their overall ranking

scheme Education18 assigns it almost as much significance at 40 The US News and World

Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just

below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)

and Netbig (15) The only other study to include reputation is Wuhan (117)

14

1 Please see Appendix B for a glossary of the various bodies referenced in this document

Appendix A Detailed Listing of Indicators and their Sources

Beginning Characteristics Indicator

Used By Source1

Incoming grades Macleans University

Percentage with grades above a set limit

Macleans University

US News and World Report

University

Performance on national standardised tests or benchmarks

Asiaweek University

Education18 3rd-party JUPAS

Financial Times Government agency 3rd-party UCAS

Guardian University Guide

Government agency 3rd-party UCAS

Melbourne Institute Government agency 3rd-party DEST

Netbig National entrance examination board

Times Good University Guide

Government agency 3rd-party UCAS

US News University

Wuhan Unknown presumed government 3rd-party

Student status La Repubblica Government agency 3rd-party MIUR

Admittance selectivity general

Asiaweek University

Admittance number of applications to places

Asiaweek University

Financial Times Government agency 3rd-party UCAS

La Repubblica Government agency 3rd-party MIUR

US News University

15

Beginning Characteristics Indicator

Used By Source1

Out-of-locality student percentage

Macleans University

International student percentages

Financial Times Government agency 3rd-party HESA

Macleans University

Shanghai Institute of Educational Science

Times World University

Wuhan Unknown presumed university

Undergraduate students among all students percentages

Netbig

Wuhan Unknown presumed university

Ethnic diversity in student body

Guardian University

Learning Inputs ndash Staff Indicator

Used By Source

Facultystudent ratio Asiaweek University

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Financial Times Government agency 3rd-party HESA

La Repubblica Government agency 3rd-party MIUR

Times Good University Guide

Government agency 3rd-party HESA

Times World University

US News University

Wuhan Unknown presumed university

16

Learning Inputs ndash Staff Indicator

Used By Source

Social science faculty student ratio

Melbourne Government agency 3rd-party DEST

Science faculty student ratio

Melbourne Government agency 3rd-party DEST

Administrative staff student ratio

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Staff student ratio (regardless of division)

Guardian Government agency 3rd-party HESA

Netbig University

Course per teacher La Repubblica Government agency 3rd-party MIUR

Per-teacher university spending

Asiaweek University

Faculty pay rates fortenured staff

Asiaweek University

US News University

Number of full-time part-time faculty

Netbig University

US News University

Wuhan Unknown presumed university

Faculty with researchprojects

Wuhan Unknown presumed university

Class size differentiation Macleans University

US News University

Classes taught by tenured faculty

Macleans University

Exchange programmes hosted

La Repubblica Government agency 3rd-party AgNaSoc

Number of classes lsquoactually taughtrsquo

La Repubblica Government agency 3rd-party MIUR

of international faculty (v faculty as a whole)

Times World University

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

Institut sur la gouvernance drsquoorganisations priveacutees et publiques 2008 3000 chemin de la Cocircte‐Sainte‐Catherine Bureau 4311 Montreacuteal (Queacutebec) H3T 2A7 Teacuteleacutephone (514) 340‐6398 Site Web wwwigopporg Deacutepocirct leacutegal ndash Mai 2008 Bibliothegraveque nationale du Queacutebec Bibliothegraveque nationale du Canada ISBN 978‐2‐9810410‐5‐0 Ce document est disponible gratuitement sur le site wwwigopporg La reproduction drsquoextraits est autoriseacutee agrave des fins non commerciales avec mention de la source Toute reproduction partielle doit ecirctre fidegravele au texte utiliseacute

wwwigopporg 2

PREMIEgraveRE PARTIE

INVENTAIRE DES INDICATEURS COURANTS

Agrave DES FINS DE PLANIFICATION DrsquoEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

Preacutesentation

Inventaire des indicateurs

Annexes quelques exemples de listes drsquoindicateurs

On peut reacutesister agrave lrsquoinvasion drsquoune armeacutee mais pas agrave une ideacutee - Victor Hugo Histoire drsquoun crime

wwwigopporg 3

La production drsquoindicateurs et leur application aux universiteacutes remontent agrave la fin des anneacutees

soixante Les premiegraveres listes apparaissent en 1968 soit il y a deacutejagrave une cinquantaine

drsquoanneacutees Les premiers indicateurs se preacutesentent sous la forme de donneacutees statistiques et

de caracteacuteristiques institutionnelles portant sur le nombre drsquoeacutetudiants la qualiteacute des

programmes et les problegravemes financiers que rencontrent les universiteacutes Les anneacutees quatre‐

vingt teacutemoignent drsquoun changement majeur lrsquointroduction dans le domaine de

lrsquoenseignement supeacuterieur des concepts et des techniques de planification strateacutegique

emprunteacutes au monde des affaires incite au deacuteveloppement drsquoautres types drsquoindicateurs et agrave

leur application agrave lrsquoensemble des domaines drsquoactiviteacutes universitaires Le livre de George

Keller publieacute en 1983 sous le titre de Academic Strategy The Management Revolution in

Higher Educations est significatif du virage majeur qui srsquoopegravere dans les universiteacutes nord‐

ameacutericaines

Nous assistons au cours de la mecircme peacuteriode agrave la naissance drsquoune veacuteritable typologie des

indicateurs Aux indicateurs simples et descriptifs viennent srsquoajouter des indicateurs

composeacutes (ratios pourcentage) les deux cateacutegories drsquoindicateurs se reacutefeacuterant de plus en plus

agrave des normes agrave des standards agrave des objectifs et agrave des cibles Les critegraveres retenus eacutevoluent

en direction drsquoindicateurs de rendement de performance de suivi La notion de laquo key

performance indicators (KPIs) raquo se retrouve au cours des derniegraveres anneacutees dans le titre de

la plupart des publications et dans la production drsquoindicateurs Plusieurs facteurs rendent

compte de cette eacutevolution Il faut retenir agrave cet eacutegard lrsquointeacutegration des indicateurs tant

qualitatifs que quantitatifs au processus de planification strateacutegique qursquoadoptent de plus les

universiteacutes lrsquoeacutevaluation par les pairs des faculteacutes et des deacutepartements des universiteacutes les

demandes de reddition de compte formuleacutees entre autres par les gouvernements les

Conseils drsquoadministration les laquo stakeholders raquo et le positionnement strateacutegique tant

national qursquointernational des universiteacutes Il est devenu eacutevident pour tous que les indicateurs

repreacutesentent un certain modegravele de production qui srsquoarticule autour de quatre cateacutegories

inter‐relieacutes de mesures que lrsquoon retrouve dans la plupart des listes et drsquoindicateurs input ndash

processus ndash output ndash reacutesultat

wwwigopporg 4

Les indicateurs font partie deacutesormais des outils indispensables auxquels ont recours les

gouvernements la communauteacute universitaire les eacutetudiants et leurs parents Leur nombre

varie entre dix et 300 indicateurs Bon nombre drsquoauteurs (Barbara Taylor (1993) Gaither amp

alii (1994) drsquoorganismes (COU) de producteurs de classements nationaux et internationaux

favorisent des listes courtes de 10 agrave 20 indicateurs ces listes courtes sont suffisantes pour

caracteacuteriser les activiteacutes drsquoune universiteacute repreacutesentent des indicateurs courants dont les

sources sont accessibles et permettent le repeacuterage (benchmarking) des universiteacutes

comparables En produisant des listes longues de 100 agrave 300 indicateurs les producteurs

preacutesentent plutocirct un reacutepertoire ou une banque drsquoindicateurs parmi lesquels les requeacuterants

comme les utilisateurs peuvent piger selon les objectifs poursuivis et les inteacuterecircts privileacutegieacutes

On trouvera dans les pages qui suivent un inventaire de quelque 175 indicateurs regroupeacutes

autour de 13 chefs de division ou domaines drsquoactiviteacutes (parmi lesquels deux seacuteries

drsquoindicateurs ndash environnement et planification et gestion strateacutegique constituent une

innovation)

Eacutetudiants

Enseignants

Personnels

Activiteacutes de recherche

Internationalisation

Inteacutegration des technologies de linformation

Bibliothegraveques

Revenus et deacutepenses

Espaces

Environnement

Reacuteputation

Planification et gestion strateacutegiques

Eacuteducation permanente

wwwigopporg 5

wwwigopporg

Tous les indicateurs des listes longues et tous les domaines drsquoactiviteacutes nrsquoont pas eacuteteacute retenus

Il srsquoagit drsquoun inventaire des indicateurs courants tels que recenseacutes dans plusieurs listes

courtes et longues et utiliseacutes agrave des fins de planification drsquoeacutevaluation de la performance et

de reddition de comptes De plus dans un contexte de mondialisation et

drsquointernationalisation il est devenu important et pertinent drsquoun point de vue strateacutegique de

retenir les indicateurs utiliseacutes dans les principaux classements nationaux (9 sur 20 possibles)

et internationaux (4) 1 Lrsquoencadreacute qui suit reacutesume ce choix

classements nationaux classements internationaux

universiteacutes canadiennes

et MEQ

USNews and

Report ampThe

Center

Macleans CHEDAAD GUARDIAN ASIAWEEKDEST

AUSTRALIE amp

MELBOURNE REPUBLICA

QS-TIMES SHANHAI TAIWAN

WEB

1 On trouvera en fin de document lrsquoensemble des reacutefeacuterences consulteacutees ainsi qursquoun annexe preacutesentant quelques exemples de listes drsquoindicateurs

6

1INVENTAIRE DES INDICATEURS COURANTS

Agrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

EacutetudiantsUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIEampMel

bourneREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre deacutetudiantsNombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input MEQ G10 DETYA THESARWUproportion deacutetudiantes input DETYA des eacutetudiants agrave temps partiel input AOTQU CHEDAAD Republicainscriptions au 1er cycle agrave temps plein input MEQ CHEDAAD inscriptions au 2e cycle input MEQ inscriptions au 3e cycle input MEQ eacutetudiants aux 2e et 3e cycles input G10 DETYA eacutetudiants internationaux input TU THES des eacutetudiants internationaux input Macleans CHEDAAD THES des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input MEQ stagiaires postdoctoraux input MEQ reacutesidents et reacutesidentes en meacutedecine input MEQ

inputrecrutement et admission inputdemandes dadmission aux programmes reacuteguliers input OTQ CHEDAAD offres dadmissiondemandes dadmission i t CHEDAAD offres dadmissiondemandes dadmission input CHEDAAD inscriptionsdemandes dadmission input T CHEDAAD Asiaweek Republica

inputdossier scolaire des nouveaux eacutetudiants (bulletin Cote R) input US NEWS Macleans Asiaweek

provenance des eacutetudiants input Macleanspopulation cible inputreacutegion meacutetropolitaine inputQueacutebec inputCanada inputInternational input

inputdiversiteacute ethnique input US NEWS Guardian

Bourses accordeacutees aux eacutetudiants - Financement moyen input TMEQpremier cycle input Macleans Republicadeuxiegraveme cycle input G10troisiegraveme cycle input G10

cheminement des eacutetudiants (temps complet) perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output QT MEQ US NEWS Macleans Republica diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output G10 MEQ US NEWS Macleans Melbourne Republica dilplomationmaicirctrise apregraves 4 ans output G10 MEQ US NEWS Macleans Republica diplomationdoctorat apregraves 6 ans output G10 MEQ US NEWS Macleans Republica

Deacutepenses de fonctionnement par eacutetudiant input CAUBOG10 US NEWS Macleans Guardian Asia weekFrais de scolariteacute input G10frais affeacuterents inputDeacutepenses pour les Services aux eacutetudiants input Macleans

taux demployabiliteacute apregraves 1 an outcome Ontario CHEDAAD Guardian de retour aux eacutetudes outcome Melbourne

Valeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome Guardian

RP gouvernance - 2008-06-23

2INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

EnseignantsUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Nombre denseignants Nombre total denseignants inputNombre de professeurs reacuteguliers input G10 MEQStatCan USNEWS Melbourne de professeures input OQ MelbourneNombre de chercheurs inputNombre de professeurs avec diplocircme international THESNombre de professeurs de clinique eacutequivalents temps complet input G10StatCanNombre de chargeacutes de cours et autres personnels eacutequivalents temps complet input USNEWSNombre dauxiliaires en eacutequivalence temps complet input

Caracteacuteristiques des professeurs reacuteguliers input des professeurs et chercheurs avec PhD (ou grade terminal) input USNEWS Macleansstructure dage du corps professoral input Asiaweek Melbourne Republicaacircge moyen des professeurs reacuteguliers inputacircge moyen des professeurs adjoints inputacircge moyen des professeurs agreacutegeacutes inputacircge moyen des professeurs titulaires input

reacutemuneacuteration moyenne des professeurs reacuteguliers input Q T U U S NEWS Asiaweek Melbournereacutemuneacuteration moyenne des professeurs reacuteguliers input QTU USNEWS Asiaweek Melbournedes professeurs adjoints inputdes professeurs agreacutegeacutes inputdes professeurs titulaire input

Activiteacutes denseignement inputnombre deacutetudiants EETC Professeur reacutegulier input G10 MEQ USNEWS Guardian Asiaweek Melbourne Republica THESnombre deacutetudiants EETC Professeur reacutegulier en sciences sociales input Melbournenombre deacutetudiants EETC Professeur reacutegulier en sciences input Melbournenombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input

Nombre de cours dispenseacutes par lInstitution (ANNUEL) input Republica cours assumeacutes par les professeurs reacuteguliers input MEQ USNEWS Macleans Republica cours assumeacutes par les chargeacutes de cours eacutequivalents temps complet input MEQTaille des groupe cours input MEQ USNEWS Macleans Guardian

RP gouvernance - 2008-06-23

3INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

PersonnelsUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Le personnel de soutientotal de tous les personnels reacuteguliers acadeacutemiques et non acd agrave temps complet inputnombre des personnels de soutien (staff) input G10 MelbourneDeacutepenses des personnel non-enseignant input MEQnombre des personnels de soutien (staff)- secteur enseignement inputnombre des personnels de soutien (staff) - secteur soutien acadeacutemique inputnombre des personnels de soutien (staff) - secteur administration inputnombre des personnels de soutien (staff) - secteur service aux eacutetudiants inputnombre des personnels de soutien (staff) - entreprises auxiliaires input

rectoratpersonnel dencadrement non acadeacutemique et cadres acadeacutemiques inputratio sur lensemble des personnels acadeacutemiques et non acadeacutemiques inputeacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input G10 Guardianpersonnel de soutien professeur reacutegulier input G10

RP gouvernance - 2008-06-23

4INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Activiteacutes de recherche USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Revenus de recherche de professeurs et chercheurs ayant fait une demande de subvention processus de professeurs et chercheurs ayant obtenu une subvention inputRevenus totaux de recherche (subventions et contrats) input CAUBO G10 The Center Asiaweek Melbournepart de la recherche contractuelle input MEQSubventions publiques input Macleans Melbournerevenus de recherche obtenus des conseils de recherche feacutedeacuteraux input AOQTU The Center Macleans Melbournerevenus moyens de recherche par professeur et chercheur input InfosourceMEQ des deacutepenses de recherchebudget total des deacutepenses input TNombre de chaires et de centre dexcellence output AFondation canadienne pour linnovation (FCI) input G10 -nombre de projets output G10 -valeur $ output G10

nombre de programmes de maicirctrise de recherche input The Center Asiaweeknombre de programmes de doctorat input The Center Asiaweek

Publications et prix outputNombre de publications output Infosource Asiaweek Melbourne

Shanghai Leiden Taiwan

Nombre de publications par professeur et chercheur output Infosource CHEDAAD ShanghaiPublications dans Nature and Science output TaiwanNombre total de citations output InfosourceT Asiaweek MelbourneNombre de citations par professeur et chercheur output InfosourceT Melbourne THESNombre de citations par publication output T CHEDAAD LeidenHighly cited researchers dans 21 secteurs disciplinaires

output

ShanghaiTaiwan Leiden

Impact des publications selon les secteurs disciplinaires output Infosource TaiwanTHES Shanghai

Domaines disciplinaires dexcellence inputTaiwanShanghai THES

Prix et meacutedailles de niveau national et international output G10 The Center Macleans Melbourne Republica ShanghaiNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur output CHEDAADnombre de brevets et de licences output G10 CHEDAADnombre de spin-off companies G10

RP gouvernance - 2008-06-23

5INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

InternationalisationUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

deacutepenses pour le secteur internationalisation input MEQnombre deacutetudiants en stage agrave leacutetrangernombre deacutetudiants temps complet input tes un cannombre deacutetudiants internationauxnombre deacutetudiants temps complet input tes un can Melbourne THESnombre deacutetudiants en cotutellesnombre deacutetudiant des cycles supeacuterieurs input CHEDAADnombre de professeurs inviteacutes internationaux processus tes un can CHEDAADprofesseurs inviteacutes agrave leacutetranger processus CHEDAADnombre deacutetudiants posdoctoraux processus tes un can CHEDAADproportion de cours de langues eacutetrangegraveres processusnombre de programmes deacutechange processus tes un can CHEDAADEntentes de partenariat outputPrix internationaux output G10 The Center Macleans Melbourne Republica Shanghai

RP gouvernance - 2008-06-23

6INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Inteacutegration des technologies de linformation USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Deacutepenses totales consacreacutees aux TIC input MEQpersonnel techniqueenseignant reacutegulier inputpersonnel techniqueeacutetudiant en eacutequivalence temps complet inputExistence dun reacuteseau informatique (accessible agrave linterne et agrave lexterne) outputaccegraves au reacuteseau avec fils et sans fils processusaccegraves aux ressources documentaires des bibliothegraveques outputService de courriel interne et externe en mode WEB outputNombre dordinateurs par professeur et par eacutetudiant outputNombre de cours en ligne processusnombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) outputAccegraves des professeurs aux banque de donneacutees et aux logiciels processus

Site WEB (eacutevaluation du site WEB) WEBOMETRICSSize (nombre de pages recenseacutees) outcomevisibilitty (consultation observeacutee selon le nobre de lien externe unique) outcomeRich files (intensiteacute de lactiviteacute acadeacutemique) outcomeScholar (activiteacute de recherche publications et citations) outcome

RP gouvernance - 2008-06-23

7INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

BibliothegravequesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre total de volumes et de documents audiovisuels et eacutelectroniques input G10 ARLacquisition de volumes (au cours de lanneacutee) input ARL Macleansnombre total de volumes eacutetudiants en eacutequivalence temps complet input G10 ARL Macleans Deacutepenses des bibliothegraveques input MEQ Macleans AsiaweekDeacutepenses des bibliothegravequesbudget de fonctionnement input ARLDeacutepenses des bibliothegravequeseacutetudiants en eacutequivalence temps complet input G10 ARLPersonnels reacuteguliers agrave temps complet input G10 ARLClassement outcome ARL

RP gouvernance - 2008-06-23

8INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Revenus et deacutepenses USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Produits et revenusSubvention du ministegravere de lEacuteducation du Queacutebec input MEQ subvMEQrevenus de fonctionnement input G10 MacleansDroits de scolariteacute input G10 droits de scolariteacute revenus de fonctionnement input G10autres produits input G10Total des revenus de fonctionnement input G10 MEQ Guardian revenus de fonctionnement eacutetudiants en eacutequivalence temps complet input CAUBO USNews MacleansRevenus de recherche input G10encaissement par nature (dons immobilisations chaires bourses) input G10 fonds de dotation input G10 CenterContribution des alumni aux encaissements et fonds de dotation input USNews Macleans fonds de dotation eacutetudiants en eacutequivalence temps complet input G10Total des produits G10

Charges et deacutepenses G10salaire des professeurs input G10 MEQsalaire des chargeacutes de cours input G10salaires des auxiliaires input MEQsalaires des auxiliaires input MEQsalaire des personnels (staff) input G10autres deacutepenses input G10total des deacutepenses input G10 deacutepenses denseignement deacutepenses totales de fonctionnement input UBC MEQ

Chargesproduits processusEacutequilibre budgeacutetaire revenus et deacutepenses de fonctionnement processus MEQ

RP gouvernance - 2008-06-23

9INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

EspacesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Nombre de megravetres carreacutes bruts input G10MCB eacutetudiants en eacutequivalence temps complet input G10megravetres carreacutes nets alloueacutes input MEQNombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input Tbesoin despace en megravetres carreacutes nets inputExistence dun plan directeur des espaces inputBudget preacutevu au plan directeur des espaces input TBudget alloueacute aux espaces input MEQdeacutepenses de fonctionnement nombre de megravetres carreacutes bruts input G10

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Environnement USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

consommation de leacutenergie (exprimeacutee en BTU processus universiteacutes canadiennes et ameacutericaines BTUpopulation totale de luniversiteacute processus voir en particulier Penn State et Umich de leacutenergie produit par des sources renouvelable processusUtilisation de leau processuslitres deau par personne processusGaz agrave effet de serre (en tonnes meacutetriques CO2 eacutequivalents) processusMT CO2 par personne processusDeacutechets solides (en tonnes) processusTonnes de deacutechets solides per capita processus de deacutechets solides recycleacutes processus

RP gouvernance - 2008-06-23

10INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

ReacuteputationUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

enquecircte meneacutee aupregraves des eacutetudiants outcome NSSE Melbourneenquecircte meneacutee aupregraves des diplocircmeacutes outcomeenquecircte meneacutee aupregraves des pairs outcome USNews Asiaweek THESenquecircte meneacutee aupregraves des employeurs outcome Macleans Melbourneenquecircte meneacutee aupregraves des stakeholders outcome Macleans Melbourneenquecircte meneacutee aupregraves du monde des affaires outcome Macleans Melbournecouverture meacutediatique outcome AT

RP gouvernance - 2008-06-23

11INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Planification et gestion strateacutegiquesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Pouvoirs des corps universitaire processus un Canliens dynamiques entre Conseil Assembleacutee universitaire commission des eacutetudes processus un Canprocessus formel de planification strateacutegique processus un Canproduction de donneacutees et indicateurs processus un Canprocessus deacutevaluation des faculteacutes deacutepartements et services processus un Canrevue systeacutematique des responsabiliteacutes et efficaciteacute des comiteacutes processus un Candeacutepenses de ladministration centrale deacutepenses de fonctionnement processus un Can

RP gouvernance - 2008-06-23

12INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Eacuteducation permanenteUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre de programmes offerts input un Cannombre deacutetudiants inscrits input un CanRevenus et deacutepenses du service input un Can

RP gouvernance - 2008-06-23

13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes

REacuteFEacuteRENCES

Aperccedilu densemble des principaux indicateurs

GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost

Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes

MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf

Indicateurs utiliseacutes dans les classements internationaux

Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml

RP gouvernance - 2008-06-23

ANNEXES

Quelques exemples de listes drsquoindicateurs

1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)

Voir page suivante

Preacutepareacute par Roland Proulx

Consultant en planification institutionnelle et veille strateacutegique

Montreacuteal mai 2008

Performance Indicators

Discussion Paper

May 2004 (Prepared by the Office of Planning amp Analysis)

Table of Contents

Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25

2

Table 1 ndash Number of Indictors by Type of Data Source

Table 1 shows that surveys are the least frequently used source of data for indicators

Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a

plurality of indicators come from this source This measure somewhat underestimates the

importance of surveys however as it does not account for the weighting given to each

indicator in each study In the THES World Rankings for instance there is only a single

survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos

Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one

indicator is worth 20 of the final score

Raw

indIcator count

Surve

y data

Third parties

Universities

AsiaweekmdashAsias Best Universities 18 - - 18

Daily Telegraph (2003) 1 - 1 -

Education18com 9 3 4 2

Excelencia 2001 71 - 71 -

Financial Times (2003) 17 - 17 -

Guangdong Institute of Management Science 17 - 14 3

GuardianmdashUniversity Guide 2005 7 - 2 5

La Repubblica 23 2 21 -

Macleans University Rankings 24 1 5 18

Melbourne Institutemdash International Standing of Australian Universities

26 3 23 -

Netbig 2004 18 1 10 7

Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15

Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities

6 - 5 1

The TimesmdashGood University Guide 2005 9 - 9 -

Times Higher Education SupplementmdashWorld University Rankings

5 1 1 3

US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11

Washington MonthlymdashCollege Rankings 2005 8 - 1 7

Wuhan University Centre for Science Evaluation 45 2 22 21

3

A Indicators of Beginning Characteristics

ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of

students at the time they begin their studies Fourteen of the 18 rankings examined in this

study use one or more indicators of the beginning characteristics of students to arrive at

their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on

these factors with 33 of the total ranking coming from this class of indicators but the

Guardian Education18 Asiaweek and the two other North American surveys also place

considerable emphasis on this category

There are six main indicators used to determine which institutions have students with

positive ldquobeginning characteristicsrdquo

The most common measure of beginning characteristics is performance on national

standardized tests with nine surveys using this as a measure Education18 and the

Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used

by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)

Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this

data is collected and standardized by national bodies it has the benefit of being seen as a

relatively impartial method of determining the relative ldquostrengthrdquo of the students entering

each institution Institutionsrsquo results can be scored by showing either averages or the

percentage of entering students meeting a particular standard

Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses

secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This

is a second-best solution made necessary by the absence of any national standardized test

in Canada (or indeed of any provincial standardized tests at the end of secondary school

in provinces other than Alberta) The lack of national standardization makes this an

undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is

truly equivalent to an ldquoArdquo in another jurisdiction

Another measure of the strength of the student body is the percentage of incoming

students receiving (third-party) scholarships which is worth 11 of the score in the

Wuhan survey One can also approach the issue by measuring institutional selectivity In

effect this method infers the strength of the student body by the proportion of

4

applicants rejected the theory being that the higher the number of rejected applicants the

stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)

Normally this measure is expressed as a straight ratio of acceptances to applications but it

can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the

greatest weight on this measure) as a ratio of enrolments to applications Within the US

there is some dispute as to what constitutes an offer of admission and whether or not late

admissions are included as noted by Ganeshananthan (2003)

Student bodies are often considered to be strong if the school is able to attract a large

number of international or out-of-district students or if they contain people from diverse

ethnic backgrounds A number of league tables use the international student indicator

(which like the selectivity indicator is arguably as much an indicator of prestige and

reputation as it is of student characteristics) although in no case does this indicator

account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a

quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data

on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian

puts a somewhat larger emphasis on this indicator in comparison to other league tables

which use similar variables

A very different take on this idea is present in the Washington Monthly which released its

first set of College Rankings in September 2005 With the declared aim of using an

institutionrsquos commitment to social mobility as a measure of quality it uses the percentage

of students from low-income backgrounds as an indicator (with percentage of students

receiving need-based government (Pell) grants used as a proxy)

Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy

statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the

percentage of the student population who are graduate students (arguably this is a

research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the

number of graduate students auditing classes is used as an indicator the assumption is

presumably that if people are auditing then the classes must be very attractive The Italian

La Repubblica ranks an institution according to the number of part-time students it has

contrary to prevailing North American views on the undesirability of part-time study the

Italian rankings see higher numbers of part-time students in a positive light as it is

evidence that an institution is becoming less rigid in its timetabling and

5

permitting students to juggle both work and study something which was nearly

impossible in that country just a few years ago

The Washington Monthly also has a final category of indicators which reflect studentsrsquo

beginning characteristics namely their likelihood of performing community service as

measured by the percentage of students in the US Peace Corps and Reserve Officer

Training Corps (ROTC) and the percentage of students involved in work-study in the

community3 Together these three indicators account for 33 of an institutionrsquos total

ranking

B Indicators of Learning InputsmdashStaff

Generally both the quantity and quality of staff are positively correlated with institutional

quality The problem of course is finding useful metrics for each of these factors

especially if one excludes as we have done here measures of research performance and

research intensity putting them in a separate category4

The simplest measure is simply the number of faculty unadjusted for things like size of

student body Most national league tables however prefer to use variations on the

concept of facultystudent ratio Others try to measure teaching intensity with measures

such as courses per teacher or hours spent in class per student (both in La Repubblica)

These kinds of measures usually account for between 2-5 of the final rankings although

in some cases (ie the Guardian) this figure can be as high as 20

Another important way of measuring how faculty resources are deployed is the measure

of average class size which is used only by Macleanrsquos and the US News and World Report

Ostensibly the reason for measuring class size is to account in some form for the degree

3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category

6

of attention devoted to actually teaching students Despite the fact that important research

(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level

the use of this indicator appears to be a spillover from the North American debates on

class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein

2000) Regardless of why the indicators are used they are extraordinarily important to

these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and

World Reportrsquos rankings respectively

A number of ranking systems try to look at staff qualifications such as the number of

PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the

Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further

than other surveys and actually looks at the proportion of classes taught by tenure-track

staff Others (ie THES) look at the number of foreign faculty based on the assumption

that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still

others (ie La Repubblica) look at the age structure of the faculty Another proxy for

institutional quality is the pay rates for tenured staff on the assumption that institutions

with higher rates of pay on average attract better faculty this measure has been used both

by the US News and World Report and Asiaweek

Finally a number of league tables rank faculty inputs on the basis of standardized third-

party evaluations 6

Education18 the Financial Times the Times the Guardian and the US

News and World Report league tables all use some sort of ranking criterion based at least in

part on this indicator or variations thereof

C Indicators of Learning InputsmdashResources

Resource inputsmdashcrudely the amount of current dollars equipment and books available

to students at an institutionmdashare widely considered an important measure of quality Yet

despite the apparent simplicity of counting dollars and measuring assets

5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each

university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future

7

the means by which institutional wealth is measured varies considerably between ranking

systems

There are a number of revenue-based measures of resources Macleanrsquos uses public funding

of institutional budgets as a factor in its analysis conversely the Financial Times uses the

private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of

total score) and the US News and World Report (5 of total score) also measure alumni

financial support as a measure of quality

For reasons that are not entirely clear league tables tend to favour measures of

expenditures rather than revenues The Guardian looks at total institutional expenditures

as an indicator Institutional expenditure on student services is used as a measure of

institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total

institutional scores respectively) Rzezspospolita does not measure student services

expenditures directly but does measure student services outputs such as number of

student athletes and number of study clubs which amounts to more or less the same thing

Macleanrsquos also gives out 433 of its total score based on institutional expenditures on

scholarships and bursaries

Various aspects of physical infrastructure are also used as measures of institutional

resources most directly in the case of La Repubblica which bases 317 of its total rank on

the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding

assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type

of physical infrastructure measured is available Internet bandwidth which was used by

Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth

roughly 3 of the total score

By some distance the infrastructure indicators most favoured by the compilers of league

tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on

this with 12 of the total quality mark being taken from various types of library

infrastructure measurements (including acquisitions per year total volumes average

number of volumes per student and yearly library expenditure outside of acquisitions)

Netbig and Education18 also use library volume holdings while Asiaweek the Financial

Times and the Times also use measures of library expenditures outside of acquisitions or

computerization of library resources as measures of institutional quality

8

One important factor to note is that most ranking systems do not normalize their resource

and infrastructure measures That is to say it is raw spending power or simple size of

assets that is usually measured rather than spending per studentprofessor or assets per

studentprofessor As a result a number of these rankings systems have inbuilt biases

towards larger institutions

D Indicators of Learning Outputs

Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge

learned over the course of a baccalaureate degreemdashshould be a basic indicator of

institutional quality Unfortunately good means of measuring these outputsmdashlike the

National Survey of Student Engagement (NSSE) and the College Learning Assessment

(CLA)mdashhave only recently become available and for the most part institutions are still

keeping their scores secret Outside of these measures only a few very crude indicators are

available which likely explains why learning outputs do not feature especially prominently

in most ranking schemes

The simplest types of measures of learning outputs are those linked to graduation and

retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong

and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7

the latter three also use rates of graduation from Masterrsquos programs as indicators In some

cases the weights on these measures can be very highmdashin the Guangdong rankings

graduation rates account for over 50 of the rankingmdashbut in most cases the weights are

10 or less Retention rates commonly meaning the progression rate of first-year students

into second year are accorded less importance The US News and World Report Macleanrsquos

the Melbourne Institute and La Repubblica all employ retention measures as indicators but

none of them are worth more than 4 of total weighting Two publications make specific

indicators for retention and graduation of international students Macleanrsquos (graduation

rates of international students) and the Melbourne Institute (retention rates of international

students) The Washington Monthly looks specifically at institutional retention rates

adjusted for the participation of lower-income students and gives higher scores to

institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores

and number of Pell Grant recipients the US

7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18

9

News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos

ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)

E Indicators of Final Outcomes

Final outcomes are indications of generalized outcomes for students after graduation

Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness

good citizenship) but given the somewhat utilitarian justifications for education that are

currently in fashion (see Wolf 2000) employment outcomes are the most commonly used

measure of final outcomes These are given particular emphasis by the Guardian (where

employment outcomes are worth 17 of the total score) but are also used by the Financial

Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the

Times are interestingly not concerned with employment per se but with ldquoemployment in

an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher

Education Statistics Agency (HESA) uses Standard Occupational Classifications to

measure the proportion of graduates in professional or white-collar jobs anyone not in

such a job is considered not to be working in an area related to their studies (it is unclear

what methodology is used by the Financial Times and the Times although we suspect their

methods are broadly similar)

The only other measure of final outcomes in use is percentage of graduates returning for

additional education which is an indicator used by both the Melbourne Institute and the

Financial Times This is a particularly important indicator for the latter as it is worth 21

of the final ranking

The lack of indicators concerning final outcomes is interesting since most government-

sponsored performance-indicator regimes around the world are very much concerned

with such measures especially with respect to employment Possibly this indicates that

ranking systems simply do not view education outcomes as relevant measures of

educational quality Alternatively it may be the case that they simply have not found a

reliable indicator of outcomes or that there are reliable indicators but that there is so little

variation between institutions that it makes no sense to rank based on the data

10

F Indicators of Research

Many of the league tables covered in this survey include sections and weightings related

to universitiesrsquo research efforts It is in this field of measurement that we see the greatest

diversity of indicators Presumably this is because research inputs and outputs lend

themselves much more easily to measurement and manipulation than other areas of

institutional activity

Three studies include research staff as part of their ranking scheme La Repubblica at 952

the Melbourne Institute (4) and Wuhan (078)

Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used

method of looking at research quality but it is not universally admired because different

disciplines use different means to communicate major advances in knowledge (leading

scientists invariably produce large numbers of journal articles leading social scientists

may produce fewer journal articles but instead have one or two long important

monographsmdashsee Hicks 2004) There is also some concern among non-English speaking

countries that they are penalized in international rankings because so many of the major

journals (notably Science and Nature) are printed in English However the one set of

rankings that uses separate indicators to monitor articles published in English and articles

published in another language (the Wuhan rankings) shows that the two indicators are

positively correlated institutions that have more Chinese publications are also likely to

have more English publications and vice versa

Several sets of league tables measure bibliometric citations in various publication indices

The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a

weight of 20 of the final total Guangdong also monitors other Chinese universities

specifically for citations in engineering publications and weights this at 29 Moreover it

tacks on an additional 1049 for citations in science-oriented indices such as the Science

Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the

same category of scientific citations while the Melbourne Institute rates science citations at

68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social

Science Citation Index which does not include the humanities) are noted in only two

league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne

Institute (32) Another way of measuring research

11

impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a

weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the

Melbourne Institute

The complement of citations is of course publications Listing the number of publications

an individual group department or whole university releases can act as a weak substitute

for citationsmdashweak because simply publishing a paper or monograph is no guarantee that

the general public or other researchers will even glance at the work Guangdong gives an

1179 weighting to publications in science-oriented indices such as the Science Citation

Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same

indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong

even has a separate category just for measuring publications in Science and Nature

although it accords it an almost derisory weighting of 06 Under publications in social

science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their

final totals For publications in other indices (where the subject indices are

undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for

Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of

rankings to include research monographs weighted at 033

In countries where there are specific third-party evaluations of research output academic

quality of research is sometimes used as a research indicator The Times puts a very large

30 weight on this indicator while the Financial Times puts it at 119

Research awards are another handy third-party measurement of quality as the number of

international and national awards won by faculty andor graduates is often considered a

useful measure of institutional success International research awardsmdash specifically the

number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator

by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as

particularly suspect in some quarters given that the points are based on where the

recipient went to school rather than on where they are or were on the

8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the

most prominent publication indices

9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences

12

faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead

for nearly a century Wuhan uses similar measures but only accords them a weight of 14

National research awards are more common as a quality indicator used by La Repubblica

(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)

Financial indicators of research are also very common Research budgets as a factor in the

overall assessment of research in universities are covered by the Financial Times (9)

Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for

total amount of research expenditure unfortunately it is unclear precisely what this

research expenditure represents or how it is determined although it is clearly indicated

that it does not represent the total number of grants or projects at a university Total

number of research-based grants and projects is weighted by Education18 at 15 and

Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science

and engineering and another 55 to those for social sciences and humanities Similarly

the Melbourne Institute gives 6 of its overall weight to public-source grants making no

distinction between areas of study

In a slightly different vein Netbig (46) and Wuhan (278) both list the number of

research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica

(095) all weigh research-basedaffiliated research institutions or centres for studies

Finally one can also measure research not simply in terms of the amount of money it

generates but also in terms of the amount of future income it will generate Both

Guangdong (245) and Wuhan (193) measure the number of patents issued to

universities as a quality indicator

A final way of measuring an institutionrsquos research intensity is to look at the range of its

course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of

doctoral and Masterrsquos programs offered as a proxy for research intensity

As with physical and financial resources few if any of the research indicators are

normalized to account for institutional size (either by student or faculty numbers) In the

world of rankings bigger almost always means better an institution with 100 faculty with

ten citations apiece will always look worse than an institution with 1001 faculty

13

with one citation each To the extent that the raw production of knowledge matters this

form of measurement is acceptable To the extent that rankings are meant to show how

well institutions are doing on a like-to-like basis or to show the efficiency of universities it

is plainly inadequate This should be of particular concern to Chinese policy-makers

whose ranking systems are especially reliant on research-based indicators

G Indicators of Reputation

The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo

Those rankings systems which use the results of reputation surveys as an indicator do so as

an indirect measure of quality based on the assumption that the employers academics and

academic administrators surveyed have opinions of institutional quality that are informed

up-to-date and impartial While these assumptions are clearly open to debate they

nevertheless form an important basis for many ranking systems Another reason for using

reputation measures is the paucity of other data availablemdashsome countries have few

independent measures of teaching effectiveness university resources or output and

reputation can thus act as a useful surrogate Reputation rankings are often criticized as

simply quantifying the common ignorance of the people being surveyed However to the

extent that the people being surveyed hold positions which have the potential to affect

large numbers of young people and whose positions actually require some knowledge of

institutional quality (ie officials in charge of graduate admissions corporate recruiters

etc) then reputation rankings make sense because they provide useful information for

students about the perceived value of the degrees that they could obtain from various

universities

The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and

the Times which both accord reputation a weighting of 50 in their overall ranking

scheme Education18 assigns it almost as much significance at 40 The US News and World

Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just

below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)

and Netbig (15) The only other study to include reputation is Wuhan (117)

14

1 Please see Appendix B for a glossary of the various bodies referenced in this document

Appendix A Detailed Listing of Indicators and their Sources

Beginning Characteristics Indicator

Used By Source1

Incoming grades Macleans University

Percentage with grades above a set limit

Macleans University

US News and World Report

University

Performance on national standardised tests or benchmarks

Asiaweek University

Education18 3rd-party JUPAS

Financial Times Government agency 3rd-party UCAS

Guardian University Guide

Government agency 3rd-party UCAS

Melbourne Institute Government agency 3rd-party DEST

Netbig National entrance examination board

Times Good University Guide

Government agency 3rd-party UCAS

US News University

Wuhan Unknown presumed government 3rd-party

Student status La Repubblica Government agency 3rd-party MIUR

Admittance selectivity general

Asiaweek University

Admittance number of applications to places

Asiaweek University

Financial Times Government agency 3rd-party UCAS

La Repubblica Government agency 3rd-party MIUR

US News University

15

Beginning Characteristics Indicator

Used By Source1

Out-of-locality student percentage

Macleans University

International student percentages

Financial Times Government agency 3rd-party HESA

Macleans University

Shanghai Institute of Educational Science

Times World University

Wuhan Unknown presumed university

Undergraduate students among all students percentages

Netbig

Wuhan Unknown presumed university

Ethnic diversity in student body

Guardian University

Learning Inputs ndash Staff Indicator

Used By Source

Facultystudent ratio Asiaweek University

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Financial Times Government agency 3rd-party HESA

La Repubblica Government agency 3rd-party MIUR

Times Good University Guide

Government agency 3rd-party HESA

Times World University

US News University

Wuhan Unknown presumed university

16

Learning Inputs ndash Staff Indicator

Used By Source

Social science faculty student ratio

Melbourne Government agency 3rd-party DEST

Science faculty student ratio

Melbourne Government agency 3rd-party DEST

Administrative staff student ratio

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Staff student ratio (regardless of division)

Guardian Government agency 3rd-party HESA

Netbig University

Course per teacher La Repubblica Government agency 3rd-party MIUR

Per-teacher university spending

Asiaweek University

Faculty pay rates fortenured staff

Asiaweek University

US News University

Number of full-time part-time faculty

Netbig University

US News University

Wuhan Unknown presumed university

Faculty with researchprojects

Wuhan Unknown presumed university

Class size differentiation Macleans University

US News University

Classes taught by tenured faculty

Macleans University

Exchange programmes hosted

La Repubblica Government agency 3rd-party AgNaSoc

Number of classes lsquoactually taughtrsquo

La Repubblica Government agency 3rd-party MIUR

of international faculty (v faculty as a whole)

Times World University

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

PREMIEgraveRE PARTIE

INVENTAIRE DES INDICATEURS COURANTS

Agrave DES FINS DE PLANIFICATION DrsquoEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

Preacutesentation

Inventaire des indicateurs

Annexes quelques exemples de listes drsquoindicateurs

On peut reacutesister agrave lrsquoinvasion drsquoune armeacutee mais pas agrave une ideacutee - Victor Hugo Histoire drsquoun crime

wwwigopporg 3

La production drsquoindicateurs et leur application aux universiteacutes remontent agrave la fin des anneacutees

soixante Les premiegraveres listes apparaissent en 1968 soit il y a deacutejagrave une cinquantaine

drsquoanneacutees Les premiers indicateurs se preacutesentent sous la forme de donneacutees statistiques et

de caracteacuteristiques institutionnelles portant sur le nombre drsquoeacutetudiants la qualiteacute des

programmes et les problegravemes financiers que rencontrent les universiteacutes Les anneacutees quatre‐

vingt teacutemoignent drsquoun changement majeur lrsquointroduction dans le domaine de

lrsquoenseignement supeacuterieur des concepts et des techniques de planification strateacutegique

emprunteacutes au monde des affaires incite au deacuteveloppement drsquoautres types drsquoindicateurs et agrave

leur application agrave lrsquoensemble des domaines drsquoactiviteacutes universitaires Le livre de George

Keller publieacute en 1983 sous le titre de Academic Strategy The Management Revolution in

Higher Educations est significatif du virage majeur qui srsquoopegravere dans les universiteacutes nord‐

ameacutericaines

Nous assistons au cours de la mecircme peacuteriode agrave la naissance drsquoune veacuteritable typologie des

indicateurs Aux indicateurs simples et descriptifs viennent srsquoajouter des indicateurs

composeacutes (ratios pourcentage) les deux cateacutegories drsquoindicateurs se reacutefeacuterant de plus en plus

agrave des normes agrave des standards agrave des objectifs et agrave des cibles Les critegraveres retenus eacutevoluent

en direction drsquoindicateurs de rendement de performance de suivi La notion de laquo key

performance indicators (KPIs) raquo se retrouve au cours des derniegraveres anneacutees dans le titre de

la plupart des publications et dans la production drsquoindicateurs Plusieurs facteurs rendent

compte de cette eacutevolution Il faut retenir agrave cet eacutegard lrsquointeacutegration des indicateurs tant

qualitatifs que quantitatifs au processus de planification strateacutegique qursquoadoptent de plus les

universiteacutes lrsquoeacutevaluation par les pairs des faculteacutes et des deacutepartements des universiteacutes les

demandes de reddition de compte formuleacutees entre autres par les gouvernements les

Conseils drsquoadministration les laquo stakeholders raquo et le positionnement strateacutegique tant

national qursquointernational des universiteacutes Il est devenu eacutevident pour tous que les indicateurs

repreacutesentent un certain modegravele de production qui srsquoarticule autour de quatre cateacutegories

inter‐relieacutes de mesures que lrsquoon retrouve dans la plupart des listes et drsquoindicateurs input ndash

processus ndash output ndash reacutesultat

wwwigopporg 4

Les indicateurs font partie deacutesormais des outils indispensables auxquels ont recours les

gouvernements la communauteacute universitaire les eacutetudiants et leurs parents Leur nombre

varie entre dix et 300 indicateurs Bon nombre drsquoauteurs (Barbara Taylor (1993) Gaither amp

alii (1994) drsquoorganismes (COU) de producteurs de classements nationaux et internationaux

favorisent des listes courtes de 10 agrave 20 indicateurs ces listes courtes sont suffisantes pour

caracteacuteriser les activiteacutes drsquoune universiteacute repreacutesentent des indicateurs courants dont les

sources sont accessibles et permettent le repeacuterage (benchmarking) des universiteacutes

comparables En produisant des listes longues de 100 agrave 300 indicateurs les producteurs

preacutesentent plutocirct un reacutepertoire ou une banque drsquoindicateurs parmi lesquels les requeacuterants

comme les utilisateurs peuvent piger selon les objectifs poursuivis et les inteacuterecircts privileacutegieacutes

On trouvera dans les pages qui suivent un inventaire de quelque 175 indicateurs regroupeacutes

autour de 13 chefs de division ou domaines drsquoactiviteacutes (parmi lesquels deux seacuteries

drsquoindicateurs ndash environnement et planification et gestion strateacutegique constituent une

innovation)

Eacutetudiants

Enseignants

Personnels

Activiteacutes de recherche

Internationalisation

Inteacutegration des technologies de linformation

Bibliothegraveques

Revenus et deacutepenses

Espaces

Environnement

Reacuteputation

Planification et gestion strateacutegiques

Eacuteducation permanente

wwwigopporg 5

wwwigopporg

Tous les indicateurs des listes longues et tous les domaines drsquoactiviteacutes nrsquoont pas eacuteteacute retenus

Il srsquoagit drsquoun inventaire des indicateurs courants tels que recenseacutes dans plusieurs listes

courtes et longues et utiliseacutes agrave des fins de planification drsquoeacutevaluation de la performance et

de reddition de comptes De plus dans un contexte de mondialisation et

drsquointernationalisation il est devenu important et pertinent drsquoun point de vue strateacutegique de

retenir les indicateurs utiliseacutes dans les principaux classements nationaux (9 sur 20 possibles)

et internationaux (4) 1 Lrsquoencadreacute qui suit reacutesume ce choix

classements nationaux classements internationaux

universiteacutes canadiennes

et MEQ

USNews and

Report ampThe

Center

Macleans CHEDAAD GUARDIAN ASIAWEEKDEST

AUSTRALIE amp

MELBOURNE REPUBLICA

QS-TIMES SHANHAI TAIWAN

WEB

1 On trouvera en fin de document lrsquoensemble des reacutefeacuterences consulteacutees ainsi qursquoun annexe preacutesentant quelques exemples de listes drsquoindicateurs

6

1INVENTAIRE DES INDICATEURS COURANTS

Agrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

EacutetudiantsUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIEampMel

bourneREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre deacutetudiantsNombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input MEQ G10 DETYA THESARWUproportion deacutetudiantes input DETYA des eacutetudiants agrave temps partiel input AOTQU CHEDAAD Republicainscriptions au 1er cycle agrave temps plein input MEQ CHEDAAD inscriptions au 2e cycle input MEQ inscriptions au 3e cycle input MEQ eacutetudiants aux 2e et 3e cycles input G10 DETYA eacutetudiants internationaux input TU THES des eacutetudiants internationaux input Macleans CHEDAAD THES des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input MEQ stagiaires postdoctoraux input MEQ reacutesidents et reacutesidentes en meacutedecine input MEQ

inputrecrutement et admission inputdemandes dadmission aux programmes reacuteguliers input OTQ CHEDAAD offres dadmissiondemandes dadmission i t CHEDAAD offres dadmissiondemandes dadmission input CHEDAAD inscriptionsdemandes dadmission input T CHEDAAD Asiaweek Republica

inputdossier scolaire des nouveaux eacutetudiants (bulletin Cote R) input US NEWS Macleans Asiaweek

provenance des eacutetudiants input Macleanspopulation cible inputreacutegion meacutetropolitaine inputQueacutebec inputCanada inputInternational input

inputdiversiteacute ethnique input US NEWS Guardian

Bourses accordeacutees aux eacutetudiants - Financement moyen input TMEQpremier cycle input Macleans Republicadeuxiegraveme cycle input G10troisiegraveme cycle input G10

cheminement des eacutetudiants (temps complet) perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output QT MEQ US NEWS Macleans Republica diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output G10 MEQ US NEWS Macleans Melbourne Republica dilplomationmaicirctrise apregraves 4 ans output G10 MEQ US NEWS Macleans Republica diplomationdoctorat apregraves 6 ans output G10 MEQ US NEWS Macleans Republica

Deacutepenses de fonctionnement par eacutetudiant input CAUBOG10 US NEWS Macleans Guardian Asia weekFrais de scolariteacute input G10frais affeacuterents inputDeacutepenses pour les Services aux eacutetudiants input Macleans

taux demployabiliteacute apregraves 1 an outcome Ontario CHEDAAD Guardian de retour aux eacutetudes outcome Melbourne

Valeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome Guardian

RP gouvernance - 2008-06-23

2INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

EnseignantsUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Nombre denseignants Nombre total denseignants inputNombre de professeurs reacuteguliers input G10 MEQStatCan USNEWS Melbourne de professeures input OQ MelbourneNombre de chercheurs inputNombre de professeurs avec diplocircme international THESNombre de professeurs de clinique eacutequivalents temps complet input G10StatCanNombre de chargeacutes de cours et autres personnels eacutequivalents temps complet input USNEWSNombre dauxiliaires en eacutequivalence temps complet input

Caracteacuteristiques des professeurs reacuteguliers input des professeurs et chercheurs avec PhD (ou grade terminal) input USNEWS Macleansstructure dage du corps professoral input Asiaweek Melbourne Republicaacircge moyen des professeurs reacuteguliers inputacircge moyen des professeurs adjoints inputacircge moyen des professeurs agreacutegeacutes inputacircge moyen des professeurs titulaires input

reacutemuneacuteration moyenne des professeurs reacuteguliers input Q T U U S NEWS Asiaweek Melbournereacutemuneacuteration moyenne des professeurs reacuteguliers input QTU USNEWS Asiaweek Melbournedes professeurs adjoints inputdes professeurs agreacutegeacutes inputdes professeurs titulaire input

Activiteacutes denseignement inputnombre deacutetudiants EETC Professeur reacutegulier input G10 MEQ USNEWS Guardian Asiaweek Melbourne Republica THESnombre deacutetudiants EETC Professeur reacutegulier en sciences sociales input Melbournenombre deacutetudiants EETC Professeur reacutegulier en sciences input Melbournenombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input

Nombre de cours dispenseacutes par lInstitution (ANNUEL) input Republica cours assumeacutes par les professeurs reacuteguliers input MEQ USNEWS Macleans Republica cours assumeacutes par les chargeacutes de cours eacutequivalents temps complet input MEQTaille des groupe cours input MEQ USNEWS Macleans Guardian

RP gouvernance - 2008-06-23

3INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

PersonnelsUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Le personnel de soutientotal de tous les personnels reacuteguliers acadeacutemiques et non acd agrave temps complet inputnombre des personnels de soutien (staff) input G10 MelbourneDeacutepenses des personnel non-enseignant input MEQnombre des personnels de soutien (staff)- secteur enseignement inputnombre des personnels de soutien (staff) - secteur soutien acadeacutemique inputnombre des personnels de soutien (staff) - secteur administration inputnombre des personnels de soutien (staff) - secteur service aux eacutetudiants inputnombre des personnels de soutien (staff) - entreprises auxiliaires input

rectoratpersonnel dencadrement non acadeacutemique et cadres acadeacutemiques inputratio sur lensemble des personnels acadeacutemiques et non acadeacutemiques inputeacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input G10 Guardianpersonnel de soutien professeur reacutegulier input G10

RP gouvernance - 2008-06-23

4INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Activiteacutes de recherche USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Revenus de recherche de professeurs et chercheurs ayant fait une demande de subvention processus de professeurs et chercheurs ayant obtenu une subvention inputRevenus totaux de recherche (subventions et contrats) input CAUBO G10 The Center Asiaweek Melbournepart de la recherche contractuelle input MEQSubventions publiques input Macleans Melbournerevenus de recherche obtenus des conseils de recherche feacutedeacuteraux input AOQTU The Center Macleans Melbournerevenus moyens de recherche par professeur et chercheur input InfosourceMEQ des deacutepenses de recherchebudget total des deacutepenses input TNombre de chaires et de centre dexcellence output AFondation canadienne pour linnovation (FCI) input G10 -nombre de projets output G10 -valeur $ output G10

nombre de programmes de maicirctrise de recherche input The Center Asiaweeknombre de programmes de doctorat input The Center Asiaweek

Publications et prix outputNombre de publications output Infosource Asiaweek Melbourne

Shanghai Leiden Taiwan

Nombre de publications par professeur et chercheur output Infosource CHEDAAD ShanghaiPublications dans Nature and Science output TaiwanNombre total de citations output InfosourceT Asiaweek MelbourneNombre de citations par professeur et chercheur output InfosourceT Melbourne THESNombre de citations par publication output T CHEDAAD LeidenHighly cited researchers dans 21 secteurs disciplinaires

output

ShanghaiTaiwan Leiden

Impact des publications selon les secteurs disciplinaires output Infosource TaiwanTHES Shanghai

Domaines disciplinaires dexcellence inputTaiwanShanghai THES

Prix et meacutedailles de niveau national et international output G10 The Center Macleans Melbourne Republica ShanghaiNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur output CHEDAADnombre de brevets et de licences output G10 CHEDAADnombre de spin-off companies G10

RP gouvernance - 2008-06-23

5INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

InternationalisationUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

deacutepenses pour le secteur internationalisation input MEQnombre deacutetudiants en stage agrave leacutetrangernombre deacutetudiants temps complet input tes un cannombre deacutetudiants internationauxnombre deacutetudiants temps complet input tes un can Melbourne THESnombre deacutetudiants en cotutellesnombre deacutetudiant des cycles supeacuterieurs input CHEDAADnombre de professeurs inviteacutes internationaux processus tes un can CHEDAADprofesseurs inviteacutes agrave leacutetranger processus CHEDAADnombre deacutetudiants posdoctoraux processus tes un can CHEDAADproportion de cours de langues eacutetrangegraveres processusnombre de programmes deacutechange processus tes un can CHEDAADEntentes de partenariat outputPrix internationaux output G10 The Center Macleans Melbourne Republica Shanghai

RP gouvernance - 2008-06-23

6INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Inteacutegration des technologies de linformation USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Deacutepenses totales consacreacutees aux TIC input MEQpersonnel techniqueenseignant reacutegulier inputpersonnel techniqueeacutetudiant en eacutequivalence temps complet inputExistence dun reacuteseau informatique (accessible agrave linterne et agrave lexterne) outputaccegraves au reacuteseau avec fils et sans fils processusaccegraves aux ressources documentaires des bibliothegraveques outputService de courriel interne et externe en mode WEB outputNombre dordinateurs par professeur et par eacutetudiant outputNombre de cours en ligne processusnombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) outputAccegraves des professeurs aux banque de donneacutees et aux logiciels processus

Site WEB (eacutevaluation du site WEB) WEBOMETRICSSize (nombre de pages recenseacutees) outcomevisibilitty (consultation observeacutee selon le nobre de lien externe unique) outcomeRich files (intensiteacute de lactiviteacute acadeacutemique) outcomeScholar (activiteacute de recherche publications et citations) outcome

RP gouvernance - 2008-06-23

7INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

BibliothegravequesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre total de volumes et de documents audiovisuels et eacutelectroniques input G10 ARLacquisition de volumes (au cours de lanneacutee) input ARL Macleansnombre total de volumes eacutetudiants en eacutequivalence temps complet input G10 ARL Macleans Deacutepenses des bibliothegraveques input MEQ Macleans AsiaweekDeacutepenses des bibliothegravequesbudget de fonctionnement input ARLDeacutepenses des bibliothegravequeseacutetudiants en eacutequivalence temps complet input G10 ARLPersonnels reacuteguliers agrave temps complet input G10 ARLClassement outcome ARL

RP gouvernance - 2008-06-23

8INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Revenus et deacutepenses USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Produits et revenusSubvention du ministegravere de lEacuteducation du Queacutebec input MEQ subvMEQrevenus de fonctionnement input G10 MacleansDroits de scolariteacute input G10 droits de scolariteacute revenus de fonctionnement input G10autres produits input G10Total des revenus de fonctionnement input G10 MEQ Guardian revenus de fonctionnement eacutetudiants en eacutequivalence temps complet input CAUBO USNews MacleansRevenus de recherche input G10encaissement par nature (dons immobilisations chaires bourses) input G10 fonds de dotation input G10 CenterContribution des alumni aux encaissements et fonds de dotation input USNews Macleans fonds de dotation eacutetudiants en eacutequivalence temps complet input G10Total des produits G10

Charges et deacutepenses G10salaire des professeurs input G10 MEQsalaire des chargeacutes de cours input G10salaires des auxiliaires input MEQsalaires des auxiliaires input MEQsalaire des personnels (staff) input G10autres deacutepenses input G10total des deacutepenses input G10 deacutepenses denseignement deacutepenses totales de fonctionnement input UBC MEQ

Chargesproduits processusEacutequilibre budgeacutetaire revenus et deacutepenses de fonctionnement processus MEQ

RP gouvernance - 2008-06-23

9INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

EspacesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Nombre de megravetres carreacutes bruts input G10MCB eacutetudiants en eacutequivalence temps complet input G10megravetres carreacutes nets alloueacutes input MEQNombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input Tbesoin despace en megravetres carreacutes nets inputExistence dun plan directeur des espaces inputBudget preacutevu au plan directeur des espaces input TBudget alloueacute aux espaces input MEQdeacutepenses de fonctionnement nombre de megravetres carreacutes bruts input G10

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Environnement USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

consommation de leacutenergie (exprimeacutee en BTU processus universiteacutes canadiennes et ameacutericaines BTUpopulation totale de luniversiteacute processus voir en particulier Penn State et Umich de leacutenergie produit par des sources renouvelable processusUtilisation de leau processuslitres deau par personne processusGaz agrave effet de serre (en tonnes meacutetriques CO2 eacutequivalents) processusMT CO2 par personne processusDeacutechets solides (en tonnes) processusTonnes de deacutechets solides per capita processus de deacutechets solides recycleacutes processus

RP gouvernance - 2008-06-23

10INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

ReacuteputationUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

enquecircte meneacutee aupregraves des eacutetudiants outcome NSSE Melbourneenquecircte meneacutee aupregraves des diplocircmeacutes outcomeenquecircte meneacutee aupregraves des pairs outcome USNews Asiaweek THESenquecircte meneacutee aupregraves des employeurs outcome Macleans Melbourneenquecircte meneacutee aupregraves des stakeholders outcome Macleans Melbourneenquecircte meneacutee aupregraves du monde des affaires outcome Macleans Melbournecouverture meacutediatique outcome AT

RP gouvernance - 2008-06-23

11INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Planification et gestion strateacutegiquesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Pouvoirs des corps universitaire processus un Canliens dynamiques entre Conseil Assembleacutee universitaire commission des eacutetudes processus un Canprocessus formel de planification strateacutegique processus un Canproduction de donneacutees et indicateurs processus un Canprocessus deacutevaluation des faculteacutes deacutepartements et services processus un Canrevue systeacutematique des responsabiliteacutes et efficaciteacute des comiteacutes processus un Candeacutepenses de ladministration centrale deacutepenses de fonctionnement processus un Can

RP gouvernance - 2008-06-23

12INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Eacuteducation permanenteUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre de programmes offerts input un Cannombre deacutetudiants inscrits input un CanRevenus et deacutepenses du service input un Can

RP gouvernance - 2008-06-23

13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes

REacuteFEacuteRENCES

Aperccedilu densemble des principaux indicateurs

GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost

Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes

MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf

Indicateurs utiliseacutes dans les classements internationaux

Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml

RP gouvernance - 2008-06-23

ANNEXES

Quelques exemples de listes drsquoindicateurs

1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)

Voir page suivante

Preacutepareacute par Roland Proulx

Consultant en planification institutionnelle et veille strateacutegique

Montreacuteal mai 2008

Performance Indicators

Discussion Paper

May 2004 (Prepared by the Office of Planning amp Analysis)

Table of Contents

Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25

2

Table 1 ndash Number of Indictors by Type of Data Source

Table 1 shows that surveys are the least frequently used source of data for indicators

Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a

plurality of indicators come from this source This measure somewhat underestimates the

importance of surveys however as it does not account for the weighting given to each

indicator in each study In the THES World Rankings for instance there is only a single

survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos

Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one

indicator is worth 20 of the final score

Raw

indIcator count

Surve

y data

Third parties

Universities

AsiaweekmdashAsias Best Universities 18 - - 18

Daily Telegraph (2003) 1 - 1 -

Education18com 9 3 4 2

Excelencia 2001 71 - 71 -

Financial Times (2003) 17 - 17 -

Guangdong Institute of Management Science 17 - 14 3

GuardianmdashUniversity Guide 2005 7 - 2 5

La Repubblica 23 2 21 -

Macleans University Rankings 24 1 5 18

Melbourne Institutemdash International Standing of Australian Universities

26 3 23 -

Netbig 2004 18 1 10 7

Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15

Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities

6 - 5 1

The TimesmdashGood University Guide 2005 9 - 9 -

Times Higher Education SupplementmdashWorld University Rankings

5 1 1 3

US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11

Washington MonthlymdashCollege Rankings 2005 8 - 1 7

Wuhan University Centre for Science Evaluation 45 2 22 21

3

A Indicators of Beginning Characteristics

ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of

students at the time they begin their studies Fourteen of the 18 rankings examined in this

study use one or more indicators of the beginning characteristics of students to arrive at

their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on

these factors with 33 of the total ranking coming from this class of indicators but the

Guardian Education18 Asiaweek and the two other North American surveys also place

considerable emphasis on this category

There are six main indicators used to determine which institutions have students with

positive ldquobeginning characteristicsrdquo

The most common measure of beginning characteristics is performance on national

standardized tests with nine surveys using this as a measure Education18 and the

Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used

by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)

Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this

data is collected and standardized by national bodies it has the benefit of being seen as a

relatively impartial method of determining the relative ldquostrengthrdquo of the students entering

each institution Institutionsrsquo results can be scored by showing either averages or the

percentage of entering students meeting a particular standard

Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses

secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This

is a second-best solution made necessary by the absence of any national standardized test

in Canada (or indeed of any provincial standardized tests at the end of secondary school

in provinces other than Alberta) The lack of national standardization makes this an

undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is

truly equivalent to an ldquoArdquo in another jurisdiction

Another measure of the strength of the student body is the percentage of incoming

students receiving (third-party) scholarships which is worth 11 of the score in the

Wuhan survey One can also approach the issue by measuring institutional selectivity In

effect this method infers the strength of the student body by the proportion of

4

applicants rejected the theory being that the higher the number of rejected applicants the

stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)

Normally this measure is expressed as a straight ratio of acceptances to applications but it

can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the

greatest weight on this measure) as a ratio of enrolments to applications Within the US

there is some dispute as to what constitutes an offer of admission and whether or not late

admissions are included as noted by Ganeshananthan (2003)

Student bodies are often considered to be strong if the school is able to attract a large

number of international or out-of-district students or if they contain people from diverse

ethnic backgrounds A number of league tables use the international student indicator

(which like the selectivity indicator is arguably as much an indicator of prestige and

reputation as it is of student characteristics) although in no case does this indicator

account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a

quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data

on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian

puts a somewhat larger emphasis on this indicator in comparison to other league tables

which use similar variables

A very different take on this idea is present in the Washington Monthly which released its

first set of College Rankings in September 2005 With the declared aim of using an

institutionrsquos commitment to social mobility as a measure of quality it uses the percentage

of students from low-income backgrounds as an indicator (with percentage of students

receiving need-based government (Pell) grants used as a proxy)

Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy

statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the

percentage of the student population who are graduate students (arguably this is a

research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the

number of graduate students auditing classes is used as an indicator the assumption is

presumably that if people are auditing then the classes must be very attractive The Italian

La Repubblica ranks an institution according to the number of part-time students it has

contrary to prevailing North American views on the undesirability of part-time study the

Italian rankings see higher numbers of part-time students in a positive light as it is

evidence that an institution is becoming less rigid in its timetabling and

5

permitting students to juggle both work and study something which was nearly

impossible in that country just a few years ago

The Washington Monthly also has a final category of indicators which reflect studentsrsquo

beginning characteristics namely their likelihood of performing community service as

measured by the percentage of students in the US Peace Corps and Reserve Officer

Training Corps (ROTC) and the percentage of students involved in work-study in the

community3 Together these three indicators account for 33 of an institutionrsquos total

ranking

B Indicators of Learning InputsmdashStaff

Generally both the quantity and quality of staff are positively correlated with institutional

quality The problem of course is finding useful metrics for each of these factors

especially if one excludes as we have done here measures of research performance and

research intensity putting them in a separate category4

The simplest measure is simply the number of faculty unadjusted for things like size of

student body Most national league tables however prefer to use variations on the

concept of facultystudent ratio Others try to measure teaching intensity with measures

such as courses per teacher or hours spent in class per student (both in La Repubblica)

These kinds of measures usually account for between 2-5 of the final rankings although

in some cases (ie the Guardian) this figure can be as high as 20

Another important way of measuring how faculty resources are deployed is the measure

of average class size which is used only by Macleanrsquos and the US News and World Report

Ostensibly the reason for measuring class size is to account in some form for the degree

3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category

6

of attention devoted to actually teaching students Despite the fact that important research

(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level

the use of this indicator appears to be a spillover from the North American debates on

class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein

2000) Regardless of why the indicators are used they are extraordinarily important to

these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and

World Reportrsquos rankings respectively

A number of ranking systems try to look at staff qualifications such as the number of

PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the

Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further

than other surveys and actually looks at the proportion of classes taught by tenure-track

staff Others (ie THES) look at the number of foreign faculty based on the assumption

that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still

others (ie La Repubblica) look at the age structure of the faculty Another proxy for

institutional quality is the pay rates for tenured staff on the assumption that institutions

with higher rates of pay on average attract better faculty this measure has been used both

by the US News and World Report and Asiaweek

Finally a number of league tables rank faculty inputs on the basis of standardized third-

party evaluations 6

Education18 the Financial Times the Times the Guardian and the US

News and World Report league tables all use some sort of ranking criterion based at least in

part on this indicator or variations thereof

C Indicators of Learning InputsmdashResources

Resource inputsmdashcrudely the amount of current dollars equipment and books available

to students at an institutionmdashare widely considered an important measure of quality Yet

despite the apparent simplicity of counting dollars and measuring assets

5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each

university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future

7

the means by which institutional wealth is measured varies considerably between ranking

systems

There are a number of revenue-based measures of resources Macleanrsquos uses public funding

of institutional budgets as a factor in its analysis conversely the Financial Times uses the

private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of

total score) and the US News and World Report (5 of total score) also measure alumni

financial support as a measure of quality

For reasons that are not entirely clear league tables tend to favour measures of

expenditures rather than revenues The Guardian looks at total institutional expenditures

as an indicator Institutional expenditure on student services is used as a measure of

institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total

institutional scores respectively) Rzezspospolita does not measure student services

expenditures directly but does measure student services outputs such as number of

student athletes and number of study clubs which amounts to more or less the same thing

Macleanrsquos also gives out 433 of its total score based on institutional expenditures on

scholarships and bursaries

Various aspects of physical infrastructure are also used as measures of institutional

resources most directly in the case of La Repubblica which bases 317 of its total rank on

the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding

assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type

of physical infrastructure measured is available Internet bandwidth which was used by

Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth

roughly 3 of the total score

By some distance the infrastructure indicators most favoured by the compilers of league

tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on

this with 12 of the total quality mark being taken from various types of library

infrastructure measurements (including acquisitions per year total volumes average

number of volumes per student and yearly library expenditure outside of acquisitions)

Netbig and Education18 also use library volume holdings while Asiaweek the Financial

Times and the Times also use measures of library expenditures outside of acquisitions or

computerization of library resources as measures of institutional quality

8

One important factor to note is that most ranking systems do not normalize their resource

and infrastructure measures That is to say it is raw spending power or simple size of

assets that is usually measured rather than spending per studentprofessor or assets per

studentprofessor As a result a number of these rankings systems have inbuilt biases

towards larger institutions

D Indicators of Learning Outputs

Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge

learned over the course of a baccalaureate degreemdashshould be a basic indicator of

institutional quality Unfortunately good means of measuring these outputsmdashlike the

National Survey of Student Engagement (NSSE) and the College Learning Assessment

(CLA)mdashhave only recently become available and for the most part institutions are still

keeping their scores secret Outside of these measures only a few very crude indicators are

available which likely explains why learning outputs do not feature especially prominently

in most ranking schemes

The simplest types of measures of learning outputs are those linked to graduation and

retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong

and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7

the latter three also use rates of graduation from Masterrsquos programs as indicators In some

cases the weights on these measures can be very highmdashin the Guangdong rankings

graduation rates account for over 50 of the rankingmdashbut in most cases the weights are

10 or less Retention rates commonly meaning the progression rate of first-year students

into second year are accorded less importance The US News and World Report Macleanrsquos

the Melbourne Institute and La Repubblica all employ retention measures as indicators but

none of them are worth more than 4 of total weighting Two publications make specific

indicators for retention and graduation of international students Macleanrsquos (graduation

rates of international students) and the Melbourne Institute (retention rates of international

students) The Washington Monthly looks specifically at institutional retention rates

adjusted for the participation of lower-income students and gives higher scores to

institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores

and number of Pell Grant recipients the US

7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18

9

News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos

ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)

E Indicators of Final Outcomes

Final outcomes are indications of generalized outcomes for students after graduation

Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness

good citizenship) but given the somewhat utilitarian justifications for education that are

currently in fashion (see Wolf 2000) employment outcomes are the most commonly used

measure of final outcomes These are given particular emphasis by the Guardian (where

employment outcomes are worth 17 of the total score) but are also used by the Financial

Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the

Times are interestingly not concerned with employment per se but with ldquoemployment in

an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher

Education Statistics Agency (HESA) uses Standard Occupational Classifications to

measure the proportion of graduates in professional or white-collar jobs anyone not in

such a job is considered not to be working in an area related to their studies (it is unclear

what methodology is used by the Financial Times and the Times although we suspect their

methods are broadly similar)

The only other measure of final outcomes in use is percentage of graduates returning for

additional education which is an indicator used by both the Melbourne Institute and the

Financial Times This is a particularly important indicator for the latter as it is worth 21

of the final ranking

The lack of indicators concerning final outcomes is interesting since most government-

sponsored performance-indicator regimes around the world are very much concerned

with such measures especially with respect to employment Possibly this indicates that

ranking systems simply do not view education outcomes as relevant measures of

educational quality Alternatively it may be the case that they simply have not found a

reliable indicator of outcomes or that there are reliable indicators but that there is so little

variation between institutions that it makes no sense to rank based on the data

10

F Indicators of Research

Many of the league tables covered in this survey include sections and weightings related

to universitiesrsquo research efforts It is in this field of measurement that we see the greatest

diversity of indicators Presumably this is because research inputs and outputs lend

themselves much more easily to measurement and manipulation than other areas of

institutional activity

Three studies include research staff as part of their ranking scheme La Repubblica at 952

the Melbourne Institute (4) and Wuhan (078)

Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used

method of looking at research quality but it is not universally admired because different

disciplines use different means to communicate major advances in knowledge (leading

scientists invariably produce large numbers of journal articles leading social scientists

may produce fewer journal articles but instead have one or two long important

monographsmdashsee Hicks 2004) There is also some concern among non-English speaking

countries that they are penalized in international rankings because so many of the major

journals (notably Science and Nature) are printed in English However the one set of

rankings that uses separate indicators to monitor articles published in English and articles

published in another language (the Wuhan rankings) shows that the two indicators are

positively correlated institutions that have more Chinese publications are also likely to

have more English publications and vice versa

Several sets of league tables measure bibliometric citations in various publication indices

The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a

weight of 20 of the final total Guangdong also monitors other Chinese universities

specifically for citations in engineering publications and weights this at 29 Moreover it

tacks on an additional 1049 for citations in science-oriented indices such as the Science

Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the

same category of scientific citations while the Melbourne Institute rates science citations at

68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social

Science Citation Index which does not include the humanities) are noted in only two

league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne

Institute (32) Another way of measuring research

11

impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a

weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the

Melbourne Institute

The complement of citations is of course publications Listing the number of publications

an individual group department or whole university releases can act as a weak substitute

for citationsmdashweak because simply publishing a paper or monograph is no guarantee that

the general public or other researchers will even glance at the work Guangdong gives an

1179 weighting to publications in science-oriented indices such as the Science Citation

Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same

indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong

even has a separate category just for measuring publications in Science and Nature

although it accords it an almost derisory weighting of 06 Under publications in social

science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their

final totals For publications in other indices (where the subject indices are

undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for

Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of

rankings to include research monographs weighted at 033

In countries where there are specific third-party evaluations of research output academic

quality of research is sometimes used as a research indicator The Times puts a very large

30 weight on this indicator while the Financial Times puts it at 119

Research awards are another handy third-party measurement of quality as the number of

international and national awards won by faculty andor graduates is often considered a

useful measure of institutional success International research awardsmdash specifically the

number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator

by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as

particularly suspect in some quarters given that the points are based on where the

recipient went to school rather than on where they are or were on the

8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the

most prominent publication indices

9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences

12

faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead

for nearly a century Wuhan uses similar measures but only accords them a weight of 14

National research awards are more common as a quality indicator used by La Repubblica

(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)

Financial indicators of research are also very common Research budgets as a factor in the

overall assessment of research in universities are covered by the Financial Times (9)

Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for

total amount of research expenditure unfortunately it is unclear precisely what this

research expenditure represents or how it is determined although it is clearly indicated

that it does not represent the total number of grants or projects at a university Total

number of research-based grants and projects is weighted by Education18 at 15 and

Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science

and engineering and another 55 to those for social sciences and humanities Similarly

the Melbourne Institute gives 6 of its overall weight to public-source grants making no

distinction between areas of study

In a slightly different vein Netbig (46) and Wuhan (278) both list the number of

research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica

(095) all weigh research-basedaffiliated research institutions or centres for studies

Finally one can also measure research not simply in terms of the amount of money it

generates but also in terms of the amount of future income it will generate Both

Guangdong (245) and Wuhan (193) measure the number of patents issued to

universities as a quality indicator

A final way of measuring an institutionrsquos research intensity is to look at the range of its

course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of

doctoral and Masterrsquos programs offered as a proxy for research intensity

As with physical and financial resources few if any of the research indicators are

normalized to account for institutional size (either by student or faculty numbers) In the

world of rankings bigger almost always means better an institution with 100 faculty with

ten citations apiece will always look worse than an institution with 1001 faculty

13

with one citation each To the extent that the raw production of knowledge matters this

form of measurement is acceptable To the extent that rankings are meant to show how

well institutions are doing on a like-to-like basis or to show the efficiency of universities it

is plainly inadequate This should be of particular concern to Chinese policy-makers

whose ranking systems are especially reliant on research-based indicators

G Indicators of Reputation

The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo

Those rankings systems which use the results of reputation surveys as an indicator do so as

an indirect measure of quality based on the assumption that the employers academics and

academic administrators surveyed have opinions of institutional quality that are informed

up-to-date and impartial While these assumptions are clearly open to debate they

nevertheless form an important basis for many ranking systems Another reason for using

reputation measures is the paucity of other data availablemdashsome countries have few

independent measures of teaching effectiveness university resources or output and

reputation can thus act as a useful surrogate Reputation rankings are often criticized as

simply quantifying the common ignorance of the people being surveyed However to the

extent that the people being surveyed hold positions which have the potential to affect

large numbers of young people and whose positions actually require some knowledge of

institutional quality (ie officials in charge of graduate admissions corporate recruiters

etc) then reputation rankings make sense because they provide useful information for

students about the perceived value of the degrees that they could obtain from various

universities

The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and

the Times which both accord reputation a weighting of 50 in their overall ranking

scheme Education18 assigns it almost as much significance at 40 The US News and World

Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just

below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)

and Netbig (15) The only other study to include reputation is Wuhan (117)

14

1 Please see Appendix B for a glossary of the various bodies referenced in this document

Appendix A Detailed Listing of Indicators and their Sources

Beginning Characteristics Indicator

Used By Source1

Incoming grades Macleans University

Percentage with grades above a set limit

Macleans University

US News and World Report

University

Performance on national standardised tests or benchmarks

Asiaweek University

Education18 3rd-party JUPAS

Financial Times Government agency 3rd-party UCAS

Guardian University Guide

Government agency 3rd-party UCAS

Melbourne Institute Government agency 3rd-party DEST

Netbig National entrance examination board

Times Good University Guide

Government agency 3rd-party UCAS

US News University

Wuhan Unknown presumed government 3rd-party

Student status La Repubblica Government agency 3rd-party MIUR

Admittance selectivity general

Asiaweek University

Admittance number of applications to places

Asiaweek University

Financial Times Government agency 3rd-party UCAS

La Repubblica Government agency 3rd-party MIUR

US News University

15

Beginning Characteristics Indicator

Used By Source1

Out-of-locality student percentage

Macleans University

International student percentages

Financial Times Government agency 3rd-party HESA

Macleans University

Shanghai Institute of Educational Science

Times World University

Wuhan Unknown presumed university

Undergraduate students among all students percentages

Netbig

Wuhan Unknown presumed university

Ethnic diversity in student body

Guardian University

Learning Inputs ndash Staff Indicator

Used By Source

Facultystudent ratio Asiaweek University

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Financial Times Government agency 3rd-party HESA

La Repubblica Government agency 3rd-party MIUR

Times Good University Guide

Government agency 3rd-party HESA

Times World University

US News University

Wuhan Unknown presumed university

16

Learning Inputs ndash Staff Indicator

Used By Source

Social science faculty student ratio

Melbourne Government agency 3rd-party DEST

Science faculty student ratio

Melbourne Government agency 3rd-party DEST

Administrative staff student ratio

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Staff student ratio (regardless of division)

Guardian Government agency 3rd-party HESA

Netbig University

Course per teacher La Repubblica Government agency 3rd-party MIUR

Per-teacher university spending

Asiaweek University

Faculty pay rates fortenured staff

Asiaweek University

US News University

Number of full-time part-time faculty

Netbig University

US News University

Wuhan Unknown presumed university

Faculty with researchprojects

Wuhan Unknown presumed university

Class size differentiation Macleans University

US News University

Classes taught by tenured faculty

Macleans University

Exchange programmes hosted

La Repubblica Government agency 3rd-party AgNaSoc

Number of classes lsquoactually taughtrsquo

La Repubblica Government agency 3rd-party MIUR

of international faculty (v faculty as a whole)

Times World University

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

La production drsquoindicateurs et leur application aux universiteacutes remontent agrave la fin des anneacutees

soixante Les premiegraveres listes apparaissent en 1968 soit il y a deacutejagrave une cinquantaine

drsquoanneacutees Les premiers indicateurs se preacutesentent sous la forme de donneacutees statistiques et

de caracteacuteristiques institutionnelles portant sur le nombre drsquoeacutetudiants la qualiteacute des

programmes et les problegravemes financiers que rencontrent les universiteacutes Les anneacutees quatre‐

vingt teacutemoignent drsquoun changement majeur lrsquointroduction dans le domaine de

lrsquoenseignement supeacuterieur des concepts et des techniques de planification strateacutegique

emprunteacutes au monde des affaires incite au deacuteveloppement drsquoautres types drsquoindicateurs et agrave

leur application agrave lrsquoensemble des domaines drsquoactiviteacutes universitaires Le livre de George

Keller publieacute en 1983 sous le titre de Academic Strategy The Management Revolution in

Higher Educations est significatif du virage majeur qui srsquoopegravere dans les universiteacutes nord‐

ameacutericaines

Nous assistons au cours de la mecircme peacuteriode agrave la naissance drsquoune veacuteritable typologie des

indicateurs Aux indicateurs simples et descriptifs viennent srsquoajouter des indicateurs

composeacutes (ratios pourcentage) les deux cateacutegories drsquoindicateurs se reacutefeacuterant de plus en plus

agrave des normes agrave des standards agrave des objectifs et agrave des cibles Les critegraveres retenus eacutevoluent

en direction drsquoindicateurs de rendement de performance de suivi La notion de laquo key

performance indicators (KPIs) raquo se retrouve au cours des derniegraveres anneacutees dans le titre de

la plupart des publications et dans la production drsquoindicateurs Plusieurs facteurs rendent

compte de cette eacutevolution Il faut retenir agrave cet eacutegard lrsquointeacutegration des indicateurs tant

qualitatifs que quantitatifs au processus de planification strateacutegique qursquoadoptent de plus les

universiteacutes lrsquoeacutevaluation par les pairs des faculteacutes et des deacutepartements des universiteacutes les

demandes de reddition de compte formuleacutees entre autres par les gouvernements les

Conseils drsquoadministration les laquo stakeholders raquo et le positionnement strateacutegique tant

national qursquointernational des universiteacutes Il est devenu eacutevident pour tous que les indicateurs

repreacutesentent un certain modegravele de production qui srsquoarticule autour de quatre cateacutegories

inter‐relieacutes de mesures que lrsquoon retrouve dans la plupart des listes et drsquoindicateurs input ndash

processus ndash output ndash reacutesultat

wwwigopporg 4

Les indicateurs font partie deacutesormais des outils indispensables auxquels ont recours les

gouvernements la communauteacute universitaire les eacutetudiants et leurs parents Leur nombre

varie entre dix et 300 indicateurs Bon nombre drsquoauteurs (Barbara Taylor (1993) Gaither amp

alii (1994) drsquoorganismes (COU) de producteurs de classements nationaux et internationaux

favorisent des listes courtes de 10 agrave 20 indicateurs ces listes courtes sont suffisantes pour

caracteacuteriser les activiteacutes drsquoune universiteacute repreacutesentent des indicateurs courants dont les

sources sont accessibles et permettent le repeacuterage (benchmarking) des universiteacutes

comparables En produisant des listes longues de 100 agrave 300 indicateurs les producteurs

preacutesentent plutocirct un reacutepertoire ou une banque drsquoindicateurs parmi lesquels les requeacuterants

comme les utilisateurs peuvent piger selon les objectifs poursuivis et les inteacuterecircts privileacutegieacutes

On trouvera dans les pages qui suivent un inventaire de quelque 175 indicateurs regroupeacutes

autour de 13 chefs de division ou domaines drsquoactiviteacutes (parmi lesquels deux seacuteries

drsquoindicateurs ndash environnement et planification et gestion strateacutegique constituent une

innovation)

Eacutetudiants

Enseignants

Personnels

Activiteacutes de recherche

Internationalisation

Inteacutegration des technologies de linformation

Bibliothegraveques

Revenus et deacutepenses

Espaces

Environnement

Reacuteputation

Planification et gestion strateacutegiques

Eacuteducation permanente

wwwigopporg 5

wwwigopporg

Tous les indicateurs des listes longues et tous les domaines drsquoactiviteacutes nrsquoont pas eacuteteacute retenus

Il srsquoagit drsquoun inventaire des indicateurs courants tels que recenseacutes dans plusieurs listes

courtes et longues et utiliseacutes agrave des fins de planification drsquoeacutevaluation de la performance et

de reddition de comptes De plus dans un contexte de mondialisation et

drsquointernationalisation il est devenu important et pertinent drsquoun point de vue strateacutegique de

retenir les indicateurs utiliseacutes dans les principaux classements nationaux (9 sur 20 possibles)

et internationaux (4) 1 Lrsquoencadreacute qui suit reacutesume ce choix

classements nationaux classements internationaux

universiteacutes canadiennes

et MEQ

USNews and

Report ampThe

Center

Macleans CHEDAAD GUARDIAN ASIAWEEKDEST

AUSTRALIE amp

MELBOURNE REPUBLICA

QS-TIMES SHANHAI TAIWAN

WEB

1 On trouvera en fin de document lrsquoensemble des reacutefeacuterences consulteacutees ainsi qursquoun annexe preacutesentant quelques exemples de listes drsquoindicateurs

6

1INVENTAIRE DES INDICATEURS COURANTS

Agrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

EacutetudiantsUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIEampMel

bourneREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre deacutetudiantsNombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input MEQ G10 DETYA THESARWUproportion deacutetudiantes input DETYA des eacutetudiants agrave temps partiel input AOTQU CHEDAAD Republicainscriptions au 1er cycle agrave temps plein input MEQ CHEDAAD inscriptions au 2e cycle input MEQ inscriptions au 3e cycle input MEQ eacutetudiants aux 2e et 3e cycles input G10 DETYA eacutetudiants internationaux input TU THES des eacutetudiants internationaux input Macleans CHEDAAD THES des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input MEQ stagiaires postdoctoraux input MEQ reacutesidents et reacutesidentes en meacutedecine input MEQ

inputrecrutement et admission inputdemandes dadmission aux programmes reacuteguliers input OTQ CHEDAAD offres dadmissiondemandes dadmission i t CHEDAAD offres dadmissiondemandes dadmission input CHEDAAD inscriptionsdemandes dadmission input T CHEDAAD Asiaweek Republica

inputdossier scolaire des nouveaux eacutetudiants (bulletin Cote R) input US NEWS Macleans Asiaweek

provenance des eacutetudiants input Macleanspopulation cible inputreacutegion meacutetropolitaine inputQueacutebec inputCanada inputInternational input

inputdiversiteacute ethnique input US NEWS Guardian

Bourses accordeacutees aux eacutetudiants - Financement moyen input TMEQpremier cycle input Macleans Republicadeuxiegraveme cycle input G10troisiegraveme cycle input G10

cheminement des eacutetudiants (temps complet) perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output QT MEQ US NEWS Macleans Republica diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output G10 MEQ US NEWS Macleans Melbourne Republica dilplomationmaicirctrise apregraves 4 ans output G10 MEQ US NEWS Macleans Republica diplomationdoctorat apregraves 6 ans output G10 MEQ US NEWS Macleans Republica

Deacutepenses de fonctionnement par eacutetudiant input CAUBOG10 US NEWS Macleans Guardian Asia weekFrais de scolariteacute input G10frais affeacuterents inputDeacutepenses pour les Services aux eacutetudiants input Macleans

taux demployabiliteacute apregraves 1 an outcome Ontario CHEDAAD Guardian de retour aux eacutetudes outcome Melbourne

Valeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome Guardian

RP gouvernance - 2008-06-23

2INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

EnseignantsUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Nombre denseignants Nombre total denseignants inputNombre de professeurs reacuteguliers input G10 MEQStatCan USNEWS Melbourne de professeures input OQ MelbourneNombre de chercheurs inputNombre de professeurs avec diplocircme international THESNombre de professeurs de clinique eacutequivalents temps complet input G10StatCanNombre de chargeacutes de cours et autres personnels eacutequivalents temps complet input USNEWSNombre dauxiliaires en eacutequivalence temps complet input

Caracteacuteristiques des professeurs reacuteguliers input des professeurs et chercheurs avec PhD (ou grade terminal) input USNEWS Macleansstructure dage du corps professoral input Asiaweek Melbourne Republicaacircge moyen des professeurs reacuteguliers inputacircge moyen des professeurs adjoints inputacircge moyen des professeurs agreacutegeacutes inputacircge moyen des professeurs titulaires input

reacutemuneacuteration moyenne des professeurs reacuteguliers input Q T U U S NEWS Asiaweek Melbournereacutemuneacuteration moyenne des professeurs reacuteguliers input QTU USNEWS Asiaweek Melbournedes professeurs adjoints inputdes professeurs agreacutegeacutes inputdes professeurs titulaire input

Activiteacutes denseignement inputnombre deacutetudiants EETC Professeur reacutegulier input G10 MEQ USNEWS Guardian Asiaweek Melbourne Republica THESnombre deacutetudiants EETC Professeur reacutegulier en sciences sociales input Melbournenombre deacutetudiants EETC Professeur reacutegulier en sciences input Melbournenombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input

Nombre de cours dispenseacutes par lInstitution (ANNUEL) input Republica cours assumeacutes par les professeurs reacuteguliers input MEQ USNEWS Macleans Republica cours assumeacutes par les chargeacutes de cours eacutequivalents temps complet input MEQTaille des groupe cours input MEQ USNEWS Macleans Guardian

RP gouvernance - 2008-06-23

3INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

PersonnelsUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Le personnel de soutientotal de tous les personnels reacuteguliers acadeacutemiques et non acd agrave temps complet inputnombre des personnels de soutien (staff) input G10 MelbourneDeacutepenses des personnel non-enseignant input MEQnombre des personnels de soutien (staff)- secteur enseignement inputnombre des personnels de soutien (staff) - secteur soutien acadeacutemique inputnombre des personnels de soutien (staff) - secteur administration inputnombre des personnels de soutien (staff) - secteur service aux eacutetudiants inputnombre des personnels de soutien (staff) - entreprises auxiliaires input

rectoratpersonnel dencadrement non acadeacutemique et cadres acadeacutemiques inputratio sur lensemble des personnels acadeacutemiques et non acadeacutemiques inputeacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input G10 Guardianpersonnel de soutien professeur reacutegulier input G10

RP gouvernance - 2008-06-23

4INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Activiteacutes de recherche USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Revenus de recherche de professeurs et chercheurs ayant fait une demande de subvention processus de professeurs et chercheurs ayant obtenu une subvention inputRevenus totaux de recherche (subventions et contrats) input CAUBO G10 The Center Asiaweek Melbournepart de la recherche contractuelle input MEQSubventions publiques input Macleans Melbournerevenus de recherche obtenus des conseils de recherche feacutedeacuteraux input AOQTU The Center Macleans Melbournerevenus moyens de recherche par professeur et chercheur input InfosourceMEQ des deacutepenses de recherchebudget total des deacutepenses input TNombre de chaires et de centre dexcellence output AFondation canadienne pour linnovation (FCI) input G10 -nombre de projets output G10 -valeur $ output G10

nombre de programmes de maicirctrise de recherche input The Center Asiaweeknombre de programmes de doctorat input The Center Asiaweek

Publications et prix outputNombre de publications output Infosource Asiaweek Melbourne

Shanghai Leiden Taiwan

Nombre de publications par professeur et chercheur output Infosource CHEDAAD ShanghaiPublications dans Nature and Science output TaiwanNombre total de citations output InfosourceT Asiaweek MelbourneNombre de citations par professeur et chercheur output InfosourceT Melbourne THESNombre de citations par publication output T CHEDAAD LeidenHighly cited researchers dans 21 secteurs disciplinaires

output

ShanghaiTaiwan Leiden

Impact des publications selon les secteurs disciplinaires output Infosource TaiwanTHES Shanghai

Domaines disciplinaires dexcellence inputTaiwanShanghai THES

Prix et meacutedailles de niveau national et international output G10 The Center Macleans Melbourne Republica ShanghaiNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur output CHEDAADnombre de brevets et de licences output G10 CHEDAADnombre de spin-off companies G10

RP gouvernance - 2008-06-23

5INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

InternationalisationUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

deacutepenses pour le secteur internationalisation input MEQnombre deacutetudiants en stage agrave leacutetrangernombre deacutetudiants temps complet input tes un cannombre deacutetudiants internationauxnombre deacutetudiants temps complet input tes un can Melbourne THESnombre deacutetudiants en cotutellesnombre deacutetudiant des cycles supeacuterieurs input CHEDAADnombre de professeurs inviteacutes internationaux processus tes un can CHEDAADprofesseurs inviteacutes agrave leacutetranger processus CHEDAADnombre deacutetudiants posdoctoraux processus tes un can CHEDAADproportion de cours de langues eacutetrangegraveres processusnombre de programmes deacutechange processus tes un can CHEDAADEntentes de partenariat outputPrix internationaux output G10 The Center Macleans Melbourne Republica Shanghai

RP gouvernance - 2008-06-23

6INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Inteacutegration des technologies de linformation USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Deacutepenses totales consacreacutees aux TIC input MEQpersonnel techniqueenseignant reacutegulier inputpersonnel techniqueeacutetudiant en eacutequivalence temps complet inputExistence dun reacuteseau informatique (accessible agrave linterne et agrave lexterne) outputaccegraves au reacuteseau avec fils et sans fils processusaccegraves aux ressources documentaires des bibliothegraveques outputService de courriel interne et externe en mode WEB outputNombre dordinateurs par professeur et par eacutetudiant outputNombre de cours en ligne processusnombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) outputAccegraves des professeurs aux banque de donneacutees et aux logiciels processus

Site WEB (eacutevaluation du site WEB) WEBOMETRICSSize (nombre de pages recenseacutees) outcomevisibilitty (consultation observeacutee selon le nobre de lien externe unique) outcomeRich files (intensiteacute de lactiviteacute acadeacutemique) outcomeScholar (activiteacute de recherche publications et citations) outcome

RP gouvernance - 2008-06-23

7INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

BibliothegravequesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre total de volumes et de documents audiovisuels et eacutelectroniques input G10 ARLacquisition de volumes (au cours de lanneacutee) input ARL Macleansnombre total de volumes eacutetudiants en eacutequivalence temps complet input G10 ARL Macleans Deacutepenses des bibliothegraveques input MEQ Macleans AsiaweekDeacutepenses des bibliothegravequesbudget de fonctionnement input ARLDeacutepenses des bibliothegravequeseacutetudiants en eacutequivalence temps complet input G10 ARLPersonnels reacuteguliers agrave temps complet input G10 ARLClassement outcome ARL

RP gouvernance - 2008-06-23

8INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Revenus et deacutepenses USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Produits et revenusSubvention du ministegravere de lEacuteducation du Queacutebec input MEQ subvMEQrevenus de fonctionnement input G10 MacleansDroits de scolariteacute input G10 droits de scolariteacute revenus de fonctionnement input G10autres produits input G10Total des revenus de fonctionnement input G10 MEQ Guardian revenus de fonctionnement eacutetudiants en eacutequivalence temps complet input CAUBO USNews MacleansRevenus de recherche input G10encaissement par nature (dons immobilisations chaires bourses) input G10 fonds de dotation input G10 CenterContribution des alumni aux encaissements et fonds de dotation input USNews Macleans fonds de dotation eacutetudiants en eacutequivalence temps complet input G10Total des produits G10

Charges et deacutepenses G10salaire des professeurs input G10 MEQsalaire des chargeacutes de cours input G10salaires des auxiliaires input MEQsalaires des auxiliaires input MEQsalaire des personnels (staff) input G10autres deacutepenses input G10total des deacutepenses input G10 deacutepenses denseignement deacutepenses totales de fonctionnement input UBC MEQ

Chargesproduits processusEacutequilibre budgeacutetaire revenus et deacutepenses de fonctionnement processus MEQ

RP gouvernance - 2008-06-23

9INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

EspacesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Nombre de megravetres carreacutes bruts input G10MCB eacutetudiants en eacutequivalence temps complet input G10megravetres carreacutes nets alloueacutes input MEQNombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input Tbesoin despace en megravetres carreacutes nets inputExistence dun plan directeur des espaces inputBudget preacutevu au plan directeur des espaces input TBudget alloueacute aux espaces input MEQdeacutepenses de fonctionnement nombre de megravetres carreacutes bruts input G10

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Environnement USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

consommation de leacutenergie (exprimeacutee en BTU processus universiteacutes canadiennes et ameacutericaines BTUpopulation totale de luniversiteacute processus voir en particulier Penn State et Umich de leacutenergie produit par des sources renouvelable processusUtilisation de leau processuslitres deau par personne processusGaz agrave effet de serre (en tonnes meacutetriques CO2 eacutequivalents) processusMT CO2 par personne processusDeacutechets solides (en tonnes) processusTonnes de deacutechets solides per capita processus de deacutechets solides recycleacutes processus

RP gouvernance - 2008-06-23

10INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

ReacuteputationUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

enquecircte meneacutee aupregraves des eacutetudiants outcome NSSE Melbourneenquecircte meneacutee aupregraves des diplocircmeacutes outcomeenquecircte meneacutee aupregraves des pairs outcome USNews Asiaweek THESenquecircte meneacutee aupregraves des employeurs outcome Macleans Melbourneenquecircte meneacutee aupregraves des stakeholders outcome Macleans Melbourneenquecircte meneacutee aupregraves du monde des affaires outcome Macleans Melbournecouverture meacutediatique outcome AT

RP gouvernance - 2008-06-23

11INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Planification et gestion strateacutegiquesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Pouvoirs des corps universitaire processus un Canliens dynamiques entre Conseil Assembleacutee universitaire commission des eacutetudes processus un Canprocessus formel de planification strateacutegique processus un Canproduction de donneacutees et indicateurs processus un Canprocessus deacutevaluation des faculteacutes deacutepartements et services processus un Canrevue systeacutematique des responsabiliteacutes et efficaciteacute des comiteacutes processus un Candeacutepenses de ladministration centrale deacutepenses de fonctionnement processus un Can

RP gouvernance - 2008-06-23

12INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Eacuteducation permanenteUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre de programmes offerts input un Cannombre deacutetudiants inscrits input un CanRevenus et deacutepenses du service input un Can

RP gouvernance - 2008-06-23

13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes

REacuteFEacuteRENCES

Aperccedilu densemble des principaux indicateurs

GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost

Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes

MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf

Indicateurs utiliseacutes dans les classements internationaux

Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml

RP gouvernance - 2008-06-23

ANNEXES

Quelques exemples de listes drsquoindicateurs

1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)

Voir page suivante

Preacutepareacute par Roland Proulx

Consultant en planification institutionnelle et veille strateacutegique

Montreacuteal mai 2008

Performance Indicators

Discussion Paper

May 2004 (Prepared by the Office of Planning amp Analysis)

Table of Contents

Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25

2

Table 1 ndash Number of Indictors by Type of Data Source

Table 1 shows that surveys are the least frequently used source of data for indicators

Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a

plurality of indicators come from this source This measure somewhat underestimates the

importance of surveys however as it does not account for the weighting given to each

indicator in each study In the THES World Rankings for instance there is only a single

survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos

Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one

indicator is worth 20 of the final score

Raw

indIcator count

Surve

y data

Third parties

Universities

AsiaweekmdashAsias Best Universities 18 - - 18

Daily Telegraph (2003) 1 - 1 -

Education18com 9 3 4 2

Excelencia 2001 71 - 71 -

Financial Times (2003) 17 - 17 -

Guangdong Institute of Management Science 17 - 14 3

GuardianmdashUniversity Guide 2005 7 - 2 5

La Repubblica 23 2 21 -

Macleans University Rankings 24 1 5 18

Melbourne Institutemdash International Standing of Australian Universities

26 3 23 -

Netbig 2004 18 1 10 7

Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15

Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities

6 - 5 1

The TimesmdashGood University Guide 2005 9 - 9 -

Times Higher Education SupplementmdashWorld University Rankings

5 1 1 3

US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11

Washington MonthlymdashCollege Rankings 2005 8 - 1 7

Wuhan University Centre for Science Evaluation 45 2 22 21

3

A Indicators of Beginning Characteristics

ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of

students at the time they begin their studies Fourteen of the 18 rankings examined in this

study use one or more indicators of the beginning characteristics of students to arrive at

their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on

these factors with 33 of the total ranking coming from this class of indicators but the

Guardian Education18 Asiaweek and the two other North American surveys also place

considerable emphasis on this category

There are six main indicators used to determine which institutions have students with

positive ldquobeginning characteristicsrdquo

The most common measure of beginning characteristics is performance on national

standardized tests with nine surveys using this as a measure Education18 and the

Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used

by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)

Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this

data is collected and standardized by national bodies it has the benefit of being seen as a

relatively impartial method of determining the relative ldquostrengthrdquo of the students entering

each institution Institutionsrsquo results can be scored by showing either averages or the

percentage of entering students meeting a particular standard

Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses

secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This

is a second-best solution made necessary by the absence of any national standardized test

in Canada (or indeed of any provincial standardized tests at the end of secondary school

in provinces other than Alberta) The lack of national standardization makes this an

undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is

truly equivalent to an ldquoArdquo in another jurisdiction

Another measure of the strength of the student body is the percentage of incoming

students receiving (third-party) scholarships which is worth 11 of the score in the

Wuhan survey One can also approach the issue by measuring institutional selectivity In

effect this method infers the strength of the student body by the proportion of

4

applicants rejected the theory being that the higher the number of rejected applicants the

stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)

Normally this measure is expressed as a straight ratio of acceptances to applications but it

can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the

greatest weight on this measure) as a ratio of enrolments to applications Within the US

there is some dispute as to what constitutes an offer of admission and whether or not late

admissions are included as noted by Ganeshananthan (2003)

Student bodies are often considered to be strong if the school is able to attract a large

number of international or out-of-district students or if they contain people from diverse

ethnic backgrounds A number of league tables use the international student indicator

(which like the selectivity indicator is arguably as much an indicator of prestige and

reputation as it is of student characteristics) although in no case does this indicator

account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a

quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data

on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian

puts a somewhat larger emphasis on this indicator in comparison to other league tables

which use similar variables

A very different take on this idea is present in the Washington Monthly which released its

first set of College Rankings in September 2005 With the declared aim of using an

institutionrsquos commitment to social mobility as a measure of quality it uses the percentage

of students from low-income backgrounds as an indicator (with percentage of students

receiving need-based government (Pell) grants used as a proxy)

Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy

statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the

percentage of the student population who are graduate students (arguably this is a

research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the

number of graduate students auditing classes is used as an indicator the assumption is

presumably that if people are auditing then the classes must be very attractive The Italian

La Repubblica ranks an institution according to the number of part-time students it has

contrary to prevailing North American views on the undesirability of part-time study the

Italian rankings see higher numbers of part-time students in a positive light as it is

evidence that an institution is becoming less rigid in its timetabling and

5

permitting students to juggle both work and study something which was nearly

impossible in that country just a few years ago

The Washington Monthly also has a final category of indicators which reflect studentsrsquo

beginning characteristics namely their likelihood of performing community service as

measured by the percentage of students in the US Peace Corps and Reserve Officer

Training Corps (ROTC) and the percentage of students involved in work-study in the

community3 Together these three indicators account for 33 of an institutionrsquos total

ranking

B Indicators of Learning InputsmdashStaff

Generally both the quantity and quality of staff are positively correlated with institutional

quality The problem of course is finding useful metrics for each of these factors

especially if one excludes as we have done here measures of research performance and

research intensity putting them in a separate category4

The simplest measure is simply the number of faculty unadjusted for things like size of

student body Most national league tables however prefer to use variations on the

concept of facultystudent ratio Others try to measure teaching intensity with measures

such as courses per teacher or hours spent in class per student (both in La Repubblica)

These kinds of measures usually account for between 2-5 of the final rankings although

in some cases (ie the Guardian) this figure can be as high as 20

Another important way of measuring how faculty resources are deployed is the measure

of average class size which is used only by Macleanrsquos and the US News and World Report

Ostensibly the reason for measuring class size is to account in some form for the degree

3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category

6

of attention devoted to actually teaching students Despite the fact that important research

(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level

the use of this indicator appears to be a spillover from the North American debates on

class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein

2000) Regardless of why the indicators are used they are extraordinarily important to

these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and

World Reportrsquos rankings respectively

A number of ranking systems try to look at staff qualifications such as the number of

PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the

Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further

than other surveys and actually looks at the proportion of classes taught by tenure-track

staff Others (ie THES) look at the number of foreign faculty based on the assumption

that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still

others (ie La Repubblica) look at the age structure of the faculty Another proxy for

institutional quality is the pay rates for tenured staff on the assumption that institutions

with higher rates of pay on average attract better faculty this measure has been used both

by the US News and World Report and Asiaweek

Finally a number of league tables rank faculty inputs on the basis of standardized third-

party evaluations 6

Education18 the Financial Times the Times the Guardian and the US

News and World Report league tables all use some sort of ranking criterion based at least in

part on this indicator or variations thereof

C Indicators of Learning InputsmdashResources

Resource inputsmdashcrudely the amount of current dollars equipment and books available

to students at an institutionmdashare widely considered an important measure of quality Yet

despite the apparent simplicity of counting dollars and measuring assets

5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each

university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future

7

the means by which institutional wealth is measured varies considerably between ranking

systems

There are a number of revenue-based measures of resources Macleanrsquos uses public funding

of institutional budgets as a factor in its analysis conversely the Financial Times uses the

private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of

total score) and the US News and World Report (5 of total score) also measure alumni

financial support as a measure of quality

For reasons that are not entirely clear league tables tend to favour measures of

expenditures rather than revenues The Guardian looks at total institutional expenditures

as an indicator Institutional expenditure on student services is used as a measure of

institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total

institutional scores respectively) Rzezspospolita does not measure student services

expenditures directly but does measure student services outputs such as number of

student athletes and number of study clubs which amounts to more or less the same thing

Macleanrsquos also gives out 433 of its total score based on institutional expenditures on

scholarships and bursaries

Various aspects of physical infrastructure are also used as measures of institutional

resources most directly in the case of La Repubblica which bases 317 of its total rank on

the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding

assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type

of physical infrastructure measured is available Internet bandwidth which was used by

Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth

roughly 3 of the total score

By some distance the infrastructure indicators most favoured by the compilers of league

tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on

this with 12 of the total quality mark being taken from various types of library

infrastructure measurements (including acquisitions per year total volumes average

number of volumes per student and yearly library expenditure outside of acquisitions)

Netbig and Education18 also use library volume holdings while Asiaweek the Financial

Times and the Times also use measures of library expenditures outside of acquisitions or

computerization of library resources as measures of institutional quality

8

One important factor to note is that most ranking systems do not normalize their resource

and infrastructure measures That is to say it is raw spending power or simple size of

assets that is usually measured rather than spending per studentprofessor or assets per

studentprofessor As a result a number of these rankings systems have inbuilt biases

towards larger institutions

D Indicators of Learning Outputs

Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge

learned over the course of a baccalaureate degreemdashshould be a basic indicator of

institutional quality Unfortunately good means of measuring these outputsmdashlike the

National Survey of Student Engagement (NSSE) and the College Learning Assessment

(CLA)mdashhave only recently become available and for the most part institutions are still

keeping their scores secret Outside of these measures only a few very crude indicators are

available which likely explains why learning outputs do not feature especially prominently

in most ranking schemes

The simplest types of measures of learning outputs are those linked to graduation and

retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong

and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7

the latter three also use rates of graduation from Masterrsquos programs as indicators In some

cases the weights on these measures can be very highmdashin the Guangdong rankings

graduation rates account for over 50 of the rankingmdashbut in most cases the weights are

10 or less Retention rates commonly meaning the progression rate of first-year students

into second year are accorded less importance The US News and World Report Macleanrsquos

the Melbourne Institute and La Repubblica all employ retention measures as indicators but

none of them are worth more than 4 of total weighting Two publications make specific

indicators for retention and graduation of international students Macleanrsquos (graduation

rates of international students) and the Melbourne Institute (retention rates of international

students) The Washington Monthly looks specifically at institutional retention rates

adjusted for the participation of lower-income students and gives higher scores to

institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores

and number of Pell Grant recipients the US

7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18

9

News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos

ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)

E Indicators of Final Outcomes

Final outcomes are indications of generalized outcomes for students after graduation

Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness

good citizenship) but given the somewhat utilitarian justifications for education that are

currently in fashion (see Wolf 2000) employment outcomes are the most commonly used

measure of final outcomes These are given particular emphasis by the Guardian (where

employment outcomes are worth 17 of the total score) but are also used by the Financial

Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the

Times are interestingly not concerned with employment per se but with ldquoemployment in

an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher

Education Statistics Agency (HESA) uses Standard Occupational Classifications to

measure the proportion of graduates in professional or white-collar jobs anyone not in

such a job is considered not to be working in an area related to their studies (it is unclear

what methodology is used by the Financial Times and the Times although we suspect their

methods are broadly similar)

The only other measure of final outcomes in use is percentage of graduates returning for

additional education which is an indicator used by both the Melbourne Institute and the

Financial Times This is a particularly important indicator for the latter as it is worth 21

of the final ranking

The lack of indicators concerning final outcomes is interesting since most government-

sponsored performance-indicator regimes around the world are very much concerned

with such measures especially with respect to employment Possibly this indicates that

ranking systems simply do not view education outcomes as relevant measures of

educational quality Alternatively it may be the case that they simply have not found a

reliable indicator of outcomes or that there are reliable indicators but that there is so little

variation between institutions that it makes no sense to rank based on the data

10

F Indicators of Research

Many of the league tables covered in this survey include sections and weightings related

to universitiesrsquo research efforts It is in this field of measurement that we see the greatest

diversity of indicators Presumably this is because research inputs and outputs lend

themselves much more easily to measurement and manipulation than other areas of

institutional activity

Three studies include research staff as part of their ranking scheme La Repubblica at 952

the Melbourne Institute (4) and Wuhan (078)

Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used

method of looking at research quality but it is not universally admired because different

disciplines use different means to communicate major advances in knowledge (leading

scientists invariably produce large numbers of journal articles leading social scientists

may produce fewer journal articles but instead have one or two long important

monographsmdashsee Hicks 2004) There is also some concern among non-English speaking

countries that they are penalized in international rankings because so many of the major

journals (notably Science and Nature) are printed in English However the one set of

rankings that uses separate indicators to monitor articles published in English and articles

published in another language (the Wuhan rankings) shows that the two indicators are

positively correlated institutions that have more Chinese publications are also likely to

have more English publications and vice versa

Several sets of league tables measure bibliometric citations in various publication indices

The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a

weight of 20 of the final total Guangdong also monitors other Chinese universities

specifically for citations in engineering publications and weights this at 29 Moreover it

tacks on an additional 1049 for citations in science-oriented indices such as the Science

Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the

same category of scientific citations while the Melbourne Institute rates science citations at

68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social

Science Citation Index which does not include the humanities) are noted in only two

league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne

Institute (32) Another way of measuring research

11

impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a

weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the

Melbourne Institute

The complement of citations is of course publications Listing the number of publications

an individual group department or whole university releases can act as a weak substitute

for citationsmdashweak because simply publishing a paper or monograph is no guarantee that

the general public or other researchers will even glance at the work Guangdong gives an

1179 weighting to publications in science-oriented indices such as the Science Citation

Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same

indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong

even has a separate category just for measuring publications in Science and Nature

although it accords it an almost derisory weighting of 06 Under publications in social

science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their

final totals For publications in other indices (where the subject indices are

undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for

Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of

rankings to include research monographs weighted at 033

In countries where there are specific third-party evaluations of research output academic

quality of research is sometimes used as a research indicator The Times puts a very large

30 weight on this indicator while the Financial Times puts it at 119

Research awards are another handy third-party measurement of quality as the number of

international and national awards won by faculty andor graduates is often considered a

useful measure of institutional success International research awardsmdash specifically the

number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator

by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as

particularly suspect in some quarters given that the points are based on where the

recipient went to school rather than on where they are or were on the

8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the

most prominent publication indices

9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences

12

faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead

for nearly a century Wuhan uses similar measures but only accords them a weight of 14

National research awards are more common as a quality indicator used by La Repubblica

(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)

Financial indicators of research are also very common Research budgets as a factor in the

overall assessment of research in universities are covered by the Financial Times (9)

Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for

total amount of research expenditure unfortunately it is unclear precisely what this

research expenditure represents or how it is determined although it is clearly indicated

that it does not represent the total number of grants or projects at a university Total

number of research-based grants and projects is weighted by Education18 at 15 and

Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science

and engineering and another 55 to those for social sciences and humanities Similarly

the Melbourne Institute gives 6 of its overall weight to public-source grants making no

distinction between areas of study

In a slightly different vein Netbig (46) and Wuhan (278) both list the number of

research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica

(095) all weigh research-basedaffiliated research institutions or centres for studies

Finally one can also measure research not simply in terms of the amount of money it

generates but also in terms of the amount of future income it will generate Both

Guangdong (245) and Wuhan (193) measure the number of patents issued to

universities as a quality indicator

A final way of measuring an institutionrsquos research intensity is to look at the range of its

course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of

doctoral and Masterrsquos programs offered as a proxy for research intensity

As with physical and financial resources few if any of the research indicators are

normalized to account for institutional size (either by student or faculty numbers) In the

world of rankings bigger almost always means better an institution with 100 faculty with

ten citations apiece will always look worse than an institution with 1001 faculty

13

with one citation each To the extent that the raw production of knowledge matters this

form of measurement is acceptable To the extent that rankings are meant to show how

well institutions are doing on a like-to-like basis or to show the efficiency of universities it

is plainly inadequate This should be of particular concern to Chinese policy-makers

whose ranking systems are especially reliant on research-based indicators

G Indicators of Reputation

The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo

Those rankings systems which use the results of reputation surveys as an indicator do so as

an indirect measure of quality based on the assumption that the employers academics and

academic administrators surveyed have opinions of institutional quality that are informed

up-to-date and impartial While these assumptions are clearly open to debate they

nevertheless form an important basis for many ranking systems Another reason for using

reputation measures is the paucity of other data availablemdashsome countries have few

independent measures of teaching effectiveness university resources or output and

reputation can thus act as a useful surrogate Reputation rankings are often criticized as

simply quantifying the common ignorance of the people being surveyed However to the

extent that the people being surveyed hold positions which have the potential to affect

large numbers of young people and whose positions actually require some knowledge of

institutional quality (ie officials in charge of graduate admissions corporate recruiters

etc) then reputation rankings make sense because they provide useful information for

students about the perceived value of the degrees that they could obtain from various

universities

The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and

the Times which both accord reputation a weighting of 50 in their overall ranking

scheme Education18 assigns it almost as much significance at 40 The US News and World

Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just

below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)

and Netbig (15) The only other study to include reputation is Wuhan (117)

14

1 Please see Appendix B for a glossary of the various bodies referenced in this document

Appendix A Detailed Listing of Indicators and their Sources

Beginning Characteristics Indicator

Used By Source1

Incoming grades Macleans University

Percentage with grades above a set limit

Macleans University

US News and World Report

University

Performance on national standardised tests or benchmarks

Asiaweek University

Education18 3rd-party JUPAS

Financial Times Government agency 3rd-party UCAS

Guardian University Guide

Government agency 3rd-party UCAS

Melbourne Institute Government agency 3rd-party DEST

Netbig National entrance examination board

Times Good University Guide

Government agency 3rd-party UCAS

US News University

Wuhan Unknown presumed government 3rd-party

Student status La Repubblica Government agency 3rd-party MIUR

Admittance selectivity general

Asiaweek University

Admittance number of applications to places

Asiaweek University

Financial Times Government agency 3rd-party UCAS

La Repubblica Government agency 3rd-party MIUR

US News University

15

Beginning Characteristics Indicator

Used By Source1

Out-of-locality student percentage

Macleans University

International student percentages

Financial Times Government agency 3rd-party HESA

Macleans University

Shanghai Institute of Educational Science

Times World University

Wuhan Unknown presumed university

Undergraduate students among all students percentages

Netbig

Wuhan Unknown presumed university

Ethnic diversity in student body

Guardian University

Learning Inputs ndash Staff Indicator

Used By Source

Facultystudent ratio Asiaweek University

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Financial Times Government agency 3rd-party HESA

La Repubblica Government agency 3rd-party MIUR

Times Good University Guide

Government agency 3rd-party HESA

Times World University

US News University

Wuhan Unknown presumed university

16

Learning Inputs ndash Staff Indicator

Used By Source

Social science faculty student ratio

Melbourne Government agency 3rd-party DEST

Science faculty student ratio

Melbourne Government agency 3rd-party DEST

Administrative staff student ratio

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Staff student ratio (regardless of division)

Guardian Government agency 3rd-party HESA

Netbig University

Course per teacher La Repubblica Government agency 3rd-party MIUR

Per-teacher university spending

Asiaweek University

Faculty pay rates fortenured staff

Asiaweek University

US News University

Number of full-time part-time faculty

Netbig University

US News University

Wuhan Unknown presumed university

Faculty with researchprojects

Wuhan Unknown presumed university

Class size differentiation Macleans University

US News University

Classes taught by tenured faculty

Macleans University

Exchange programmes hosted

La Repubblica Government agency 3rd-party AgNaSoc

Number of classes lsquoactually taughtrsquo

La Repubblica Government agency 3rd-party MIUR

of international faculty (v faculty as a whole)

Times World University

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

Les indicateurs font partie deacutesormais des outils indispensables auxquels ont recours les

gouvernements la communauteacute universitaire les eacutetudiants et leurs parents Leur nombre

varie entre dix et 300 indicateurs Bon nombre drsquoauteurs (Barbara Taylor (1993) Gaither amp

alii (1994) drsquoorganismes (COU) de producteurs de classements nationaux et internationaux

favorisent des listes courtes de 10 agrave 20 indicateurs ces listes courtes sont suffisantes pour

caracteacuteriser les activiteacutes drsquoune universiteacute repreacutesentent des indicateurs courants dont les

sources sont accessibles et permettent le repeacuterage (benchmarking) des universiteacutes

comparables En produisant des listes longues de 100 agrave 300 indicateurs les producteurs

preacutesentent plutocirct un reacutepertoire ou une banque drsquoindicateurs parmi lesquels les requeacuterants

comme les utilisateurs peuvent piger selon les objectifs poursuivis et les inteacuterecircts privileacutegieacutes

On trouvera dans les pages qui suivent un inventaire de quelque 175 indicateurs regroupeacutes

autour de 13 chefs de division ou domaines drsquoactiviteacutes (parmi lesquels deux seacuteries

drsquoindicateurs ndash environnement et planification et gestion strateacutegique constituent une

innovation)

Eacutetudiants

Enseignants

Personnels

Activiteacutes de recherche

Internationalisation

Inteacutegration des technologies de linformation

Bibliothegraveques

Revenus et deacutepenses

Espaces

Environnement

Reacuteputation

Planification et gestion strateacutegiques

Eacuteducation permanente

wwwigopporg 5

wwwigopporg

Tous les indicateurs des listes longues et tous les domaines drsquoactiviteacutes nrsquoont pas eacuteteacute retenus

Il srsquoagit drsquoun inventaire des indicateurs courants tels que recenseacutes dans plusieurs listes

courtes et longues et utiliseacutes agrave des fins de planification drsquoeacutevaluation de la performance et

de reddition de comptes De plus dans un contexte de mondialisation et

drsquointernationalisation il est devenu important et pertinent drsquoun point de vue strateacutegique de

retenir les indicateurs utiliseacutes dans les principaux classements nationaux (9 sur 20 possibles)

et internationaux (4) 1 Lrsquoencadreacute qui suit reacutesume ce choix

classements nationaux classements internationaux

universiteacutes canadiennes

et MEQ

USNews and

Report ampThe

Center

Macleans CHEDAAD GUARDIAN ASIAWEEKDEST

AUSTRALIE amp

MELBOURNE REPUBLICA

QS-TIMES SHANHAI TAIWAN

WEB

1 On trouvera en fin de document lrsquoensemble des reacutefeacuterences consulteacutees ainsi qursquoun annexe preacutesentant quelques exemples de listes drsquoindicateurs

6

1INVENTAIRE DES INDICATEURS COURANTS

Agrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

EacutetudiantsUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIEampMel

bourneREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre deacutetudiantsNombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input MEQ G10 DETYA THESARWUproportion deacutetudiantes input DETYA des eacutetudiants agrave temps partiel input AOTQU CHEDAAD Republicainscriptions au 1er cycle agrave temps plein input MEQ CHEDAAD inscriptions au 2e cycle input MEQ inscriptions au 3e cycle input MEQ eacutetudiants aux 2e et 3e cycles input G10 DETYA eacutetudiants internationaux input TU THES des eacutetudiants internationaux input Macleans CHEDAAD THES des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input MEQ stagiaires postdoctoraux input MEQ reacutesidents et reacutesidentes en meacutedecine input MEQ

inputrecrutement et admission inputdemandes dadmission aux programmes reacuteguliers input OTQ CHEDAAD offres dadmissiondemandes dadmission i t CHEDAAD offres dadmissiondemandes dadmission input CHEDAAD inscriptionsdemandes dadmission input T CHEDAAD Asiaweek Republica

inputdossier scolaire des nouveaux eacutetudiants (bulletin Cote R) input US NEWS Macleans Asiaweek

provenance des eacutetudiants input Macleanspopulation cible inputreacutegion meacutetropolitaine inputQueacutebec inputCanada inputInternational input

inputdiversiteacute ethnique input US NEWS Guardian

Bourses accordeacutees aux eacutetudiants - Financement moyen input TMEQpremier cycle input Macleans Republicadeuxiegraveme cycle input G10troisiegraveme cycle input G10

cheminement des eacutetudiants (temps complet) perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output QT MEQ US NEWS Macleans Republica diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output G10 MEQ US NEWS Macleans Melbourne Republica dilplomationmaicirctrise apregraves 4 ans output G10 MEQ US NEWS Macleans Republica diplomationdoctorat apregraves 6 ans output G10 MEQ US NEWS Macleans Republica

Deacutepenses de fonctionnement par eacutetudiant input CAUBOG10 US NEWS Macleans Guardian Asia weekFrais de scolariteacute input G10frais affeacuterents inputDeacutepenses pour les Services aux eacutetudiants input Macleans

taux demployabiliteacute apregraves 1 an outcome Ontario CHEDAAD Guardian de retour aux eacutetudes outcome Melbourne

Valeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome Guardian

RP gouvernance - 2008-06-23

2INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

EnseignantsUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Nombre denseignants Nombre total denseignants inputNombre de professeurs reacuteguliers input G10 MEQStatCan USNEWS Melbourne de professeures input OQ MelbourneNombre de chercheurs inputNombre de professeurs avec diplocircme international THESNombre de professeurs de clinique eacutequivalents temps complet input G10StatCanNombre de chargeacutes de cours et autres personnels eacutequivalents temps complet input USNEWSNombre dauxiliaires en eacutequivalence temps complet input

Caracteacuteristiques des professeurs reacuteguliers input des professeurs et chercheurs avec PhD (ou grade terminal) input USNEWS Macleansstructure dage du corps professoral input Asiaweek Melbourne Republicaacircge moyen des professeurs reacuteguliers inputacircge moyen des professeurs adjoints inputacircge moyen des professeurs agreacutegeacutes inputacircge moyen des professeurs titulaires input

reacutemuneacuteration moyenne des professeurs reacuteguliers input Q T U U S NEWS Asiaweek Melbournereacutemuneacuteration moyenne des professeurs reacuteguliers input QTU USNEWS Asiaweek Melbournedes professeurs adjoints inputdes professeurs agreacutegeacutes inputdes professeurs titulaire input

Activiteacutes denseignement inputnombre deacutetudiants EETC Professeur reacutegulier input G10 MEQ USNEWS Guardian Asiaweek Melbourne Republica THESnombre deacutetudiants EETC Professeur reacutegulier en sciences sociales input Melbournenombre deacutetudiants EETC Professeur reacutegulier en sciences input Melbournenombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input

Nombre de cours dispenseacutes par lInstitution (ANNUEL) input Republica cours assumeacutes par les professeurs reacuteguliers input MEQ USNEWS Macleans Republica cours assumeacutes par les chargeacutes de cours eacutequivalents temps complet input MEQTaille des groupe cours input MEQ USNEWS Macleans Guardian

RP gouvernance - 2008-06-23

3INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

PersonnelsUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Le personnel de soutientotal de tous les personnels reacuteguliers acadeacutemiques et non acd agrave temps complet inputnombre des personnels de soutien (staff) input G10 MelbourneDeacutepenses des personnel non-enseignant input MEQnombre des personnels de soutien (staff)- secteur enseignement inputnombre des personnels de soutien (staff) - secteur soutien acadeacutemique inputnombre des personnels de soutien (staff) - secteur administration inputnombre des personnels de soutien (staff) - secteur service aux eacutetudiants inputnombre des personnels de soutien (staff) - entreprises auxiliaires input

rectoratpersonnel dencadrement non acadeacutemique et cadres acadeacutemiques inputratio sur lensemble des personnels acadeacutemiques et non acadeacutemiques inputeacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input G10 Guardianpersonnel de soutien professeur reacutegulier input G10

RP gouvernance - 2008-06-23

4INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Activiteacutes de recherche USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Revenus de recherche de professeurs et chercheurs ayant fait une demande de subvention processus de professeurs et chercheurs ayant obtenu une subvention inputRevenus totaux de recherche (subventions et contrats) input CAUBO G10 The Center Asiaweek Melbournepart de la recherche contractuelle input MEQSubventions publiques input Macleans Melbournerevenus de recherche obtenus des conseils de recherche feacutedeacuteraux input AOQTU The Center Macleans Melbournerevenus moyens de recherche par professeur et chercheur input InfosourceMEQ des deacutepenses de recherchebudget total des deacutepenses input TNombre de chaires et de centre dexcellence output AFondation canadienne pour linnovation (FCI) input G10 -nombre de projets output G10 -valeur $ output G10

nombre de programmes de maicirctrise de recherche input The Center Asiaweeknombre de programmes de doctorat input The Center Asiaweek

Publications et prix outputNombre de publications output Infosource Asiaweek Melbourne

Shanghai Leiden Taiwan

Nombre de publications par professeur et chercheur output Infosource CHEDAAD ShanghaiPublications dans Nature and Science output TaiwanNombre total de citations output InfosourceT Asiaweek MelbourneNombre de citations par professeur et chercheur output InfosourceT Melbourne THESNombre de citations par publication output T CHEDAAD LeidenHighly cited researchers dans 21 secteurs disciplinaires

output

ShanghaiTaiwan Leiden

Impact des publications selon les secteurs disciplinaires output Infosource TaiwanTHES Shanghai

Domaines disciplinaires dexcellence inputTaiwanShanghai THES

Prix et meacutedailles de niveau national et international output G10 The Center Macleans Melbourne Republica ShanghaiNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur output CHEDAADnombre de brevets et de licences output G10 CHEDAADnombre de spin-off companies G10

RP gouvernance - 2008-06-23

5INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

InternationalisationUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

deacutepenses pour le secteur internationalisation input MEQnombre deacutetudiants en stage agrave leacutetrangernombre deacutetudiants temps complet input tes un cannombre deacutetudiants internationauxnombre deacutetudiants temps complet input tes un can Melbourne THESnombre deacutetudiants en cotutellesnombre deacutetudiant des cycles supeacuterieurs input CHEDAADnombre de professeurs inviteacutes internationaux processus tes un can CHEDAADprofesseurs inviteacutes agrave leacutetranger processus CHEDAADnombre deacutetudiants posdoctoraux processus tes un can CHEDAADproportion de cours de langues eacutetrangegraveres processusnombre de programmes deacutechange processus tes un can CHEDAADEntentes de partenariat outputPrix internationaux output G10 The Center Macleans Melbourne Republica Shanghai

RP gouvernance - 2008-06-23

6INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Inteacutegration des technologies de linformation USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Deacutepenses totales consacreacutees aux TIC input MEQpersonnel techniqueenseignant reacutegulier inputpersonnel techniqueeacutetudiant en eacutequivalence temps complet inputExistence dun reacuteseau informatique (accessible agrave linterne et agrave lexterne) outputaccegraves au reacuteseau avec fils et sans fils processusaccegraves aux ressources documentaires des bibliothegraveques outputService de courriel interne et externe en mode WEB outputNombre dordinateurs par professeur et par eacutetudiant outputNombre de cours en ligne processusnombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) outputAccegraves des professeurs aux banque de donneacutees et aux logiciels processus

Site WEB (eacutevaluation du site WEB) WEBOMETRICSSize (nombre de pages recenseacutees) outcomevisibilitty (consultation observeacutee selon le nobre de lien externe unique) outcomeRich files (intensiteacute de lactiviteacute acadeacutemique) outcomeScholar (activiteacute de recherche publications et citations) outcome

RP gouvernance - 2008-06-23

7INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

BibliothegravequesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre total de volumes et de documents audiovisuels et eacutelectroniques input G10 ARLacquisition de volumes (au cours de lanneacutee) input ARL Macleansnombre total de volumes eacutetudiants en eacutequivalence temps complet input G10 ARL Macleans Deacutepenses des bibliothegraveques input MEQ Macleans AsiaweekDeacutepenses des bibliothegravequesbudget de fonctionnement input ARLDeacutepenses des bibliothegravequeseacutetudiants en eacutequivalence temps complet input G10 ARLPersonnels reacuteguliers agrave temps complet input G10 ARLClassement outcome ARL

RP gouvernance - 2008-06-23

8INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Revenus et deacutepenses USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Produits et revenusSubvention du ministegravere de lEacuteducation du Queacutebec input MEQ subvMEQrevenus de fonctionnement input G10 MacleansDroits de scolariteacute input G10 droits de scolariteacute revenus de fonctionnement input G10autres produits input G10Total des revenus de fonctionnement input G10 MEQ Guardian revenus de fonctionnement eacutetudiants en eacutequivalence temps complet input CAUBO USNews MacleansRevenus de recherche input G10encaissement par nature (dons immobilisations chaires bourses) input G10 fonds de dotation input G10 CenterContribution des alumni aux encaissements et fonds de dotation input USNews Macleans fonds de dotation eacutetudiants en eacutequivalence temps complet input G10Total des produits G10

Charges et deacutepenses G10salaire des professeurs input G10 MEQsalaire des chargeacutes de cours input G10salaires des auxiliaires input MEQsalaires des auxiliaires input MEQsalaire des personnels (staff) input G10autres deacutepenses input G10total des deacutepenses input G10 deacutepenses denseignement deacutepenses totales de fonctionnement input UBC MEQ

Chargesproduits processusEacutequilibre budgeacutetaire revenus et deacutepenses de fonctionnement processus MEQ

RP gouvernance - 2008-06-23

9INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

EspacesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Nombre de megravetres carreacutes bruts input G10MCB eacutetudiants en eacutequivalence temps complet input G10megravetres carreacutes nets alloueacutes input MEQNombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input Tbesoin despace en megravetres carreacutes nets inputExistence dun plan directeur des espaces inputBudget preacutevu au plan directeur des espaces input TBudget alloueacute aux espaces input MEQdeacutepenses de fonctionnement nombre de megravetres carreacutes bruts input G10

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Environnement USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

consommation de leacutenergie (exprimeacutee en BTU processus universiteacutes canadiennes et ameacutericaines BTUpopulation totale de luniversiteacute processus voir en particulier Penn State et Umich de leacutenergie produit par des sources renouvelable processusUtilisation de leau processuslitres deau par personne processusGaz agrave effet de serre (en tonnes meacutetriques CO2 eacutequivalents) processusMT CO2 par personne processusDeacutechets solides (en tonnes) processusTonnes de deacutechets solides per capita processus de deacutechets solides recycleacutes processus

RP gouvernance - 2008-06-23

10INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

ReacuteputationUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

enquecircte meneacutee aupregraves des eacutetudiants outcome NSSE Melbourneenquecircte meneacutee aupregraves des diplocircmeacutes outcomeenquecircte meneacutee aupregraves des pairs outcome USNews Asiaweek THESenquecircte meneacutee aupregraves des employeurs outcome Macleans Melbourneenquecircte meneacutee aupregraves des stakeholders outcome Macleans Melbourneenquecircte meneacutee aupregraves du monde des affaires outcome Macleans Melbournecouverture meacutediatique outcome AT

RP gouvernance - 2008-06-23

11INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Planification et gestion strateacutegiquesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Pouvoirs des corps universitaire processus un Canliens dynamiques entre Conseil Assembleacutee universitaire commission des eacutetudes processus un Canprocessus formel de planification strateacutegique processus un Canproduction de donneacutees et indicateurs processus un Canprocessus deacutevaluation des faculteacutes deacutepartements et services processus un Canrevue systeacutematique des responsabiliteacutes et efficaciteacute des comiteacutes processus un Candeacutepenses de ladministration centrale deacutepenses de fonctionnement processus un Can

RP gouvernance - 2008-06-23

12INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Eacuteducation permanenteUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre de programmes offerts input un Cannombre deacutetudiants inscrits input un CanRevenus et deacutepenses du service input un Can

RP gouvernance - 2008-06-23

13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes

REacuteFEacuteRENCES

Aperccedilu densemble des principaux indicateurs

GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost

Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes

MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf

Indicateurs utiliseacutes dans les classements internationaux

Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml

RP gouvernance - 2008-06-23

ANNEXES

Quelques exemples de listes drsquoindicateurs

1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)

Voir page suivante

Preacutepareacute par Roland Proulx

Consultant en planification institutionnelle et veille strateacutegique

Montreacuteal mai 2008

Performance Indicators

Discussion Paper

May 2004 (Prepared by the Office of Planning amp Analysis)

Table of Contents

Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25

2

Table 1 ndash Number of Indictors by Type of Data Source

Table 1 shows that surveys are the least frequently used source of data for indicators

Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a

plurality of indicators come from this source This measure somewhat underestimates the

importance of surveys however as it does not account for the weighting given to each

indicator in each study In the THES World Rankings for instance there is only a single

survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos

Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one

indicator is worth 20 of the final score

Raw

indIcator count

Surve

y data

Third parties

Universities

AsiaweekmdashAsias Best Universities 18 - - 18

Daily Telegraph (2003) 1 - 1 -

Education18com 9 3 4 2

Excelencia 2001 71 - 71 -

Financial Times (2003) 17 - 17 -

Guangdong Institute of Management Science 17 - 14 3

GuardianmdashUniversity Guide 2005 7 - 2 5

La Repubblica 23 2 21 -

Macleans University Rankings 24 1 5 18

Melbourne Institutemdash International Standing of Australian Universities

26 3 23 -

Netbig 2004 18 1 10 7

Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15

Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities

6 - 5 1

The TimesmdashGood University Guide 2005 9 - 9 -

Times Higher Education SupplementmdashWorld University Rankings

5 1 1 3

US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11

Washington MonthlymdashCollege Rankings 2005 8 - 1 7

Wuhan University Centre for Science Evaluation 45 2 22 21

3

A Indicators of Beginning Characteristics

ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of

students at the time they begin their studies Fourteen of the 18 rankings examined in this

study use one or more indicators of the beginning characteristics of students to arrive at

their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on

these factors with 33 of the total ranking coming from this class of indicators but the

Guardian Education18 Asiaweek and the two other North American surveys also place

considerable emphasis on this category

There are six main indicators used to determine which institutions have students with

positive ldquobeginning characteristicsrdquo

The most common measure of beginning characteristics is performance on national

standardized tests with nine surveys using this as a measure Education18 and the

Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used

by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)

Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this

data is collected and standardized by national bodies it has the benefit of being seen as a

relatively impartial method of determining the relative ldquostrengthrdquo of the students entering

each institution Institutionsrsquo results can be scored by showing either averages or the

percentage of entering students meeting a particular standard

Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses

secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This

is a second-best solution made necessary by the absence of any national standardized test

in Canada (or indeed of any provincial standardized tests at the end of secondary school

in provinces other than Alberta) The lack of national standardization makes this an

undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is

truly equivalent to an ldquoArdquo in another jurisdiction

Another measure of the strength of the student body is the percentage of incoming

students receiving (third-party) scholarships which is worth 11 of the score in the

Wuhan survey One can also approach the issue by measuring institutional selectivity In

effect this method infers the strength of the student body by the proportion of

4

applicants rejected the theory being that the higher the number of rejected applicants the

stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)

Normally this measure is expressed as a straight ratio of acceptances to applications but it

can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the

greatest weight on this measure) as a ratio of enrolments to applications Within the US

there is some dispute as to what constitutes an offer of admission and whether or not late

admissions are included as noted by Ganeshananthan (2003)

Student bodies are often considered to be strong if the school is able to attract a large

number of international or out-of-district students or if they contain people from diverse

ethnic backgrounds A number of league tables use the international student indicator

(which like the selectivity indicator is arguably as much an indicator of prestige and

reputation as it is of student characteristics) although in no case does this indicator

account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a

quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data

on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian

puts a somewhat larger emphasis on this indicator in comparison to other league tables

which use similar variables

A very different take on this idea is present in the Washington Monthly which released its

first set of College Rankings in September 2005 With the declared aim of using an

institutionrsquos commitment to social mobility as a measure of quality it uses the percentage

of students from low-income backgrounds as an indicator (with percentage of students

receiving need-based government (Pell) grants used as a proxy)

Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy

statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the

percentage of the student population who are graduate students (arguably this is a

research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the

number of graduate students auditing classes is used as an indicator the assumption is

presumably that if people are auditing then the classes must be very attractive The Italian

La Repubblica ranks an institution according to the number of part-time students it has

contrary to prevailing North American views on the undesirability of part-time study the

Italian rankings see higher numbers of part-time students in a positive light as it is

evidence that an institution is becoming less rigid in its timetabling and

5

permitting students to juggle both work and study something which was nearly

impossible in that country just a few years ago

The Washington Monthly also has a final category of indicators which reflect studentsrsquo

beginning characteristics namely their likelihood of performing community service as

measured by the percentage of students in the US Peace Corps and Reserve Officer

Training Corps (ROTC) and the percentage of students involved in work-study in the

community3 Together these three indicators account for 33 of an institutionrsquos total

ranking

B Indicators of Learning InputsmdashStaff

Generally both the quantity and quality of staff are positively correlated with institutional

quality The problem of course is finding useful metrics for each of these factors

especially if one excludes as we have done here measures of research performance and

research intensity putting them in a separate category4

The simplest measure is simply the number of faculty unadjusted for things like size of

student body Most national league tables however prefer to use variations on the

concept of facultystudent ratio Others try to measure teaching intensity with measures

such as courses per teacher or hours spent in class per student (both in La Repubblica)

These kinds of measures usually account for between 2-5 of the final rankings although

in some cases (ie the Guardian) this figure can be as high as 20

Another important way of measuring how faculty resources are deployed is the measure

of average class size which is used only by Macleanrsquos and the US News and World Report

Ostensibly the reason for measuring class size is to account in some form for the degree

3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category

6

of attention devoted to actually teaching students Despite the fact that important research

(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level

the use of this indicator appears to be a spillover from the North American debates on

class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein

2000) Regardless of why the indicators are used they are extraordinarily important to

these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and

World Reportrsquos rankings respectively

A number of ranking systems try to look at staff qualifications such as the number of

PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the

Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further

than other surveys and actually looks at the proportion of classes taught by tenure-track

staff Others (ie THES) look at the number of foreign faculty based on the assumption

that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still

others (ie La Repubblica) look at the age structure of the faculty Another proxy for

institutional quality is the pay rates for tenured staff on the assumption that institutions

with higher rates of pay on average attract better faculty this measure has been used both

by the US News and World Report and Asiaweek

Finally a number of league tables rank faculty inputs on the basis of standardized third-

party evaluations 6

Education18 the Financial Times the Times the Guardian and the US

News and World Report league tables all use some sort of ranking criterion based at least in

part on this indicator or variations thereof

C Indicators of Learning InputsmdashResources

Resource inputsmdashcrudely the amount of current dollars equipment and books available

to students at an institutionmdashare widely considered an important measure of quality Yet

despite the apparent simplicity of counting dollars and measuring assets

5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each

university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future

7

the means by which institutional wealth is measured varies considerably between ranking

systems

There are a number of revenue-based measures of resources Macleanrsquos uses public funding

of institutional budgets as a factor in its analysis conversely the Financial Times uses the

private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of

total score) and the US News and World Report (5 of total score) also measure alumni

financial support as a measure of quality

For reasons that are not entirely clear league tables tend to favour measures of

expenditures rather than revenues The Guardian looks at total institutional expenditures

as an indicator Institutional expenditure on student services is used as a measure of

institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total

institutional scores respectively) Rzezspospolita does not measure student services

expenditures directly but does measure student services outputs such as number of

student athletes and number of study clubs which amounts to more or less the same thing

Macleanrsquos also gives out 433 of its total score based on institutional expenditures on

scholarships and bursaries

Various aspects of physical infrastructure are also used as measures of institutional

resources most directly in the case of La Repubblica which bases 317 of its total rank on

the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding

assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type

of physical infrastructure measured is available Internet bandwidth which was used by

Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth

roughly 3 of the total score

By some distance the infrastructure indicators most favoured by the compilers of league

tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on

this with 12 of the total quality mark being taken from various types of library

infrastructure measurements (including acquisitions per year total volumes average

number of volumes per student and yearly library expenditure outside of acquisitions)

Netbig and Education18 also use library volume holdings while Asiaweek the Financial

Times and the Times also use measures of library expenditures outside of acquisitions or

computerization of library resources as measures of institutional quality

8

One important factor to note is that most ranking systems do not normalize their resource

and infrastructure measures That is to say it is raw spending power or simple size of

assets that is usually measured rather than spending per studentprofessor or assets per

studentprofessor As a result a number of these rankings systems have inbuilt biases

towards larger institutions

D Indicators of Learning Outputs

Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge

learned over the course of a baccalaureate degreemdashshould be a basic indicator of

institutional quality Unfortunately good means of measuring these outputsmdashlike the

National Survey of Student Engagement (NSSE) and the College Learning Assessment

(CLA)mdashhave only recently become available and for the most part institutions are still

keeping their scores secret Outside of these measures only a few very crude indicators are

available which likely explains why learning outputs do not feature especially prominently

in most ranking schemes

The simplest types of measures of learning outputs are those linked to graduation and

retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong

and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7

the latter three also use rates of graduation from Masterrsquos programs as indicators In some

cases the weights on these measures can be very highmdashin the Guangdong rankings

graduation rates account for over 50 of the rankingmdashbut in most cases the weights are

10 or less Retention rates commonly meaning the progression rate of first-year students

into second year are accorded less importance The US News and World Report Macleanrsquos

the Melbourne Institute and La Repubblica all employ retention measures as indicators but

none of them are worth more than 4 of total weighting Two publications make specific

indicators for retention and graduation of international students Macleanrsquos (graduation

rates of international students) and the Melbourne Institute (retention rates of international

students) The Washington Monthly looks specifically at institutional retention rates

adjusted for the participation of lower-income students and gives higher scores to

institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores

and number of Pell Grant recipients the US

7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18

9

News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos

ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)

E Indicators of Final Outcomes

Final outcomes are indications of generalized outcomes for students after graduation

Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness

good citizenship) but given the somewhat utilitarian justifications for education that are

currently in fashion (see Wolf 2000) employment outcomes are the most commonly used

measure of final outcomes These are given particular emphasis by the Guardian (where

employment outcomes are worth 17 of the total score) but are also used by the Financial

Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the

Times are interestingly not concerned with employment per se but with ldquoemployment in

an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher

Education Statistics Agency (HESA) uses Standard Occupational Classifications to

measure the proportion of graduates in professional or white-collar jobs anyone not in

such a job is considered not to be working in an area related to their studies (it is unclear

what methodology is used by the Financial Times and the Times although we suspect their

methods are broadly similar)

The only other measure of final outcomes in use is percentage of graduates returning for

additional education which is an indicator used by both the Melbourne Institute and the

Financial Times This is a particularly important indicator for the latter as it is worth 21

of the final ranking

The lack of indicators concerning final outcomes is interesting since most government-

sponsored performance-indicator regimes around the world are very much concerned

with such measures especially with respect to employment Possibly this indicates that

ranking systems simply do not view education outcomes as relevant measures of

educational quality Alternatively it may be the case that they simply have not found a

reliable indicator of outcomes or that there are reliable indicators but that there is so little

variation between institutions that it makes no sense to rank based on the data

10

F Indicators of Research

Many of the league tables covered in this survey include sections and weightings related

to universitiesrsquo research efforts It is in this field of measurement that we see the greatest

diversity of indicators Presumably this is because research inputs and outputs lend

themselves much more easily to measurement and manipulation than other areas of

institutional activity

Three studies include research staff as part of their ranking scheme La Repubblica at 952

the Melbourne Institute (4) and Wuhan (078)

Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used

method of looking at research quality but it is not universally admired because different

disciplines use different means to communicate major advances in knowledge (leading

scientists invariably produce large numbers of journal articles leading social scientists

may produce fewer journal articles but instead have one or two long important

monographsmdashsee Hicks 2004) There is also some concern among non-English speaking

countries that they are penalized in international rankings because so many of the major

journals (notably Science and Nature) are printed in English However the one set of

rankings that uses separate indicators to monitor articles published in English and articles

published in another language (the Wuhan rankings) shows that the two indicators are

positively correlated institutions that have more Chinese publications are also likely to

have more English publications and vice versa

Several sets of league tables measure bibliometric citations in various publication indices

The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a

weight of 20 of the final total Guangdong also monitors other Chinese universities

specifically for citations in engineering publications and weights this at 29 Moreover it

tacks on an additional 1049 for citations in science-oriented indices such as the Science

Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the

same category of scientific citations while the Melbourne Institute rates science citations at

68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social

Science Citation Index which does not include the humanities) are noted in only two

league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne

Institute (32) Another way of measuring research

11

impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a

weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the

Melbourne Institute

The complement of citations is of course publications Listing the number of publications

an individual group department or whole university releases can act as a weak substitute

for citationsmdashweak because simply publishing a paper or monograph is no guarantee that

the general public or other researchers will even glance at the work Guangdong gives an

1179 weighting to publications in science-oriented indices such as the Science Citation

Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same

indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong

even has a separate category just for measuring publications in Science and Nature

although it accords it an almost derisory weighting of 06 Under publications in social

science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their

final totals For publications in other indices (where the subject indices are

undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for

Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of

rankings to include research monographs weighted at 033

In countries where there are specific third-party evaluations of research output academic

quality of research is sometimes used as a research indicator The Times puts a very large

30 weight on this indicator while the Financial Times puts it at 119

Research awards are another handy third-party measurement of quality as the number of

international and national awards won by faculty andor graduates is often considered a

useful measure of institutional success International research awardsmdash specifically the

number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator

by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as

particularly suspect in some quarters given that the points are based on where the

recipient went to school rather than on where they are or were on the

8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the

most prominent publication indices

9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences

12

faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead

for nearly a century Wuhan uses similar measures but only accords them a weight of 14

National research awards are more common as a quality indicator used by La Repubblica

(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)

Financial indicators of research are also very common Research budgets as a factor in the

overall assessment of research in universities are covered by the Financial Times (9)

Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for

total amount of research expenditure unfortunately it is unclear precisely what this

research expenditure represents or how it is determined although it is clearly indicated

that it does not represent the total number of grants or projects at a university Total

number of research-based grants and projects is weighted by Education18 at 15 and

Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science

and engineering and another 55 to those for social sciences and humanities Similarly

the Melbourne Institute gives 6 of its overall weight to public-source grants making no

distinction between areas of study

In a slightly different vein Netbig (46) and Wuhan (278) both list the number of

research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica

(095) all weigh research-basedaffiliated research institutions or centres for studies

Finally one can also measure research not simply in terms of the amount of money it

generates but also in terms of the amount of future income it will generate Both

Guangdong (245) and Wuhan (193) measure the number of patents issued to

universities as a quality indicator

A final way of measuring an institutionrsquos research intensity is to look at the range of its

course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of

doctoral and Masterrsquos programs offered as a proxy for research intensity

As with physical and financial resources few if any of the research indicators are

normalized to account for institutional size (either by student or faculty numbers) In the

world of rankings bigger almost always means better an institution with 100 faculty with

ten citations apiece will always look worse than an institution with 1001 faculty

13

with one citation each To the extent that the raw production of knowledge matters this

form of measurement is acceptable To the extent that rankings are meant to show how

well institutions are doing on a like-to-like basis or to show the efficiency of universities it

is plainly inadequate This should be of particular concern to Chinese policy-makers

whose ranking systems are especially reliant on research-based indicators

G Indicators of Reputation

The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo

Those rankings systems which use the results of reputation surveys as an indicator do so as

an indirect measure of quality based on the assumption that the employers academics and

academic administrators surveyed have opinions of institutional quality that are informed

up-to-date and impartial While these assumptions are clearly open to debate they

nevertheless form an important basis for many ranking systems Another reason for using

reputation measures is the paucity of other data availablemdashsome countries have few

independent measures of teaching effectiveness university resources or output and

reputation can thus act as a useful surrogate Reputation rankings are often criticized as

simply quantifying the common ignorance of the people being surveyed However to the

extent that the people being surveyed hold positions which have the potential to affect

large numbers of young people and whose positions actually require some knowledge of

institutional quality (ie officials in charge of graduate admissions corporate recruiters

etc) then reputation rankings make sense because they provide useful information for

students about the perceived value of the degrees that they could obtain from various

universities

The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and

the Times which both accord reputation a weighting of 50 in their overall ranking

scheme Education18 assigns it almost as much significance at 40 The US News and World

Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just

below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)

and Netbig (15) The only other study to include reputation is Wuhan (117)

14

1 Please see Appendix B for a glossary of the various bodies referenced in this document

Appendix A Detailed Listing of Indicators and their Sources

Beginning Characteristics Indicator

Used By Source1

Incoming grades Macleans University

Percentage with grades above a set limit

Macleans University

US News and World Report

University

Performance on national standardised tests or benchmarks

Asiaweek University

Education18 3rd-party JUPAS

Financial Times Government agency 3rd-party UCAS

Guardian University Guide

Government agency 3rd-party UCAS

Melbourne Institute Government agency 3rd-party DEST

Netbig National entrance examination board

Times Good University Guide

Government agency 3rd-party UCAS

US News University

Wuhan Unknown presumed government 3rd-party

Student status La Repubblica Government agency 3rd-party MIUR

Admittance selectivity general

Asiaweek University

Admittance number of applications to places

Asiaweek University

Financial Times Government agency 3rd-party UCAS

La Repubblica Government agency 3rd-party MIUR

US News University

15

Beginning Characteristics Indicator

Used By Source1

Out-of-locality student percentage

Macleans University

International student percentages

Financial Times Government agency 3rd-party HESA

Macleans University

Shanghai Institute of Educational Science

Times World University

Wuhan Unknown presumed university

Undergraduate students among all students percentages

Netbig

Wuhan Unknown presumed university

Ethnic diversity in student body

Guardian University

Learning Inputs ndash Staff Indicator

Used By Source

Facultystudent ratio Asiaweek University

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Financial Times Government agency 3rd-party HESA

La Repubblica Government agency 3rd-party MIUR

Times Good University Guide

Government agency 3rd-party HESA

Times World University

US News University

Wuhan Unknown presumed university

16

Learning Inputs ndash Staff Indicator

Used By Source

Social science faculty student ratio

Melbourne Government agency 3rd-party DEST

Science faculty student ratio

Melbourne Government agency 3rd-party DEST

Administrative staff student ratio

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Staff student ratio (regardless of division)

Guardian Government agency 3rd-party HESA

Netbig University

Course per teacher La Repubblica Government agency 3rd-party MIUR

Per-teacher university spending

Asiaweek University

Faculty pay rates fortenured staff

Asiaweek University

US News University

Number of full-time part-time faculty

Netbig University

US News University

Wuhan Unknown presumed university

Faculty with researchprojects

Wuhan Unknown presumed university

Class size differentiation Macleans University

US News University

Classes taught by tenured faculty

Macleans University

Exchange programmes hosted

La Repubblica Government agency 3rd-party AgNaSoc

Number of classes lsquoactually taughtrsquo

La Repubblica Government agency 3rd-party MIUR

of international faculty (v faculty as a whole)

Times World University

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

wwwigopporg

Tous les indicateurs des listes longues et tous les domaines drsquoactiviteacutes nrsquoont pas eacuteteacute retenus

Il srsquoagit drsquoun inventaire des indicateurs courants tels que recenseacutes dans plusieurs listes

courtes et longues et utiliseacutes agrave des fins de planification drsquoeacutevaluation de la performance et

de reddition de comptes De plus dans un contexte de mondialisation et

drsquointernationalisation il est devenu important et pertinent drsquoun point de vue strateacutegique de

retenir les indicateurs utiliseacutes dans les principaux classements nationaux (9 sur 20 possibles)

et internationaux (4) 1 Lrsquoencadreacute qui suit reacutesume ce choix

classements nationaux classements internationaux

universiteacutes canadiennes

et MEQ

USNews and

Report ampThe

Center

Macleans CHEDAAD GUARDIAN ASIAWEEKDEST

AUSTRALIE amp

MELBOURNE REPUBLICA

QS-TIMES SHANHAI TAIWAN

WEB

1 On trouvera en fin de document lrsquoensemble des reacutefeacuterences consulteacutees ainsi qursquoun annexe preacutesentant quelques exemples de listes drsquoindicateurs

6

1INVENTAIRE DES INDICATEURS COURANTS

Agrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

EacutetudiantsUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIEampMel

bourneREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre deacutetudiantsNombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input MEQ G10 DETYA THESARWUproportion deacutetudiantes input DETYA des eacutetudiants agrave temps partiel input AOTQU CHEDAAD Republicainscriptions au 1er cycle agrave temps plein input MEQ CHEDAAD inscriptions au 2e cycle input MEQ inscriptions au 3e cycle input MEQ eacutetudiants aux 2e et 3e cycles input G10 DETYA eacutetudiants internationaux input TU THES des eacutetudiants internationaux input Macleans CHEDAAD THES des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input MEQ stagiaires postdoctoraux input MEQ reacutesidents et reacutesidentes en meacutedecine input MEQ

inputrecrutement et admission inputdemandes dadmission aux programmes reacuteguliers input OTQ CHEDAAD offres dadmissiondemandes dadmission i t CHEDAAD offres dadmissiondemandes dadmission input CHEDAAD inscriptionsdemandes dadmission input T CHEDAAD Asiaweek Republica

inputdossier scolaire des nouveaux eacutetudiants (bulletin Cote R) input US NEWS Macleans Asiaweek

provenance des eacutetudiants input Macleanspopulation cible inputreacutegion meacutetropolitaine inputQueacutebec inputCanada inputInternational input

inputdiversiteacute ethnique input US NEWS Guardian

Bourses accordeacutees aux eacutetudiants - Financement moyen input TMEQpremier cycle input Macleans Republicadeuxiegraveme cycle input G10troisiegraveme cycle input G10

cheminement des eacutetudiants (temps complet) perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output QT MEQ US NEWS Macleans Republica diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output G10 MEQ US NEWS Macleans Melbourne Republica dilplomationmaicirctrise apregraves 4 ans output G10 MEQ US NEWS Macleans Republica diplomationdoctorat apregraves 6 ans output G10 MEQ US NEWS Macleans Republica

Deacutepenses de fonctionnement par eacutetudiant input CAUBOG10 US NEWS Macleans Guardian Asia weekFrais de scolariteacute input G10frais affeacuterents inputDeacutepenses pour les Services aux eacutetudiants input Macleans

taux demployabiliteacute apregraves 1 an outcome Ontario CHEDAAD Guardian de retour aux eacutetudes outcome Melbourne

Valeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome Guardian

RP gouvernance - 2008-06-23

2INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

EnseignantsUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Nombre denseignants Nombre total denseignants inputNombre de professeurs reacuteguliers input G10 MEQStatCan USNEWS Melbourne de professeures input OQ MelbourneNombre de chercheurs inputNombre de professeurs avec diplocircme international THESNombre de professeurs de clinique eacutequivalents temps complet input G10StatCanNombre de chargeacutes de cours et autres personnels eacutequivalents temps complet input USNEWSNombre dauxiliaires en eacutequivalence temps complet input

Caracteacuteristiques des professeurs reacuteguliers input des professeurs et chercheurs avec PhD (ou grade terminal) input USNEWS Macleansstructure dage du corps professoral input Asiaweek Melbourne Republicaacircge moyen des professeurs reacuteguliers inputacircge moyen des professeurs adjoints inputacircge moyen des professeurs agreacutegeacutes inputacircge moyen des professeurs titulaires input

reacutemuneacuteration moyenne des professeurs reacuteguliers input Q T U U S NEWS Asiaweek Melbournereacutemuneacuteration moyenne des professeurs reacuteguliers input QTU USNEWS Asiaweek Melbournedes professeurs adjoints inputdes professeurs agreacutegeacutes inputdes professeurs titulaire input

Activiteacutes denseignement inputnombre deacutetudiants EETC Professeur reacutegulier input G10 MEQ USNEWS Guardian Asiaweek Melbourne Republica THESnombre deacutetudiants EETC Professeur reacutegulier en sciences sociales input Melbournenombre deacutetudiants EETC Professeur reacutegulier en sciences input Melbournenombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input

Nombre de cours dispenseacutes par lInstitution (ANNUEL) input Republica cours assumeacutes par les professeurs reacuteguliers input MEQ USNEWS Macleans Republica cours assumeacutes par les chargeacutes de cours eacutequivalents temps complet input MEQTaille des groupe cours input MEQ USNEWS Macleans Guardian

RP gouvernance - 2008-06-23

3INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

PersonnelsUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Le personnel de soutientotal de tous les personnels reacuteguliers acadeacutemiques et non acd agrave temps complet inputnombre des personnels de soutien (staff) input G10 MelbourneDeacutepenses des personnel non-enseignant input MEQnombre des personnels de soutien (staff)- secteur enseignement inputnombre des personnels de soutien (staff) - secteur soutien acadeacutemique inputnombre des personnels de soutien (staff) - secteur administration inputnombre des personnels de soutien (staff) - secteur service aux eacutetudiants inputnombre des personnels de soutien (staff) - entreprises auxiliaires input

rectoratpersonnel dencadrement non acadeacutemique et cadres acadeacutemiques inputratio sur lensemble des personnels acadeacutemiques et non acadeacutemiques inputeacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input G10 Guardianpersonnel de soutien professeur reacutegulier input G10

RP gouvernance - 2008-06-23

4INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Activiteacutes de recherche USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Revenus de recherche de professeurs et chercheurs ayant fait une demande de subvention processus de professeurs et chercheurs ayant obtenu une subvention inputRevenus totaux de recherche (subventions et contrats) input CAUBO G10 The Center Asiaweek Melbournepart de la recherche contractuelle input MEQSubventions publiques input Macleans Melbournerevenus de recherche obtenus des conseils de recherche feacutedeacuteraux input AOQTU The Center Macleans Melbournerevenus moyens de recherche par professeur et chercheur input InfosourceMEQ des deacutepenses de recherchebudget total des deacutepenses input TNombre de chaires et de centre dexcellence output AFondation canadienne pour linnovation (FCI) input G10 -nombre de projets output G10 -valeur $ output G10

nombre de programmes de maicirctrise de recherche input The Center Asiaweeknombre de programmes de doctorat input The Center Asiaweek

Publications et prix outputNombre de publications output Infosource Asiaweek Melbourne

Shanghai Leiden Taiwan

Nombre de publications par professeur et chercheur output Infosource CHEDAAD ShanghaiPublications dans Nature and Science output TaiwanNombre total de citations output InfosourceT Asiaweek MelbourneNombre de citations par professeur et chercheur output InfosourceT Melbourne THESNombre de citations par publication output T CHEDAAD LeidenHighly cited researchers dans 21 secteurs disciplinaires

output

ShanghaiTaiwan Leiden

Impact des publications selon les secteurs disciplinaires output Infosource TaiwanTHES Shanghai

Domaines disciplinaires dexcellence inputTaiwanShanghai THES

Prix et meacutedailles de niveau national et international output G10 The Center Macleans Melbourne Republica ShanghaiNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur output CHEDAADnombre de brevets et de licences output G10 CHEDAADnombre de spin-off companies G10

RP gouvernance - 2008-06-23

5INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

InternationalisationUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

deacutepenses pour le secteur internationalisation input MEQnombre deacutetudiants en stage agrave leacutetrangernombre deacutetudiants temps complet input tes un cannombre deacutetudiants internationauxnombre deacutetudiants temps complet input tes un can Melbourne THESnombre deacutetudiants en cotutellesnombre deacutetudiant des cycles supeacuterieurs input CHEDAADnombre de professeurs inviteacutes internationaux processus tes un can CHEDAADprofesseurs inviteacutes agrave leacutetranger processus CHEDAADnombre deacutetudiants posdoctoraux processus tes un can CHEDAADproportion de cours de langues eacutetrangegraveres processusnombre de programmes deacutechange processus tes un can CHEDAADEntentes de partenariat outputPrix internationaux output G10 The Center Macleans Melbourne Republica Shanghai

RP gouvernance - 2008-06-23

6INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Inteacutegration des technologies de linformation USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Deacutepenses totales consacreacutees aux TIC input MEQpersonnel techniqueenseignant reacutegulier inputpersonnel techniqueeacutetudiant en eacutequivalence temps complet inputExistence dun reacuteseau informatique (accessible agrave linterne et agrave lexterne) outputaccegraves au reacuteseau avec fils et sans fils processusaccegraves aux ressources documentaires des bibliothegraveques outputService de courriel interne et externe en mode WEB outputNombre dordinateurs par professeur et par eacutetudiant outputNombre de cours en ligne processusnombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) outputAccegraves des professeurs aux banque de donneacutees et aux logiciels processus

Site WEB (eacutevaluation du site WEB) WEBOMETRICSSize (nombre de pages recenseacutees) outcomevisibilitty (consultation observeacutee selon le nobre de lien externe unique) outcomeRich files (intensiteacute de lactiviteacute acadeacutemique) outcomeScholar (activiteacute de recherche publications et citations) outcome

RP gouvernance - 2008-06-23

7INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

BibliothegravequesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre total de volumes et de documents audiovisuels et eacutelectroniques input G10 ARLacquisition de volumes (au cours de lanneacutee) input ARL Macleansnombre total de volumes eacutetudiants en eacutequivalence temps complet input G10 ARL Macleans Deacutepenses des bibliothegraveques input MEQ Macleans AsiaweekDeacutepenses des bibliothegravequesbudget de fonctionnement input ARLDeacutepenses des bibliothegravequeseacutetudiants en eacutequivalence temps complet input G10 ARLPersonnels reacuteguliers agrave temps complet input G10 ARLClassement outcome ARL

RP gouvernance - 2008-06-23

8INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Revenus et deacutepenses USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Produits et revenusSubvention du ministegravere de lEacuteducation du Queacutebec input MEQ subvMEQrevenus de fonctionnement input G10 MacleansDroits de scolariteacute input G10 droits de scolariteacute revenus de fonctionnement input G10autres produits input G10Total des revenus de fonctionnement input G10 MEQ Guardian revenus de fonctionnement eacutetudiants en eacutequivalence temps complet input CAUBO USNews MacleansRevenus de recherche input G10encaissement par nature (dons immobilisations chaires bourses) input G10 fonds de dotation input G10 CenterContribution des alumni aux encaissements et fonds de dotation input USNews Macleans fonds de dotation eacutetudiants en eacutequivalence temps complet input G10Total des produits G10

Charges et deacutepenses G10salaire des professeurs input G10 MEQsalaire des chargeacutes de cours input G10salaires des auxiliaires input MEQsalaires des auxiliaires input MEQsalaire des personnels (staff) input G10autres deacutepenses input G10total des deacutepenses input G10 deacutepenses denseignement deacutepenses totales de fonctionnement input UBC MEQ

Chargesproduits processusEacutequilibre budgeacutetaire revenus et deacutepenses de fonctionnement processus MEQ

RP gouvernance - 2008-06-23

9INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

EspacesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Nombre de megravetres carreacutes bruts input G10MCB eacutetudiants en eacutequivalence temps complet input G10megravetres carreacutes nets alloueacutes input MEQNombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input Tbesoin despace en megravetres carreacutes nets inputExistence dun plan directeur des espaces inputBudget preacutevu au plan directeur des espaces input TBudget alloueacute aux espaces input MEQdeacutepenses de fonctionnement nombre de megravetres carreacutes bruts input G10

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Environnement USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

consommation de leacutenergie (exprimeacutee en BTU processus universiteacutes canadiennes et ameacutericaines BTUpopulation totale de luniversiteacute processus voir en particulier Penn State et Umich de leacutenergie produit par des sources renouvelable processusUtilisation de leau processuslitres deau par personne processusGaz agrave effet de serre (en tonnes meacutetriques CO2 eacutequivalents) processusMT CO2 par personne processusDeacutechets solides (en tonnes) processusTonnes de deacutechets solides per capita processus de deacutechets solides recycleacutes processus

RP gouvernance - 2008-06-23

10INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

ReacuteputationUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

enquecircte meneacutee aupregraves des eacutetudiants outcome NSSE Melbourneenquecircte meneacutee aupregraves des diplocircmeacutes outcomeenquecircte meneacutee aupregraves des pairs outcome USNews Asiaweek THESenquecircte meneacutee aupregraves des employeurs outcome Macleans Melbourneenquecircte meneacutee aupregraves des stakeholders outcome Macleans Melbourneenquecircte meneacutee aupregraves du monde des affaires outcome Macleans Melbournecouverture meacutediatique outcome AT

RP gouvernance - 2008-06-23

11INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Planification et gestion strateacutegiquesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Pouvoirs des corps universitaire processus un Canliens dynamiques entre Conseil Assembleacutee universitaire commission des eacutetudes processus un Canprocessus formel de planification strateacutegique processus un Canproduction de donneacutees et indicateurs processus un Canprocessus deacutevaluation des faculteacutes deacutepartements et services processus un Canrevue systeacutematique des responsabiliteacutes et efficaciteacute des comiteacutes processus un Candeacutepenses de ladministration centrale deacutepenses de fonctionnement processus un Can

RP gouvernance - 2008-06-23

12INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Eacuteducation permanenteUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre de programmes offerts input un Cannombre deacutetudiants inscrits input un CanRevenus et deacutepenses du service input un Can

RP gouvernance - 2008-06-23

13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes

REacuteFEacuteRENCES

Aperccedilu densemble des principaux indicateurs

GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost

Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes

MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf

Indicateurs utiliseacutes dans les classements internationaux

Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml

RP gouvernance - 2008-06-23

ANNEXES

Quelques exemples de listes drsquoindicateurs

1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)

Voir page suivante

Preacutepareacute par Roland Proulx

Consultant en planification institutionnelle et veille strateacutegique

Montreacuteal mai 2008

Performance Indicators

Discussion Paper

May 2004 (Prepared by the Office of Planning amp Analysis)

Table of Contents

Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25

2

Table 1 ndash Number of Indictors by Type of Data Source

Table 1 shows that surveys are the least frequently used source of data for indicators

Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a

plurality of indicators come from this source This measure somewhat underestimates the

importance of surveys however as it does not account for the weighting given to each

indicator in each study In the THES World Rankings for instance there is only a single

survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos

Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one

indicator is worth 20 of the final score

Raw

indIcator count

Surve

y data

Third parties

Universities

AsiaweekmdashAsias Best Universities 18 - - 18

Daily Telegraph (2003) 1 - 1 -

Education18com 9 3 4 2

Excelencia 2001 71 - 71 -

Financial Times (2003) 17 - 17 -

Guangdong Institute of Management Science 17 - 14 3

GuardianmdashUniversity Guide 2005 7 - 2 5

La Repubblica 23 2 21 -

Macleans University Rankings 24 1 5 18

Melbourne Institutemdash International Standing of Australian Universities

26 3 23 -

Netbig 2004 18 1 10 7

Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15

Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities

6 - 5 1

The TimesmdashGood University Guide 2005 9 - 9 -

Times Higher Education SupplementmdashWorld University Rankings

5 1 1 3

US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11

Washington MonthlymdashCollege Rankings 2005 8 - 1 7

Wuhan University Centre for Science Evaluation 45 2 22 21

3

A Indicators of Beginning Characteristics

ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of

students at the time they begin their studies Fourteen of the 18 rankings examined in this

study use one or more indicators of the beginning characteristics of students to arrive at

their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on

these factors with 33 of the total ranking coming from this class of indicators but the

Guardian Education18 Asiaweek and the two other North American surveys also place

considerable emphasis on this category

There are six main indicators used to determine which institutions have students with

positive ldquobeginning characteristicsrdquo

The most common measure of beginning characteristics is performance on national

standardized tests with nine surveys using this as a measure Education18 and the

Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used

by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)

Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this

data is collected and standardized by national bodies it has the benefit of being seen as a

relatively impartial method of determining the relative ldquostrengthrdquo of the students entering

each institution Institutionsrsquo results can be scored by showing either averages or the

percentage of entering students meeting a particular standard

Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses

secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This

is a second-best solution made necessary by the absence of any national standardized test

in Canada (or indeed of any provincial standardized tests at the end of secondary school

in provinces other than Alberta) The lack of national standardization makes this an

undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is

truly equivalent to an ldquoArdquo in another jurisdiction

Another measure of the strength of the student body is the percentage of incoming

students receiving (third-party) scholarships which is worth 11 of the score in the

Wuhan survey One can also approach the issue by measuring institutional selectivity In

effect this method infers the strength of the student body by the proportion of

4

applicants rejected the theory being that the higher the number of rejected applicants the

stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)

Normally this measure is expressed as a straight ratio of acceptances to applications but it

can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the

greatest weight on this measure) as a ratio of enrolments to applications Within the US

there is some dispute as to what constitutes an offer of admission and whether or not late

admissions are included as noted by Ganeshananthan (2003)

Student bodies are often considered to be strong if the school is able to attract a large

number of international or out-of-district students or if they contain people from diverse

ethnic backgrounds A number of league tables use the international student indicator

(which like the selectivity indicator is arguably as much an indicator of prestige and

reputation as it is of student characteristics) although in no case does this indicator

account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a

quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data

on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian

puts a somewhat larger emphasis on this indicator in comparison to other league tables

which use similar variables

A very different take on this idea is present in the Washington Monthly which released its

first set of College Rankings in September 2005 With the declared aim of using an

institutionrsquos commitment to social mobility as a measure of quality it uses the percentage

of students from low-income backgrounds as an indicator (with percentage of students

receiving need-based government (Pell) grants used as a proxy)

Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy

statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the

percentage of the student population who are graduate students (arguably this is a

research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the

number of graduate students auditing classes is used as an indicator the assumption is

presumably that if people are auditing then the classes must be very attractive The Italian

La Repubblica ranks an institution according to the number of part-time students it has

contrary to prevailing North American views on the undesirability of part-time study the

Italian rankings see higher numbers of part-time students in a positive light as it is

evidence that an institution is becoming less rigid in its timetabling and

5

permitting students to juggle both work and study something which was nearly

impossible in that country just a few years ago

The Washington Monthly also has a final category of indicators which reflect studentsrsquo

beginning characteristics namely their likelihood of performing community service as

measured by the percentage of students in the US Peace Corps and Reserve Officer

Training Corps (ROTC) and the percentage of students involved in work-study in the

community3 Together these three indicators account for 33 of an institutionrsquos total

ranking

B Indicators of Learning InputsmdashStaff

Generally both the quantity and quality of staff are positively correlated with institutional

quality The problem of course is finding useful metrics for each of these factors

especially if one excludes as we have done here measures of research performance and

research intensity putting them in a separate category4

The simplest measure is simply the number of faculty unadjusted for things like size of

student body Most national league tables however prefer to use variations on the

concept of facultystudent ratio Others try to measure teaching intensity with measures

such as courses per teacher or hours spent in class per student (both in La Repubblica)

These kinds of measures usually account for between 2-5 of the final rankings although

in some cases (ie the Guardian) this figure can be as high as 20

Another important way of measuring how faculty resources are deployed is the measure

of average class size which is used only by Macleanrsquos and the US News and World Report

Ostensibly the reason for measuring class size is to account in some form for the degree

3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category

6

of attention devoted to actually teaching students Despite the fact that important research

(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level

the use of this indicator appears to be a spillover from the North American debates on

class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein

2000) Regardless of why the indicators are used they are extraordinarily important to

these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and

World Reportrsquos rankings respectively

A number of ranking systems try to look at staff qualifications such as the number of

PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the

Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further

than other surveys and actually looks at the proportion of classes taught by tenure-track

staff Others (ie THES) look at the number of foreign faculty based on the assumption

that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still

others (ie La Repubblica) look at the age structure of the faculty Another proxy for

institutional quality is the pay rates for tenured staff on the assumption that institutions

with higher rates of pay on average attract better faculty this measure has been used both

by the US News and World Report and Asiaweek

Finally a number of league tables rank faculty inputs on the basis of standardized third-

party evaluations 6

Education18 the Financial Times the Times the Guardian and the US

News and World Report league tables all use some sort of ranking criterion based at least in

part on this indicator or variations thereof

C Indicators of Learning InputsmdashResources

Resource inputsmdashcrudely the amount of current dollars equipment and books available

to students at an institutionmdashare widely considered an important measure of quality Yet

despite the apparent simplicity of counting dollars and measuring assets

5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each

university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future

7

the means by which institutional wealth is measured varies considerably between ranking

systems

There are a number of revenue-based measures of resources Macleanrsquos uses public funding

of institutional budgets as a factor in its analysis conversely the Financial Times uses the

private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of

total score) and the US News and World Report (5 of total score) also measure alumni

financial support as a measure of quality

For reasons that are not entirely clear league tables tend to favour measures of

expenditures rather than revenues The Guardian looks at total institutional expenditures

as an indicator Institutional expenditure on student services is used as a measure of

institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total

institutional scores respectively) Rzezspospolita does not measure student services

expenditures directly but does measure student services outputs such as number of

student athletes and number of study clubs which amounts to more or less the same thing

Macleanrsquos also gives out 433 of its total score based on institutional expenditures on

scholarships and bursaries

Various aspects of physical infrastructure are also used as measures of institutional

resources most directly in the case of La Repubblica which bases 317 of its total rank on

the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding

assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type

of physical infrastructure measured is available Internet bandwidth which was used by

Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth

roughly 3 of the total score

By some distance the infrastructure indicators most favoured by the compilers of league

tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on

this with 12 of the total quality mark being taken from various types of library

infrastructure measurements (including acquisitions per year total volumes average

number of volumes per student and yearly library expenditure outside of acquisitions)

Netbig and Education18 also use library volume holdings while Asiaweek the Financial

Times and the Times also use measures of library expenditures outside of acquisitions or

computerization of library resources as measures of institutional quality

8

One important factor to note is that most ranking systems do not normalize their resource

and infrastructure measures That is to say it is raw spending power or simple size of

assets that is usually measured rather than spending per studentprofessor or assets per

studentprofessor As a result a number of these rankings systems have inbuilt biases

towards larger institutions

D Indicators of Learning Outputs

Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge

learned over the course of a baccalaureate degreemdashshould be a basic indicator of

institutional quality Unfortunately good means of measuring these outputsmdashlike the

National Survey of Student Engagement (NSSE) and the College Learning Assessment

(CLA)mdashhave only recently become available and for the most part institutions are still

keeping their scores secret Outside of these measures only a few very crude indicators are

available which likely explains why learning outputs do not feature especially prominently

in most ranking schemes

The simplest types of measures of learning outputs are those linked to graduation and

retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong

and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7

the latter three also use rates of graduation from Masterrsquos programs as indicators In some

cases the weights on these measures can be very highmdashin the Guangdong rankings

graduation rates account for over 50 of the rankingmdashbut in most cases the weights are

10 or less Retention rates commonly meaning the progression rate of first-year students

into second year are accorded less importance The US News and World Report Macleanrsquos

the Melbourne Institute and La Repubblica all employ retention measures as indicators but

none of them are worth more than 4 of total weighting Two publications make specific

indicators for retention and graduation of international students Macleanrsquos (graduation

rates of international students) and the Melbourne Institute (retention rates of international

students) The Washington Monthly looks specifically at institutional retention rates

adjusted for the participation of lower-income students and gives higher scores to

institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores

and number of Pell Grant recipients the US

7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18

9

News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos

ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)

E Indicators of Final Outcomes

Final outcomes are indications of generalized outcomes for students after graduation

Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness

good citizenship) but given the somewhat utilitarian justifications for education that are

currently in fashion (see Wolf 2000) employment outcomes are the most commonly used

measure of final outcomes These are given particular emphasis by the Guardian (where

employment outcomes are worth 17 of the total score) but are also used by the Financial

Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the

Times are interestingly not concerned with employment per se but with ldquoemployment in

an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher

Education Statistics Agency (HESA) uses Standard Occupational Classifications to

measure the proportion of graduates in professional or white-collar jobs anyone not in

such a job is considered not to be working in an area related to their studies (it is unclear

what methodology is used by the Financial Times and the Times although we suspect their

methods are broadly similar)

The only other measure of final outcomes in use is percentage of graduates returning for

additional education which is an indicator used by both the Melbourne Institute and the

Financial Times This is a particularly important indicator for the latter as it is worth 21

of the final ranking

The lack of indicators concerning final outcomes is interesting since most government-

sponsored performance-indicator regimes around the world are very much concerned

with such measures especially with respect to employment Possibly this indicates that

ranking systems simply do not view education outcomes as relevant measures of

educational quality Alternatively it may be the case that they simply have not found a

reliable indicator of outcomes or that there are reliable indicators but that there is so little

variation between institutions that it makes no sense to rank based on the data

10

F Indicators of Research

Many of the league tables covered in this survey include sections and weightings related

to universitiesrsquo research efforts It is in this field of measurement that we see the greatest

diversity of indicators Presumably this is because research inputs and outputs lend

themselves much more easily to measurement and manipulation than other areas of

institutional activity

Three studies include research staff as part of their ranking scheme La Repubblica at 952

the Melbourne Institute (4) and Wuhan (078)

Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used

method of looking at research quality but it is not universally admired because different

disciplines use different means to communicate major advances in knowledge (leading

scientists invariably produce large numbers of journal articles leading social scientists

may produce fewer journal articles but instead have one or two long important

monographsmdashsee Hicks 2004) There is also some concern among non-English speaking

countries that they are penalized in international rankings because so many of the major

journals (notably Science and Nature) are printed in English However the one set of

rankings that uses separate indicators to monitor articles published in English and articles

published in another language (the Wuhan rankings) shows that the two indicators are

positively correlated institutions that have more Chinese publications are also likely to

have more English publications and vice versa

Several sets of league tables measure bibliometric citations in various publication indices

The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a

weight of 20 of the final total Guangdong also monitors other Chinese universities

specifically for citations in engineering publications and weights this at 29 Moreover it

tacks on an additional 1049 for citations in science-oriented indices such as the Science

Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the

same category of scientific citations while the Melbourne Institute rates science citations at

68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social

Science Citation Index which does not include the humanities) are noted in only two

league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne

Institute (32) Another way of measuring research

11

impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a

weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the

Melbourne Institute

The complement of citations is of course publications Listing the number of publications

an individual group department or whole university releases can act as a weak substitute

for citationsmdashweak because simply publishing a paper or monograph is no guarantee that

the general public or other researchers will even glance at the work Guangdong gives an

1179 weighting to publications in science-oriented indices such as the Science Citation

Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same

indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong

even has a separate category just for measuring publications in Science and Nature

although it accords it an almost derisory weighting of 06 Under publications in social

science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their

final totals For publications in other indices (where the subject indices are

undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for

Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of

rankings to include research monographs weighted at 033

In countries where there are specific third-party evaluations of research output academic

quality of research is sometimes used as a research indicator The Times puts a very large

30 weight on this indicator while the Financial Times puts it at 119

Research awards are another handy third-party measurement of quality as the number of

international and national awards won by faculty andor graduates is often considered a

useful measure of institutional success International research awardsmdash specifically the

number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator

by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as

particularly suspect in some quarters given that the points are based on where the

recipient went to school rather than on where they are or were on the

8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the

most prominent publication indices

9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences

12

faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead

for nearly a century Wuhan uses similar measures but only accords them a weight of 14

National research awards are more common as a quality indicator used by La Repubblica

(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)

Financial indicators of research are also very common Research budgets as a factor in the

overall assessment of research in universities are covered by the Financial Times (9)

Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for

total amount of research expenditure unfortunately it is unclear precisely what this

research expenditure represents or how it is determined although it is clearly indicated

that it does not represent the total number of grants or projects at a university Total

number of research-based grants and projects is weighted by Education18 at 15 and

Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science

and engineering and another 55 to those for social sciences and humanities Similarly

the Melbourne Institute gives 6 of its overall weight to public-source grants making no

distinction between areas of study

In a slightly different vein Netbig (46) and Wuhan (278) both list the number of

research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica

(095) all weigh research-basedaffiliated research institutions or centres for studies

Finally one can also measure research not simply in terms of the amount of money it

generates but also in terms of the amount of future income it will generate Both

Guangdong (245) and Wuhan (193) measure the number of patents issued to

universities as a quality indicator

A final way of measuring an institutionrsquos research intensity is to look at the range of its

course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of

doctoral and Masterrsquos programs offered as a proxy for research intensity

As with physical and financial resources few if any of the research indicators are

normalized to account for institutional size (either by student or faculty numbers) In the

world of rankings bigger almost always means better an institution with 100 faculty with

ten citations apiece will always look worse than an institution with 1001 faculty

13

with one citation each To the extent that the raw production of knowledge matters this

form of measurement is acceptable To the extent that rankings are meant to show how

well institutions are doing on a like-to-like basis or to show the efficiency of universities it

is plainly inadequate This should be of particular concern to Chinese policy-makers

whose ranking systems are especially reliant on research-based indicators

G Indicators of Reputation

The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo

Those rankings systems which use the results of reputation surveys as an indicator do so as

an indirect measure of quality based on the assumption that the employers academics and

academic administrators surveyed have opinions of institutional quality that are informed

up-to-date and impartial While these assumptions are clearly open to debate they

nevertheless form an important basis for many ranking systems Another reason for using

reputation measures is the paucity of other data availablemdashsome countries have few

independent measures of teaching effectiveness university resources or output and

reputation can thus act as a useful surrogate Reputation rankings are often criticized as

simply quantifying the common ignorance of the people being surveyed However to the

extent that the people being surveyed hold positions which have the potential to affect

large numbers of young people and whose positions actually require some knowledge of

institutional quality (ie officials in charge of graduate admissions corporate recruiters

etc) then reputation rankings make sense because they provide useful information for

students about the perceived value of the degrees that they could obtain from various

universities

The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and

the Times which both accord reputation a weighting of 50 in their overall ranking

scheme Education18 assigns it almost as much significance at 40 The US News and World

Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just

below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)

and Netbig (15) The only other study to include reputation is Wuhan (117)

14

1 Please see Appendix B for a glossary of the various bodies referenced in this document

Appendix A Detailed Listing of Indicators and their Sources

Beginning Characteristics Indicator

Used By Source1

Incoming grades Macleans University

Percentage with grades above a set limit

Macleans University

US News and World Report

University

Performance on national standardised tests or benchmarks

Asiaweek University

Education18 3rd-party JUPAS

Financial Times Government agency 3rd-party UCAS

Guardian University Guide

Government agency 3rd-party UCAS

Melbourne Institute Government agency 3rd-party DEST

Netbig National entrance examination board

Times Good University Guide

Government agency 3rd-party UCAS

US News University

Wuhan Unknown presumed government 3rd-party

Student status La Repubblica Government agency 3rd-party MIUR

Admittance selectivity general

Asiaweek University

Admittance number of applications to places

Asiaweek University

Financial Times Government agency 3rd-party UCAS

La Repubblica Government agency 3rd-party MIUR

US News University

15

Beginning Characteristics Indicator

Used By Source1

Out-of-locality student percentage

Macleans University

International student percentages

Financial Times Government agency 3rd-party HESA

Macleans University

Shanghai Institute of Educational Science

Times World University

Wuhan Unknown presumed university

Undergraduate students among all students percentages

Netbig

Wuhan Unknown presumed university

Ethnic diversity in student body

Guardian University

Learning Inputs ndash Staff Indicator

Used By Source

Facultystudent ratio Asiaweek University

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Financial Times Government agency 3rd-party HESA

La Repubblica Government agency 3rd-party MIUR

Times Good University Guide

Government agency 3rd-party HESA

Times World University

US News University

Wuhan Unknown presumed university

16

Learning Inputs ndash Staff Indicator

Used By Source

Social science faculty student ratio

Melbourne Government agency 3rd-party DEST

Science faculty student ratio

Melbourne Government agency 3rd-party DEST

Administrative staff student ratio

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Staff student ratio (regardless of division)

Guardian Government agency 3rd-party HESA

Netbig University

Course per teacher La Repubblica Government agency 3rd-party MIUR

Per-teacher university spending

Asiaweek University

Faculty pay rates fortenured staff

Asiaweek University

US News University

Number of full-time part-time faculty

Netbig University

US News University

Wuhan Unknown presumed university

Faculty with researchprojects

Wuhan Unknown presumed university

Class size differentiation Macleans University

US News University

Classes taught by tenured faculty

Macleans University

Exchange programmes hosted

La Repubblica Government agency 3rd-party AgNaSoc

Number of classes lsquoactually taughtrsquo

La Repubblica Government agency 3rd-party MIUR

of international faculty (v faculty as a whole)

Times World University

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

1INVENTAIRE DES INDICATEURS COURANTS

Agrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

EacutetudiantsUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIEampMel

bourneREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre deacutetudiantsNombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input MEQ G10 DETYA THESARWUproportion deacutetudiantes input DETYA des eacutetudiants agrave temps partiel input AOTQU CHEDAAD Republicainscriptions au 1er cycle agrave temps plein input MEQ CHEDAAD inscriptions au 2e cycle input MEQ inscriptions au 3e cycle input MEQ eacutetudiants aux 2e et 3e cycles input G10 DETYA eacutetudiants internationaux input TU THES des eacutetudiants internationaux input Macleans CHEDAAD THES des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input MEQ stagiaires postdoctoraux input MEQ reacutesidents et reacutesidentes en meacutedecine input MEQ

inputrecrutement et admission inputdemandes dadmission aux programmes reacuteguliers input OTQ CHEDAAD offres dadmissiondemandes dadmission i t CHEDAAD offres dadmissiondemandes dadmission input CHEDAAD inscriptionsdemandes dadmission input T CHEDAAD Asiaweek Republica

inputdossier scolaire des nouveaux eacutetudiants (bulletin Cote R) input US NEWS Macleans Asiaweek

provenance des eacutetudiants input Macleanspopulation cible inputreacutegion meacutetropolitaine inputQueacutebec inputCanada inputInternational input

inputdiversiteacute ethnique input US NEWS Guardian

Bourses accordeacutees aux eacutetudiants - Financement moyen input TMEQpremier cycle input Macleans Republicadeuxiegraveme cycle input G10troisiegraveme cycle input G10

cheminement des eacutetudiants (temps complet) perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output QT MEQ US NEWS Macleans Republica diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output G10 MEQ US NEWS Macleans Melbourne Republica dilplomationmaicirctrise apregraves 4 ans output G10 MEQ US NEWS Macleans Republica diplomationdoctorat apregraves 6 ans output G10 MEQ US NEWS Macleans Republica

Deacutepenses de fonctionnement par eacutetudiant input CAUBOG10 US NEWS Macleans Guardian Asia weekFrais de scolariteacute input G10frais affeacuterents inputDeacutepenses pour les Services aux eacutetudiants input Macleans

taux demployabiliteacute apregraves 1 an outcome Ontario CHEDAAD Guardian de retour aux eacutetudes outcome Melbourne

Valeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome Guardian

RP gouvernance - 2008-06-23

2INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

EnseignantsUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Nombre denseignants Nombre total denseignants inputNombre de professeurs reacuteguliers input G10 MEQStatCan USNEWS Melbourne de professeures input OQ MelbourneNombre de chercheurs inputNombre de professeurs avec diplocircme international THESNombre de professeurs de clinique eacutequivalents temps complet input G10StatCanNombre de chargeacutes de cours et autres personnels eacutequivalents temps complet input USNEWSNombre dauxiliaires en eacutequivalence temps complet input

Caracteacuteristiques des professeurs reacuteguliers input des professeurs et chercheurs avec PhD (ou grade terminal) input USNEWS Macleansstructure dage du corps professoral input Asiaweek Melbourne Republicaacircge moyen des professeurs reacuteguliers inputacircge moyen des professeurs adjoints inputacircge moyen des professeurs agreacutegeacutes inputacircge moyen des professeurs titulaires input

reacutemuneacuteration moyenne des professeurs reacuteguliers input Q T U U S NEWS Asiaweek Melbournereacutemuneacuteration moyenne des professeurs reacuteguliers input QTU USNEWS Asiaweek Melbournedes professeurs adjoints inputdes professeurs agreacutegeacutes inputdes professeurs titulaire input

Activiteacutes denseignement inputnombre deacutetudiants EETC Professeur reacutegulier input G10 MEQ USNEWS Guardian Asiaweek Melbourne Republica THESnombre deacutetudiants EETC Professeur reacutegulier en sciences sociales input Melbournenombre deacutetudiants EETC Professeur reacutegulier en sciences input Melbournenombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input

Nombre de cours dispenseacutes par lInstitution (ANNUEL) input Republica cours assumeacutes par les professeurs reacuteguliers input MEQ USNEWS Macleans Republica cours assumeacutes par les chargeacutes de cours eacutequivalents temps complet input MEQTaille des groupe cours input MEQ USNEWS Macleans Guardian

RP gouvernance - 2008-06-23

3INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

PersonnelsUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Le personnel de soutientotal de tous les personnels reacuteguliers acadeacutemiques et non acd agrave temps complet inputnombre des personnels de soutien (staff) input G10 MelbourneDeacutepenses des personnel non-enseignant input MEQnombre des personnels de soutien (staff)- secteur enseignement inputnombre des personnels de soutien (staff) - secteur soutien acadeacutemique inputnombre des personnels de soutien (staff) - secteur administration inputnombre des personnels de soutien (staff) - secteur service aux eacutetudiants inputnombre des personnels de soutien (staff) - entreprises auxiliaires input

rectoratpersonnel dencadrement non acadeacutemique et cadres acadeacutemiques inputratio sur lensemble des personnels acadeacutemiques et non acadeacutemiques inputeacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input G10 Guardianpersonnel de soutien professeur reacutegulier input G10

RP gouvernance - 2008-06-23

4INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Activiteacutes de recherche USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Revenus de recherche de professeurs et chercheurs ayant fait une demande de subvention processus de professeurs et chercheurs ayant obtenu une subvention inputRevenus totaux de recherche (subventions et contrats) input CAUBO G10 The Center Asiaweek Melbournepart de la recherche contractuelle input MEQSubventions publiques input Macleans Melbournerevenus de recherche obtenus des conseils de recherche feacutedeacuteraux input AOQTU The Center Macleans Melbournerevenus moyens de recherche par professeur et chercheur input InfosourceMEQ des deacutepenses de recherchebudget total des deacutepenses input TNombre de chaires et de centre dexcellence output AFondation canadienne pour linnovation (FCI) input G10 -nombre de projets output G10 -valeur $ output G10

nombre de programmes de maicirctrise de recherche input The Center Asiaweeknombre de programmes de doctorat input The Center Asiaweek

Publications et prix outputNombre de publications output Infosource Asiaweek Melbourne

Shanghai Leiden Taiwan

Nombre de publications par professeur et chercheur output Infosource CHEDAAD ShanghaiPublications dans Nature and Science output TaiwanNombre total de citations output InfosourceT Asiaweek MelbourneNombre de citations par professeur et chercheur output InfosourceT Melbourne THESNombre de citations par publication output T CHEDAAD LeidenHighly cited researchers dans 21 secteurs disciplinaires

output

ShanghaiTaiwan Leiden

Impact des publications selon les secteurs disciplinaires output Infosource TaiwanTHES Shanghai

Domaines disciplinaires dexcellence inputTaiwanShanghai THES

Prix et meacutedailles de niveau national et international output G10 The Center Macleans Melbourne Republica ShanghaiNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur output CHEDAADnombre de brevets et de licences output G10 CHEDAADnombre de spin-off companies G10

RP gouvernance - 2008-06-23

5INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

InternationalisationUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

deacutepenses pour le secteur internationalisation input MEQnombre deacutetudiants en stage agrave leacutetrangernombre deacutetudiants temps complet input tes un cannombre deacutetudiants internationauxnombre deacutetudiants temps complet input tes un can Melbourne THESnombre deacutetudiants en cotutellesnombre deacutetudiant des cycles supeacuterieurs input CHEDAADnombre de professeurs inviteacutes internationaux processus tes un can CHEDAADprofesseurs inviteacutes agrave leacutetranger processus CHEDAADnombre deacutetudiants posdoctoraux processus tes un can CHEDAADproportion de cours de langues eacutetrangegraveres processusnombre de programmes deacutechange processus tes un can CHEDAADEntentes de partenariat outputPrix internationaux output G10 The Center Macleans Melbourne Republica Shanghai

RP gouvernance - 2008-06-23

6INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Inteacutegration des technologies de linformation USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Deacutepenses totales consacreacutees aux TIC input MEQpersonnel techniqueenseignant reacutegulier inputpersonnel techniqueeacutetudiant en eacutequivalence temps complet inputExistence dun reacuteseau informatique (accessible agrave linterne et agrave lexterne) outputaccegraves au reacuteseau avec fils et sans fils processusaccegraves aux ressources documentaires des bibliothegraveques outputService de courriel interne et externe en mode WEB outputNombre dordinateurs par professeur et par eacutetudiant outputNombre de cours en ligne processusnombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) outputAccegraves des professeurs aux banque de donneacutees et aux logiciels processus

Site WEB (eacutevaluation du site WEB) WEBOMETRICSSize (nombre de pages recenseacutees) outcomevisibilitty (consultation observeacutee selon le nobre de lien externe unique) outcomeRich files (intensiteacute de lactiviteacute acadeacutemique) outcomeScholar (activiteacute de recherche publications et citations) outcome

RP gouvernance - 2008-06-23

7INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

BibliothegravequesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre total de volumes et de documents audiovisuels et eacutelectroniques input G10 ARLacquisition de volumes (au cours de lanneacutee) input ARL Macleansnombre total de volumes eacutetudiants en eacutequivalence temps complet input G10 ARL Macleans Deacutepenses des bibliothegraveques input MEQ Macleans AsiaweekDeacutepenses des bibliothegravequesbudget de fonctionnement input ARLDeacutepenses des bibliothegravequeseacutetudiants en eacutequivalence temps complet input G10 ARLPersonnels reacuteguliers agrave temps complet input G10 ARLClassement outcome ARL

RP gouvernance - 2008-06-23

8INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Revenus et deacutepenses USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Produits et revenusSubvention du ministegravere de lEacuteducation du Queacutebec input MEQ subvMEQrevenus de fonctionnement input G10 MacleansDroits de scolariteacute input G10 droits de scolariteacute revenus de fonctionnement input G10autres produits input G10Total des revenus de fonctionnement input G10 MEQ Guardian revenus de fonctionnement eacutetudiants en eacutequivalence temps complet input CAUBO USNews MacleansRevenus de recherche input G10encaissement par nature (dons immobilisations chaires bourses) input G10 fonds de dotation input G10 CenterContribution des alumni aux encaissements et fonds de dotation input USNews Macleans fonds de dotation eacutetudiants en eacutequivalence temps complet input G10Total des produits G10

Charges et deacutepenses G10salaire des professeurs input G10 MEQsalaire des chargeacutes de cours input G10salaires des auxiliaires input MEQsalaires des auxiliaires input MEQsalaire des personnels (staff) input G10autres deacutepenses input G10total des deacutepenses input G10 deacutepenses denseignement deacutepenses totales de fonctionnement input UBC MEQ

Chargesproduits processusEacutequilibre budgeacutetaire revenus et deacutepenses de fonctionnement processus MEQ

RP gouvernance - 2008-06-23

9INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

EspacesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Nombre de megravetres carreacutes bruts input G10MCB eacutetudiants en eacutequivalence temps complet input G10megravetres carreacutes nets alloueacutes input MEQNombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input Tbesoin despace en megravetres carreacutes nets inputExistence dun plan directeur des espaces inputBudget preacutevu au plan directeur des espaces input TBudget alloueacute aux espaces input MEQdeacutepenses de fonctionnement nombre de megravetres carreacutes bruts input G10

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Environnement USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

consommation de leacutenergie (exprimeacutee en BTU processus universiteacutes canadiennes et ameacutericaines BTUpopulation totale de luniversiteacute processus voir en particulier Penn State et Umich de leacutenergie produit par des sources renouvelable processusUtilisation de leau processuslitres deau par personne processusGaz agrave effet de serre (en tonnes meacutetriques CO2 eacutequivalents) processusMT CO2 par personne processusDeacutechets solides (en tonnes) processusTonnes de deacutechets solides per capita processus de deacutechets solides recycleacutes processus

RP gouvernance - 2008-06-23

10INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

ReacuteputationUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

enquecircte meneacutee aupregraves des eacutetudiants outcome NSSE Melbourneenquecircte meneacutee aupregraves des diplocircmeacutes outcomeenquecircte meneacutee aupregraves des pairs outcome USNews Asiaweek THESenquecircte meneacutee aupregraves des employeurs outcome Macleans Melbourneenquecircte meneacutee aupregraves des stakeholders outcome Macleans Melbourneenquecircte meneacutee aupregraves du monde des affaires outcome Macleans Melbournecouverture meacutediatique outcome AT

RP gouvernance - 2008-06-23

11INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Planification et gestion strateacutegiquesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Pouvoirs des corps universitaire processus un Canliens dynamiques entre Conseil Assembleacutee universitaire commission des eacutetudes processus un Canprocessus formel de planification strateacutegique processus un Canproduction de donneacutees et indicateurs processus un Canprocessus deacutevaluation des faculteacutes deacutepartements et services processus un Canrevue systeacutematique des responsabiliteacutes et efficaciteacute des comiteacutes processus un Candeacutepenses de ladministration centrale deacutepenses de fonctionnement processus un Can

RP gouvernance - 2008-06-23

12INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Eacuteducation permanenteUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre de programmes offerts input un Cannombre deacutetudiants inscrits input un CanRevenus et deacutepenses du service input un Can

RP gouvernance - 2008-06-23

13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes

REacuteFEacuteRENCES

Aperccedilu densemble des principaux indicateurs

GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost

Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes

MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf

Indicateurs utiliseacutes dans les classements internationaux

Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml

RP gouvernance - 2008-06-23

ANNEXES

Quelques exemples de listes drsquoindicateurs

1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)

Voir page suivante

Preacutepareacute par Roland Proulx

Consultant en planification institutionnelle et veille strateacutegique

Montreacuteal mai 2008

Performance Indicators

Discussion Paper

May 2004 (Prepared by the Office of Planning amp Analysis)

Table of Contents

Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25

2

Table 1 ndash Number of Indictors by Type of Data Source

Table 1 shows that surveys are the least frequently used source of data for indicators

Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a

plurality of indicators come from this source This measure somewhat underestimates the

importance of surveys however as it does not account for the weighting given to each

indicator in each study In the THES World Rankings for instance there is only a single

survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos

Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one

indicator is worth 20 of the final score

Raw

indIcator count

Surve

y data

Third parties

Universities

AsiaweekmdashAsias Best Universities 18 - - 18

Daily Telegraph (2003) 1 - 1 -

Education18com 9 3 4 2

Excelencia 2001 71 - 71 -

Financial Times (2003) 17 - 17 -

Guangdong Institute of Management Science 17 - 14 3

GuardianmdashUniversity Guide 2005 7 - 2 5

La Repubblica 23 2 21 -

Macleans University Rankings 24 1 5 18

Melbourne Institutemdash International Standing of Australian Universities

26 3 23 -

Netbig 2004 18 1 10 7

Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15

Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities

6 - 5 1

The TimesmdashGood University Guide 2005 9 - 9 -

Times Higher Education SupplementmdashWorld University Rankings

5 1 1 3

US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11

Washington MonthlymdashCollege Rankings 2005 8 - 1 7

Wuhan University Centre for Science Evaluation 45 2 22 21

3

A Indicators of Beginning Characteristics

ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of

students at the time they begin their studies Fourteen of the 18 rankings examined in this

study use one or more indicators of the beginning characteristics of students to arrive at

their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on

these factors with 33 of the total ranking coming from this class of indicators but the

Guardian Education18 Asiaweek and the two other North American surveys also place

considerable emphasis on this category

There are six main indicators used to determine which institutions have students with

positive ldquobeginning characteristicsrdquo

The most common measure of beginning characteristics is performance on national

standardized tests with nine surveys using this as a measure Education18 and the

Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used

by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)

Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this

data is collected and standardized by national bodies it has the benefit of being seen as a

relatively impartial method of determining the relative ldquostrengthrdquo of the students entering

each institution Institutionsrsquo results can be scored by showing either averages or the

percentage of entering students meeting a particular standard

Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses

secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This

is a second-best solution made necessary by the absence of any national standardized test

in Canada (or indeed of any provincial standardized tests at the end of secondary school

in provinces other than Alberta) The lack of national standardization makes this an

undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is

truly equivalent to an ldquoArdquo in another jurisdiction

Another measure of the strength of the student body is the percentage of incoming

students receiving (third-party) scholarships which is worth 11 of the score in the

Wuhan survey One can also approach the issue by measuring institutional selectivity In

effect this method infers the strength of the student body by the proportion of

4

applicants rejected the theory being that the higher the number of rejected applicants the

stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)

Normally this measure is expressed as a straight ratio of acceptances to applications but it

can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the

greatest weight on this measure) as a ratio of enrolments to applications Within the US

there is some dispute as to what constitutes an offer of admission and whether or not late

admissions are included as noted by Ganeshananthan (2003)

Student bodies are often considered to be strong if the school is able to attract a large

number of international or out-of-district students or if they contain people from diverse

ethnic backgrounds A number of league tables use the international student indicator

(which like the selectivity indicator is arguably as much an indicator of prestige and

reputation as it is of student characteristics) although in no case does this indicator

account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a

quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data

on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian

puts a somewhat larger emphasis on this indicator in comparison to other league tables

which use similar variables

A very different take on this idea is present in the Washington Monthly which released its

first set of College Rankings in September 2005 With the declared aim of using an

institutionrsquos commitment to social mobility as a measure of quality it uses the percentage

of students from low-income backgrounds as an indicator (with percentage of students

receiving need-based government (Pell) grants used as a proxy)

Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy

statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the

percentage of the student population who are graduate students (arguably this is a

research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the

number of graduate students auditing classes is used as an indicator the assumption is

presumably that if people are auditing then the classes must be very attractive The Italian

La Repubblica ranks an institution according to the number of part-time students it has

contrary to prevailing North American views on the undesirability of part-time study the

Italian rankings see higher numbers of part-time students in a positive light as it is

evidence that an institution is becoming less rigid in its timetabling and

5

permitting students to juggle both work and study something which was nearly

impossible in that country just a few years ago

The Washington Monthly also has a final category of indicators which reflect studentsrsquo

beginning characteristics namely their likelihood of performing community service as

measured by the percentage of students in the US Peace Corps and Reserve Officer

Training Corps (ROTC) and the percentage of students involved in work-study in the

community3 Together these three indicators account for 33 of an institutionrsquos total

ranking

B Indicators of Learning InputsmdashStaff

Generally both the quantity and quality of staff are positively correlated with institutional

quality The problem of course is finding useful metrics for each of these factors

especially if one excludes as we have done here measures of research performance and

research intensity putting them in a separate category4

The simplest measure is simply the number of faculty unadjusted for things like size of

student body Most national league tables however prefer to use variations on the

concept of facultystudent ratio Others try to measure teaching intensity with measures

such as courses per teacher or hours spent in class per student (both in La Repubblica)

These kinds of measures usually account for between 2-5 of the final rankings although

in some cases (ie the Guardian) this figure can be as high as 20

Another important way of measuring how faculty resources are deployed is the measure

of average class size which is used only by Macleanrsquos and the US News and World Report

Ostensibly the reason for measuring class size is to account in some form for the degree

3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category

6

of attention devoted to actually teaching students Despite the fact that important research

(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level

the use of this indicator appears to be a spillover from the North American debates on

class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein

2000) Regardless of why the indicators are used they are extraordinarily important to

these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and

World Reportrsquos rankings respectively

A number of ranking systems try to look at staff qualifications such as the number of

PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the

Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further

than other surveys and actually looks at the proportion of classes taught by tenure-track

staff Others (ie THES) look at the number of foreign faculty based on the assumption

that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still

others (ie La Repubblica) look at the age structure of the faculty Another proxy for

institutional quality is the pay rates for tenured staff on the assumption that institutions

with higher rates of pay on average attract better faculty this measure has been used both

by the US News and World Report and Asiaweek

Finally a number of league tables rank faculty inputs on the basis of standardized third-

party evaluations 6

Education18 the Financial Times the Times the Guardian and the US

News and World Report league tables all use some sort of ranking criterion based at least in

part on this indicator or variations thereof

C Indicators of Learning InputsmdashResources

Resource inputsmdashcrudely the amount of current dollars equipment and books available

to students at an institutionmdashare widely considered an important measure of quality Yet

despite the apparent simplicity of counting dollars and measuring assets

5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each

university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future

7

the means by which institutional wealth is measured varies considerably between ranking

systems

There are a number of revenue-based measures of resources Macleanrsquos uses public funding

of institutional budgets as a factor in its analysis conversely the Financial Times uses the

private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of

total score) and the US News and World Report (5 of total score) also measure alumni

financial support as a measure of quality

For reasons that are not entirely clear league tables tend to favour measures of

expenditures rather than revenues The Guardian looks at total institutional expenditures

as an indicator Institutional expenditure on student services is used as a measure of

institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total

institutional scores respectively) Rzezspospolita does not measure student services

expenditures directly but does measure student services outputs such as number of

student athletes and number of study clubs which amounts to more or less the same thing

Macleanrsquos also gives out 433 of its total score based on institutional expenditures on

scholarships and bursaries

Various aspects of physical infrastructure are also used as measures of institutional

resources most directly in the case of La Repubblica which bases 317 of its total rank on

the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding

assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type

of physical infrastructure measured is available Internet bandwidth which was used by

Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth

roughly 3 of the total score

By some distance the infrastructure indicators most favoured by the compilers of league

tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on

this with 12 of the total quality mark being taken from various types of library

infrastructure measurements (including acquisitions per year total volumes average

number of volumes per student and yearly library expenditure outside of acquisitions)

Netbig and Education18 also use library volume holdings while Asiaweek the Financial

Times and the Times also use measures of library expenditures outside of acquisitions or

computerization of library resources as measures of institutional quality

8

One important factor to note is that most ranking systems do not normalize their resource

and infrastructure measures That is to say it is raw spending power or simple size of

assets that is usually measured rather than spending per studentprofessor or assets per

studentprofessor As a result a number of these rankings systems have inbuilt biases

towards larger institutions

D Indicators of Learning Outputs

Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge

learned over the course of a baccalaureate degreemdashshould be a basic indicator of

institutional quality Unfortunately good means of measuring these outputsmdashlike the

National Survey of Student Engagement (NSSE) and the College Learning Assessment

(CLA)mdashhave only recently become available and for the most part institutions are still

keeping their scores secret Outside of these measures only a few very crude indicators are

available which likely explains why learning outputs do not feature especially prominently

in most ranking schemes

The simplest types of measures of learning outputs are those linked to graduation and

retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong

and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7

the latter three also use rates of graduation from Masterrsquos programs as indicators In some

cases the weights on these measures can be very highmdashin the Guangdong rankings

graduation rates account for over 50 of the rankingmdashbut in most cases the weights are

10 or less Retention rates commonly meaning the progression rate of first-year students

into second year are accorded less importance The US News and World Report Macleanrsquos

the Melbourne Institute and La Repubblica all employ retention measures as indicators but

none of them are worth more than 4 of total weighting Two publications make specific

indicators for retention and graduation of international students Macleanrsquos (graduation

rates of international students) and the Melbourne Institute (retention rates of international

students) The Washington Monthly looks specifically at institutional retention rates

adjusted for the participation of lower-income students and gives higher scores to

institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores

and number of Pell Grant recipients the US

7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18

9

News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos

ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)

E Indicators of Final Outcomes

Final outcomes are indications of generalized outcomes for students after graduation

Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness

good citizenship) but given the somewhat utilitarian justifications for education that are

currently in fashion (see Wolf 2000) employment outcomes are the most commonly used

measure of final outcomes These are given particular emphasis by the Guardian (where

employment outcomes are worth 17 of the total score) but are also used by the Financial

Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the

Times are interestingly not concerned with employment per se but with ldquoemployment in

an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher

Education Statistics Agency (HESA) uses Standard Occupational Classifications to

measure the proportion of graduates in professional or white-collar jobs anyone not in

such a job is considered not to be working in an area related to their studies (it is unclear

what methodology is used by the Financial Times and the Times although we suspect their

methods are broadly similar)

The only other measure of final outcomes in use is percentage of graduates returning for

additional education which is an indicator used by both the Melbourne Institute and the

Financial Times This is a particularly important indicator for the latter as it is worth 21

of the final ranking

The lack of indicators concerning final outcomes is interesting since most government-

sponsored performance-indicator regimes around the world are very much concerned

with such measures especially with respect to employment Possibly this indicates that

ranking systems simply do not view education outcomes as relevant measures of

educational quality Alternatively it may be the case that they simply have not found a

reliable indicator of outcomes or that there are reliable indicators but that there is so little

variation between institutions that it makes no sense to rank based on the data

10

F Indicators of Research

Many of the league tables covered in this survey include sections and weightings related

to universitiesrsquo research efforts It is in this field of measurement that we see the greatest

diversity of indicators Presumably this is because research inputs and outputs lend

themselves much more easily to measurement and manipulation than other areas of

institutional activity

Three studies include research staff as part of their ranking scheme La Repubblica at 952

the Melbourne Institute (4) and Wuhan (078)

Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used

method of looking at research quality but it is not universally admired because different

disciplines use different means to communicate major advances in knowledge (leading

scientists invariably produce large numbers of journal articles leading social scientists

may produce fewer journal articles but instead have one or two long important

monographsmdashsee Hicks 2004) There is also some concern among non-English speaking

countries that they are penalized in international rankings because so many of the major

journals (notably Science and Nature) are printed in English However the one set of

rankings that uses separate indicators to monitor articles published in English and articles

published in another language (the Wuhan rankings) shows that the two indicators are

positively correlated institutions that have more Chinese publications are also likely to

have more English publications and vice versa

Several sets of league tables measure bibliometric citations in various publication indices

The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a

weight of 20 of the final total Guangdong also monitors other Chinese universities

specifically for citations in engineering publications and weights this at 29 Moreover it

tacks on an additional 1049 for citations in science-oriented indices such as the Science

Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the

same category of scientific citations while the Melbourne Institute rates science citations at

68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social

Science Citation Index which does not include the humanities) are noted in only two

league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne

Institute (32) Another way of measuring research

11

impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a

weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the

Melbourne Institute

The complement of citations is of course publications Listing the number of publications

an individual group department or whole university releases can act as a weak substitute

for citationsmdashweak because simply publishing a paper or monograph is no guarantee that

the general public or other researchers will even glance at the work Guangdong gives an

1179 weighting to publications in science-oriented indices such as the Science Citation

Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same

indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong

even has a separate category just for measuring publications in Science and Nature

although it accords it an almost derisory weighting of 06 Under publications in social

science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their

final totals For publications in other indices (where the subject indices are

undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for

Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of

rankings to include research monographs weighted at 033

In countries where there are specific third-party evaluations of research output academic

quality of research is sometimes used as a research indicator The Times puts a very large

30 weight on this indicator while the Financial Times puts it at 119

Research awards are another handy third-party measurement of quality as the number of

international and national awards won by faculty andor graduates is often considered a

useful measure of institutional success International research awardsmdash specifically the

number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator

by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as

particularly suspect in some quarters given that the points are based on where the

recipient went to school rather than on where they are or were on the

8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the

most prominent publication indices

9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences

12

faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead

for nearly a century Wuhan uses similar measures but only accords them a weight of 14

National research awards are more common as a quality indicator used by La Repubblica

(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)

Financial indicators of research are also very common Research budgets as a factor in the

overall assessment of research in universities are covered by the Financial Times (9)

Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for

total amount of research expenditure unfortunately it is unclear precisely what this

research expenditure represents or how it is determined although it is clearly indicated

that it does not represent the total number of grants or projects at a university Total

number of research-based grants and projects is weighted by Education18 at 15 and

Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science

and engineering and another 55 to those for social sciences and humanities Similarly

the Melbourne Institute gives 6 of its overall weight to public-source grants making no

distinction between areas of study

In a slightly different vein Netbig (46) and Wuhan (278) both list the number of

research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica

(095) all weigh research-basedaffiliated research institutions or centres for studies

Finally one can also measure research not simply in terms of the amount of money it

generates but also in terms of the amount of future income it will generate Both

Guangdong (245) and Wuhan (193) measure the number of patents issued to

universities as a quality indicator

A final way of measuring an institutionrsquos research intensity is to look at the range of its

course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of

doctoral and Masterrsquos programs offered as a proxy for research intensity

As with physical and financial resources few if any of the research indicators are

normalized to account for institutional size (either by student or faculty numbers) In the

world of rankings bigger almost always means better an institution with 100 faculty with

ten citations apiece will always look worse than an institution with 1001 faculty

13

with one citation each To the extent that the raw production of knowledge matters this

form of measurement is acceptable To the extent that rankings are meant to show how

well institutions are doing on a like-to-like basis or to show the efficiency of universities it

is plainly inadequate This should be of particular concern to Chinese policy-makers

whose ranking systems are especially reliant on research-based indicators

G Indicators of Reputation

The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo

Those rankings systems which use the results of reputation surveys as an indicator do so as

an indirect measure of quality based on the assumption that the employers academics and

academic administrators surveyed have opinions of institutional quality that are informed

up-to-date and impartial While these assumptions are clearly open to debate they

nevertheless form an important basis for many ranking systems Another reason for using

reputation measures is the paucity of other data availablemdashsome countries have few

independent measures of teaching effectiveness university resources or output and

reputation can thus act as a useful surrogate Reputation rankings are often criticized as

simply quantifying the common ignorance of the people being surveyed However to the

extent that the people being surveyed hold positions which have the potential to affect

large numbers of young people and whose positions actually require some knowledge of

institutional quality (ie officials in charge of graduate admissions corporate recruiters

etc) then reputation rankings make sense because they provide useful information for

students about the perceived value of the degrees that they could obtain from various

universities

The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and

the Times which both accord reputation a weighting of 50 in their overall ranking

scheme Education18 assigns it almost as much significance at 40 The US News and World

Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just

below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)

and Netbig (15) The only other study to include reputation is Wuhan (117)

14

1 Please see Appendix B for a glossary of the various bodies referenced in this document

Appendix A Detailed Listing of Indicators and their Sources

Beginning Characteristics Indicator

Used By Source1

Incoming grades Macleans University

Percentage with grades above a set limit

Macleans University

US News and World Report

University

Performance on national standardised tests or benchmarks

Asiaweek University

Education18 3rd-party JUPAS

Financial Times Government agency 3rd-party UCAS

Guardian University Guide

Government agency 3rd-party UCAS

Melbourne Institute Government agency 3rd-party DEST

Netbig National entrance examination board

Times Good University Guide

Government agency 3rd-party UCAS

US News University

Wuhan Unknown presumed government 3rd-party

Student status La Repubblica Government agency 3rd-party MIUR

Admittance selectivity general

Asiaweek University

Admittance number of applications to places

Asiaweek University

Financial Times Government agency 3rd-party UCAS

La Repubblica Government agency 3rd-party MIUR

US News University

15

Beginning Characteristics Indicator

Used By Source1

Out-of-locality student percentage

Macleans University

International student percentages

Financial Times Government agency 3rd-party HESA

Macleans University

Shanghai Institute of Educational Science

Times World University

Wuhan Unknown presumed university

Undergraduate students among all students percentages

Netbig

Wuhan Unknown presumed university

Ethnic diversity in student body

Guardian University

Learning Inputs ndash Staff Indicator

Used By Source

Facultystudent ratio Asiaweek University

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Financial Times Government agency 3rd-party HESA

La Repubblica Government agency 3rd-party MIUR

Times Good University Guide

Government agency 3rd-party HESA

Times World University

US News University

Wuhan Unknown presumed university

16

Learning Inputs ndash Staff Indicator

Used By Source

Social science faculty student ratio

Melbourne Government agency 3rd-party DEST

Science faculty student ratio

Melbourne Government agency 3rd-party DEST

Administrative staff student ratio

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Staff student ratio (regardless of division)

Guardian Government agency 3rd-party HESA

Netbig University

Course per teacher La Repubblica Government agency 3rd-party MIUR

Per-teacher university spending

Asiaweek University

Faculty pay rates fortenured staff

Asiaweek University

US News University

Number of full-time part-time faculty

Netbig University

US News University

Wuhan Unknown presumed university

Faculty with researchprojects

Wuhan Unknown presumed university

Class size differentiation Macleans University

US News University

Classes taught by tenured faculty

Macleans University

Exchange programmes hosted

La Repubblica Government agency 3rd-party AgNaSoc

Number of classes lsquoactually taughtrsquo

La Repubblica Government agency 3rd-party MIUR

of international faculty (v faculty as a whole)

Times World University

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

2INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

EnseignantsUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Nombre denseignants Nombre total denseignants inputNombre de professeurs reacuteguliers input G10 MEQStatCan USNEWS Melbourne de professeures input OQ MelbourneNombre de chercheurs inputNombre de professeurs avec diplocircme international THESNombre de professeurs de clinique eacutequivalents temps complet input G10StatCanNombre de chargeacutes de cours et autres personnels eacutequivalents temps complet input USNEWSNombre dauxiliaires en eacutequivalence temps complet input

Caracteacuteristiques des professeurs reacuteguliers input des professeurs et chercheurs avec PhD (ou grade terminal) input USNEWS Macleansstructure dage du corps professoral input Asiaweek Melbourne Republicaacircge moyen des professeurs reacuteguliers inputacircge moyen des professeurs adjoints inputacircge moyen des professeurs agreacutegeacutes inputacircge moyen des professeurs titulaires input

reacutemuneacuteration moyenne des professeurs reacuteguliers input Q T U U S NEWS Asiaweek Melbournereacutemuneacuteration moyenne des professeurs reacuteguliers input QTU USNEWS Asiaweek Melbournedes professeurs adjoints inputdes professeurs agreacutegeacutes inputdes professeurs titulaire input

Activiteacutes denseignement inputnombre deacutetudiants EETC Professeur reacutegulier input G10 MEQ USNEWS Guardian Asiaweek Melbourne Republica THESnombre deacutetudiants EETC Professeur reacutegulier en sciences sociales input Melbournenombre deacutetudiants EETC Professeur reacutegulier en sciences input Melbournenombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input

Nombre de cours dispenseacutes par lInstitution (ANNUEL) input Republica cours assumeacutes par les professeurs reacuteguliers input MEQ USNEWS Macleans Republica cours assumeacutes par les chargeacutes de cours eacutequivalents temps complet input MEQTaille des groupe cours input MEQ USNEWS Macleans Guardian

RP gouvernance - 2008-06-23

3INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

PersonnelsUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Le personnel de soutientotal de tous les personnels reacuteguliers acadeacutemiques et non acd agrave temps complet inputnombre des personnels de soutien (staff) input G10 MelbourneDeacutepenses des personnel non-enseignant input MEQnombre des personnels de soutien (staff)- secteur enseignement inputnombre des personnels de soutien (staff) - secteur soutien acadeacutemique inputnombre des personnels de soutien (staff) - secteur administration inputnombre des personnels de soutien (staff) - secteur service aux eacutetudiants inputnombre des personnels de soutien (staff) - entreprises auxiliaires input

rectoratpersonnel dencadrement non acadeacutemique et cadres acadeacutemiques inputratio sur lensemble des personnels acadeacutemiques et non acadeacutemiques inputeacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input G10 Guardianpersonnel de soutien professeur reacutegulier input G10

RP gouvernance - 2008-06-23

4INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Activiteacutes de recherche USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Revenus de recherche de professeurs et chercheurs ayant fait une demande de subvention processus de professeurs et chercheurs ayant obtenu une subvention inputRevenus totaux de recherche (subventions et contrats) input CAUBO G10 The Center Asiaweek Melbournepart de la recherche contractuelle input MEQSubventions publiques input Macleans Melbournerevenus de recherche obtenus des conseils de recherche feacutedeacuteraux input AOQTU The Center Macleans Melbournerevenus moyens de recherche par professeur et chercheur input InfosourceMEQ des deacutepenses de recherchebudget total des deacutepenses input TNombre de chaires et de centre dexcellence output AFondation canadienne pour linnovation (FCI) input G10 -nombre de projets output G10 -valeur $ output G10

nombre de programmes de maicirctrise de recherche input The Center Asiaweeknombre de programmes de doctorat input The Center Asiaweek

Publications et prix outputNombre de publications output Infosource Asiaweek Melbourne

Shanghai Leiden Taiwan

Nombre de publications par professeur et chercheur output Infosource CHEDAAD ShanghaiPublications dans Nature and Science output TaiwanNombre total de citations output InfosourceT Asiaweek MelbourneNombre de citations par professeur et chercheur output InfosourceT Melbourne THESNombre de citations par publication output T CHEDAAD LeidenHighly cited researchers dans 21 secteurs disciplinaires

output

ShanghaiTaiwan Leiden

Impact des publications selon les secteurs disciplinaires output Infosource TaiwanTHES Shanghai

Domaines disciplinaires dexcellence inputTaiwanShanghai THES

Prix et meacutedailles de niveau national et international output G10 The Center Macleans Melbourne Republica ShanghaiNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur output CHEDAADnombre de brevets et de licences output G10 CHEDAADnombre de spin-off companies G10

RP gouvernance - 2008-06-23

5INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

InternationalisationUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

deacutepenses pour le secteur internationalisation input MEQnombre deacutetudiants en stage agrave leacutetrangernombre deacutetudiants temps complet input tes un cannombre deacutetudiants internationauxnombre deacutetudiants temps complet input tes un can Melbourne THESnombre deacutetudiants en cotutellesnombre deacutetudiant des cycles supeacuterieurs input CHEDAADnombre de professeurs inviteacutes internationaux processus tes un can CHEDAADprofesseurs inviteacutes agrave leacutetranger processus CHEDAADnombre deacutetudiants posdoctoraux processus tes un can CHEDAADproportion de cours de langues eacutetrangegraveres processusnombre de programmes deacutechange processus tes un can CHEDAADEntentes de partenariat outputPrix internationaux output G10 The Center Macleans Melbourne Republica Shanghai

RP gouvernance - 2008-06-23

6INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Inteacutegration des technologies de linformation USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Deacutepenses totales consacreacutees aux TIC input MEQpersonnel techniqueenseignant reacutegulier inputpersonnel techniqueeacutetudiant en eacutequivalence temps complet inputExistence dun reacuteseau informatique (accessible agrave linterne et agrave lexterne) outputaccegraves au reacuteseau avec fils et sans fils processusaccegraves aux ressources documentaires des bibliothegraveques outputService de courriel interne et externe en mode WEB outputNombre dordinateurs par professeur et par eacutetudiant outputNombre de cours en ligne processusnombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) outputAccegraves des professeurs aux banque de donneacutees et aux logiciels processus

Site WEB (eacutevaluation du site WEB) WEBOMETRICSSize (nombre de pages recenseacutees) outcomevisibilitty (consultation observeacutee selon le nobre de lien externe unique) outcomeRich files (intensiteacute de lactiviteacute acadeacutemique) outcomeScholar (activiteacute de recherche publications et citations) outcome

RP gouvernance - 2008-06-23

7INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

BibliothegravequesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre total de volumes et de documents audiovisuels et eacutelectroniques input G10 ARLacquisition de volumes (au cours de lanneacutee) input ARL Macleansnombre total de volumes eacutetudiants en eacutequivalence temps complet input G10 ARL Macleans Deacutepenses des bibliothegraveques input MEQ Macleans AsiaweekDeacutepenses des bibliothegravequesbudget de fonctionnement input ARLDeacutepenses des bibliothegravequeseacutetudiants en eacutequivalence temps complet input G10 ARLPersonnels reacuteguliers agrave temps complet input G10 ARLClassement outcome ARL

RP gouvernance - 2008-06-23

8INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Revenus et deacutepenses USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Produits et revenusSubvention du ministegravere de lEacuteducation du Queacutebec input MEQ subvMEQrevenus de fonctionnement input G10 MacleansDroits de scolariteacute input G10 droits de scolariteacute revenus de fonctionnement input G10autres produits input G10Total des revenus de fonctionnement input G10 MEQ Guardian revenus de fonctionnement eacutetudiants en eacutequivalence temps complet input CAUBO USNews MacleansRevenus de recherche input G10encaissement par nature (dons immobilisations chaires bourses) input G10 fonds de dotation input G10 CenterContribution des alumni aux encaissements et fonds de dotation input USNews Macleans fonds de dotation eacutetudiants en eacutequivalence temps complet input G10Total des produits G10

Charges et deacutepenses G10salaire des professeurs input G10 MEQsalaire des chargeacutes de cours input G10salaires des auxiliaires input MEQsalaires des auxiliaires input MEQsalaire des personnels (staff) input G10autres deacutepenses input G10total des deacutepenses input G10 deacutepenses denseignement deacutepenses totales de fonctionnement input UBC MEQ

Chargesproduits processusEacutequilibre budgeacutetaire revenus et deacutepenses de fonctionnement processus MEQ

RP gouvernance - 2008-06-23

9INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

EspacesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Nombre de megravetres carreacutes bruts input G10MCB eacutetudiants en eacutequivalence temps complet input G10megravetres carreacutes nets alloueacutes input MEQNombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input Tbesoin despace en megravetres carreacutes nets inputExistence dun plan directeur des espaces inputBudget preacutevu au plan directeur des espaces input TBudget alloueacute aux espaces input MEQdeacutepenses de fonctionnement nombre de megravetres carreacutes bruts input G10

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Environnement USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

consommation de leacutenergie (exprimeacutee en BTU processus universiteacutes canadiennes et ameacutericaines BTUpopulation totale de luniversiteacute processus voir en particulier Penn State et Umich de leacutenergie produit par des sources renouvelable processusUtilisation de leau processuslitres deau par personne processusGaz agrave effet de serre (en tonnes meacutetriques CO2 eacutequivalents) processusMT CO2 par personne processusDeacutechets solides (en tonnes) processusTonnes de deacutechets solides per capita processus de deacutechets solides recycleacutes processus

RP gouvernance - 2008-06-23

10INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

ReacuteputationUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

enquecircte meneacutee aupregraves des eacutetudiants outcome NSSE Melbourneenquecircte meneacutee aupregraves des diplocircmeacutes outcomeenquecircte meneacutee aupregraves des pairs outcome USNews Asiaweek THESenquecircte meneacutee aupregraves des employeurs outcome Macleans Melbourneenquecircte meneacutee aupregraves des stakeholders outcome Macleans Melbourneenquecircte meneacutee aupregraves du monde des affaires outcome Macleans Melbournecouverture meacutediatique outcome AT

RP gouvernance - 2008-06-23

11INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Planification et gestion strateacutegiquesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Pouvoirs des corps universitaire processus un Canliens dynamiques entre Conseil Assembleacutee universitaire commission des eacutetudes processus un Canprocessus formel de planification strateacutegique processus un Canproduction de donneacutees et indicateurs processus un Canprocessus deacutevaluation des faculteacutes deacutepartements et services processus un Canrevue systeacutematique des responsabiliteacutes et efficaciteacute des comiteacutes processus un Candeacutepenses de ladministration centrale deacutepenses de fonctionnement processus un Can

RP gouvernance - 2008-06-23

12INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Eacuteducation permanenteUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre de programmes offerts input un Cannombre deacutetudiants inscrits input un CanRevenus et deacutepenses du service input un Can

RP gouvernance - 2008-06-23

13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes

REacuteFEacuteRENCES

Aperccedilu densemble des principaux indicateurs

GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost

Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes

MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf

Indicateurs utiliseacutes dans les classements internationaux

Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml

RP gouvernance - 2008-06-23

ANNEXES

Quelques exemples de listes drsquoindicateurs

1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)

Voir page suivante

Preacutepareacute par Roland Proulx

Consultant en planification institutionnelle et veille strateacutegique

Montreacuteal mai 2008

Performance Indicators

Discussion Paper

May 2004 (Prepared by the Office of Planning amp Analysis)

Table of Contents

Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25

2

Table 1 ndash Number of Indictors by Type of Data Source

Table 1 shows that surveys are the least frequently used source of data for indicators

Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a

plurality of indicators come from this source This measure somewhat underestimates the

importance of surveys however as it does not account for the weighting given to each

indicator in each study In the THES World Rankings for instance there is only a single

survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos

Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one

indicator is worth 20 of the final score

Raw

indIcator count

Surve

y data

Third parties

Universities

AsiaweekmdashAsias Best Universities 18 - - 18

Daily Telegraph (2003) 1 - 1 -

Education18com 9 3 4 2

Excelencia 2001 71 - 71 -

Financial Times (2003) 17 - 17 -

Guangdong Institute of Management Science 17 - 14 3

GuardianmdashUniversity Guide 2005 7 - 2 5

La Repubblica 23 2 21 -

Macleans University Rankings 24 1 5 18

Melbourne Institutemdash International Standing of Australian Universities

26 3 23 -

Netbig 2004 18 1 10 7

Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15

Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities

6 - 5 1

The TimesmdashGood University Guide 2005 9 - 9 -

Times Higher Education SupplementmdashWorld University Rankings

5 1 1 3

US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11

Washington MonthlymdashCollege Rankings 2005 8 - 1 7

Wuhan University Centre for Science Evaluation 45 2 22 21

3

A Indicators of Beginning Characteristics

ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of

students at the time they begin their studies Fourteen of the 18 rankings examined in this

study use one or more indicators of the beginning characteristics of students to arrive at

their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on

these factors with 33 of the total ranking coming from this class of indicators but the

Guardian Education18 Asiaweek and the two other North American surveys also place

considerable emphasis on this category

There are six main indicators used to determine which institutions have students with

positive ldquobeginning characteristicsrdquo

The most common measure of beginning characteristics is performance on national

standardized tests with nine surveys using this as a measure Education18 and the

Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used

by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)

Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this

data is collected and standardized by national bodies it has the benefit of being seen as a

relatively impartial method of determining the relative ldquostrengthrdquo of the students entering

each institution Institutionsrsquo results can be scored by showing either averages or the

percentage of entering students meeting a particular standard

Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses

secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This

is a second-best solution made necessary by the absence of any national standardized test

in Canada (or indeed of any provincial standardized tests at the end of secondary school

in provinces other than Alberta) The lack of national standardization makes this an

undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is

truly equivalent to an ldquoArdquo in another jurisdiction

Another measure of the strength of the student body is the percentage of incoming

students receiving (third-party) scholarships which is worth 11 of the score in the

Wuhan survey One can also approach the issue by measuring institutional selectivity In

effect this method infers the strength of the student body by the proportion of

4

applicants rejected the theory being that the higher the number of rejected applicants the

stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)

Normally this measure is expressed as a straight ratio of acceptances to applications but it

can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the

greatest weight on this measure) as a ratio of enrolments to applications Within the US

there is some dispute as to what constitutes an offer of admission and whether or not late

admissions are included as noted by Ganeshananthan (2003)

Student bodies are often considered to be strong if the school is able to attract a large

number of international or out-of-district students or if they contain people from diverse

ethnic backgrounds A number of league tables use the international student indicator

(which like the selectivity indicator is arguably as much an indicator of prestige and

reputation as it is of student characteristics) although in no case does this indicator

account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a

quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data

on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian

puts a somewhat larger emphasis on this indicator in comparison to other league tables

which use similar variables

A very different take on this idea is present in the Washington Monthly which released its

first set of College Rankings in September 2005 With the declared aim of using an

institutionrsquos commitment to social mobility as a measure of quality it uses the percentage

of students from low-income backgrounds as an indicator (with percentage of students

receiving need-based government (Pell) grants used as a proxy)

Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy

statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the

percentage of the student population who are graduate students (arguably this is a

research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the

number of graduate students auditing classes is used as an indicator the assumption is

presumably that if people are auditing then the classes must be very attractive The Italian

La Repubblica ranks an institution according to the number of part-time students it has

contrary to prevailing North American views on the undesirability of part-time study the

Italian rankings see higher numbers of part-time students in a positive light as it is

evidence that an institution is becoming less rigid in its timetabling and

5

permitting students to juggle both work and study something which was nearly

impossible in that country just a few years ago

The Washington Monthly also has a final category of indicators which reflect studentsrsquo

beginning characteristics namely their likelihood of performing community service as

measured by the percentage of students in the US Peace Corps and Reserve Officer

Training Corps (ROTC) and the percentage of students involved in work-study in the

community3 Together these three indicators account for 33 of an institutionrsquos total

ranking

B Indicators of Learning InputsmdashStaff

Generally both the quantity and quality of staff are positively correlated with institutional

quality The problem of course is finding useful metrics for each of these factors

especially if one excludes as we have done here measures of research performance and

research intensity putting them in a separate category4

The simplest measure is simply the number of faculty unadjusted for things like size of

student body Most national league tables however prefer to use variations on the

concept of facultystudent ratio Others try to measure teaching intensity with measures

such as courses per teacher or hours spent in class per student (both in La Repubblica)

These kinds of measures usually account for between 2-5 of the final rankings although

in some cases (ie the Guardian) this figure can be as high as 20

Another important way of measuring how faculty resources are deployed is the measure

of average class size which is used only by Macleanrsquos and the US News and World Report

Ostensibly the reason for measuring class size is to account in some form for the degree

3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category

6

of attention devoted to actually teaching students Despite the fact that important research

(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level

the use of this indicator appears to be a spillover from the North American debates on

class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein

2000) Regardless of why the indicators are used they are extraordinarily important to

these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and

World Reportrsquos rankings respectively

A number of ranking systems try to look at staff qualifications such as the number of

PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the

Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further

than other surveys and actually looks at the proportion of classes taught by tenure-track

staff Others (ie THES) look at the number of foreign faculty based on the assumption

that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still

others (ie La Repubblica) look at the age structure of the faculty Another proxy for

institutional quality is the pay rates for tenured staff on the assumption that institutions

with higher rates of pay on average attract better faculty this measure has been used both

by the US News and World Report and Asiaweek

Finally a number of league tables rank faculty inputs on the basis of standardized third-

party evaluations 6

Education18 the Financial Times the Times the Guardian and the US

News and World Report league tables all use some sort of ranking criterion based at least in

part on this indicator or variations thereof

C Indicators of Learning InputsmdashResources

Resource inputsmdashcrudely the amount of current dollars equipment and books available

to students at an institutionmdashare widely considered an important measure of quality Yet

despite the apparent simplicity of counting dollars and measuring assets

5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each

university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future

7

the means by which institutional wealth is measured varies considerably between ranking

systems

There are a number of revenue-based measures of resources Macleanrsquos uses public funding

of institutional budgets as a factor in its analysis conversely the Financial Times uses the

private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of

total score) and the US News and World Report (5 of total score) also measure alumni

financial support as a measure of quality

For reasons that are not entirely clear league tables tend to favour measures of

expenditures rather than revenues The Guardian looks at total institutional expenditures

as an indicator Institutional expenditure on student services is used as a measure of

institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total

institutional scores respectively) Rzezspospolita does not measure student services

expenditures directly but does measure student services outputs such as number of

student athletes and number of study clubs which amounts to more or less the same thing

Macleanrsquos also gives out 433 of its total score based on institutional expenditures on

scholarships and bursaries

Various aspects of physical infrastructure are also used as measures of institutional

resources most directly in the case of La Repubblica which bases 317 of its total rank on

the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding

assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type

of physical infrastructure measured is available Internet bandwidth which was used by

Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth

roughly 3 of the total score

By some distance the infrastructure indicators most favoured by the compilers of league

tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on

this with 12 of the total quality mark being taken from various types of library

infrastructure measurements (including acquisitions per year total volumes average

number of volumes per student and yearly library expenditure outside of acquisitions)

Netbig and Education18 also use library volume holdings while Asiaweek the Financial

Times and the Times also use measures of library expenditures outside of acquisitions or

computerization of library resources as measures of institutional quality

8

One important factor to note is that most ranking systems do not normalize their resource

and infrastructure measures That is to say it is raw spending power or simple size of

assets that is usually measured rather than spending per studentprofessor or assets per

studentprofessor As a result a number of these rankings systems have inbuilt biases

towards larger institutions

D Indicators of Learning Outputs

Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge

learned over the course of a baccalaureate degreemdashshould be a basic indicator of

institutional quality Unfortunately good means of measuring these outputsmdashlike the

National Survey of Student Engagement (NSSE) and the College Learning Assessment

(CLA)mdashhave only recently become available and for the most part institutions are still

keeping their scores secret Outside of these measures only a few very crude indicators are

available which likely explains why learning outputs do not feature especially prominently

in most ranking schemes

The simplest types of measures of learning outputs are those linked to graduation and

retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong

and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7

the latter three also use rates of graduation from Masterrsquos programs as indicators In some

cases the weights on these measures can be very highmdashin the Guangdong rankings

graduation rates account for over 50 of the rankingmdashbut in most cases the weights are

10 or less Retention rates commonly meaning the progression rate of first-year students

into second year are accorded less importance The US News and World Report Macleanrsquos

the Melbourne Institute and La Repubblica all employ retention measures as indicators but

none of them are worth more than 4 of total weighting Two publications make specific

indicators for retention and graduation of international students Macleanrsquos (graduation

rates of international students) and the Melbourne Institute (retention rates of international

students) The Washington Monthly looks specifically at institutional retention rates

adjusted for the participation of lower-income students and gives higher scores to

institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores

and number of Pell Grant recipients the US

7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18

9

News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos

ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)

E Indicators of Final Outcomes

Final outcomes are indications of generalized outcomes for students after graduation

Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness

good citizenship) but given the somewhat utilitarian justifications for education that are

currently in fashion (see Wolf 2000) employment outcomes are the most commonly used

measure of final outcomes These are given particular emphasis by the Guardian (where

employment outcomes are worth 17 of the total score) but are also used by the Financial

Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the

Times are interestingly not concerned with employment per se but with ldquoemployment in

an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher

Education Statistics Agency (HESA) uses Standard Occupational Classifications to

measure the proportion of graduates in professional or white-collar jobs anyone not in

such a job is considered not to be working in an area related to their studies (it is unclear

what methodology is used by the Financial Times and the Times although we suspect their

methods are broadly similar)

The only other measure of final outcomes in use is percentage of graduates returning for

additional education which is an indicator used by both the Melbourne Institute and the

Financial Times This is a particularly important indicator for the latter as it is worth 21

of the final ranking

The lack of indicators concerning final outcomes is interesting since most government-

sponsored performance-indicator regimes around the world are very much concerned

with such measures especially with respect to employment Possibly this indicates that

ranking systems simply do not view education outcomes as relevant measures of

educational quality Alternatively it may be the case that they simply have not found a

reliable indicator of outcomes or that there are reliable indicators but that there is so little

variation between institutions that it makes no sense to rank based on the data

10

F Indicators of Research

Many of the league tables covered in this survey include sections and weightings related

to universitiesrsquo research efforts It is in this field of measurement that we see the greatest

diversity of indicators Presumably this is because research inputs and outputs lend

themselves much more easily to measurement and manipulation than other areas of

institutional activity

Three studies include research staff as part of their ranking scheme La Repubblica at 952

the Melbourne Institute (4) and Wuhan (078)

Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used

method of looking at research quality but it is not universally admired because different

disciplines use different means to communicate major advances in knowledge (leading

scientists invariably produce large numbers of journal articles leading social scientists

may produce fewer journal articles but instead have one or two long important

monographsmdashsee Hicks 2004) There is also some concern among non-English speaking

countries that they are penalized in international rankings because so many of the major

journals (notably Science and Nature) are printed in English However the one set of

rankings that uses separate indicators to monitor articles published in English and articles

published in another language (the Wuhan rankings) shows that the two indicators are

positively correlated institutions that have more Chinese publications are also likely to

have more English publications and vice versa

Several sets of league tables measure bibliometric citations in various publication indices

The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a

weight of 20 of the final total Guangdong also monitors other Chinese universities

specifically for citations in engineering publications and weights this at 29 Moreover it

tacks on an additional 1049 for citations in science-oriented indices such as the Science

Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the

same category of scientific citations while the Melbourne Institute rates science citations at

68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social

Science Citation Index which does not include the humanities) are noted in only two

league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne

Institute (32) Another way of measuring research

11

impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a

weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the

Melbourne Institute

The complement of citations is of course publications Listing the number of publications

an individual group department or whole university releases can act as a weak substitute

for citationsmdashweak because simply publishing a paper or monograph is no guarantee that

the general public or other researchers will even glance at the work Guangdong gives an

1179 weighting to publications in science-oriented indices such as the Science Citation

Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same

indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong

even has a separate category just for measuring publications in Science and Nature

although it accords it an almost derisory weighting of 06 Under publications in social

science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their

final totals For publications in other indices (where the subject indices are

undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for

Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of

rankings to include research monographs weighted at 033

In countries where there are specific third-party evaluations of research output academic

quality of research is sometimes used as a research indicator The Times puts a very large

30 weight on this indicator while the Financial Times puts it at 119

Research awards are another handy third-party measurement of quality as the number of

international and national awards won by faculty andor graduates is often considered a

useful measure of institutional success International research awardsmdash specifically the

number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator

by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as

particularly suspect in some quarters given that the points are based on where the

recipient went to school rather than on where they are or were on the

8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the

most prominent publication indices

9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences

12

faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead

for nearly a century Wuhan uses similar measures but only accords them a weight of 14

National research awards are more common as a quality indicator used by La Repubblica

(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)

Financial indicators of research are also very common Research budgets as a factor in the

overall assessment of research in universities are covered by the Financial Times (9)

Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for

total amount of research expenditure unfortunately it is unclear precisely what this

research expenditure represents or how it is determined although it is clearly indicated

that it does not represent the total number of grants or projects at a university Total

number of research-based grants and projects is weighted by Education18 at 15 and

Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science

and engineering and another 55 to those for social sciences and humanities Similarly

the Melbourne Institute gives 6 of its overall weight to public-source grants making no

distinction between areas of study

In a slightly different vein Netbig (46) and Wuhan (278) both list the number of

research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica

(095) all weigh research-basedaffiliated research institutions or centres for studies

Finally one can also measure research not simply in terms of the amount of money it

generates but also in terms of the amount of future income it will generate Both

Guangdong (245) and Wuhan (193) measure the number of patents issued to

universities as a quality indicator

A final way of measuring an institutionrsquos research intensity is to look at the range of its

course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of

doctoral and Masterrsquos programs offered as a proxy for research intensity

As with physical and financial resources few if any of the research indicators are

normalized to account for institutional size (either by student or faculty numbers) In the

world of rankings bigger almost always means better an institution with 100 faculty with

ten citations apiece will always look worse than an institution with 1001 faculty

13

with one citation each To the extent that the raw production of knowledge matters this

form of measurement is acceptable To the extent that rankings are meant to show how

well institutions are doing on a like-to-like basis or to show the efficiency of universities it

is plainly inadequate This should be of particular concern to Chinese policy-makers

whose ranking systems are especially reliant on research-based indicators

G Indicators of Reputation

The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo

Those rankings systems which use the results of reputation surveys as an indicator do so as

an indirect measure of quality based on the assumption that the employers academics and

academic administrators surveyed have opinions of institutional quality that are informed

up-to-date and impartial While these assumptions are clearly open to debate they

nevertheless form an important basis for many ranking systems Another reason for using

reputation measures is the paucity of other data availablemdashsome countries have few

independent measures of teaching effectiveness university resources or output and

reputation can thus act as a useful surrogate Reputation rankings are often criticized as

simply quantifying the common ignorance of the people being surveyed However to the

extent that the people being surveyed hold positions which have the potential to affect

large numbers of young people and whose positions actually require some knowledge of

institutional quality (ie officials in charge of graduate admissions corporate recruiters

etc) then reputation rankings make sense because they provide useful information for

students about the perceived value of the degrees that they could obtain from various

universities

The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and

the Times which both accord reputation a weighting of 50 in their overall ranking

scheme Education18 assigns it almost as much significance at 40 The US News and World

Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just

below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)

and Netbig (15) The only other study to include reputation is Wuhan (117)

14

1 Please see Appendix B for a glossary of the various bodies referenced in this document

Appendix A Detailed Listing of Indicators and their Sources

Beginning Characteristics Indicator

Used By Source1

Incoming grades Macleans University

Percentage with grades above a set limit

Macleans University

US News and World Report

University

Performance on national standardised tests or benchmarks

Asiaweek University

Education18 3rd-party JUPAS

Financial Times Government agency 3rd-party UCAS

Guardian University Guide

Government agency 3rd-party UCAS

Melbourne Institute Government agency 3rd-party DEST

Netbig National entrance examination board

Times Good University Guide

Government agency 3rd-party UCAS

US News University

Wuhan Unknown presumed government 3rd-party

Student status La Repubblica Government agency 3rd-party MIUR

Admittance selectivity general

Asiaweek University

Admittance number of applications to places

Asiaweek University

Financial Times Government agency 3rd-party UCAS

La Repubblica Government agency 3rd-party MIUR

US News University

15

Beginning Characteristics Indicator

Used By Source1

Out-of-locality student percentage

Macleans University

International student percentages

Financial Times Government agency 3rd-party HESA

Macleans University

Shanghai Institute of Educational Science

Times World University

Wuhan Unknown presumed university

Undergraduate students among all students percentages

Netbig

Wuhan Unknown presumed university

Ethnic diversity in student body

Guardian University

Learning Inputs ndash Staff Indicator

Used By Source

Facultystudent ratio Asiaweek University

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Financial Times Government agency 3rd-party HESA

La Repubblica Government agency 3rd-party MIUR

Times Good University Guide

Government agency 3rd-party HESA

Times World University

US News University

Wuhan Unknown presumed university

16

Learning Inputs ndash Staff Indicator

Used By Source

Social science faculty student ratio

Melbourne Government agency 3rd-party DEST

Science faculty student ratio

Melbourne Government agency 3rd-party DEST

Administrative staff student ratio

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Staff student ratio (regardless of division)

Guardian Government agency 3rd-party HESA

Netbig University

Course per teacher La Repubblica Government agency 3rd-party MIUR

Per-teacher university spending

Asiaweek University

Faculty pay rates fortenured staff

Asiaweek University

US News University

Number of full-time part-time faculty

Netbig University

US News University

Wuhan Unknown presumed university

Faculty with researchprojects

Wuhan Unknown presumed university

Class size differentiation Macleans University

US News University

Classes taught by tenured faculty

Macleans University

Exchange programmes hosted

La Repubblica Government agency 3rd-party AgNaSoc

Number of classes lsquoactually taughtrsquo

La Repubblica Government agency 3rd-party MIUR

of international faculty (v faculty as a whole)

Times World University

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

3INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

PersonnelsUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Le personnel de soutientotal de tous les personnels reacuteguliers acadeacutemiques et non acd agrave temps complet inputnombre des personnels de soutien (staff) input G10 MelbourneDeacutepenses des personnel non-enseignant input MEQnombre des personnels de soutien (staff)- secteur enseignement inputnombre des personnels de soutien (staff) - secteur soutien acadeacutemique inputnombre des personnels de soutien (staff) - secteur administration inputnombre des personnels de soutien (staff) - secteur service aux eacutetudiants inputnombre des personnels de soutien (staff) - entreprises auxiliaires input

rectoratpersonnel dencadrement non acadeacutemique et cadres acadeacutemiques inputratio sur lensemble des personnels acadeacutemiques et non acadeacutemiques inputeacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input G10 Guardianpersonnel de soutien professeur reacutegulier input G10

RP gouvernance - 2008-06-23

4INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Activiteacutes de recherche USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Revenus de recherche de professeurs et chercheurs ayant fait une demande de subvention processus de professeurs et chercheurs ayant obtenu une subvention inputRevenus totaux de recherche (subventions et contrats) input CAUBO G10 The Center Asiaweek Melbournepart de la recherche contractuelle input MEQSubventions publiques input Macleans Melbournerevenus de recherche obtenus des conseils de recherche feacutedeacuteraux input AOQTU The Center Macleans Melbournerevenus moyens de recherche par professeur et chercheur input InfosourceMEQ des deacutepenses de recherchebudget total des deacutepenses input TNombre de chaires et de centre dexcellence output AFondation canadienne pour linnovation (FCI) input G10 -nombre de projets output G10 -valeur $ output G10

nombre de programmes de maicirctrise de recherche input The Center Asiaweeknombre de programmes de doctorat input The Center Asiaweek

Publications et prix outputNombre de publications output Infosource Asiaweek Melbourne

Shanghai Leiden Taiwan

Nombre de publications par professeur et chercheur output Infosource CHEDAAD ShanghaiPublications dans Nature and Science output TaiwanNombre total de citations output InfosourceT Asiaweek MelbourneNombre de citations par professeur et chercheur output InfosourceT Melbourne THESNombre de citations par publication output T CHEDAAD LeidenHighly cited researchers dans 21 secteurs disciplinaires

output

ShanghaiTaiwan Leiden

Impact des publications selon les secteurs disciplinaires output Infosource TaiwanTHES Shanghai

Domaines disciplinaires dexcellence inputTaiwanShanghai THES

Prix et meacutedailles de niveau national et international output G10 The Center Macleans Melbourne Republica ShanghaiNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur output CHEDAADnombre de brevets et de licences output G10 CHEDAADnombre de spin-off companies G10

RP gouvernance - 2008-06-23

5INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

InternationalisationUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

deacutepenses pour le secteur internationalisation input MEQnombre deacutetudiants en stage agrave leacutetrangernombre deacutetudiants temps complet input tes un cannombre deacutetudiants internationauxnombre deacutetudiants temps complet input tes un can Melbourne THESnombre deacutetudiants en cotutellesnombre deacutetudiant des cycles supeacuterieurs input CHEDAADnombre de professeurs inviteacutes internationaux processus tes un can CHEDAADprofesseurs inviteacutes agrave leacutetranger processus CHEDAADnombre deacutetudiants posdoctoraux processus tes un can CHEDAADproportion de cours de langues eacutetrangegraveres processusnombre de programmes deacutechange processus tes un can CHEDAADEntentes de partenariat outputPrix internationaux output G10 The Center Macleans Melbourne Republica Shanghai

RP gouvernance - 2008-06-23

6INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Inteacutegration des technologies de linformation USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Deacutepenses totales consacreacutees aux TIC input MEQpersonnel techniqueenseignant reacutegulier inputpersonnel techniqueeacutetudiant en eacutequivalence temps complet inputExistence dun reacuteseau informatique (accessible agrave linterne et agrave lexterne) outputaccegraves au reacuteseau avec fils et sans fils processusaccegraves aux ressources documentaires des bibliothegraveques outputService de courriel interne et externe en mode WEB outputNombre dordinateurs par professeur et par eacutetudiant outputNombre de cours en ligne processusnombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) outputAccegraves des professeurs aux banque de donneacutees et aux logiciels processus

Site WEB (eacutevaluation du site WEB) WEBOMETRICSSize (nombre de pages recenseacutees) outcomevisibilitty (consultation observeacutee selon le nobre de lien externe unique) outcomeRich files (intensiteacute de lactiviteacute acadeacutemique) outcomeScholar (activiteacute de recherche publications et citations) outcome

RP gouvernance - 2008-06-23

7INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

BibliothegravequesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre total de volumes et de documents audiovisuels et eacutelectroniques input G10 ARLacquisition de volumes (au cours de lanneacutee) input ARL Macleansnombre total de volumes eacutetudiants en eacutequivalence temps complet input G10 ARL Macleans Deacutepenses des bibliothegraveques input MEQ Macleans AsiaweekDeacutepenses des bibliothegravequesbudget de fonctionnement input ARLDeacutepenses des bibliothegravequeseacutetudiants en eacutequivalence temps complet input G10 ARLPersonnels reacuteguliers agrave temps complet input G10 ARLClassement outcome ARL

RP gouvernance - 2008-06-23

8INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Revenus et deacutepenses USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Produits et revenusSubvention du ministegravere de lEacuteducation du Queacutebec input MEQ subvMEQrevenus de fonctionnement input G10 MacleansDroits de scolariteacute input G10 droits de scolariteacute revenus de fonctionnement input G10autres produits input G10Total des revenus de fonctionnement input G10 MEQ Guardian revenus de fonctionnement eacutetudiants en eacutequivalence temps complet input CAUBO USNews MacleansRevenus de recherche input G10encaissement par nature (dons immobilisations chaires bourses) input G10 fonds de dotation input G10 CenterContribution des alumni aux encaissements et fonds de dotation input USNews Macleans fonds de dotation eacutetudiants en eacutequivalence temps complet input G10Total des produits G10

Charges et deacutepenses G10salaire des professeurs input G10 MEQsalaire des chargeacutes de cours input G10salaires des auxiliaires input MEQsalaires des auxiliaires input MEQsalaire des personnels (staff) input G10autres deacutepenses input G10total des deacutepenses input G10 deacutepenses denseignement deacutepenses totales de fonctionnement input UBC MEQ

Chargesproduits processusEacutequilibre budgeacutetaire revenus et deacutepenses de fonctionnement processus MEQ

RP gouvernance - 2008-06-23

9INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

EspacesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Nombre de megravetres carreacutes bruts input G10MCB eacutetudiants en eacutequivalence temps complet input G10megravetres carreacutes nets alloueacutes input MEQNombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input Tbesoin despace en megravetres carreacutes nets inputExistence dun plan directeur des espaces inputBudget preacutevu au plan directeur des espaces input TBudget alloueacute aux espaces input MEQdeacutepenses de fonctionnement nombre de megravetres carreacutes bruts input G10

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Environnement USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

consommation de leacutenergie (exprimeacutee en BTU processus universiteacutes canadiennes et ameacutericaines BTUpopulation totale de luniversiteacute processus voir en particulier Penn State et Umich de leacutenergie produit par des sources renouvelable processusUtilisation de leau processuslitres deau par personne processusGaz agrave effet de serre (en tonnes meacutetriques CO2 eacutequivalents) processusMT CO2 par personne processusDeacutechets solides (en tonnes) processusTonnes de deacutechets solides per capita processus de deacutechets solides recycleacutes processus

RP gouvernance - 2008-06-23

10INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

ReacuteputationUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

enquecircte meneacutee aupregraves des eacutetudiants outcome NSSE Melbourneenquecircte meneacutee aupregraves des diplocircmeacutes outcomeenquecircte meneacutee aupregraves des pairs outcome USNews Asiaweek THESenquecircte meneacutee aupregraves des employeurs outcome Macleans Melbourneenquecircte meneacutee aupregraves des stakeholders outcome Macleans Melbourneenquecircte meneacutee aupregraves du monde des affaires outcome Macleans Melbournecouverture meacutediatique outcome AT

RP gouvernance - 2008-06-23

11INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Planification et gestion strateacutegiquesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Pouvoirs des corps universitaire processus un Canliens dynamiques entre Conseil Assembleacutee universitaire commission des eacutetudes processus un Canprocessus formel de planification strateacutegique processus un Canproduction de donneacutees et indicateurs processus un Canprocessus deacutevaluation des faculteacutes deacutepartements et services processus un Canrevue systeacutematique des responsabiliteacutes et efficaciteacute des comiteacutes processus un Candeacutepenses de ladministration centrale deacutepenses de fonctionnement processus un Can

RP gouvernance - 2008-06-23

12INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Eacuteducation permanenteUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre de programmes offerts input un Cannombre deacutetudiants inscrits input un CanRevenus et deacutepenses du service input un Can

RP gouvernance - 2008-06-23

13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes

REacuteFEacuteRENCES

Aperccedilu densemble des principaux indicateurs

GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost

Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes

MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf

Indicateurs utiliseacutes dans les classements internationaux

Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml

RP gouvernance - 2008-06-23

ANNEXES

Quelques exemples de listes drsquoindicateurs

1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)

Voir page suivante

Preacutepareacute par Roland Proulx

Consultant en planification institutionnelle et veille strateacutegique

Montreacuteal mai 2008

Performance Indicators

Discussion Paper

May 2004 (Prepared by the Office of Planning amp Analysis)

Table of Contents

Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25

2

Table 1 ndash Number of Indictors by Type of Data Source

Table 1 shows that surveys are the least frequently used source of data for indicators

Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a

plurality of indicators come from this source This measure somewhat underestimates the

importance of surveys however as it does not account for the weighting given to each

indicator in each study In the THES World Rankings for instance there is only a single

survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos

Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one

indicator is worth 20 of the final score

Raw

indIcator count

Surve

y data

Third parties

Universities

AsiaweekmdashAsias Best Universities 18 - - 18

Daily Telegraph (2003) 1 - 1 -

Education18com 9 3 4 2

Excelencia 2001 71 - 71 -

Financial Times (2003) 17 - 17 -

Guangdong Institute of Management Science 17 - 14 3

GuardianmdashUniversity Guide 2005 7 - 2 5

La Repubblica 23 2 21 -

Macleans University Rankings 24 1 5 18

Melbourne Institutemdash International Standing of Australian Universities

26 3 23 -

Netbig 2004 18 1 10 7

Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15

Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities

6 - 5 1

The TimesmdashGood University Guide 2005 9 - 9 -

Times Higher Education SupplementmdashWorld University Rankings

5 1 1 3

US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11

Washington MonthlymdashCollege Rankings 2005 8 - 1 7

Wuhan University Centre for Science Evaluation 45 2 22 21

3

A Indicators of Beginning Characteristics

ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of

students at the time they begin their studies Fourteen of the 18 rankings examined in this

study use one or more indicators of the beginning characteristics of students to arrive at

their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on

these factors with 33 of the total ranking coming from this class of indicators but the

Guardian Education18 Asiaweek and the two other North American surveys also place

considerable emphasis on this category

There are six main indicators used to determine which institutions have students with

positive ldquobeginning characteristicsrdquo

The most common measure of beginning characteristics is performance on national

standardized tests with nine surveys using this as a measure Education18 and the

Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used

by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)

Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this

data is collected and standardized by national bodies it has the benefit of being seen as a

relatively impartial method of determining the relative ldquostrengthrdquo of the students entering

each institution Institutionsrsquo results can be scored by showing either averages or the

percentage of entering students meeting a particular standard

Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses

secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This

is a second-best solution made necessary by the absence of any national standardized test

in Canada (or indeed of any provincial standardized tests at the end of secondary school

in provinces other than Alberta) The lack of national standardization makes this an

undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is

truly equivalent to an ldquoArdquo in another jurisdiction

Another measure of the strength of the student body is the percentage of incoming

students receiving (third-party) scholarships which is worth 11 of the score in the

Wuhan survey One can also approach the issue by measuring institutional selectivity In

effect this method infers the strength of the student body by the proportion of

4

applicants rejected the theory being that the higher the number of rejected applicants the

stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)

Normally this measure is expressed as a straight ratio of acceptances to applications but it

can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the

greatest weight on this measure) as a ratio of enrolments to applications Within the US

there is some dispute as to what constitutes an offer of admission and whether or not late

admissions are included as noted by Ganeshananthan (2003)

Student bodies are often considered to be strong if the school is able to attract a large

number of international or out-of-district students or if they contain people from diverse

ethnic backgrounds A number of league tables use the international student indicator

(which like the selectivity indicator is arguably as much an indicator of prestige and

reputation as it is of student characteristics) although in no case does this indicator

account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a

quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data

on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian

puts a somewhat larger emphasis on this indicator in comparison to other league tables

which use similar variables

A very different take on this idea is present in the Washington Monthly which released its

first set of College Rankings in September 2005 With the declared aim of using an

institutionrsquos commitment to social mobility as a measure of quality it uses the percentage

of students from low-income backgrounds as an indicator (with percentage of students

receiving need-based government (Pell) grants used as a proxy)

Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy

statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the

percentage of the student population who are graduate students (arguably this is a

research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the

number of graduate students auditing classes is used as an indicator the assumption is

presumably that if people are auditing then the classes must be very attractive The Italian

La Repubblica ranks an institution according to the number of part-time students it has

contrary to prevailing North American views on the undesirability of part-time study the

Italian rankings see higher numbers of part-time students in a positive light as it is

evidence that an institution is becoming less rigid in its timetabling and

5

permitting students to juggle both work and study something which was nearly

impossible in that country just a few years ago

The Washington Monthly also has a final category of indicators which reflect studentsrsquo

beginning characteristics namely their likelihood of performing community service as

measured by the percentage of students in the US Peace Corps and Reserve Officer

Training Corps (ROTC) and the percentage of students involved in work-study in the

community3 Together these three indicators account for 33 of an institutionrsquos total

ranking

B Indicators of Learning InputsmdashStaff

Generally both the quantity and quality of staff are positively correlated with institutional

quality The problem of course is finding useful metrics for each of these factors

especially if one excludes as we have done here measures of research performance and

research intensity putting them in a separate category4

The simplest measure is simply the number of faculty unadjusted for things like size of

student body Most national league tables however prefer to use variations on the

concept of facultystudent ratio Others try to measure teaching intensity with measures

such as courses per teacher or hours spent in class per student (both in La Repubblica)

These kinds of measures usually account for between 2-5 of the final rankings although

in some cases (ie the Guardian) this figure can be as high as 20

Another important way of measuring how faculty resources are deployed is the measure

of average class size which is used only by Macleanrsquos and the US News and World Report

Ostensibly the reason for measuring class size is to account in some form for the degree

3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category

6

of attention devoted to actually teaching students Despite the fact that important research

(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level

the use of this indicator appears to be a spillover from the North American debates on

class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein

2000) Regardless of why the indicators are used they are extraordinarily important to

these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and

World Reportrsquos rankings respectively

A number of ranking systems try to look at staff qualifications such as the number of

PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the

Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further

than other surveys and actually looks at the proportion of classes taught by tenure-track

staff Others (ie THES) look at the number of foreign faculty based on the assumption

that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still

others (ie La Repubblica) look at the age structure of the faculty Another proxy for

institutional quality is the pay rates for tenured staff on the assumption that institutions

with higher rates of pay on average attract better faculty this measure has been used both

by the US News and World Report and Asiaweek

Finally a number of league tables rank faculty inputs on the basis of standardized third-

party evaluations 6

Education18 the Financial Times the Times the Guardian and the US

News and World Report league tables all use some sort of ranking criterion based at least in

part on this indicator or variations thereof

C Indicators of Learning InputsmdashResources

Resource inputsmdashcrudely the amount of current dollars equipment and books available

to students at an institutionmdashare widely considered an important measure of quality Yet

despite the apparent simplicity of counting dollars and measuring assets

5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each

university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future

7

the means by which institutional wealth is measured varies considerably between ranking

systems

There are a number of revenue-based measures of resources Macleanrsquos uses public funding

of institutional budgets as a factor in its analysis conversely the Financial Times uses the

private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of

total score) and the US News and World Report (5 of total score) also measure alumni

financial support as a measure of quality

For reasons that are not entirely clear league tables tend to favour measures of

expenditures rather than revenues The Guardian looks at total institutional expenditures

as an indicator Institutional expenditure on student services is used as a measure of

institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total

institutional scores respectively) Rzezspospolita does not measure student services

expenditures directly but does measure student services outputs such as number of

student athletes and number of study clubs which amounts to more or less the same thing

Macleanrsquos also gives out 433 of its total score based on institutional expenditures on

scholarships and bursaries

Various aspects of physical infrastructure are also used as measures of institutional

resources most directly in the case of La Repubblica which bases 317 of its total rank on

the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding

assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type

of physical infrastructure measured is available Internet bandwidth which was used by

Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth

roughly 3 of the total score

By some distance the infrastructure indicators most favoured by the compilers of league

tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on

this with 12 of the total quality mark being taken from various types of library

infrastructure measurements (including acquisitions per year total volumes average

number of volumes per student and yearly library expenditure outside of acquisitions)

Netbig and Education18 also use library volume holdings while Asiaweek the Financial

Times and the Times also use measures of library expenditures outside of acquisitions or

computerization of library resources as measures of institutional quality

8

One important factor to note is that most ranking systems do not normalize their resource

and infrastructure measures That is to say it is raw spending power or simple size of

assets that is usually measured rather than spending per studentprofessor or assets per

studentprofessor As a result a number of these rankings systems have inbuilt biases

towards larger institutions

D Indicators of Learning Outputs

Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge

learned over the course of a baccalaureate degreemdashshould be a basic indicator of

institutional quality Unfortunately good means of measuring these outputsmdashlike the

National Survey of Student Engagement (NSSE) and the College Learning Assessment

(CLA)mdashhave only recently become available and for the most part institutions are still

keeping their scores secret Outside of these measures only a few very crude indicators are

available which likely explains why learning outputs do not feature especially prominently

in most ranking schemes

The simplest types of measures of learning outputs are those linked to graduation and

retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong

and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7

the latter three also use rates of graduation from Masterrsquos programs as indicators In some

cases the weights on these measures can be very highmdashin the Guangdong rankings

graduation rates account for over 50 of the rankingmdashbut in most cases the weights are

10 or less Retention rates commonly meaning the progression rate of first-year students

into second year are accorded less importance The US News and World Report Macleanrsquos

the Melbourne Institute and La Repubblica all employ retention measures as indicators but

none of them are worth more than 4 of total weighting Two publications make specific

indicators for retention and graduation of international students Macleanrsquos (graduation

rates of international students) and the Melbourne Institute (retention rates of international

students) The Washington Monthly looks specifically at institutional retention rates

adjusted for the participation of lower-income students and gives higher scores to

institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores

and number of Pell Grant recipients the US

7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18

9

News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos

ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)

E Indicators of Final Outcomes

Final outcomes are indications of generalized outcomes for students after graduation

Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness

good citizenship) but given the somewhat utilitarian justifications for education that are

currently in fashion (see Wolf 2000) employment outcomes are the most commonly used

measure of final outcomes These are given particular emphasis by the Guardian (where

employment outcomes are worth 17 of the total score) but are also used by the Financial

Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the

Times are interestingly not concerned with employment per se but with ldquoemployment in

an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher

Education Statistics Agency (HESA) uses Standard Occupational Classifications to

measure the proportion of graduates in professional or white-collar jobs anyone not in

such a job is considered not to be working in an area related to their studies (it is unclear

what methodology is used by the Financial Times and the Times although we suspect their

methods are broadly similar)

The only other measure of final outcomes in use is percentage of graduates returning for

additional education which is an indicator used by both the Melbourne Institute and the

Financial Times This is a particularly important indicator for the latter as it is worth 21

of the final ranking

The lack of indicators concerning final outcomes is interesting since most government-

sponsored performance-indicator regimes around the world are very much concerned

with such measures especially with respect to employment Possibly this indicates that

ranking systems simply do not view education outcomes as relevant measures of

educational quality Alternatively it may be the case that they simply have not found a

reliable indicator of outcomes or that there are reliable indicators but that there is so little

variation between institutions that it makes no sense to rank based on the data

10

F Indicators of Research

Many of the league tables covered in this survey include sections and weightings related

to universitiesrsquo research efforts It is in this field of measurement that we see the greatest

diversity of indicators Presumably this is because research inputs and outputs lend

themselves much more easily to measurement and manipulation than other areas of

institutional activity

Three studies include research staff as part of their ranking scheme La Repubblica at 952

the Melbourne Institute (4) and Wuhan (078)

Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used

method of looking at research quality but it is not universally admired because different

disciplines use different means to communicate major advances in knowledge (leading

scientists invariably produce large numbers of journal articles leading social scientists

may produce fewer journal articles but instead have one or two long important

monographsmdashsee Hicks 2004) There is also some concern among non-English speaking

countries that they are penalized in international rankings because so many of the major

journals (notably Science and Nature) are printed in English However the one set of

rankings that uses separate indicators to monitor articles published in English and articles

published in another language (the Wuhan rankings) shows that the two indicators are

positively correlated institutions that have more Chinese publications are also likely to

have more English publications and vice versa

Several sets of league tables measure bibliometric citations in various publication indices

The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a

weight of 20 of the final total Guangdong also monitors other Chinese universities

specifically for citations in engineering publications and weights this at 29 Moreover it

tacks on an additional 1049 for citations in science-oriented indices such as the Science

Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the

same category of scientific citations while the Melbourne Institute rates science citations at

68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social

Science Citation Index which does not include the humanities) are noted in only two

league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne

Institute (32) Another way of measuring research

11

impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a

weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the

Melbourne Institute

The complement of citations is of course publications Listing the number of publications

an individual group department or whole university releases can act as a weak substitute

for citationsmdashweak because simply publishing a paper or monograph is no guarantee that

the general public or other researchers will even glance at the work Guangdong gives an

1179 weighting to publications in science-oriented indices such as the Science Citation

Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same

indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong

even has a separate category just for measuring publications in Science and Nature

although it accords it an almost derisory weighting of 06 Under publications in social

science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their

final totals For publications in other indices (where the subject indices are

undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for

Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of

rankings to include research monographs weighted at 033

In countries where there are specific third-party evaluations of research output academic

quality of research is sometimes used as a research indicator The Times puts a very large

30 weight on this indicator while the Financial Times puts it at 119

Research awards are another handy third-party measurement of quality as the number of

international and national awards won by faculty andor graduates is often considered a

useful measure of institutional success International research awardsmdash specifically the

number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator

by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as

particularly suspect in some quarters given that the points are based on where the

recipient went to school rather than on where they are or were on the

8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the

most prominent publication indices

9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences

12

faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead

for nearly a century Wuhan uses similar measures but only accords them a weight of 14

National research awards are more common as a quality indicator used by La Repubblica

(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)

Financial indicators of research are also very common Research budgets as a factor in the

overall assessment of research in universities are covered by the Financial Times (9)

Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for

total amount of research expenditure unfortunately it is unclear precisely what this

research expenditure represents or how it is determined although it is clearly indicated

that it does not represent the total number of grants or projects at a university Total

number of research-based grants and projects is weighted by Education18 at 15 and

Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science

and engineering and another 55 to those for social sciences and humanities Similarly

the Melbourne Institute gives 6 of its overall weight to public-source grants making no

distinction between areas of study

In a slightly different vein Netbig (46) and Wuhan (278) both list the number of

research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica

(095) all weigh research-basedaffiliated research institutions or centres for studies

Finally one can also measure research not simply in terms of the amount of money it

generates but also in terms of the amount of future income it will generate Both

Guangdong (245) and Wuhan (193) measure the number of patents issued to

universities as a quality indicator

A final way of measuring an institutionrsquos research intensity is to look at the range of its

course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of

doctoral and Masterrsquos programs offered as a proxy for research intensity

As with physical and financial resources few if any of the research indicators are

normalized to account for institutional size (either by student or faculty numbers) In the

world of rankings bigger almost always means better an institution with 100 faculty with

ten citations apiece will always look worse than an institution with 1001 faculty

13

with one citation each To the extent that the raw production of knowledge matters this

form of measurement is acceptable To the extent that rankings are meant to show how

well institutions are doing on a like-to-like basis or to show the efficiency of universities it

is plainly inadequate This should be of particular concern to Chinese policy-makers

whose ranking systems are especially reliant on research-based indicators

G Indicators of Reputation

The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo

Those rankings systems which use the results of reputation surveys as an indicator do so as

an indirect measure of quality based on the assumption that the employers academics and

academic administrators surveyed have opinions of institutional quality that are informed

up-to-date and impartial While these assumptions are clearly open to debate they

nevertheless form an important basis for many ranking systems Another reason for using

reputation measures is the paucity of other data availablemdashsome countries have few

independent measures of teaching effectiveness university resources or output and

reputation can thus act as a useful surrogate Reputation rankings are often criticized as

simply quantifying the common ignorance of the people being surveyed However to the

extent that the people being surveyed hold positions which have the potential to affect

large numbers of young people and whose positions actually require some knowledge of

institutional quality (ie officials in charge of graduate admissions corporate recruiters

etc) then reputation rankings make sense because they provide useful information for

students about the perceived value of the degrees that they could obtain from various

universities

The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and

the Times which both accord reputation a weighting of 50 in their overall ranking

scheme Education18 assigns it almost as much significance at 40 The US News and World

Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just

below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)

and Netbig (15) The only other study to include reputation is Wuhan (117)

14

1 Please see Appendix B for a glossary of the various bodies referenced in this document

Appendix A Detailed Listing of Indicators and their Sources

Beginning Characteristics Indicator

Used By Source1

Incoming grades Macleans University

Percentage with grades above a set limit

Macleans University

US News and World Report

University

Performance on national standardised tests or benchmarks

Asiaweek University

Education18 3rd-party JUPAS

Financial Times Government agency 3rd-party UCAS

Guardian University Guide

Government agency 3rd-party UCAS

Melbourne Institute Government agency 3rd-party DEST

Netbig National entrance examination board

Times Good University Guide

Government agency 3rd-party UCAS

US News University

Wuhan Unknown presumed government 3rd-party

Student status La Repubblica Government agency 3rd-party MIUR

Admittance selectivity general

Asiaweek University

Admittance number of applications to places

Asiaweek University

Financial Times Government agency 3rd-party UCAS

La Repubblica Government agency 3rd-party MIUR

US News University

15

Beginning Characteristics Indicator

Used By Source1

Out-of-locality student percentage

Macleans University

International student percentages

Financial Times Government agency 3rd-party HESA

Macleans University

Shanghai Institute of Educational Science

Times World University

Wuhan Unknown presumed university

Undergraduate students among all students percentages

Netbig

Wuhan Unknown presumed university

Ethnic diversity in student body

Guardian University

Learning Inputs ndash Staff Indicator

Used By Source

Facultystudent ratio Asiaweek University

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Financial Times Government agency 3rd-party HESA

La Repubblica Government agency 3rd-party MIUR

Times Good University Guide

Government agency 3rd-party HESA

Times World University

US News University

Wuhan Unknown presumed university

16

Learning Inputs ndash Staff Indicator

Used By Source

Social science faculty student ratio

Melbourne Government agency 3rd-party DEST

Science faculty student ratio

Melbourne Government agency 3rd-party DEST

Administrative staff student ratio

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Staff student ratio (regardless of division)

Guardian Government agency 3rd-party HESA

Netbig University

Course per teacher La Repubblica Government agency 3rd-party MIUR

Per-teacher university spending

Asiaweek University

Faculty pay rates fortenured staff

Asiaweek University

US News University

Number of full-time part-time faculty

Netbig University

US News University

Wuhan Unknown presumed university

Faculty with researchprojects

Wuhan Unknown presumed university

Class size differentiation Macleans University

US News University

Classes taught by tenured faculty

Macleans University

Exchange programmes hosted

La Repubblica Government agency 3rd-party AgNaSoc

Number of classes lsquoactually taughtrsquo

La Repubblica Government agency 3rd-party MIUR

of international faculty (v faculty as a whole)

Times World University

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

4INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Activiteacutes de recherche USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Revenus de recherche de professeurs et chercheurs ayant fait une demande de subvention processus de professeurs et chercheurs ayant obtenu une subvention inputRevenus totaux de recherche (subventions et contrats) input CAUBO G10 The Center Asiaweek Melbournepart de la recherche contractuelle input MEQSubventions publiques input Macleans Melbournerevenus de recherche obtenus des conseils de recherche feacutedeacuteraux input AOQTU The Center Macleans Melbournerevenus moyens de recherche par professeur et chercheur input InfosourceMEQ des deacutepenses de recherchebudget total des deacutepenses input TNombre de chaires et de centre dexcellence output AFondation canadienne pour linnovation (FCI) input G10 -nombre de projets output G10 -valeur $ output G10

nombre de programmes de maicirctrise de recherche input The Center Asiaweeknombre de programmes de doctorat input The Center Asiaweek

Publications et prix outputNombre de publications output Infosource Asiaweek Melbourne

Shanghai Leiden Taiwan

Nombre de publications par professeur et chercheur output Infosource CHEDAAD ShanghaiPublications dans Nature and Science output TaiwanNombre total de citations output InfosourceT Asiaweek MelbourneNombre de citations par professeur et chercheur output InfosourceT Melbourne THESNombre de citations par publication output T CHEDAAD LeidenHighly cited researchers dans 21 secteurs disciplinaires

output

ShanghaiTaiwan Leiden

Impact des publications selon les secteurs disciplinaires output Infosource TaiwanTHES Shanghai

Domaines disciplinaires dexcellence inputTaiwanShanghai THES

Prix et meacutedailles de niveau national et international output G10 The Center Macleans Melbourne Republica ShanghaiNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur output CHEDAADnombre de brevets et de licences output G10 CHEDAADnombre de spin-off companies G10

RP gouvernance - 2008-06-23

5INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

InternationalisationUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

deacutepenses pour le secteur internationalisation input MEQnombre deacutetudiants en stage agrave leacutetrangernombre deacutetudiants temps complet input tes un cannombre deacutetudiants internationauxnombre deacutetudiants temps complet input tes un can Melbourne THESnombre deacutetudiants en cotutellesnombre deacutetudiant des cycles supeacuterieurs input CHEDAADnombre de professeurs inviteacutes internationaux processus tes un can CHEDAADprofesseurs inviteacutes agrave leacutetranger processus CHEDAADnombre deacutetudiants posdoctoraux processus tes un can CHEDAADproportion de cours de langues eacutetrangegraveres processusnombre de programmes deacutechange processus tes un can CHEDAADEntentes de partenariat outputPrix internationaux output G10 The Center Macleans Melbourne Republica Shanghai

RP gouvernance - 2008-06-23

6INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Inteacutegration des technologies de linformation USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Deacutepenses totales consacreacutees aux TIC input MEQpersonnel techniqueenseignant reacutegulier inputpersonnel techniqueeacutetudiant en eacutequivalence temps complet inputExistence dun reacuteseau informatique (accessible agrave linterne et agrave lexterne) outputaccegraves au reacuteseau avec fils et sans fils processusaccegraves aux ressources documentaires des bibliothegraveques outputService de courriel interne et externe en mode WEB outputNombre dordinateurs par professeur et par eacutetudiant outputNombre de cours en ligne processusnombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) outputAccegraves des professeurs aux banque de donneacutees et aux logiciels processus

Site WEB (eacutevaluation du site WEB) WEBOMETRICSSize (nombre de pages recenseacutees) outcomevisibilitty (consultation observeacutee selon le nobre de lien externe unique) outcomeRich files (intensiteacute de lactiviteacute acadeacutemique) outcomeScholar (activiteacute de recherche publications et citations) outcome

RP gouvernance - 2008-06-23

7INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

BibliothegravequesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre total de volumes et de documents audiovisuels et eacutelectroniques input G10 ARLacquisition de volumes (au cours de lanneacutee) input ARL Macleansnombre total de volumes eacutetudiants en eacutequivalence temps complet input G10 ARL Macleans Deacutepenses des bibliothegraveques input MEQ Macleans AsiaweekDeacutepenses des bibliothegravequesbudget de fonctionnement input ARLDeacutepenses des bibliothegravequeseacutetudiants en eacutequivalence temps complet input G10 ARLPersonnels reacuteguliers agrave temps complet input G10 ARLClassement outcome ARL

RP gouvernance - 2008-06-23

8INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Revenus et deacutepenses USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Produits et revenusSubvention du ministegravere de lEacuteducation du Queacutebec input MEQ subvMEQrevenus de fonctionnement input G10 MacleansDroits de scolariteacute input G10 droits de scolariteacute revenus de fonctionnement input G10autres produits input G10Total des revenus de fonctionnement input G10 MEQ Guardian revenus de fonctionnement eacutetudiants en eacutequivalence temps complet input CAUBO USNews MacleansRevenus de recherche input G10encaissement par nature (dons immobilisations chaires bourses) input G10 fonds de dotation input G10 CenterContribution des alumni aux encaissements et fonds de dotation input USNews Macleans fonds de dotation eacutetudiants en eacutequivalence temps complet input G10Total des produits G10

Charges et deacutepenses G10salaire des professeurs input G10 MEQsalaire des chargeacutes de cours input G10salaires des auxiliaires input MEQsalaires des auxiliaires input MEQsalaire des personnels (staff) input G10autres deacutepenses input G10total des deacutepenses input G10 deacutepenses denseignement deacutepenses totales de fonctionnement input UBC MEQ

Chargesproduits processusEacutequilibre budgeacutetaire revenus et deacutepenses de fonctionnement processus MEQ

RP gouvernance - 2008-06-23

9INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

EspacesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Nombre de megravetres carreacutes bruts input G10MCB eacutetudiants en eacutequivalence temps complet input G10megravetres carreacutes nets alloueacutes input MEQNombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input Tbesoin despace en megravetres carreacutes nets inputExistence dun plan directeur des espaces inputBudget preacutevu au plan directeur des espaces input TBudget alloueacute aux espaces input MEQdeacutepenses de fonctionnement nombre de megravetres carreacutes bruts input G10

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Environnement USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

consommation de leacutenergie (exprimeacutee en BTU processus universiteacutes canadiennes et ameacutericaines BTUpopulation totale de luniversiteacute processus voir en particulier Penn State et Umich de leacutenergie produit par des sources renouvelable processusUtilisation de leau processuslitres deau par personne processusGaz agrave effet de serre (en tonnes meacutetriques CO2 eacutequivalents) processusMT CO2 par personne processusDeacutechets solides (en tonnes) processusTonnes de deacutechets solides per capita processus de deacutechets solides recycleacutes processus

RP gouvernance - 2008-06-23

10INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

ReacuteputationUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

enquecircte meneacutee aupregraves des eacutetudiants outcome NSSE Melbourneenquecircte meneacutee aupregraves des diplocircmeacutes outcomeenquecircte meneacutee aupregraves des pairs outcome USNews Asiaweek THESenquecircte meneacutee aupregraves des employeurs outcome Macleans Melbourneenquecircte meneacutee aupregraves des stakeholders outcome Macleans Melbourneenquecircte meneacutee aupregraves du monde des affaires outcome Macleans Melbournecouverture meacutediatique outcome AT

RP gouvernance - 2008-06-23

11INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Planification et gestion strateacutegiquesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Pouvoirs des corps universitaire processus un Canliens dynamiques entre Conseil Assembleacutee universitaire commission des eacutetudes processus un Canprocessus formel de planification strateacutegique processus un Canproduction de donneacutees et indicateurs processus un Canprocessus deacutevaluation des faculteacutes deacutepartements et services processus un Canrevue systeacutematique des responsabiliteacutes et efficaciteacute des comiteacutes processus un Candeacutepenses de ladministration centrale deacutepenses de fonctionnement processus un Can

RP gouvernance - 2008-06-23

12INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Eacuteducation permanenteUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre de programmes offerts input un Cannombre deacutetudiants inscrits input un CanRevenus et deacutepenses du service input un Can

RP gouvernance - 2008-06-23

13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes

REacuteFEacuteRENCES

Aperccedilu densemble des principaux indicateurs

GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost

Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes

MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf

Indicateurs utiliseacutes dans les classements internationaux

Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml

RP gouvernance - 2008-06-23

ANNEXES

Quelques exemples de listes drsquoindicateurs

1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)

Voir page suivante

Preacutepareacute par Roland Proulx

Consultant en planification institutionnelle et veille strateacutegique

Montreacuteal mai 2008

Performance Indicators

Discussion Paper

May 2004 (Prepared by the Office of Planning amp Analysis)

Table of Contents

Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25

2

Table 1 ndash Number of Indictors by Type of Data Source

Table 1 shows that surveys are the least frequently used source of data for indicators

Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a

plurality of indicators come from this source This measure somewhat underestimates the

importance of surveys however as it does not account for the weighting given to each

indicator in each study In the THES World Rankings for instance there is only a single

survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos

Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one

indicator is worth 20 of the final score

Raw

indIcator count

Surve

y data

Third parties

Universities

AsiaweekmdashAsias Best Universities 18 - - 18

Daily Telegraph (2003) 1 - 1 -

Education18com 9 3 4 2

Excelencia 2001 71 - 71 -

Financial Times (2003) 17 - 17 -

Guangdong Institute of Management Science 17 - 14 3

GuardianmdashUniversity Guide 2005 7 - 2 5

La Repubblica 23 2 21 -

Macleans University Rankings 24 1 5 18

Melbourne Institutemdash International Standing of Australian Universities

26 3 23 -

Netbig 2004 18 1 10 7

Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15

Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities

6 - 5 1

The TimesmdashGood University Guide 2005 9 - 9 -

Times Higher Education SupplementmdashWorld University Rankings

5 1 1 3

US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11

Washington MonthlymdashCollege Rankings 2005 8 - 1 7

Wuhan University Centre for Science Evaluation 45 2 22 21

3

A Indicators of Beginning Characteristics

ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of

students at the time they begin their studies Fourteen of the 18 rankings examined in this

study use one or more indicators of the beginning characteristics of students to arrive at

their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on

these factors with 33 of the total ranking coming from this class of indicators but the

Guardian Education18 Asiaweek and the two other North American surveys also place

considerable emphasis on this category

There are six main indicators used to determine which institutions have students with

positive ldquobeginning characteristicsrdquo

The most common measure of beginning characteristics is performance on national

standardized tests with nine surveys using this as a measure Education18 and the

Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used

by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)

Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this

data is collected and standardized by national bodies it has the benefit of being seen as a

relatively impartial method of determining the relative ldquostrengthrdquo of the students entering

each institution Institutionsrsquo results can be scored by showing either averages or the

percentage of entering students meeting a particular standard

Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses

secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This

is a second-best solution made necessary by the absence of any national standardized test

in Canada (or indeed of any provincial standardized tests at the end of secondary school

in provinces other than Alberta) The lack of national standardization makes this an

undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is

truly equivalent to an ldquoArdquo in another jurisdiction

Another measure of the strength of the student body is the percentage of incoming

students receiving (third-party) scholarships which is worth 11 of the score in the

Wuhan survey One can also approach the issue by measuring institutional selectivity In

effect this method infers the strength of the student body by the proportion of

4

applicants rejected the theory being that the higher the number of rejected applicants the

stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)

Normally this measure is expressed as a straight ratio of acceptances to applications but it

can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the

greatest weight on this measure) as a ratio of enrolments to applications Within the US

there is some dispute as to what constitutes an offer of admission and whether or not late

admissions are included as noted by Ganeshananthan (2003)

Student bodies are often considered to be strong if the school is able to attract a large

number of international or out-of-district students or if they contain people from diverse

ethnic backgrounds A number of league tables use the international student indicator

(which like the selectivity indicator is arguably as much an indicator of prestige and

reputation as it is of student characteristics) although in no case does this indicator

account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a

quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data

on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian

puts a somewhat larger emphasis on this indicator in comparison to other league tables

which use similar variables

A very different take on this idea is present in the Washington Monthly which released its

first set of College Rankings in September 2005 With the declared aim of using an

institutionrsquos commitment to social mobility as a measure of quality it uses the percentage

of students from low-income backgrounds as an indicator (with percentage of students

receiving need-based government (Pell) grants used as a proxy)

Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy

statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the

percentage of the student population who are graduate students (arguably this is a

research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the

number of graduate students auditing classes is used as an indicator the assumption is

presumably that if people are auditing then the classes must be very attractive The Italian

La Repubblica ranks an institution according to the number of part-time students it has

contrary to prevailing North American views on the undesirability of part-time study the

Italian rankings see higher numbers of part-time students in a positive light as it is

evidence that an institution is becoming less rigid in its timetabling and

5

permitting students to juggle both work and study something which was nearly

impossible in that country just a few years ago

The Washington Monthly also has a final category of indicators which reflect studentsrsquo

beginning characteristics namely their likelihood of performing community service as

measured by the percentage of students in the US Peace Corps and Reserve Officer

Training Corps (ROTC) and the percentage of students involved in work-study in the

community3 Together these three indicators account for 33 of an institutionrsquos total

ranking

B Indicators of Learning InputsmdashStaff

Generally both the quantity and quality of staff are positively correlated with institutional

quality The problem of course is finding useful metrics for each of these factors

especially if one excludes as we have done here measures of research performance and

research intensity putting them in a separate category4

The simplest measure is simply the number of faculty unadjusted for things like size of

student body Most national league tables however prefer to use variations on the

concept of facultystudent ratio Others try to measure teaching intensity with measures

such as courses per teacher or hours spent in class per student (both in La Repubblica)

These kinds of measures usually account for between 2-5 of the final rankings although

in some cases (ie the Guardian) this figure can be as high as 20

Another important way of measuring how faculty resources are deployed is the measure

of average class size which is used only by Macleanrsquos and the US News and World Report

Ostensibly the reason for measuring class size is to account in some form for the degree

3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category

6

of attention devoted to actually teaching students Despite the fact that important research

(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level

the use of this indicator appears to be a spillover from the North American debates on

class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein

2000) Regardless of why the indicators are used they are extraordinarily important to

these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and

World Reportrsquos rankings respectively

A number of ranking systems try to look at staff qualifications such as the number of

PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the

Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further

than other surveys and actually looks at the proportion of classes taught by tenure-track

staff Others (ie THES) look at the number of foreign faculty based on the assumption

that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still

others (ie La Repubblica) look at the age structure of the faculty Another proxy for

institutional quality is the pay rates for tenured staff on the assumption that institutions

with higher rates of pay on average attract better faculty this measure has been used both

by the US News and World Report and Asiaweek

Finally a number of league tables rank faculty inputs on the basis of standardized third-

party evaluations 6

Education18 the Financial Times the Times the Guardian and the US

News and World Report league tables all use some sort of ranking criterion based at least in

part on this indicator or variations thereof

C Indicators of Learning InputsmdashResources

Resource inputsmdashcrudely the amount of current dollars equipment and books available

to students at an institutionmdashare widely considered an important measure of quality Yet

despite the apparent simplicity of counting dollars and measuring assets

5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each

university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future

7

the means by which institutional wealth is measured varies considerably between ranking

systems

There are a number of revenue-based measures of resources Macleanrsquos uses public funding

of institutional budgets as a factor in its analysis conversely the Financial Times uses the

private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of

total score) and the US News and World Report (5 of total score) also measure alumni

financial support as a measure of quality

For reasons that are not entirely clear league tables tend to favour measures of

expenditures rather than revenues The Guardian looks at total institutional expenditures

as an indicator Institutional expenditure on student services is used as a measure of

institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total

institutional scores respectively) Rzezspospolita does not measure student services

expenditures directly but does measure student services outputs such as number of

student athletes and number of study clubs which amounts to more or less the same thing

Macleanrsquos also gives out 433 of its total score based on institutional expenditures on

scholarships and bursaries

Various aspects of physical infrastructure are also used as measures of institutional

resources most directly in the case of La Repubblica which bases 317 of its total rank on

the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding

assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type

of physical infrastructure measured is available Internet bandwidth which was used by

Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth

roughly 3 of the total score

By some distance the infrastructure indicators most favoured by the compilers of league

tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on

this with 12 of the total quality mark being taken from various types of library

infrastructure measurements (including acquisitions per year total volumes average

number of volumes per student and yearly library expenditure outside of acquisitions)

Netbig and Education18 also use library volume holdings while Asiaweek the Financial

Times and the Times also use measures of library expenditures outside of acquisitions or

computerization of library resources as measures of institutional quality

8

One important factor to note is that most ranking systems do not normalize their resource

and infrastructure measures That is to say it is raw spending power or simple size of

assets that is usually measured rather than spending per studentprofessor or assets per

studentprofessor As a result a number of these rankings systems have inbuilt biases

towards larger institutions

D Indicators of Learning Outputs

Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge

learned over the course of a baccalaureate degreemdashshould be a basic indicator of

institutional quality Unfortunately good means of measuring these outputsmdashlike the

National Survey of Student Engagement (NSSE) and the College Learning Assessment

(CLA)mdashhave only recently become available and for the most part institutions are still

keeping their scores secret Outside of these measures only a few very crude indicators are

available which likely explains why learning outputs do not feature especially prominently

in most ranking schemes

The simplest types of measures of learning outputs are those linked to graduation and

retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong

and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7

the latter three also use rates of graduation from Masterrsquos programs as indicators In some

cases the weights on these measures can be very highmdashin the Guangdong rankings

graduation rates account for over 50 of the rankingmdashbut in most cases the weights are

10 or less Retention rates commonly meaning the progression rate of first-year students

into second year are accorded less importance The US News and World Report Macleanrsquos

the Melbourne Institute and La Repubblica all employ retention measures as indicators but

none of them are worth more than 4 of total weighting Two publications make specific

indicators for retention and graduation of international students Macleanrsquos (graduation

rates of international students) and the Melbourne Institute (retention rates of international

students) The Washington Monthly looks specifically at institutional retention rates

adjusted for the participation of lower-income students and gives higher scores to

institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores

and number of Pell Grant recipients the US

7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18

9

News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos

ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)

E Indicators of Final Outcomes

Final outcomes are indications of generalized outcomes for students after graduation

Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness

good citizenship) but given the somewhat utilitarian justifications for education that are

currently in fashion (see Wolf 2000) employment outcomes are the most commonly used

measure of final outcomes These are given particular emphasis by the Guardian (where

employment outcomes are worth 17 of the total score) but are also used by the Financial

Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the

Times are interestingly not concerned with employment per se but with ldquoemployment in

an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher

Education Statistics Agency (HESA) uses Standard Occupational Classifications to

measure the proportion of graduates in professional or white-collar jobs anyone not in

such a job is considered not to be working in an area related to their studies (it is unclear

what methodology is used by the Financial Times and the Times although we suspect their

methods are broadly similar)

The only other measure of final outcomes in use is percentage of graduates returning for

additional education which is an indicator used by both the Melbourne Institute and the

Financial Times This is a particularly important indicator for the latter as it is worth 21

of the final ranking

The lack of indicators concerning final outcomes is interesting since most government-

sponsored performance-indicator regimes around the world are very much concerned

with such measures especially with respect to employment Possibly this indicates that

ranking systems simply do not view education outcomes as relevant measures of

educational quality Alternatively it may be the case that they simply have not found a

reliable indicator of outcomes or that there are reliable indicators but that there is so little

variation between institutions that it makes no sense to rank based on the data

10

F Indicators of Research

Many of the league tables covered in this survey include sections and weightings related

to universitiesrsquo research efforts It is in this field of measurement that we see the greatest

diversity of indicators Presumably this is because research inputs and outputs lend

themselves much more easily to measurement and manipulation than other areas of

institutional activity

Three studies include research staff as part of their ranking scheme La Repubblica at 952

the Melbourne Institute (4) and Wuhan (078)

Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used

method of looking at research quality but it is not universally admired because different

disciplines use different means to communicate major advances in knowledge (leading

scientists invariably produce large numbers of journal articles leading social scientists

may produce fewer journal articles but instead have one or two long important

monographsmdashsee Hicks 2004) There is also some concern among non-English speaking

countries that they are penalized in international rankings because so many of the major

journals (notably Science and Nature) are printed in English However the one set of

rankings that uses separate indicators to monitor articles published in English and articles

published in another language (the Wuhan rankings) shows that the two indicators are

positively correlated institutions that have more Chinese publications are also likely to

have more English publications and vice versa

Several sets of league tables measure bibliometric citations in various publication indices

The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a

weight of 20 of the final total Guangdong also monitors other Chinese universities

specifically for citations in engineering publications and weights this at 29 Moreover it

tacks on an additional 1049 for citations in science-oriented indices such as the Science

Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the

same category of scientific citations while the Melbourne Institute rates science citations at

68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social

Science Citation Index which does not include the humanities) are noted in only two

league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne

Institute (32) Another way of measuring research

11

impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a

weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the

Melbourne Institute

The complement of citations is of course publications Listing the number of publications

an individual group department or whole university releases can act as a weak substitute

for citationsmdashweak because simply publishing a paper or monograph is no guarantee that

the general public or other researchers will even glance at the work Guangdong gives an

1179 weighting to publications in science-oriented indices such as the Science Citation

Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same

indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong

even has a separate category just for measuring publications in Science and Nature

although it accords it an almost derisory weighting of 06 Under publications in social

science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their

final totals For publications in other indices (where the subject indices are

undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for

Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of

rankings to include research monographs weighted at 033

In countries where there are specific third-party evaluations of research output academic

quality of research is sometimes used as a research indicator The Times puts a very large

30 weight on this indicator while the Financial Times puts it at 119

Research awards are another handy third-party measurement of quality as the number of

international and national awards won by faculty andor graduates is often considered a

useful measure of institutional success International research awardsmdash specifically the

number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator

by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as

particularly suspect in some quarters given that the points are based on where the

recipient went to school rather than on where they are or were on the

8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the

most prominent publication indices

9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences

12

faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead

for nearly a century Wuhan uses similar measures but only accords them a weight of 14

National research awards are more common as a quality indicator used by La Repubblica

(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)

Financial indicators of research are also very common Research budgets as a factor in the

overall assessment of research in universities are covered by the Financial Times (9)

Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for

total amount of research expenditure unfortunately it is unclear precisely what this

research expenditure represents or how it is determined although it is clearly indicated

that it does not represent the total number of grants or projects at a university Total

number of research-based grants and projects is weighted by Education18 at 15 and

Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science

and engineering and another 55 to those for social sciences and humanities Similarly

the Melbourne Institute gives 6 of its overall weight to public-source grants making no

distinction between areas of study

In a slightly different vein Netbig (46) and Wuhan (278) both list the number of

research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica

(095) all weigh research-basedaffiliated research institutions or centres for studies

Finally one can also measure research not simply in terms of the amount of money it

generates but also in terms of the amount of future income it will generate Both

Guangdong (245) and Wuhan (193) measure the number of patents issued to

universities as a quality indicator

A final way of measuring an institutionrsquos research intensity is to look at the range of its

course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of

doctoral and Masterrsquos programs offered as a proxy for research intensity

As with physical and financial resources few if any of the research indicators are

normalized to account for institutional size (either by student or faculty numbers) In the

world of rankings bigger almost always means better an institution with 100 faculty with

ten citations apiece will always look worse than an institution with 1001 faculty

13

with one citation each To the extent that the raw production of knowledge matters this

form of measurement is acceptable To the extent that rankings are meant to show how

well institutions are doing on a like-to-like basis or to show the efficiency of universities it

is plainly inadequate This should be of particular concern to Chinese policy-makers

whose ranking systems are especially reliant on research-based indicators

G Indicators of Reputation

The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo

Those rankings systems which use the results of reputation surveys as an indicator do so as

an indirect measure of quality based on the assumption that the employers academics and

academic administrators surveyed have opinions of institutional quality that are informed

up-to-date and impartial While these assumptions are clearly open to debate they

nevertheless form an important basis for many ranking systems Another reason for using

reputation measures is the paucity of other data availablemdashsome countries have few

independent measures of teaching effectiveness university resources or output and

reputation can thus act as a useful surrogate Reputation rankings are often criticized as

simply quantifying the common ignorance of the people being surveyed However to the

extent that the people being surveyed hold positions which have the potential to affect

large numbers of young people and whose positions actually require some knowledge of

institutional quality (ie officials in charge of graduate admissions corporate recruiters

etc) then reputation rankings make sense because they provide useful information for

students about the perceived value of the degrees that they could obtain from various

universities

The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and

the Times which both accord reputation a weighting of 50 in their overall ranking

scheme Education18 assigns it almost as much significance at 40 The US News and World

Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just

below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)

and Netbig (15) The only other study to include reputation is Wuhan (117)

14

1 Please see Appendix B for a glossary of the various bodies referenced in this document

Appendix A Detailed Listing of Indicators and their Sources

Beginning Characteristics Indicator

Used By Source1

Incoming grades Macleans University

Percentage with grades above a set limit

Macleans University

US News and World Report

University

Performance on national standardised tests or benchmarks

Asiaweek University

Education18 3rd-party JUPAS

Financial Times Government agency 3rd-party UCAS

Guardian University Guide

Government agency 3rd-party UCAS

Melbourne Institute Government agency 3rd-party DEST

Netbig National entrance examination board

Times Good University Guide

Government agency 3rd-party UCAS

US News University

Wuhan Unknown presumed government 3rd-party

Student status La Repubblica Government agency 3rd-party MIUR

Admittance selectivity general

Asiaweek University

Admittance number of applications to places

Asiaweek University

Financial Times Government agency 3rd-party UCAS

La Repubblica Government agency 3rd-party MIUR

US News University

15

Beginning Characteristics Indicator

Used By Source1

Out-of-locality student percentage

Macleans University

International student percentages

Financial Times Government agency 3rd-party HESA

Macleans University

Shanghai Institute of Educational Science

Times World University

Wuhan Unknown presumed university

Undergraduate students among all students percentages

Netbig

Wuhan Unknown presumed university

Ethnic diversity in student body

Guardian University

Learning Inputs ndash Staff Indicator

Used By Source

Facultystudent ratio Asiaweek University

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Financial Times Government agency 3rd-party HESA

La Repubblica Government agency 3rd-party MIUR

Times Good University Guide

Government agency 3rd-party HESA

Times World University

US News University

Wuhan Unknown presumed university

16

Learning Inputs ndash Staff Indicator

Used By Source

Social science faculty student ratio

Melbourne Government agency 3rd-party DEST

Science faculty student ratio

Melbourne Government agency 3rd-party DEST

Administrative staff student ratio

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Staff student ratio (regardless of division)

Guardian Government agency 3rd-party HESA

Netbig University

Course per teacher La Repubblica Government agency 3rd-party MIUR

Per-teacher university spending

Asiaweek University

Faculty pay rates fortenured staff

Asiaweek University

US News University

Number of full-time part-time faculty

Netbig University

US News University

Wuhan Unknown presumed university

Faculty with researchprojects

Wuhan Unknown presumed university

Class size differentiation Macleans University

US News University

Classes taught by tenured faculty

Macleans University

Exchange programmes hosted

La Repubblica Government agency 3rd-party AgNaSoc

Number of classes lsquoactually taughtrsquo

La Repubblica Government agency 3rd-party MIUR

of international faculty (v faculty as a whole)

Times World University

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

5INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

InternationalisationUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

deacutepenses pour le secteur internationalisation input MEQnombre deacutetudiants en stage agrave leacutetrangernombre deacutetudiants temps complet input tes un cannombre deacutetudiants internationauxnombre deacutetudiants temps complet input tes un can Melbourne THESnombre deacutetudiants en cotutellesnombre deacutetudiant des cycles supeacuterieurs input CHEDAADnombre de professeurs inviteacutes internationaux processus tes un can CHEDAADprofesseurs inviteacutes agrave leacutetranger processus CHEDAADnombre deacutetudiants posdoctoraux processus tes un can CHEDAADproportion de cours de langues eacutetrangegraveres processusnombre de programmes deacutechange processus tes un can CHEDAADEntentes de partenariat outputPrix internationaux output G10 The Center Macleans Melbourne Republica Shanghai

RP gouvernance - 2008-06-23

6INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Inteacutegration des technologies de linformation USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Deacutepenses totales consacreacutees aux TIC input MEQpersonnel techniqueenseignant reacutegulier inputpersonnel techniqueeacutetudiant en eacutequivalence temps complet inputExistence dun reacuteseau informatique (accessible agrave linterne et agrave lexterne) outputaccegraves au reacuteseau avec fils et sans fils processusaccegraves aux ressources documentaires des bibliothegraveques outputService de courriel interne et externe en mode WEB outputNombre dordinateurs par professeur et par eacutetudiant outputNombre de cours en ligne processusnombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) outputAccegraves des professeurs aux banque de donneacutees et aux logiciels processus

Site WEB (eacutevaluation du site WEB) WEBOMETRICSSize (nombre de pages recenseacutees) outcomevisibilitty (consultation observeacutee selon le nobre de lien externe unique) outcomeRich files (intensiteacute de lactiviteacute acadeacutemique) outcomeScholar (activiteacute de recherche publications et citations) outcome

RP gouvernance - 2008-06-23

7INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

BibliothegravequesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre total de volumes et de documents audiovisuels et eacutelectroniques input G10 ARLacquisition de volumes (au cours de lanneacutee) input ARL Macleansnombre total de volumes eacutetudiants en eacutequivalence temps complet input G10 ARL Macleans Deacutepenses des bibliothegraveques input MEQ Macleans AsiaweekDeacutepenses des bibliothegravequesbudget de fonctionnement input ARLDeacutepenses des bibliothegravequeseacutetudiants en eacutequivalence temps complet input G10 ARLPersonnels reacuteguliers agrave temps complet input G10 ARLClassement outcome ARL

RP gouvernance - 2008-06-23

8INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Revenus et deacutepenses USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Produits et revenusSubvention du ministegravere de lEacuteducation du Queacutebec input MEQ subvMEQrevenus de fonctionnement input G10 MacleansDroits de scolariteacute input G10 droits de scolariteacute revenus de fonctionnement input G10autres produits input G10Total des revenus de fonctionnement input G10 MEQ Guardian revenus de fonctionnement eacutetudiants en eacutequivalence temps complet input CAUBO USNews MacleansRevenus de recherche input G10encaissement par nature (dons immobilisations chaires bourses) input G10 fonds de dotation input G10 CenterContribution des alumni aux encaissements et fonds de dotation input USNews Macleans fonds de dotation eacutetudiants en eacutequivalence temps complet input G10Total des produits G10

Charges et deacutepenses G10salaire des professeurs input G10 MEQsalaire des chargeacutes de cours input G10salaires des auxiliaires input MEQsalaires des auxiliaires input MEQsalaire des personnels (staff) input G10autres deacutepenses input G10total des deacutepenses input G10 deacutepenses denseignement deacutepenses totales de fonctionnement input UBC MEQ

Chargesproduits processusEacutequilibre budgeacutetaire revenus et deacutepenses de fonctionnement processus MEQ

RP gouvernance - 2008-06-23

9INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

EspacesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Nombre de megravetres carreacutes bruts input G10MCB eacutetudiants en eacutequivalence temps complet input G10megravetres carreacutes nets alloueacutes input MEQNombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input Tbesoin despace en megravetres carreacutes nets inputExistence dun plan directeur des espaces inputBudget preacutevu au plan directeur des espaces input TBudget alloueacute aux espaces input MEQdeacutepenses de fonctionnement nombre de megravetres carreacutes bruts input G10

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Environnement USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

consommation de leacutenergie (exprimeacutee en BTU processus universiteacutes canadiennes et ameacutericaines BTUpopulation totale de luniversiteacute processus voir en particulier Penn State et Umich de leacutenergie produit par des sources renouvelable processusUtilisation de leau processuslitres deau par personne processusGaz agrave effet de serre (en tonnes meacutetriques CO2 eacutequivalents) processusMT CO2 par personne processusDeacutechets solides (en tonnes) processusTonnes de deacutechets solides per capita processus de deacutechets solides recycleacutes processus

RP gouvernance - 2008-06-23

10INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

ReacuteputationUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

enquecircte meneacutee aupregraves des eacutetudiants outcome NSSE Melbourneenquecircte meneacutee aupregraves des diplocircmeacutes outcomeenquecircte meneacutee aupregraves des pairs outcome USNews Asiaweek THESenquecircte meneacutee aupregraves des employeurs outcome Macleans Melbourneenquecircte meneacutee aupregraves des stakeholders outcome Macleans Melbourneenquecircte meneacutee aupregraves du monde des affaires outcome Macleans Melbournecouverture meacutediatique outcome AT

RP gouvernance - 2008-06-23

11INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Planification et gestion strateacutegiquesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Pouvoirs des corps universitaire processus un Canliens dynamiques entre Conseil Assembleacutee universitaire commission des eacutetudes processus un Canprocessus formel de planification strateacutegique processus un Canproduction de donneacutees et indicateurs processus un Canprocessus deacutevaluation des faculteacutes deacutepartements et services processus un Canrevue systeacutematique des responsabiliteacutes et efficaciteacute des comiteacutes processus un Candeacutepenses de ladministration centrale deacutepenses de fonctionnement processus un Can

RP gouvernance - 2008-06-23

12INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Eacuteducation permanenteUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre de programmes offerts input un Cannombre deacutetudiants inscrits input un CanRevenus et deacutepenses du service input un Can

RP gouvernance - 2008-06-23

13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes

REacuteFEacuteRENCES

Aperccedilu densemble des principaux indicateurs

GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost

Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes

MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf

Indicateurs utiliseacutes dans les classements internationaux

Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml

RP gouvernance - 2008-06-23

ANNEXES

Quelques exemples de listes drsquoindicateurs

1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)

Voir page suivante

Preacutepareacute par Roland Proulx

Consultant en planification institutionnelle et veille strateacutegique

Montreacuteal mai 2008

Performance Indicators

Discussion Paper

May 2004 (Prepared by the Office of Planning amp Analysis)

Table of Contents

Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25

2

Table 1 ndash Number of Indictors by Type of Data Source

Table 1 shows that surveys are the least frequently used source of data for indicators

Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a

plurality of indicators come from this source This measure somewhat underestimates the

importance of surveys however as it does not account for the weighting given to each

indicator in each study In the THES World Rankings for instance there is only a single

survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos

Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one

indicator is worth 20 of the final score

Raw

indIcator count

Surve

y data

Third parties

Universities

AsiaweekmdashAsias Best Universities 18 - - 18

Daily Telegraph (2003) 1 - 1 -

Education18com 9 3 4 2

Excelencia 2001 71 - 71 -

Financial Times (2003) 17 - 17 -

Guangdong Institute of Management Science 17 - 14 3

GuardianmdashUniversity Guide 2005 7 - 2 5

La Repubblica 23 2 21 -

Macleans University Rankings 24 1 5 18

Melbourne Institutemdash International Standing of Australian Universities

26 3 23 -

Netbig 2004 18 1 10 7

Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15

Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities

6 - 5 1

The TimesmdashGood University Guide 2005 9 - 9 -

Times Higher Education SupplementmdashWorld University Rankings

5 1 1 3

US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11

Washington MonthlymdashCollege Rankings 2005 8 - 1 7

Wuhan University Centre for Science Evaluation 45 2 22 21

3

A Indicators of Beginning Characteristics

ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of

students at the time they begin their studies Fourteen of the 18 rankings examined in this

study use one or more indicators of the beginning characteristics of students to arrive at

their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on

these factors with 33 of the total ranking coming from this class of indicators but the

Guardian Education18 Asiaweek and the two other North American surveys also place

considerable emphasis on this category

There are six main indicators used to determine which institutions have students with

positive ldquobeginning characteristicsrdquo

The most common measure of beginning characteristics is performance on national

standardized tests with nine surveys using this as a measure Education18 and the

Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used

by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)

Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this

data is collected and standardized by national bodies it has the benefit of being seen as a

relatively impartial method of determining the relative ldquostrengthrdquo of the students entering

each institution Institutionsrsquo results can be scored by showing either averages or the

percentage of entering students meeting a particular standard

Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses

secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This

is a second-best solution made necessary by the absence of any national standardized test

in Canada (or indeed of any provincial standardized tests at the end of secondary school

in provinces other than Alberta) The lack of national standardization makes this an

undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is

truly equivalent to an ldquoArdquo in another jurisdiction

Another measure of the strength of the student body is the percentage of incoming

students receiving (third-party) scholarships which is worth 11 of the score in the

Wuhan survey One can also approach the issue by measuring institutional selectivity In

effect this method infers the strength of the student body by the proportion of

4

applicants rejected the theory being that the higher the number of rejected applicants the

stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)

Normally this measure is expressed as a straight ratio of acceptances to applications but it

can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the

greatest weight on this measure) as a ratio of enrolments to applications Within the US

there is some dispute as to what constitutes an offer of admission and whether or not late

admissions are included as noted by Ganeshananthan (2003)

Student bodies are often considered to be strong if the school is able to attract a large

number of international or out-of-district students or if they contain people from diverse

ethnic backgrounds A number of league tables use the international student indicator

(which like the selectivity indicator is arguably as much an indicator of prestige and

reputation as it is of student characteristics) although in no case does this indicator

account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a

quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data

on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian

puts a somewhat larger emphasis on this indicator in comparison to other league tables

which use similar variables

A very different take on this idea is present in the Washington Monthly which released its

first set of College Rankings in September 2005 With the declared aim of using an

institutionrsquos commitment to social mobility as a measure of quality it uses the percentage

of students from low-income backgrounds as an indicator (with percentage of students

receiving need-based government (Pell) grants used as a proxy)

Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy

statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the

percentage of the student population who are graduate students (arguably this is a

research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the

number of graduate students auditing classes is used as an indicator the assumption is

presumably that if people are auditing then the classes must be very attractive The Italian

La Repubblica ranks an institution according to the number of part-time students it has

contrary to prevailing North American views on the undesirability of part-time study the

Italian rankings see higher numbers of part-time students in a positive light as it is

evidence that an institution is becoming less rigid in its timetabling and

5

permitting students to juggle both work and study something which was nearly

impossible in that country just a few years ago

The Washington Monthly also has a final category of indicators which reflect studentsrsquo

beginning characteristics namely their likelihood of performing community service as

measured by the percentage of students in the US Peace Corps and Reserve Officer

Training Corps (ROTC) and the percentage of students involved in work-study in the

community3 Together these three indicators account for 33 of an institutionrsquos total

ranking

B Indicators of Learning InputsmdashStaff

Generally both the quantity and quality of staff are positively correlated with institutional

quality The problem of course is finding useful metrics for each of these factors

especially if one excludes as we have done here measures of research performance and

research intensity putting them in a separate category4

The simplest measure is simply the number of faculty unadjusted for things like size of

student body Most national league tables however prefer to use variations on the

concept of facultystudent ratio Others try to measure teaching intensity with measures

such as courses per teacher or hours spent in class per student (both in La Repubblica)

These kinds of measures usually account for between 2-5 of the final rankings although

in some cases (ie the Guardian) this figure can be as high as 20

Another important way of measuring how faculty resources are deployed is the measure

of average class size which is used only by Macleanrsquos and the US News and World Report

Ostensibly the reason for measuring class size is to account in some form for the degree

3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category

6

of attention devoted to actually teaching students Despite the fact that important research

(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level

the use of this indicator appears to be a spillover from the North American debates on

class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein

2000) Regardless of why the indicators are used they are extraordinarily important to

these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and

World Reportrsquos rankings respectively

A number of ranking systems try to look at staff qualifications such as the number of

PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the

Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further

than other surveys and actually looks at the proportion of classes taught by tenure-track

staff Others (ie THES) look at the number of foreign faculty based on the assumption

that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still

others (ie La Repubblica) look at the age structure of the faculty Another proxy for

institutional quality is the pay rates for tenured staff on the assumption that institutions

with higher rates of pay on average attract better faculty this measure has been used both

by the US News and World Report and Asiaweek

Finally a number of league tables rank faculty inputs on the basis of standardized third-

party evaluations 6

Education18 the Financial Times the Times the Guardian and the US

News and World Report league tables all use some sort of ranking criterion based at least in

part on this indicator or variations thereof

C Indicators of Learning InputsmdashResources

Resource inputsmdashcrudely the amount of current dollars equipment and books available

to students at an institutionmdashare widely considered an important measure of quality Yet

despite the apparent simplicity of counting dollars and measuring assets

5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each

university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future

7

the means by which institutional wealth is measured varies considerably between ranking

systems

There are a number of revenue-based measures of resources Macleanrsquos uses public funding

of institutional budgets as a factor in its analysis conversely the Financial Times uses the

private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of

total score) and the US News and World Report (5 of total score) also measure alumni

financial support as a measure of quality

For reasons that are not entirely clear league tables tend to favour measures of

expenditures rather than revenues The Guardian looks at total institutional expenditures

as an indicator Institutional expenditure on student services is used as a measure of

institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total

institutional scores respectively) Rzezspospolita does not measure student services

expenditures directly but does measure student services outputs such as number of

student athletes and number of study clubs which amounts to more or less the same thing

Macleanrsquos also gives out 433 of its total score based on institutional expenditures on

scholarships and bursaries

Various aspects of physical infrastructure are also used as measures of institutional

resources most directly in the case of La Repubblica which bases 317 of its total rank on

the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding

assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type

of physical infrastructure measured is available Internet bandwidth which was used by

Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth

roughly 3 of the total score

By some distance the infrastructure indicators most favoured by the compilers of league

tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on

this with 12 of the total quality mark being taken from various types of library

infrastructure measurements (including acquisitions per year total volumes average

number of volumes per student and yearly library expenditure outside of acquisitions)

Netbig and Education18 also use library volume holdings while Asiaweek the Financial

Times and the Times also use measures of library expenditures outside of acquisitions or

computerization of library resources as measures of institutional quality

8

One important factor to note is that most ranking systems do not normalize their resource

and infrastructure measures That is to say it is raw spending power or simple size of

assets that is usually measured rather than spending per studentprofessor or assets per

studentprofessor As a result a number of these rankings systems have inbuilt biases

towards larger institutions

D Indicators of Learning Outputs

Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge

learned over the course of a baccalaureate degreemdashshould be a basic indicator of

institutional quality Unfortunately good means of measuring these outputsmdashlike the

National Survey of Student Engagement (NSSE) and the College Learning Assessment

(CLA)mdashhave only recently become available and for the most part institutions are still

keeping their scores secret Outside of these measures only a few very crude indicators are

available which likely explains why learning outputs do not feature especially prominently

in most ranking schemes

The simplest types of measures of learning outputs are those linked to graduation and

retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong

and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7

the latter three also use rates of graduation from Masterrsquos programs as indicators In some

cases the weights on these measures can be very highmdashin the Guangdong rankings

graduation rates account for over 50 of the rankingmdashbut in most cases the weights are

10 or less Retention rates commonly meaning the progression rate of first-year students

into second year are accorded less importance The US News and World Report Macleanrsquos

the Melbourne Institute and La Repubblica all employ retention measures as indicators but

none of them are worth more than 4 of total weighting Two publications make specific

indicators for retention and graduation of international students Macleanrsquos (graduation

rates of international students) and the Melbourne Institute (retention rates of international

students) The Washington Monthly looks specifically at institutional retention rates

adjusted for the participation of lower-income students and gives higher scores to

institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores

and number of Pell Grant recipients the US

7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18

9

News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos

ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)

E Indicators of Final Outcomes

Final outcomes are indications of generalized outcomes for students after graduation

Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness

good citizenship) but given the somewhat utilitarian justifications for education that are

currently in fashion (see Wolf 2000) employment outcomes are the most commonly used

measure of final outcomes These are given particular emphasis by the Guardian (where

employment outcomes are worth 17 of the total score) but are also used by the Financial

Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the

Times are interestingly not concerned with employment per se but with ldquoemployment in

an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher

Education Statistics Agency (HESA) uses Standard Occupational Classifications to

measure the proportion of graduates in professional or white-collar jobs anyone not in

such a job is considered not to be working in an area related to their studies (it is unclear

what methodology is used by the Financial Times and the Times although we suspect their

methods are broadly similar)

The only other measure of final outcomes in use is percentage of graduates returning for

additional education which is an indicator used by both the Melbourne Institute and the

Financial Times This is a particularly important indicator for the latter as it is worth 21

of the final ranking

The lack of indicators concerning final outcomes is interesting since most government-

sponsored performance-indicator regimes around the world are very much concerned

with such measures especially with respect to employment Possibly this indicates that

ranking systems simply do not view education outcomes as relevant measures of

educational quality Alternatively it may be the case that they simply have not found a

reliable indicator of outcomes or that there are reliable indicators but that there is so little

variation between institutions that it makes no sense to rank based on the data

10

F Indicators of Research

Many of the league tables covered in this survey include sections and weightings related

to universitiesrsquo research efforts It is in this field of measurement that we see the greatest

diversity of indicators Presumably this is because research inputs and outputs lend

themselves much more easily to measurement and manipulation than other areas of

institutional activity

Three studies include research staff as part of their ranking scheme La Repubblica at 952

the Melbourne Institute (4) and Wuhan (078)

Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used

method of looking at research quality but it is not universally admired because different

disciplines use different means to communicate major advances in knowledge (leading

scientists invariably produce large numbers of journal articles leading social scientists

may produce fewer journal articles but instead have one or two long important

monographsmdashsee Hicks 2004) There is also some concern among non-English speaking

countries that they are penalized in international rankings because so many of the major

journals (notably Science and Nature) are printed in English However the one set of

rankings that uses separate indicators to monitor articles published in English and articles

published in another language (the Wuhan rankings) shows that the two indicators are

positively correlated institutions that have more Chinese publications are also likely to

have more English publications and vice versa

Several sets of league tables measure bibliometric citations in various publication indices

The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a

weight of 20 of the final total Guangdong also monitors other Chinese universities

specifically for citations in engineering publications and weights this at 29 Moreover it

tacks on an additional 1049 for citations in science-oriented indices such as the Science

Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the

same category of scientific citations while the Melbourne Institute rates science citations at

68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social

Science Citation Index which does not include the humanities) are noted in only two

league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne

Institute (32) Another way of measuring research

11

impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a

weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the

Melbourne Institute

The complement of citations is of course publications Listing the number of publications

an individual group department or whole university releases can act as a weak substitute

for citationsmdashweak because simply publishing a paper or monograph is no guarantee that

the general public or other researchers will even glance at the work Guangdong gives an

1179 weighting to publications in science-oriented indices such as the Science Citation

Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same

indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong

even has a separate category just for measuring publications in Science and Nature

although it accords it an almost derisory weighting of 06 Under publications in social

science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their

final totals For publications in other indices (where the subject indices are

undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for

Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of

rankings to include research monographs weighted at 033

In countries where there are specific third-party evaluations of research output academic

quality of research is sometimes used as a research indicator The Times puts a very large

30 weight on this indicator while the Financial Times puts it at 119

Research awards are another handy third-party measurement of quality as the number of

international and national awards won by faculty andor graduates is often considered a

useful measure of institutional success International research awardsmdash specifically the

number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator

by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as

particularly suspect in some quarters given that the points are based on where the

recipient went to school rather than on where they are or were on the

8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the

most prominent publication indices

9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences

12

faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead

for nearly a century Wuhan uses similar measures but only accords them a weight of 14

National research awards are more common as a quality indicator used by La Repubblica

(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)

Financial indicators of research are also very common Research budgets as a factor in the

overall assessment of research in universities are covered by the Financial Times (9)

Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for

total amount of research expenditure unfortunately it is unclear precisely what this

research expenditure represents or how it is determined although it is clearly indicated

that it does not represent the total number of grants or projects at a university Total

number of research-based grants and projects is weighted by Education18 at 15 and

Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science

and engineering and another 55 to those for social sciences and humanities Similarly

the Melbourne Institute gives 6 of its overall weight to public-source grants making no

distinction between areas of study

In a slightly different vein Netbig (46) and Wuhan (278) both list the number of

research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica

(095) all weigh research-basedaffiliated research institutions or centres for studies

Finally one can also measure research not simply in terms of the amount of money it

generates but also in terms of the amount of future income it will generate Both

Guangdong (245) and Wuhan (193) measure the number of patents issued to

universities as a quality indicator

A final way of measuring an institutionrsquos research intensity is to look at the range of its

course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of

doctoral and Masterrsquos programs offered as a proxy for research intensity

As with physical and financial resources few if any of the research indicators are

normalized to account for institutional size (either by student or faculty numbers) In the

world of rankings bigger almost always means better an institution with 100 faculty with

ten citations apiece will always look worse than an institution with 1001 faculty

13

with one citation each To the extent that the raw production of knowledge matters this

form of measurement is acceptable To the extent that rankings are meant to show how

well institutions are doing on a like-to-like basis or to show the efficiency of universities it

is plainly inadequate This should be of particular concern to Chinese policy-makers

whose ranking systems are especially reliant on research-based indicators

G Indicators of Reputation

The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo

Those rankings systems which use the results of reputation surveys as an indicator do so as

an indirect measure of quality based on the assumption that the employers academics and

academic administrators surveyed have opinions of institutional quality that are informed

up-to-date and impartial While these assumptions are clearly open to debate they

nevertheless form an important basis for many ranking systems Another reason for using

reputation measures is the paucity of other data availablemdashsome countries have few

independent measures of teaching effectiveness university resources or output and

reputation can thus act as a useful surrogate Reputation rankings are often criticized as

simply quantifying the common ignorance of the people being surveyed However to the

extent that the people being surveyed hold positions which have the potential to affect

large numbers of young people and whose positions actually require some knowledge of

institutional quality (ie officials in charge of graduate admissions corporate recruiters

etc) then reputation rankings make sense because they provide useful information for

students about the perceived value of the degrees that they could obtain from various

universities

The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and

the Times which both accord reputation a weighting of 50 in their overall ranking

scheme Education18 assigns it almost as much significance at 40 The US News and World

Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just

below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)

and Netbig (15) The only other study to include reputation is Wuhan (117)

14

1 Please see Appendix B for a glossary of the various bodies referenced in this document

Appendix A Detailed Listing of Indicators and their Sources

Beginning Characteristics Indicator

Used By Source1

Incoming grades Macleans University

Percentage with grades above a set limit

Macleans University

US News and World Report

University

Performance on national standardised tests or benchmarks

Asiaweek University

Education18 3rd-party JUPAS

Financial Times Government agency 3rd-party UCAS

Guardian University Guide

Government agency 3rd-party UCAS

Melbourne Institute Government agency 3rd-party DEST

Netbig National entrance examination board

Times Good University Guide

Government agency 3rd-party UCAS

US News University

Wuhan Unknown presumed government 3rd-party

Student status La Repubblica Government agency 3rd-party MIUR

Admittance selectivity general

Asiaweek University

Admittance number of applications to places

Asiaweek University

Financial Times Government agency 3rd-party UCAS

La Repubblica Government agency 3rd-party MIUR

US News University

15

Beginning Characteristics Indicator

Used By Source1

Out-of-locality student percentage

Macleans University

International student percentages

Financial Times Government agency 3rd-party HESA

Macleans University

Shanghai Institute of Educational Science

Times World University

Wuhan Unknown presumed university

Undergraduate students among all students percentages

Netbig

Wuhan Unknown presumed university

Ethnic diversity in student body

Guardian University

Learning Inputs ndash Staff Indicator

Used By Source

Facultystudent ratio Asiaweek University

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Financial Times Government agency 3rd-party HESA

La Repubblica Government agency 3rd-party MIUR

Times Good University Guide

Government agency 3rd-party HESA

Times World University

US News University

Wuhan Unknown presumed university

16

Learning Inputs ndash Staff Indicator

Used By Source

Social science faculty student ratio

Melbourne Government agency 3rd-party DEST

Science faculty student ratio

Melbourne Government agency 3rd-party DEST

Administrative staff student ratio

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Staff student ratio (regardless of division)

Guardian Government agency 3rd-party HESA

Netbig University

Course per teacher La Repubblica Government agency 3rd-party MIUR

Per-teacher university spending

Asiaweek University

Faculty pay rates fortenured staff

Asiaweek University

US News University

Number of full-time part-time faculty

Netbig University

US News University

Wuhan Unknown presumed university

Faculty with researchprojects

Wuhan Unknown presumed university

Class size differentiation Macleans University

US News University

Classes taught by tenured faculty

Macleans University

Exchange programmes hosted

La Repubblica Government agency 3rd-party AgNaSoc

Number of classes lsquoactually taughtrsquo

La Repubblica Government agency 3rd-party MIUR

of international faculty (v faculty as a whole)

Times World University

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

6INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Inteacutegration des technologies de linformation USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Deacutepenses totales consacreacutees aux TIC input MEQpersonnel techniqueenseignant reacutegulier inputpersonnel techniqueeacutetudiant en eacutequivalence temps complet inputExistence dun reacuteseau informatique (accessible agrave linterne et agrave lexterne) outputaccegraves au reacuteseau avec fils et sans fils processusaccegraves aux ressources documentaires des bibliothegraveques outputService de courriel interne et externe en mode WEB outputNombre dordinateurs par professeur et par eacutetudiant outputNombre de cours en ligne processusnombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) outputAccegraves des professeurs aux banque de donneacutees et aux logiciels processus

Site WEB (eacutevaluation du site WEB) WEBOMETRICSSize (nombre de pages recenseacutees) outcomevisibilitty (consultation observeacutee selon le nobre de lien externe unique) outcomeRich files (intensiteacute de lactiviteacute acadeacutemique) outcomeScholar (activiteacute de recherche publications et citations) outcome

RP gouvernance - 2008-06-23

7INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

BibliothegravequesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre total de volumes et de documents audiovisuels et eacutelectroniques input G10 ARLacquisition de volumes (au cours de lanneacutee) input ARL Macleansnombre total de volumes eacutetudiants en eacutequivalence temps complet input G10 ARL Macleans Deacutepenses des bibliothegraveques input MEQ Macleans AsiaweekDeacutepenses des bibliothegravequesbudget de fonctionnement input ARLDeacutepenses des bibliothegravequeseacutetudiants en eacutequivalence temps complet input G10 ARLPersonnels reacuteguliers agrave temps complet input G10 ARLClassement outcome ARL

RP gouvernance - 2008-06-23

8INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Revenus et deacutepenses USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Produits et revenusSubvention du ministegravere de lEacuteducation du Queacutebec input MEQ subvMEQrevenus de fonctionnement input G10 MacleansDroits de scolariteacute input G10 droits de scolariteacute revenus de fonctionnement input G10autres produits input G10Total des revenus de fonctionnement input G10 MEQ Guardian revenus de fonctionnement eacutetudiants en eacutequivalence temps complet input CAUBO USNews MacleansRevenus de recherche input G10encaissement par nature (dons immobilisations chaires bourses) input G10 fonds de dotation input G10 CenterContribution des alumni aux encaissements et fonds de dotation input USNews Macleans fonds de dotation eacutetudiants en eacutequivalence temps complet input G10Total des produits G10

Charges et deacutepenses G10salaire des professeurs input G10 MEQsalaire des chargeacutes de cours input G10salaires des auxiliaires input MEQsalaires des auxiliaires input MEQsalaire des personnels (staff) input G10autres deacutepenses input G10total des deacutepenses input G10 deacutepenses denseignement deacutepenses totales de fonctionnement input UBC MEQ

Chargesproduits processusEacutequilibre budgeacutetaire revenus et deacutepenses de fonctionnement processus MEQ

RP gouvernance - 2008-06-23

9INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

EspacesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Nombre de megravetres carreacutes bruts input G10MCB eacutetudiants en eacutequivalence temps complet input G10megravetres carreacutes nets alloueacutes input MEQNombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input Tbesoin despace en megravetres carreacutes nets inputExistence dun plan directeur des espaces inputBudget preacutevu au plan directeur des espaces input TBudget alloueacute aux espaces input MEQdeacutepenses de fonctionnement nombre de megravetres carreacutes bruts input G10

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Environnement USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

consommation de leacutenergie (exprimeacutee en BTU processus universiteacutes canadiennes et ameacutericaines BTUpopulation totale de luniversiteacute processus voir en particulier Penn State et Umich de leacutenergie produit par des sources renouvelable processusUtilisation de leau processuslitres deau par personne processusGaz agrave effet de serre (en tonnes meacutetriques CO2 eacutequivalents) processusMT CO2 par personne processusDeacutechets solides (en tonnes) processusTonnes de deacutechets solides per capita processus de deacutechets solides recycleacutes processus

RP gouvernance - 2008-06-23

10INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

ReacuteputationUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

enquecircte meneacutee aupregraves des eacutetudiants outcome NSSE Melbourneenquecircte meneacutee aupregraves des diplocircmeacutes outcomeenquecircte meneacutee aupregraves des pairs outcome USNews Asiaweek THESenquecircte meneacutee aupregraves des employeurs outcome Macleans Melbourneenquecircte meneacutee aupregraves des stakeholders outcome Macleans Melbourneenquecircte meneacutee aupregraves du monde des affaires outcome Macleans Melbournecouverture meacutediatique outcome AT

RP gouvernance - 2008-06-23

11INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Planification et gestion strateacutegiquesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Pouvoirs des corps universitaire processus un Canliens dynamiques entre Conseil Assembleacutee universitaire commission des eacutetudes processus un Canprocessus formel de planification strateacutegique processus un Canproduction de donneacutees et indicateurs processus un Canprocessus deacutevaluation des faculteacutes deacutepartements et services processus un Canrevue systeacutematique des responsabiliteacutes et efficaciteacute des comiteacutes processus un Candeacutepenses de ladministration centrale deacutepenses de fonctionnement processus un Can

RP gouvernance - 2008-06-23

12INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Eacuteducation permanenteUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre de programmes offerts input un Cannombre deacutetudiants inscrits input un CanRevenus et deacutepenses du service input un Can

RP gouvernance - 2008-06-23

13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes

REacuteFEacuteRENCES

Aperccedilu densemble des principaux indicateurs

GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost

Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes

MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf

Indicateurs utiliseacutes dans les classements internationaux

Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml

RP gouvernance - 2008-06-23

ANNEXES

Quelques exemples de listes drsquoindicateurs

1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)

Voir page suivante

Preacutepareacute par Roland Proulx

Consultant en planification institutionnelle et veille strateacutegique

Montreacuteal mai 2008

Performance Indicators

Discussion Paper

May 2004 (Prepared by the Office of Planning amp Analysis)

Table of Contents

Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25

2

Table 1 ndash Number of Indictors by Type of Data Source

Table 1 shows that surveys are the least frequently used source of data for indicators

Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a

plurality of indicators come from this source This measure somewhat underestimates the

importance of surveys however as it does not account for the weighting given to each

indicator in each study In the THES World Rankings for instance there is only a single

survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos

Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one

indicator is worth 20 of the final score

Raw

indIcator count

Surve

y data

Third parties

Universities

AsiaweekmdashAsias Best Universities 18 - - 18

Daily Telegraph (2003) 1 - 1 -

Education18com 9 3 4 2

Excelencia 2001 71 - 71 -

Financial Times (2003) 17 - 17 -

Guangdong Institute of Management Science 17 - 14 3

GuardianmdashUniversity Guide 2005 7 - 2 5

La Repubblica 23 2 21 -

Macleans University Rankings 24 1 5 18

Melbourne Institutemdash International Standing of Australian Universities

26 3 23 -

Netbig 2004 18 1 10 7

Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15

Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities

6 - 5 1

The TimesmdashGood University Guide 2005 9 - 9 -

Times Higher Education SupplementmdashWorld University Rankings

5 1 1 3

US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11

Washington MonthlymdashCollege Rankings 2005 8 - 1 7

Wuhan University Centre for Science Evaluation 45 2 22 21

3

A Indicators of Beginning Characteristics

ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of

students at the time they begin their studies Fourteen of the 18 rankings examined in this

study use one or more indicators of the beginning characteristics of students to arrive at

their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on

these factors with 33 of the total ranking coming from this class of indicators but the

Guardian Education18 Asiaweek and the two other North American surveys also place

considerable emphasis on this category

There are six main indicators used to determine which institutions have students with

positive ldquobeginning characteristicsrdquo

The most common measure of beginning characteristics is performance on national

standardized tests with nine surveys using this as a measure Education18 and the

Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used

by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)

Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this

data is collected and standardized by national bodies it has the benefit of being seen as a

relatively impartial method of determining the relative ldquostrengthrdquo of the students entering

each institution Institutionsrsquo results can be scored by showing either averages or the

percentage of entering students meeting a particular standard

Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses

secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This

is a second-best solution made necessary by the absence of any national standardized test

in Canada (or indeed of any provincial standardized tests at the end of secondary school

in provinces other than Alberta) The lack of national standardization makes this an

undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is

truly equivalent to an ldquoArdquo in another jurisdiction

Another measure of the strength of the student body is the percentage of incoming

students receiving (third-party) scholarships which is worth 11 of the score in the

Wuhan survey One can also approach the issue by measuring institutional selectivity In

effect this method infers the strength of the student body by the proportion of

4

applicants rejected the theory being that the higher the number of rejected applicants the

stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)

Normally this measure is expressed as a straight ratio of acceptances to applications but it

can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the

greatest weight on this measure) as a ratio of enrolments to applications Within the US

there is some dispute as to what constitutes an offer of admission and whether or not late

admissions are included as noted by Ganeshananthan (2003)

Student bodies are often considered to be strong if the school is able to attract a large

number of international or out-of-district students or if they contain people from diverse

ethnic backgrounds A number of league tables use the international student indicator

(which like the selectivity indicator is arguably as much an indicator of prestige and

reputation as it is of student characteristics) although in no case does this indicator

account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a

quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data

on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian

puts a somewhat larger emphasis on this indicator in comparison to other league tables

which use similar variables

A very different take on this idea is present in the Washington Monthly which released its

first set of College Rankings in September 2005 With the declared aim of using an

institutionrsquos commitment to social mobility as a measure of quality it uses the percentage

of students from low-income backgrounds as an indicator (with percentage of students

receiving need-based government (Pell) grants used as a proxy)

Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy

statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the

percentage of the student population who are graduate students (arguably this is a

research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the

number of graduate students auditing classes is used as an indicator the assumption is

presumably that if people are auditing then the classes must be very attractive The Italian

La Repubblica ranks an institution according to the number of part-time students it has

contrary to prevailing North American views on the undesirability of part-time study the

Italian rankings see higher numbers of part-time students in a positive light as it is

evidence that an institution is becoming less rigid in its timetabling and

5

permitting students to juggle both work and study something which was nearly

impossible in that country just a few years ago

The Washington Monthly also has a final category of indicators which reflect studentsrsquo

beginning characteristics namely their likelihood of performing community service as

measured by the percentage of students in the US Peace Corps and Reserve Officer

Training Corps (ROTC) and the percentage of students involved in work-study in the

community3 Together these three indicators account for 33 of an institutionrsquos total

ranking

B Indicators of Learning InputsmdashStaff

Generally both the quantity and quality of staff are positively correlated with institutional

quality The problem of course is finding useful metrics for each of these factors

especially if one excludes as we have done here measures of research performance and

research intensity putting them in a separate category4

The simplest measure is simply the number of faculty unadjusted for things like size of

student body Most national league tables however prefer to use variations on the

concept of facultystudent ratio Others try to measure teaching intensity with measures

such as courses per teacher or hours spent in class per student (both in La Repubblica)

These kinds of measures usually account for between 2-5 of the final rankings although

in some cases (ie the Guardian) this figure can be as high as 20

Another important way of measuring how faculty resources are deployed is the measure

of average class size which is used only by Macleanrsquos and the US News and World Report

Ostensibly the reason for measuring class size is to account in some form for the degree

3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category

6

of attention devoted to actually teaching students Despite the fact that important research

(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level

the use of this indicator appears to be a spillover from the North American debates on

class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein

2000) Regardless of why the indicators are used they are extraordinarily important to

these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and

World Reportrsquos rankings respectively

A number of ranking systems try to look at staff qualifications such as the number of

PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the

Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further

than other surveys and actually looks at the proportion of classes taught by tenure-track

staff Others (ie THES) look at the number of foreign faculty based on the assumption

that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still

others (ie La Repubblica) look at the age structure of the faculty Another proxy for

institutional quality is the pay rates for tenured staff on the assumption that institutions

with higher rates of pay on average attract better faculty this measure has been used both

by the US News and World Report and Asiaweek

Finally a number of league tables rank faculty inputs on the basis of standardized third-

party evaluations 6

Education18 the Financial Times the Times the Guardian and the US

News and World Report league tables all use some sort of ranking criterion based at least in

part on this indicator or variations thereof

C Indicators of Learning InputsmdashResources

Resource inputsmdashcrudely the amount of current dollars equipment and books available

to students at an institutionmdashare widely considered an important measure of quality Yet

despite the apparent simplicity of counting dollars and measuring assets

5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each

university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future

7

the means by which institutional wealth is measured varies considerably between ranking

systems

There are a number of revenue-based measures of resources Macleanrsquos uses public funding

of institutional budgets as a factor in its analysis conversely the Financial Times uses the

private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of

total score) and the US News and World Report (5 of total score) also measure alumni

financial support as a measure of quality

For reasons that are not entirely clear league tables tend to favour measures of

expenditures rather than revenues The Guardian looks at total institutional expenditures

as an indicator Institutional expenditure on student services is used as a measure of

institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total

institutional scores respectively) Rzezspospolita does not measure student services

expenditures directly but does measure student services outputs such as number of

student athletes and number of study clubs which amounts to more or less the same thing

Macleanrsquos also gives out 433 of its total score based on institutional expenditures on

scholarships and bursaries

Various aspects of physical infrastructure are also used as measures of institutional

resources most directly in the case of La Repubblica which bases 317 of its total rank on

the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding

assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type

of physical infrastructure measured is available Internet bandwidth which was used by

Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth

roughly 3 of the total score

By some distance the infrastructure indicators most favoured by the compilers of league

tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on

this with 12 of the total quality mark being taken from various types of library

infrastructure measurements (including acquisitions per year total volumes average

number of volumes per student and yearly library expenditure outside of acquisitions)

Netbig and Education18 also use library volume holdings while Asiaweek the Financial

Times and the Times also use measures of library expenditures outside of acquisitions or

computerization of library resources as measures of institutional quality

8

One important factor to note is that most ranking systems do not normalize their resource

and infrastructure measures That is to say it is raw spending power or simple size of

assets that is usually measured rather than spending per studentprofessor or assets per

studentprofessor As a result a number of these rankings systems have inbuilt biases

towards larger institutions

D Indicators of Learning Outputs

Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge

learned over the course of a baccalaureate degreemdashshould be a basic indicator of

institutional quality Unfortunately good means of measuring these outputsmdashlike the

National Survey of Student Engagement (NSSE) and the College Learning Assessment

(CLA)mdashhave only recently become available and for the most part institutions are still

keeping their scores secret Outside of these measures only a few very crude indicators are

available which likely explains why learning outputs do not feature especially prominently

in most ranking schemes

The simplest types of measures of learning outputs are those linked to graduation and

retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong

and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7

the latter three also use rates of graduation from Masterrsquos programs as indicators In some

cases the weights on these measures can be very highmdashin the Guangdong rankings

graduation rates account for over 50 of the rankingmdashbut in most cases the weights are

10 or less Retention rates commonly meaning the progression rate of first-year students

into second year are accorded less importance The US News and World Report Macleanrsquos

the Melbourne Institute and La Repubblica all employ retention measures as indicators but

none of them are worth more than 4 of total weighting Two publications make specific

indicators for retention and graduation of international students Macleanrsquos (graduation

rates of international students) and the Melbourne Institute (retention rates of international

students) The Washington Monthly looks specifically at institutional retention rates

adjusted for the participation of lower-income students and gives higher scores to

institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores

and number of Pell Grant recipients the US

7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18

9

News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos

ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)

E Indicators of Final Outcomes

Final outcomes are indications of generalized outcomes for students after graduation

Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness

good citizenship) but given the somewhat utilitarian justifications for education that are

currently in fashion (see Wolf 2000) employment outcomes are the most commonly used

measure of final outcomes These are given particular emphasis by the Guardian (where

employment outcomes are worth 17 of the total score) but are also used by the Financial

Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the

Times are interestingly not concerned with employment per se but with ldquoemployment in

an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher

Education Statistics Agency (HESA) uses Standard Occupational Classifications to

measure the proportion of graduates in professional or white-collar jobs anyone not in

such a job is considered not to be working in an area related to their studies (it is unclear

what methodology is used by the Financial Times and the Times although we suspect their

methods are broadly similar)

The only other measure of final outcomes in use is percentage of graduates returning for

additional education which is an indicator used by both the Melbourne Institute and the

Financial Times This is a particularly important indicator for the latter as it is worth 21

of the final ranking

The lack of indicators concerning final outcomes is interesting since most government-

sponsored performance-indicator regimes around the world are very much concerned

with such measures especially with respect to employment Possibly this indicates that

ranking systems simply do not view education outcomes as relevant measures of

educational quality Alternatively it may be the case that they simply have not found a

reliable indicator of outcomes or that there are reliable indicators but that there is so little

variation between institutions that it makes no sense to rank based on the data

10

F Indicators of Research

Many of the league tables covered in this survey include sections and weightings related

to universitiesrsquo research efforts It is in this field of measurement that we see the greatest

diversity of indicators Presumably this is because research inputs and outputs lend

themselves much more easily to measurement and manipulation than other areas of

institutional activity

Three studies include research staff as part of their ranking scheme La Repubblica at 952

the Melbourne Institute (4) and Wuhan (078)

Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used

method of looking at research quality but it is not universally admired because different

disciplines use different means to communicate major advances in knowledge (leading

scientists invariably produce large numbers of journal articles leading social scientists

may produce fewer journal articles but instead have one or two long important

monographsmdashsee Hicks 2004) There is also some concern among non-English speaking

countries that they are penalized in international rankings because so many of the major

journals (notably Science and Nature) are printed in English However the one set of

rankings that uses separate indicators to monitor articles published in English and articles

published in another language (the Wuhan rankings) shows that the two indicators are

positively correlated institutions that have more Chinese publications are also likely to

have more English publications and vice versa

Several sets of league tables measure bibliometric citations in various publication indices

The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a

weight of 20 of the final total Guangdong also monitors other Chinese universities

specifically for citations in engineering publications and weights this at 29 Moreover it

tacks on an additional 1049 for citations in science-oriented indices such as the Science

Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the

same category of scientific citations while the Melbourne Institute rates science citations at

68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social

Science Citation Index which does not include the humanities) are noted in only two

league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne

Institute (32) Another way of measuring research

11

impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a

weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the

Melbourne Institute

The complement of citations is of course publications Listing the number of publications

an individual group department or whole university releases can act as a weak substitute

for citationsmdashweak because simply publishing a paper or monograph is no guarantee that

the general public or other researchers will even glance at the work Guangdong gives an

1179 weighting to publications in science-oriented indices such as the Science Citation

Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same

indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong

even has a separate category just for measuring publications in Science and Nature

although it accords it an almost derisory weighting of 06 Under publications in social

science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their

final totals For publications in other indices (where the subject indices are

undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for

Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of

rankings to include research monographs weighted at 033

In countries where there are specific third-party evaluations of research output academic

quality of research is sometimes used as a research indicator The Times puts a very large

30 weight on this indicator while the Financial Times puts it at 119

Research awards are another handy third-party measurement of quality as the number of

international and national awards won by faculty andor graduates is often considered a

useful measure of institutional success International research awardsmdash specifically the

number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator

by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as

particularly suspect in some quarters given that the points are based on where the

recipient went to school rather than on where they are or were on the

8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the

most prominent publication indices

9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences

12

faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead

for nearly a century Wuhan uses similar measures but only accords them a weight of 14

National research awards are more common as a quality indicator used by La Repubblica

(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)

Financial indicators of research are also very common Research budgets as a factor in the

overall assessment of research in universities are covered by the Financial Times (9)

Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for

total amount of research expenditure unfortunately it is unclear precisely what this

research expenditure represents or how it is determined although it is clearly indicated

that it does not represent the total number of grants or projects at a university Total

number of research-based grants and projects is weighted by Education18 at 15 and

Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science

and engineering and another 55 to those for social sciences and humanities Similarly

the Melbourne Institute gives 6 of its overall weight to public-source grants making no

distinction between areas of study

In a slightly different vein Netbig (46) and Wuhan (278) both list the number of

research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica

(095) all weigh research-basedaffiliated research institutions or centres for studies

Finally one can also measure research not simply in terms of the amount of money it

generates but also in terms of the amount of future income it will generate Both

Guangdong (245) and Wuhan (193) measure the number of patents issued to

universities as a quality indicator

A final way of measuring an institutionrsquos research intensity is to look at the range of its

course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of

doctoral and Masterrsquos programs offered as a proxy for research intensity

As with physical and financial resources few if any of the research indicators are

normalized to account for institutional size (either by student or faculty numbers) In the

world of rankings bigger almost always means better an institution with 100 faculty with

ten citations apiece will always look worse than an institution with 1001 faculty

13

with one citation each To the extent that the raw production of knowledge matters this

form of measurement is acceptable To the extent that rankings are meant to show how

well institutions are doing on a like-to-like basis or to show the efficiency of universities it

is plainly inadequate This should be of particular concern to Chinese policy-makers

whose ranking systems are especially reliant on research-based indicators

G Indicators of Reputation

The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo

Those rankings systems which use the results of reputation surveys as an indicator do so as

an indirect measure of quality based on the assumption that the employers academics and

academic administrators surveyed have opinions of institutional quality that are informed

up-to-date and impartial While these assumptions are clearly open to debate they

nevertheless form an important basis for many ranking systems Another reason for using

reputation measures is the paucity of other data availablemdashsome countries have few

independent measures of teaching effectiveness university resources or output and

reputation can thus act as a useful surrogate Reputation rankings are often criticized as

simply quantifying the common ignorance of the people being surveyed However to the

extent that the people being surveyed hold positions which have the potential to affect

large numbers of young people and whose positions actually require some knowledge of

institutional quality (ie officials in charge of graduate admissions corporate recruiters

etc) then reputation rankings make sense because they provide useful information for

students about the perceived value of the degrees that they could obtain from various

universities

The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and

the Times which both accord reputation a weighting of 50 in their overall ranking

scheme Education18 assigns it almost as much significance at 40 The US News and World

Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just

below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)

and Netbig (15) The only other study to include reputation is Wuhan (117)

14

1 Please see Appendix B for a glossary of the various bodies referenced in this document

Appendix A Detailed Listing of Indicators and their Sources

Beginning Characteristics Indicator

Used By Source1

Incoming grades Macleans University

Percentage with grades above a set limit

Macleans University

US News and World Report

University

Performance on national standardised tests or benchmarks

Asiaweek University

Education18 3rd-party JUPAS

Financial Times Government agency 3rd-party UCAS

Guardian University Guide

Government agency 3rd-party UCAS

Melbourne Institute Government agency 3rd-party DEST

Netbig National entrance examination board

Times Good University Guide

Government agency 3rd-party UCAS

US News University

Wuhan Unknown presumed government 3rd-party

Student status La Repubblica Government agency 3rd-party MIUR

Admittance selectivity general

Asiaweek University

Admittance number of applications to places

Asiaweek University

Financial Times Government agency 3rd-party UCAS

La Repubblica Government agency 3rd-party MIUR

US News University

15

Beginning Characteristics Indicator

Used By Source1

Out-of-locality student percentage

Macleans University

International student percentages

Financial Times Government agency 3rd-party HESA

Macleans University

Shanghai Institute of Educational Science

Times World University

Wuhan Unknown presumed university

Undergraduate students among all students percentages

Netbig

Wuhan Unknown presumed university

Ethnic diversity in student body

Guardian University

Learning Inputs ndash Staff Indicator

Used By Source

Facultystudent ratio Asiaweek University

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Financial Times Government agency 3rd-party HESA

La Repubblica Government agency 3rd-party MIUR

Times Good University Guide

Government agency 3rd-party HESA

Times World University

US News University

Wuhan Unknown presumed university

16

Learning Inputs ndash Staff Indicator

Used By Source

Social science faculty student ratio

Melbourne Government agency 3rd-party DEST

Science faculty student ratio

Melbourne Government agency 3rd-party DEST

Administrative staff student ratio

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Staff student ratio (regardless of division)

Guardian Government agency 3rd-party HESA

Netbig University

Course per teacher La Repubblica Government agency 3rd-party MIUR

Per-teacher university spending

Asiaweek University

Faculty pay rates fortenured staff

Asiaweek University

US News University

Number of full-time part-time faculty

Netbig University

US News University

Wuhan Unknown presumed university

Faculty with researchprojects

Wuhan Unknown presumed university

Class size differentiation Macleans University

US News University

Classes taught by tenured faculty

Macleans University

Exchange programmes hosted

La Repubblica Government agency 3rd-party AgNaSoc

Number of classes lsquoactually taughtrsquo

La Repubblica Government agency 3rd-party MIUR

of international faculty (v faculty as a whole)

Times World University

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

7INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

BibliothegravequesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre total de volumes et de documents audiovisuels et eacutelectroniques input G10 ARLacquisition de volumes (au cours de lanneacutee) input ARL Macleansnombre total de volumes eacutetudiants en eacutequivalence temps complet input G10 ARL Macleans Deacutepenses des bibliothegraveques input MEQ Macleans AsiaweekDeacutepenses des bibliothegravequesbudget de fonctionnement input ARLDeacutepenses des bibliothegravequeseacutetudiants en eacutequivalence temps complet input G10 ARLPersonnels reacuteguliers agrave temps complet input G10 ARLClassement outcome ARL

RP gouvernance - 2008-06-23

8INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Revenus et deacutepenses USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Produits et revenusSubvention du ministegravere de lEacuteducation du Queacutebec input MEQ subvMEQrevenus de fonctionnement input G10 MacleansDroits de scolariteacute input G10 droits de scolariteacute revenus de fonctionnement input G10autres produits input G10Total des revenus de fonctionnement input G10 MEQ Guardian revenus de fonctionnement eacutetudiants en eacutequivalence temps complet input CAUBO USNews MacleansRevenus de recherche input G10encaissement par nature (dons immobilisations chaires bourses) input G10 fonds de dotation input G10 CenterContribution des alumni aux encaissements et fonds de dotation input USNews Macleans fonds de dotation eacutetudiants en eacutequivalence temps complet input G10Total des produits G10

Charges et deacutepenses G10salaire des professeurs input G10 MEQsalaire des chargeacutes de cours input G10salaires des auxiliaires input MEQsalaires des auxiliaires input MEQsalaire des personnels (staff) input G10autres deacutepenses input G10total des deacutepenses input G10 deacutepenses denseignement deacutepenses totales de fonctionnement input UBC MEQ

Chargesproduits processusEacutequilibre budgeacutetaire revenus et deacutepenses de fonctionnement processus MEQ

RP gouvernance - 2008-06-23

9INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

EspacesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Nombre de megravetres carreacutes bruts input G10MCB eacutetudiants en eacutequivalence temps complet input G10megravetres carreacutes nets alloueacutes input MEQNombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input Tbesoin despace en megravetres carreacutes nets inputExistence dun plan directeur des espaces inputBudget preacutevu au plan directeur des espaces input TBudget alloueacute aux espaces input MEQdeacutepenses de fonctionnement nombre de megravetres carreacutes bruts input G10

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Environnement USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

consommation de leacutenergie (exprimeacutee en BTU processus universiteacutes canadiennes et ameacutericaines BTUpopulation totale de luniversiteacute processus voir en particulier Penn State et Umich de leacutenergie produit par des sources renouvelable processusUtilisation de leau processuslitres deau par personne processusGaz agrave effet de serre (en tonnes meacutetriques CO2 eacutequivalents) processusMT CO2 par personne processusDeacutechets solides (en tonnes) processusTonnes de deacutechets solides per capita processus de deacutechets solides recycleacutes processus

RP gouvernance - 2008-06-23

10INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

ReacuteputationUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

enquecircte meneacutee aupregraves des eacutetudiants outcome NSSE Melbourneenquecircte meneacutee aupregraves des diplocircmeacutes outcomeenquecircte meneacutee aupregraves des pairs outcome USNews Asiaweek THESenquecircte meneacutee aupregraves des employeurs outcome Macleans Melbourneenquecircte meneacutee aupregraves des stakeholders outcome Macleans Melbourneenquecircte meneacutee aupregraves du monde des affaires outcome Macleans Melbournecouverture meacutediatique outcome AT

RP gouvernance - 2008-06-23

11INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Planification et gestion strateacutegiquesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Pouvoirs des corps universitaire processus un Canliens dynamiques entre Conseil Assembleacutee universitaire commission des eacutetudes processus un Canprocessus formel de planification strateacutegique processus un Canproduction de donneacutees et indicateurs processus un Canprocessus deacutevaluation des faculteacutes deacutepartements et services processus un Canrevue systeacutematique des responsabiliteacutes et efficaciteacute des comiteacutes processus un Candeacutepenses de ladministration centrale deacutepenses de fonctionnement processus un Can

RP gouvernance - 2008-06-23

12INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Eacuteducation permanenteUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre de programmes offerts input un Cannombre deacutetudiants inscrits input un CanRevenus et deacutepenses du service input un Can

RP gouvernance - 2008-06-23

13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes

REacuteFEacuteRENCES

Aperccedilu densemble des principaux indicateurs

GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost

Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes

MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf

Indicateurs utiliseacutes dans les classements internationaux

Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml

RP gouvernance - 2008-06-23

ANNEXES

Quelques exemples de listes drsquoindicateurs

1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)

Voir page suivante

Preacutepareacute par Roland Proulx

Consultant en planification institutionnelle et veille strateacutegique

Montreacuteal mai 2008

Performance Indicators

Discussion Paper

May 2004 (Prepared by the Office of Planning amp Analysis)

Table of Contents

Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25

2

Table 1 ndash Number of Indictors by Type of Data Source

Table 1 shows that surveys are the least frequently used source of data for indicators

Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a

plurality of indicators come from this source This measure somewhat underestimates the

importance of surveys however as it does not account for the weighting given to each

indicator in each study In the THES World Rankings for instance there is only a single

survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos

Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one

indicator is worth 20 of the final score

Raw

indIcator count

Surve

y data

Third parties

Universities

AsiaweekmdashAsias Best Universities 18 - - 18

Daily Telegraph (2003) 1 - 1 -

Education18com 9 3 4 2

Excelencia 2001 71 - 71 -

Financial Times (2003) 17 - 17 -

Guangdong Institute of Management Science 17 - 14 3

GuardianmdashUniversity Guide 2005 7 - 2 5

La Repubblica 23 2 21 -

Macleans University Rankings 24 1 5 18

Melbourne Institutemdash International Standing of Australian Universities

26 3 23 -

Netbig 2004 18 1 10 7

Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15

Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities

6 - 5 1

The TimesmdashGood University Guide 2005 9 - 9 -

Times Higher Education SupplementmdashWorld University Rankings

5 1 1 3

US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11

Washington MonthlymdashCollege Rankings 2005 8 - 1 7

Wuhan University Centre for Science Evaluation 45 2 22 21

3

A Indicators of Beginning Characteristics

ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of

students at the time they begin their studies Fourteen of the 18 rankings examined in this

study use one or more indicators of the beginning characteristics of students to arrive at

their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on

these factors with 33 of the total ranking coming from this class of indicators but the

Guardian Education18 Asiaweek and the two other North American surveys also place

considerable emphasis on this category

There are six main indicators used to determine which institutions have students with

positive ldquobeginning characteristicsrdquo

The most common measure of beginning characteristics is performance on national

standardized tests with nine surveys using this as a measure Education18 and the

Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used

by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)

Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this

data is collected and standardized by national bodies it has the benefit of being seen as a

relatively impartial method of determining the relative ldquostrengthrdquo of the students entering

each institution Institutionsrsquo results can be scored by showing either averages or the

percentage of entering students meeting a particular standard

Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses

secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This

is a second-best solution made necessary by the absence of any national standardized test

in Canada (or indeed of any provincial standardized tests at the end of secondary school

in provinces other than Alberta) The lack of national standardization makes this an

undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is

truly equivalent to an ldquoArdquo in another jurisdiction

Another measure of the strength of the student body is the percentage of incoming

students receiving (third-party) scholarships which is worth 11 of the score in the

Wuhan survey One can also approach the issue by measuring institutional selectivity In

effect this method infers the strength of the student body by the proportion of

4

applicants rejected the theory being that the higher the number of rejected applicants the

stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)

Normally this measure is expressed as a straight ratio of acceptances to applications but it

can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the

greatest weight on this measure) as a ratio of enrolments to applications Within the US

there is some dispute as to what constitutes an offer of admission and whether or not late

admissions are included as noted by Ganeshananthan (2003)

Student bodies are often considered to be strong if the school is able to attract a large

number of international or out-of-district students or if they contain people from diverse

ethnic backgrounds A number of league tables use the international student indicator

(which like the selectivity indicator is arguably as much an indicator of prestige and

reputation as it is of student characteristics) although in no case does this indicator

account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a

quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data

on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian

puts a somewhat larger emphasis on this indicator in comparison to other league tables

which use similar variables

A very different take on this idea is present in the Washington Monthly which released its

first set of College Rankings in September 2005 With the declared aim of using an

institutionrsquos commitment to social mobility as a measure of quality it uses the percentage

of students from low-income backgrounds as an indicator (with percentage of students

receiving need-based government (Pell) grants used as a proxy)

Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy

statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the

percentage of the student population who are graduate students (arguably this is a

research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the

number of graduate students auditing classes is used as an indicator the assumption is

presumably that if people are auditing then the classes must be very attractive The Italian

La Repubblica ranks an institution according to the number of part-time students it has

contrary to prevailing North American views on the undesirability of part-time study the

Italian rankings see higher numbers of part-time students in a positive light as it is

evidence that an institution is becoming less rigid in its timetabling and

5

permitting students to juggle both work and study something which was nearly

impossible in that country just a few years ago

The Washington Monthly also has a final category of indicators which reflect studentsrsquo

beginning characteristics namely their likelihood of performing community service as

measured by the percentage of students in the US Peace Corps and Reserve Officer

Training Corps (ROTC) and the percentage of students involved in work-study in the

community3 Together these three indicators account for 33 of an institutionrsquos total

ranking

B Indicators of Learning InputsmdashStaff

Generally both the quantity and quality of staff are positively correlated with institutional

quality The problem of course is finding useful metrics for each of these factors

especially if one excludes as we have done here measures of research performance and

research intensity putting them in a separate category4

The simplest measure is simply the number of faculty unadjusted for things like size of

student body Most national league tables however prefer to use variations on the

concept of facultystudent ratio Others try to measure teaching intensity with measures

such as courses per teacher or hours spent in class per student (both in La Repubblica)

These kinds of measures usually account for between 2-5 of the final rankings although

in some cases (ie the Guardian) this figure can be as high as 20

Another important way of measuring how faculty resources are deployed is the measure

of average class size which is used only by Macleanrsquos and the US News and World Report

Ostensibly the reason for measuring class size is to account in some form for the degree

3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category

6

of attention devoted to actually teaching students Despite the fact that important research

(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level

the use of this indicator appears to be a spillover from the North American debates on

class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein

2000) Regardless of why the indicators are used they are extraordinarily important to

these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and

World Reportrsquos rankings respectively

A number of ranking systems try to look at staff qualifications such as the number of

PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the

Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further

than other surveys and actually looks at the proportion of classes taught by tenure-track

staff Others (ie THES) look at the number of foreign faculty based on the assumption

that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still

others (ie La Repubblica) look at the age structure of the faculty Another proxy for

institutional quality is the pay rates for tenured staff on the assumption that institutions

with higher rates of pay on average attract better faculty this measure has been used both

by the US News and World Report and Asiaweek

Finally a number of league tables rank faculty inputs on the basis of standardized third-

party evaluations 6

Education18 the Financial Times the Times the Guardian and the US

News and World Report league tables all use some sort of ranking criterion based at least in

part on this indicator or variations thereof

C Indicators of Learning InputsmdashResources

Resource inputsmdashcrudely the amount of current dollars equipment and books available

to students at an institutionmdashare widely considered an important measure of quality Yet

despite the apparent simplicity of counting dollars and measuring assets

5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each

university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future

7

the means by which institutional wealth is measured varies considerably between ranking

systems

There are a number of revenue-based measures of resources Macleanrsquos uses public funding

of institutional budgets as a factor in its analysis conversely the Financial Times uses the

private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of

total score) and the US News and World Report (5 of total score) also measure alumni

financial support as a measure of quality

For reasons that are not entirely clear league tables tend to favour measures of

expenditures rather than revenues The Guardian looks at total institutional expenditures

as an indicator Institutional expenditure on student services is used as a measure of

institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total

institutional scores respectively) Rzezspospolita does not measure student services

expenditures directly but does measure student services outputs such as number of

student athletes and number of study clubs which amounts to more or less the same thing

Macleanrsquos also gives out 433 of its total score based on institutional expenditures on

scholarships and bursaries

Various aspects of physical infrastructure are also used as measures of institutional

resources most directly in the case of La Repubblica which bases 317 of its total rank on

the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding

assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type

of physical infrastructure measured is available Internet bandwidth which was used by

Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth

roughly 3 of the total score

By some distance the infrastructure indicators most favoured by the compilers of league

tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on

this with 12 of the total quality mark being taken from various types of library

infrastructure measurements (including acquisitions per year total volumes average

number of volumes per student and yearly library expenditure outside of acquisitions)

Netbig and Education18 also use library volume holdings while Asiaweek the Financial

Times and the Times also use measures of library expenditures outside of acquisitions or

computerization of library resources as measures of institutional quality

8

One important factor to note is that most ranking systems do not normalize their resource

and infrastructure measures That is to say it is raw spending power or simple size of

assets that is usually measured rather than spending per studentprofessor or assets per

studentprofessor As a result a number of these rankings systems have inbuilt biases

towards larger institutions

D Indicators of Learning Outputs

Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge

learned over the course of a baccalaureate degreemdashshould be a basic indicator of

institutional quality Unfortunately good means of measuring these outputsmdashlike the

National Survey of Student Engagement (NSSE) and the College Learning Assessment

(CLA)mdashhave only recently become available and for the most part institutions are still

keeping their scores secret Outside of these measures only a few very crude indicators are

available which likely explains why learning outputs do not feature especially prominently

in most ranking schemes

The simplest types of measures of learning outputs are those linked to graduation and

retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong

and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7

the latter three also use rates of graduation from Masterrsquos programs as indicators In some

cases the weights on these measures can be very highmdashin the Guangdong rankings

graduation rates account for over 50 of the rankingmdashbut in most cases the weights are

10 or less Retention rates commonly meaning the progression rate of first-year students

into second year are accorded less importance The US News and World Report Macleanrsquos

the Melbourne Institute and La Repubblica all employ retention measures as indicators but

none of them are worth more than 4 of total weighting Two publications make specific

indicators for retention and graduation of international students Macleanrsquos (graduation

rates of international students) and the Melbourne Institute (retention rates of international

students) The Washington Monthly looks specifically at institutional retention rates

adjusted for the participation of lower-income students and gives higher scores to

institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores

and number of Pell Grant recipients the US

7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18

9

News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos

ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)

E Indicators of Final Outcomes

Final outcomes are indications of generalized outcomes for students after graduation

Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness

good citizenship) but given the somewhat utilitarian justifications for education that are

currently in fashion (see Wolf 2000) employment outcomes are the most commonly used

measure of final outcomes These are given particular emphasis by the Guardian (where

employment outcomes are worth 17 of the total score) but are also used by the Financial

Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the

Times are interestingly not concerned with employment per se but with ldquoemployment in

an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher

Education Statistics Agency (HESA) uses Standard Occupational Classifications to

measure the proportion of graduates in professional or white-collar jobs anyone not in

such a job is considered not to be working in an area related to their studies (it is unclear

what methodology is used by the Financial Times and the Times although we suspect their

methods are broadly similar)

The only other measure of final outcomes in use is percentage of graduates returning for

additional education which is an indicator used by both the Melbourne Institute and the

Financial Times This is a particularly important indicator for the latter as it is worth 21

of the final ranking

The lack of indicators concerning final outcomes is interesting since most government-

sponsored performance-indicator regimes around the world are very much concerned

with such measures especially with respect to employment Possibly this indicates that

ranking systems simply do not view education outcomes as relevant measures of

educational quality Alternatively it may be the case that they simply have not found a

reliable indicator of outcomes or that there are reliable indicators but that there is so little

variation between institutions that it makes no sense to rank based on the data

10

F Indicators of Research

Many of the league tables covered in this survey include sections and weightings related

to universitiesrsquo research efforts It is in this field of measurement that we see the greatest

diversity of indicators Presumably this is because research inputs and outputs lend

themselves much more easily to measurement and manipulation than other areas of

institutional activity

Three studies include research staff as part of their ranking scheme La Repubblica at 952

the Melbourne Institute (4) and Wuhan (078)

Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used

method of looking at research quality but it is not universally admired because different

disciplines use different means to communicate major advances in knowledge (leading

scientists invariably produce large numbers of journal articles leading social scientists

may produce fewer journal articles but instead have one or two long important

monographsmdashsee Hicks 2004) There is also some concern among non-English speaking

countries that they are penalized in international rankings because so many of the major

journals (notably Science and Nature) are printed in English However the one set of

rankings that uses separate indicators to monitor articles published in English and articles

published in another language (the Wuhan rankings) shows that the two indicators are

positively correlated institutions that have more Chinese publications are also likely to

have more English publications and vice versa

Several sets of league tables measure bibliometric citations in various publication indices

The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a

weight of 20 of the final total Guangdong also monitors other Chinese universities

specifically for citations in engineering publications and weights this at 29 Moreover it

tacks on an additional 1049 for citations in science-oriented indices such as the Science

Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the

same category of scientific citations while the Melbourne Institute rates science citations at

68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social

Science Citation Index which does not include the humanities) are noted in only two

league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne

Institute (32) Another way of measuring research

11

impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a

weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the

Melbourne Institute

The complement of citations is of course publications Listing the number of publications

an individual group department or whole university releases can act as a weak substitute

for citationsmdashweak because simply publishing a paper or monograph is no guarantee that

the general public or other researchers will even glance at the work Guangdong gives an

1179 weighting to publications in science-oriented indices such as the Science Citation

Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same

indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong

even has a separate category just for measuring publications in Science and Nature

although it accords it an almost derisory weighting of 06 Under publications in social

science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their

final totals For publications in other indices (where the subject indices are

undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for

Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of

rankings to include research monographs weighted at 033

In countries where there are specific third-party evaluations of research output academic

quality of research is sometimes used as a research indicator The Times puts a very large

30 weight on this indicator while the Financial Times puts it at 119

Research awards are another handy third-party measurement of quality as the number of

international and national awards won by faculty andor graduates is often considered a

useful measure of institutional success International research awardsmdash specifically the

number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator

by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as

particularly suspect in some quarters given that the points are based on where the

recipient went to school rather than on where they are or were on the

8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the

most prominent publication indices

9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences

12

faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead

for nearly a century Wuhan uses similar measures but only accords them a weight of 14

National research awards are more common as a quality indicator used by La Repubblica

(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)

Financial indicators of research are also very common Research budgets as a factor in the

overall assessment of research in universities are covered by the Financial Times (9)

Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for

total amount of research expenditure unfortunately it is unclear precisely what this

research expenditure represents or how it is determined although it is clearly indicated

that it does not represent the total number of grants or projects at a university Total

number of research-based grants and projects is weighted by Education18 at 15 and

Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science

and engineering and another 55 to those for social sciences and humanities Similarly

the Melbourne Institute gives 6 of its overall weight to public-source grants making no

distinction between areas of study

In a slightly different vein Netbig (46) and Wuhan (278) both list the number of

research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica

(095) all weigh research-basedaffiliated research institutions or centres for studies

Finally one can also measure research not simply in terms of the amount of money it

generates but also in terms of the amount of future income it will generate Both

Guangdong (245) and Wuhan (193) measure the number of patents issued to

universities as a quality indicator

A final way of measuring an institutionrsquos research intensity is to look at the range of its

course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of

doctoral and Masterrsquos programs offered as a proxy for research intensity

As with physical and financial resources few if any of the research indicators are

normalized to account for institutional size (either by student or faculty numbers) In the

world of rankings bigger almost always means better an institution with 100 faculty with

ten citations apiece will always look worse than an institution with 1001 faculty

13

with one citation each To the extent that the raw production of knowledge matters this

form of measurement is acceptable To the extent that rankings are meant to show how

well institutions are doing on a like-to-like basis or to show the efficiency of universities it

is plainly inadequate This should be of particular concern to Chinese policy-makers

whose ranking systems are especially reliant on research-based indicators

G Indicators of Reputation

The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo

Those rankings systems which use the results of reputation surveys as an indicator do so as

an indirect measure of quality based on the assumption that the employers academics and

academic administrators surveyed have opinions of institutional quality that are informed

up-to-date and impartial While these assumptions are clearly open to debate they

nevertheless form an important basis for many ranking systems Another reason for using

reputation measures is the paucity of other data availablemdashsome countries have few

independent measures of teaching effectiveness university resources or output and

reputation can thus act as a useful surrogate Reputation rankings are often criticized as

simply quantifying the common ignorance of the people being surveyed However to the

extent that the people being surveyed hold positions which have the potential to affect

large numbers of young people and whose positions actually require some knowledge of

institutional quality (ie officials in charge of graduate admissions corporate recruiters

etc) then reputation rankings make sense because they provide useful information for

students about the perceived value of the degrees that they could obtain from various

universities

The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and

the Times which both accord reputation a weighting of 50 in their overall ranking

scheme Education18 assigns it almost as much significance at 40 The US News and World

Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just

below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)

and Netbig (15) The only other study to include reputation is Wuhan (117)

14

1 Please see Appendix B for a glossary of the various bodies referenced in this document

Appendix A Detailed Listing of Indicators and their Sources

Beginning Characteristics Indicator

Used By Source1

Incoming grades Macleans University

Percentage with grades above a set limit

Macleans University

US News and World Report

University

Performance on national standardised tests or benchmarks

Asiaweek University

Education18 3rd-party JUPAS

Financial Times Government agency 3rd-party UCAS

Guardian University Guide

Government agency 3rd-party UCAS

Melbourne Institute Government agency 3rd-party DEST

Netbig National entrance examination board

Times Good University Guide

Government agency 3rd-party UCAS

US News University

Wuhan Unknown presumed government 3rd-party

Student status La Repubblica Government agency 3rd-party MIUR

Admittance selectivity general

Asiaweek University

Admittance number of applications to places

Asiaweek University

Financial Times Government agency 3rd-party UCAS

La Repubblica Government agency 3rd-party MIUR

US News University

15

Beginning Characteristics Indicator

Used By Source1

Out-of-locality student percentage

Macleans University

International student percentages

Financial Times Government agency 3rd-party HESA

Macleans University

Shanghai Institute of Educational Science

Times World University

Wuhan Unknown presumed university

Undergraduate students among all students percentages

Netbig

Wuhan Unknown presumed university

Ethnic diversity in student body

Guardian University

Learning Inputs ndash Staff Indicator

Used By Source

Facultystudent ratio Asiaweek University

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Financial Times Government agency 3rd-party HESA

La Repubblica Government agency 3rd-party MIUR

Times Good University Guide

Government agency 3rd-party HESA

Times World University

US News University

Wuhan Unknown presumed university

16

Learning Inputs ndash Staff Indicator

Used By Source

Social science faculty student ratio

Melbourne Government agency 3rd-party DEST

Science faculty student ratio

Melbourne Government agency 3rd-party DEST

Administrative staff student ratio

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Staff student ratio (regardless of division)

Guardian Government agency 3rd-party HESA

Netbig University

Course per teacher La Repubblica Government agency 3rd-party MIUR

Per-teacher university spending

Asiaweek University

Faculty pay rates fortenured staff

Asiaweek University

US News University

Number of full-time part-time faculty

Netbig University

US News University

Wuhan Unknown presumed university

Faculty with researchprojects

Wuhan Unknown presumed university

Class size differentiation Macleans University

US News University

Classes taught by tenured faculty

Macleans University

Exchange programmes hosted

La Repubblica Government agency 3rd-party AgNaSoc

Number of classes lsquoactually taughtrsquo

La Repubblica Government agency 3rd-party MIUR

of international faculty (v faculty as a whole)

Times World University

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

8INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Revenus et deacutepenses USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Produits et revenusSubvention du ministegravere de lEacuteducation du Queacutebec input MEQ subvMEQrevenus de fonctionnement input G10 MacleansDroits de scolariteacute input G10 droits de scolariteacute revenus de fonctionnement input G10autres produits input G10Total des revenus de fonctionnement input G10 MEQ Guardian revenus de fonctionnement eacutetudiants en eacutequivalence temps complet input CAUBO USNews MacleansRevenus de recherche input G10encaissement par nature (dons immobilisations chaires bourses) input G10 fonds de dotation input G10 CenterContribution des alumni aux encaissements et fonds de dotation input USNews Macleans fonds de dotation eacutetudiants en eacutequivalence temps complet input G10Total des produits G10

Charges et deacutepenses G10salaire des professeurs input G10 MEQsalaire des chargeacutes de cours input G10salaires des auxiliaires input MEQsalaires des auxiliaires input MEQsalaire des personnels (staff) input G10autres deacutepenses input G10total des deacutepenses input G10 deacutepenses denseignement deacutepenses totales de fonctionnement input UBC MEQ

Chargesproduits processusEacutequilibre budgeacutetaire revenus et deacutepenses de fonctionnement processus MEQ

RP gouvernance - 2008-06-23

9INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

EspacesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Nombre de megravetres carreacutes bruts input G10MCB eacutetudiants en eacutequivalence temps complet input G10megravetres carreacutes nets alloueacutes input MEQNombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input Tbesoin despace en megravetres carreacutes nets inputExistence dun plan directeur des espaces inputBudget preacutevu au plan directeur des espaces input TBudget alloueacute aux espaces input MEQdeacutepenses de fonctionnement nombre de megravetres carreacutes bruts input G10

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Environnement USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

consommation de leacutenergie (exprimeacutee en BTU processus universiteacutes canadiennes et ameacutericaines BTUpopulation totale de luniversiteacute processus voir en particulier Penn State et Umich de leacutenergie produit par des sources renouvelable processusUtilisation de leau processuslitres deau par personne processusGaz agrave effet de serre (en tonnes meacutetriques CO2 eacutequivalents) processusMT CO2 par personne processusDeacutechets solides (en tonnes) processusTonnes de deacutechets solides per capita processus de deacutechets solides recycleacutes processus

RP gouvernance - 2008-06-23

10INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

ReacuteputationUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

enquecircte meneacutee aupregraves des eacutetudiants outcome NSSE Melbourneenquecircte meneacutee aupregraves des diplocircmeacutes outcomeenquecircte meneacutee aupregraves des pairs outcome USNews Asiaweek THESenquecircte meneacutee aupregraves des employeurs outcome Macleans Melbourneenquecircte meneacutee aupregraves des stakeholders outcome Macleans Melbourneenquecircte meneacutee aupregraves du monde des affaires outcome Macleans Melbournecouverture meacutediatique outcome AT

RP gouvernance - 2008-06-23

11INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Planification et gestion strateacutegiquesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Pouvoirs des corps universitaire processus un Canliens dynamiques entre Conseil Assembleacutee universitaire commission des eacutetudes processus un Canprocessus formel de planification strateacutegique processus un Canproduction de donneacutees et indicateurs processus un Canprocessus deacutevaluation des faculteacutes deacutepartements et services processus un Canrevue systeacutematique des responsabiliteacutes et efficaciteacute des comiteacutes processus un Candeacutepenses de ladministration centrale deacutepenses de fonctionnement processus un Can

RP gouvernance - 2008-06-23

12INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Eacuteducation permanenteUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre de programmes offerts input un Cannombre deacutetudiants inscrits input un CanRevenus et deacutepenses du service input un Can

RP gouvernance - 2008-06-23

13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes

REacuteFEacuteRENCES

Aperccedilu densemble des principaux indicateurs

GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost

Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes

MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf

Indicateurs utiliseacutes dans les classements internationaux

Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml

RP gouvernance - 2008-06-23

ANNEXES

Quelques exemples de listes drsquoindicateurs

1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)

Voir page suivante

Preacutepareacute par Roland Proulx

Consultant en planification institutionnelle et veille strateacutegique

Montreacuteal mai 2008

Performance Indicators

Discussion Paper

May 2004 (Prepared by the Office of Planning amp Analysis)

Table of Contents

Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25

2

Table 1 ndash Number of Indictors by Type of Data Source

Table 1 shows that surveys are the least frequently used source of data for indicators

Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a

plurality of indicators come from this source This measure somewhat underestimates the

importance of surveys however as it does not account for the weighting given to each

indicator in each study In the THES World Rankings for instance there is only a single

survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos

Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one

indicator is worth 20 of the final score

Raw

indIcator count

Surve

y data

Third parties

Universities

AsiaweekmdashAsias Best Universities 18 - - 18

Daily Telegraph (2003) 1 - 1 -

Education18com 9 3 4 2

Excelencia 2001 71 - 71 -

Financial Times (2003) 17 - 17 -

Guangdong Institute of Management Science 17 - 14 3

GuardianmdashUniversity Guide 2005 7 - 2 5

La Repubblica 23 2 21 -

Macleans University Rankings 24 1 5 18

Melbourne Institutemdash International Standing of Australian Universities

26 3 23 -

Netbig 2004 18 1 10 7

Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15

Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities

6 - 5 1

The TimesmdashGood University Guide 2005 9 - 9 -

Times Higher Education SupplementmdashWorld University Rankings

5 1 1 3

US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11

Washington MonthlymdashCollege Rankings 2005 8 - 1 7

Wuhan University Centre for Science Evaluation 45 2 22 21

3

A Indicators of Beginning Characteristics

ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of

students at the time they begin their studies Fourteen of the 18 rankings examined in this

study use one or more indicators of the beginning characteristics of students to arrive at

their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on

these factors with 33 of the total ranking coming from this class of indicators but the

Guardian Education18 Asiaweek and the two other North American surveys also place

considerable emphasis on this category

There are six main indicators used to determine which institutions have students with

positive ldquobeginning characteristicsrdquo

The most common measure of beginning characteristics is performance on national

standardized tests with nine surveys using this as a measure Education18 and the

Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used

by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)

Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this

data is collected and standardized by national bodies it has the benefit of being seen as a

relatively impartial method of determining the relative ldquostrengthrdquo of the students entering

each institution Institutionsrsquo results can be scored by showing either averages or the

percentage of entering students meeting a particular standard

Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses

secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This

is a second-best solution made necessary by the absence of any national standardized test

in Canada (or indeed of any provincial standardized tests at the end of secondary school

in provinces other than Alberta) The lack of national standardization makes this an

undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is

truly equivalent to an ldquoArdquo in another jurisdiction

Another measure of the strength of the student body is the percentage of incoming

students receiving (third-party) scholarships which is worth 11 of the score in the

Wuhan survey One can also approach the issue by measuring institutional selectivity In

effect this method infers the strength of the student body by the proportion of

4

applicants rejected the theory being that the higher the number of rejected applicants the

stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)

Normally this measure is expressed as a straight ratio of acceptances to applications but it

can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the

greatest weight on this measure) as a ratio of enrolments to applications Within the US

there is some dispute as to what constitutes an offer of admission and whether or not late

admissions are included as noted by Ganeshananthan (2003)

Student bodies are often considered to be strong if the school is able to attract a large

number of international or out-of-district students or if they contain people from diverse

ethnic backgrounds A number of league tables use the international student indicator

(which like the selectivity indicator is arguably as much an indicator of prestige and

reputation as it is of student characteristics) although in no case does this indicator

account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a

quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data

on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian

puts a somewhat larger emphasis on this indicator in comparison to other league tables

which use similar variables

A very different take on this idea is present in the Washington Monthly which released its

first set of College Rankings in September 2005 With the declared aim of using an

institutionrsquos commitment to social mobility as a measure of quality it uses the percentage

of students from low-income backgrounds as an indicator (with percentage of students

receiving need-based government (Pell) grants used as a proxy)

Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy

statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the

percentage of the student population who are graduate students (arguably this is a

research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the

number of graduate students auditing classes is used as an indicator the assumption is

presumably that if people are auditing then the classes must be very attractive The Italian

La Repubblica ranks an institution according to the number of part-time students it has

contrary to prevailing North American views on the undesirability of part-time study the

Italian rankings see higher numbers of part-time students in a positive light as it is

evidence that an institution is becoming less rigid in its timetabling and

5

permitting students to juggle both work and study something which was nearly

impossible in that country just a few years ago

The Washington Monthly also has a final category of indicators which reflect studentsrsquo

beginning characteristics namely their likelihood of performing community service as

measured by the percentage of students in the US Peace Corps and Reserve Officer

Training Corps (ROTC) and the percentage of students involved in work-study in the

community3 Together these three indicators account for 33 of an institutionrsquos total

ranking

B Indicators of Learning InputsmdashStaff

Generally both the quantity and quality of staff are positively correlated with institutional

quality The problem of course is finding useful metrics for each of these factors

especially if one excludes as we have done here measures of research performance and

research intensity putting them in a separate category4

The simplest measure is simply the number of faculty unadjusted for things like size of

student body Most national league tables however prefer to use variations on the

concept of facultystudent ratio Others try to measure teaching intensity with measures

such as courses per teacher or hours spent in class per student (both in La Repubblica)

These kinds of measures usually account for between 2-5 of the final rankings although

in some cases (ie the Guardian) this figure can be as high as 20

Another important way of measuring how faculty resources are deployed is the measure

of average class size which is used only by Macleanrsquos and the US News and World Report

Ostensibly the reason for measuring class size is to account in some form for the degree

3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category

6

of attention devoted to actually teaching students Despite the fact that important research

(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level

the use of this indicator appears to be a spillover from the North American debates on

class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein

2000) Regardless of why the indicators are used they are extraordinarily important to

these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and

World Reportrsquos rankings respectively

A number of ranking systems try to look at staff qualifications such as the number of

PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the

Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further

than other surveys and actually looks at the proportion of classes taught by tenure-track

staff Others (ie THES) look at the number of foreign faculty based on the assumption

that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still

others (ie La Repubblica) look at the age structure of the faculty Another proxy for

institutional quality is the pay rates for tenured staff on the assumption that institutions

with higher rates of pay on average attract better faculty this measure has been used both

by the US News and World Report and Asiaweek

Finally a number of league tables rank faculty inputs on the basis of standardized third-

party evaluations 6

Education18 the Financial Times the Times the Guardian and the US

News and World Report league tables all use some sort of ranking criterion based at least in

part on this indicator or variations thereof

C Indicators of Learning InputsmdashResources

Resource inputsmdashcrudely the amount of current dollars equipment and books available

to students at an institutionmdashare widely considered an important measure of quality Yet

despite the apparent simplicity of counting dollars and measuring assets

5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each

university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future

7

the means by which institutional wealth is measured varies considerably between ranking

systems

There are a number of revenue-based measures of resources Macleanrsquos uses public funding

of institutional budgets as a factor in its analysis conversely the Financial Times uses the

private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of

total score) and the US News and World Report (5 of total score) also measure alumni

financial support as a measure of quality

For reasons that are not entirely clear league tables tend to favour measures of

expenditures rather than revenues The Guardian looks at total institutional expenditures

as an indicator Institutional expenditure on student services is used as a measure of

institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total

institutional scores respectively) Rzezspospolita does not measure student services

expenditures directly but does measure student services outputs such as number of

student athletes and number of study clubs which amounts to more or less the same thing

Macleanrsquos also gives out 433 of its total score based on institutional expenditures on

scholarships and bursaries

Various aspects of physical infrastructure are also used as measures of institutional

resources most directly in the case of La Repubblica which bases 317 of its total rank on

the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding

assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type

of physical infrastructure measured is available Internet bandwidth which was used by

Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth

roughly 3 of the total score

By some distance the infrastructure indicators most favoured by the compilers of league

tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on

this with 12 of the total quality mark being taken from various types of library

infrastructure measurements (including acquisitions per year total volumes average

number of volumes per student and yearly library expenditure outside of acquisitions)

Netbig and Education18 also use library volume holdings while Asiaweek the Financial

Times and the Times also use measures of library expenditures outside of acquisitions or

computerization of library resources as measures of institutional quality

8

One important factor to note is that most ranking systems do not normalize their resource

and infrastructure measures That is to say it is raw spending power or simple size of

assets that is usually measured rather than spending per studentprofessor or assets per

studentprofessor As a result a number of these rankings systems have inbuilt biases

towards larger institutions

D Indicators of Learning Outputs

Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge

learned over the course of a baccalaureate degreemdashshould be a basic indicator of

institutional quality Unfortunately good means of measuring these outputsmdashlike the

National Survey of Student Engagement (NSSE) and the College Learning Assessment

(CLA)mdashhave only recently become available and for the most part institutions are still

keeping their scores secret Outside of these measures only a few very crude indicators are

available which likely explains why learning outputs do not feature especially prominently

in most ranking schemes

The simplest types of measures of learning outputs are those linked to graduation and

retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong

and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7

the latter three also use rates of graduation from Masterrsquos programs as indicators In some

cases the weights on these measures can be very highmdashin the Guangdong rankings

graduation rates account for over 50 of the rankingmdashbut in most cases the weights are

10 or less Retention rates commonly meaning the progression rate of first-year students

into second year are accorded less importance The US News and World Report Macleanrsquos

the Melbourne Institute and La Repubblica all employ retention measures as indicators but

none of them are worth more than 4 of total weighting Two publications make specific

indicators for retention and graduation of international students Macleanrsquos (graduation

rates of international students) and the Melbourne Institute (retention rates of international

students) The Washington Monthly looks specifically at institutional retention rates

adjusted for the participation of lower-income students and gives higher scores to

institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores

and number of Pell Grant recipients the US

7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18

9

News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos

ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)

E Indicators of Final Outcomes

Final outcomes are indications of generalized outcomes for students after graduation

Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness

good citizenship) but given the somewhat utilitarian justifications for education that are

currently in fashion (see Wolf 2000) employment outcomes are the most commonly used

measure of final outcomes These are given particular emphasis by the Guardian (where

employment outcomes are worth 17 of the total score) but are also used by the Financial

Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the

Times are interestingly not concerned with employment per se but with ldquoemployment in

an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher

Education Statistics Agency (HESA) uses Standard Occupational Classifications to

measure the proportion of graduates in professional or white-collar jobs anyone not in

such a job is considered not to be working in an area related to their studies (it is unclear

what methodology is used by the Financial Times and the Times although we suspect their

methods are broadly similar)

The only other measure of final outcomes in use is percentage of graduates returning for

additional education which is an indicator used by both the Melbourne Institute and the

Financial Times This is a particularly important indicator for the latter as it is worth 21

of the final ranking

The lack of indicators concerning final outcomes is interesting since most government-

sponsored performance-indicator regimes around the world are very much concerned

with such measures especially with respect to employment Possibly this indicates that

ranking systems simply do not view education outcomes as relevant measures of

educational quality Alternatively it may be the case that they simply have not found a

reliable indicator of outcomes or that there are reliable indicators but that there is so little

variation between institutions that it makes no sense to rank based on the data

10

F Indicators of Research

Many of the league tables covered in this survey include sections and weightings related

to universitiesrsquo research efforts It is in this field of measurement that we see the greatest

diversity of indicators Presumably this is because research inputs and outputs lend

themselves much more easily to measurement and manipulation than other areas of

institutional activity

Three studies include research staff as part of their ranking scheme La Repubblica at 952

the Melbourne Institute (4) and Wuhan (078)

Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used

method of looking at research quality but it is not universally admired because different

disciplines use different means to communicate major advances in knowledge (leading

scientists invariably produce large numbers of journal articles leading social scientists

may produce fewer journal articles but instead have one or two long important

monographsmdashsee Hicks 2004) There is also some concern among non-English speaking

countries that they are penalized in international rankings because so many of the major

journals (notably Science and Nature) are printed in English However the one set of

rankings that uses separate indicators to monitor articles published in English and articles

published in another language (the Wuhan rankings) shows that the two indicators are

positively correlated institutions that have more Chinese publications are also likely to

have more English publications and vice versa

Several sets of league tables measure bibliometric citations in various publication indices

The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a

weight of 20 of the final total Guangdong also monitors other Chinese universities

specifically for citations in engineering publications and weights this at 29 Moreover it

tacks on an additional 1049 for citations in science-oriented indices such as the Science

Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the

same category of scientific citations while the Melbourne Institute rates science citations at

68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social

Science Citation Index which does not include the humanities) are noted in only two

league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne

Institute (32) Another way of measuring research

11

impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a

weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the

Melbourne Institute

The complement of citations is of course publications Listing the number of publications

an individual group department or whole university releases can act as a weak substitute

for citationsmdashweak because simply publishing a paper or monograph is no guarantee that

the general public or other researchers will even glance at the work Guangdong gives an

1179 weighting to publications in science-oriented indices such as the Science Citation

Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same

indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong

even has a separate category just for measuring publications in Science and Nature

although it accords it an almost derisory weighting of 06 Under publications in social

science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their

final totals For publications in other indices (where the subject indices are

undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for

Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of

rankings to include research monographs weighted at 033

In countries where there are specific third-party evaluations of research output academic

quality of research is sometimes used as a research indicator The Times puts a very large

30 weight on this indicator while the Financial Times puts it at 119

Research awards are another handy third-party measurement of quality as the number of

international and national awards won by faculty andor graduates is often considered a

useful measure of institutional success International research awardsmdash specifically the

number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator

by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as

particularly suspect in some quarters given that the points are based on where the

recipient went to school rather than on where they are or were on the

8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the

most prominent publication indices

9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences

12

faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead

for nearly a century Wuhan uses similar measures but only accords them a weight of 14

National research awards are more common as a quality indicator used by La Repubblica

(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)

Financial indicators of research are also very common Research budgets as a factor in the

overall assessment of research in universities are covered by the Financial Times (9)

Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for

total amount of research expenditure unfortunately it is unclear precisely what this

research expenditure represents or how it is determined although it is clearly indicated

that it does not represent the total number of grants or projects at a university Total

number of research-based grants and projects is weighted by Education18 at 15 and

Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science

and engineering and another 55 to those for social sciences and humanities Similarly

the Melbourne Institute gives 6 of its overall weight to public-source grants making no

distinction between areas of study

In a slightly different vein Netbig (46) and Wuhan (278) both list the number of

research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica

(095) all weigh research-basedaffiliated research institutions or centres for studies

Finally one can also measure research not simply in terms of the amount of money it

generates but also in terms of the amount of future income it will generate Both

Guangdong (245) and Wuhan (193) measure the number of patents issued to

universities as a quality indicator

A final way of measuring an institutionrsquos research intensity is to look at the range of its

course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of

doctoral and Masterrsquos programs offered as a proxy for research intensity

As with physical and financial resources few if any of the research indicators are

normalized to account for institutional size (either by student or faculty numbers) In the

world of rankings bigger almost always means better an institution with 100 faculty with

ten citations apiece will always look worse than an institution with 1001 faculty

13

with one citation each To the extent that the raw production of knowledge matters this

form of measurement is acceptable To the extent that rankings are meant to show how

well institutions are doing on a like-to-like basis or to show the efficiency of universities it

is plainly inadequate This should be of particular concern to Chinese policy-makers

whose ranking systems are especially reliant on research-based indicators

G Indicators of Reputation

The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo

Those rankings systems which use the results of reputation surveys as an indicator do so as

an indirect measure of quality based on the assumption that the employers academics and

academic administrators surveyed have opinions of institutional quality that are informed

up-to-date and impartial While these assumptions are clearly open to debate they

nevertheless form an important basis for many ranking systems Another reason for using

reputation measures is the paucity of other data availablemdashsome countries have few

independent measures of teaching effectiveness university resources or output and

reputation can thus act as a useful surrogate Reputation rankings are often criticized as

simply quantifying the common ignorance of the people being surveyed However to the

extent that the people being surveyed hold positions which have the potential to affect

large numbers of young people and whose positions actually require some knowledge of

institutional quality (ie officials in charge of graduate admissions corporate recruiters

etc) then reputation rankings make sense because they provide useful information for

students about the perceived value of the degrees that they could obtain from various

universities

The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and

the Times which both accord reputation a weighting of 50 in their overall ranking

scheme Education18 assigns it almost as much significance at 40 The US News and World

Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just

below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)

and Netbig (15) The only other study to include reputation is Wuhan (117)

14

1 Please see Appendix B for a glossary of the various bodies referenced in this document

Appendix A Detailed Listing of Indicators and their Sources

Beginning Characteristics Indicator

Used By Source1

Incoming grades Macleans University

Percentage with grades above a set limit

Macleans University

US News and World Report

University

Performance on national standardised tests or benchmarks

Asiaweek University

Education18 3rd-party JUPAS

Financial Times Government agency 3rd-party UCAS

Guardian University Guide

Government agency 3rd-party UCAS

Melbourne Institute Government agency 3rd-party DEST

Netbig National entrance examination board

Times Good University Guide

Government agency 3rd-party UCAS

US News University

Wuhan Unknown presumed government 3rd-party

Student status La Repubblica Government agency 3rd-party MIUR

Admittance selectivity general

Asiaweek University

Admittance number of applications to places

Asiaweek University

Financial Times Government agency 3rd-party UCAS

La Repubblica Government agency 3rd-party MIUR

US News University

15

Beginning Characteristics Indicator

Used By Source1

Out-of-locality student percentage

Macleans University

International student percentages

Financial Times Government agency 3rd-party HESA

Macleans University

Shanghai Institute of Educational Science

Times World University

Wuhan Unknown presumed university

Undergraduate students among all students percentages

Netbig

Wuhan Unknown presumed university

Ethnic diversity in student body

Guardian University

Learning Inputs ndash Staff Indicator

Used By Source

Facultystudent ratio Asiaweek University

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Financial Times Government agency 3rd-party HESA

La Repubblica Government agency 3rd-party MIUR

Times Good University Guide

Government agency 3rd-party HESA

Times World University

US News University

Wuhan Unknown presumed university

16

Learning Inputs ndash Staff Indicator

Used By Source

Social science faculty student ratio

Melbourne Government agency 3rd-party DEST

Science faculty student ratio

Melbourne Government agency 3rd-party DEST

Administrative staff student ratio

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Staff student ratio (regardless of division)

Guardian Government agency 3rd-party HESA

Netbig University

Course per teacher La Repubblica Government agency 3rd-party MIUR

Per-teacher university spending

Asiaweek University

Faculty pay rates fortenured staff

Asiaweek University

US News University

Number of full-time part-time faculty

Netbig University

US News University

Wuhan Unknown presumed university

Faculty with researchprojects

Wuhan Unknown presumed university

Class size differentiation Macleans University

US News University

Classes taught by tenured faculty

Macleans University

Exchange programmes hosted

La Repubblica Government agency 3rd-party AgNaSoc

Number of classes lsquoactually taughtrsquo

La Repubblica Government agency 3rd-party MIUR

of international faculty (v faculty as a whole)

Times World University

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

9INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

EspacesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Nombre de megravetres carreacutes bruts input G10MCB eacutetudiants en eacutequivalence temps complet input G10megravetres carreacutes nets alloueacutes input MEQNombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input Tbesoin despace en megravetres carreacutes nets inputExistence dun plan directeur des espaces inputBudget preacutevu au plan directeur des espaces input TBudget alloueacute aux espaces input MEQdeacutepenses de fonctionnement nombre de megravetres carreacutes bruts input G10

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Environnement USNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

consommation de leacutenergie (exprimeacutee en BTU processus universiteacutes canadiennes et ameacutericaines BTUpopulation totale de luniversiteacute processus voir en particulier Penn State et Umich de leacutenergie produit par des sources renouvelable processusUtilisation de leau processuslitres deau par personne processusGaz agrave effet de serre (en tonnes meacutetriques CO2 eacutequivalents) processusMT CO2 par personne processusDeacutechets solides (en tonnes) processusTonnes de deacutechets solides per capita processus de deacutechets solides recycleacutes processus

RP gouvernance - 2008-06-23

10INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

ReacuteputationUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

enquecircte meneacutee aupregraves des eacutetudiants outcome NSSE Melbourneenquecircte meneacutee aupregraves des diplocircmeacutes outcomeenquecircte meneacutee aupregraves des pairs outcome USNews Asiaweek THESenquecircte meneacutee aupregraves des employeurs outcome Macleans Melbourneenquecircte meneacutee aupregraves des stakeholders outcome Macleans Melbourneenquecircte meneacutee aupregraves du monde des affaires outcome Macleans Melbournecouverture meacutediatique outcome AT

RP gouvernance - 2008-06-23

11INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Planification et gestion strateacutegiquesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Pouvoirs des corps universitaire processus un Canliens dynamiques entre Conseil Assembleacutee universitaire commission des eacutetudes processus un Canprocessus formel de planification strateacutegique processus un Canproduction de donneacutees et indicateurs processus un Canprocessus deacutevaluation des faculteacutes deacutepartements et services processus un Canrevue systeacutematique des responsabiliteacutes et efficaciteacute des comiteacutes processus un Candeacutepenses de ladministration centrale deacutepenses de fonctionnement processus un Can

RP gouvernance - 2008-06-23

12INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Eacuteducation permanenteUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre de programmes offerts input un Cannombre deacutetudiants inscrits input un CanRevenus et deacutepenses du service input un Can

RP gouvernance - 2008-06-23

13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes

REacuteFEacuteRENCES

Aperccedilu densemble des principaux indicateurs

GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost

Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes

MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf

Indicateurs utiliseacutes dans les classements internationaux

Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml

RP gouvernance - 2008-06-23

ANNEXES

Quelques exemples de listes drsquoindicateurs

1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)

Voir page suivante

Preacutepareacute par Roland Proulx

Consultant en planification institutionnelle et veille strateacutegique

Montreacuteal mai 2008

Performance Indicators

Discussion Paper

May 2004 (Prepared by the Office of Planning amp Analysis)

Table of Contents

Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25

2

Table 1 ndash Number of Indictors by Type of Data Source

Table 1 shows that surveys are the least frequently used source of data for indicators

Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a

plurality of indicators come from this source This measure somewhat underestimates the

importance of surveys however as it does not account for the weighting given to each

indicator in each study In the THES World Rankings for instance there is only a single

survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos

Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one

indicator is worth 20 of the final score

Raw

indIcator count

Surve

y data

Third parties

Universities

AsiaweekmdashAsias Best Universities 18 - - 18

Daily Telegraph (2003) 1 - 1 -

Education18com 9 3 4 2

Excelencia 2001 71 - 71 -

Financial Times (2003) 17 - 17 -

Guangdong Institute of Management Science 17 - 14 3

GuardianmdashUniversity Guide 2005 7 - 2 5

La Repubblica 23 2 21 -

Macleans University Rankings 24 1 5 18

Melbourne Institutemdash International Standing of Australian Universities

26 3 23 -

Netbig 2004 18 1 10 7

Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15

Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities

6 - 5 1

The TimesmdashGood University Guide 2005 9 - 9 -

Times Higher Education SupplementmdashWorld University Rankings

5 1 1 3

US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11

Washington MonthlymdashCollege Rankings 2005 8 - 1 7

Wuhan University Centre for Science Evaluation 45 2 22 21

3

A Indicators of Beginning Characteristics

ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of

students at the time they begin their studies Fourteen of the 18 rankings examined in this

study use one or more indicators of the beginning characteristics of students to arrive at

their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on

these factors with 33 of the total ranking coming from this class of indicators but the

Guardian Education18 Asiaweek and the two other North American surveys also place

considerable emphasis on this category

There are six main indicators used to determine which institutions have students with

positive ldquobeginning characteristicsrdquo

The most common measure of beginning characteristics is performance on national

standardized tests with nine surveys using this as a measure Education18 and the

Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used

by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)

Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this

data is collected and standardized by national bodies it has the benefit of being seen as a

relatively impartial method of determining the relative ldquostrengthrdquo of the students entering

each institution Institutionsrsquo results can be scored by showing either averages or the

percentage of entering students meeting a particular standard

Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses

secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This

is a second-best solution made necessary by the absence of any national standardized test

in Canada (or indeed of any provincial standardized tests at the end of secondary school

in provinces other than Alberta) The lack of national standardization makes this an

undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is

truly equivalent to an ldquoArdquo in another jurisdiction

Another measure of the strength of the student body is the percentage of incoming

students receiving (third-party) scholarships which is worth 11 of the score in the

Wuhan survey One can also approach the issue by measuring institutional selectivity In

effect this method infers the strength of the student body by the proportion of

4

applicants rejected the theory being that the higher the number of rejected applicants the

stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)

Normally this measure is expressed as a straight ratio of acceptances to applications but it

can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the

greatest weight on this measure) as a ratio of enrolments to applications Within the US

there is some dispute as to what constitutes an offer of admission and whether or not late

admissions are included as noted by Ganeshananthan (2003)

Student bodies are often considered to be strong if the school is able to attract a large

number of international or out-of-district students or if they contain people from diverse

ethnic backgrounds A number of league tables use the international student indicator

(which like the selectivity indicator is arguably as much an indicator of prestige and

reputation as it is of student characteristics) although in no case does this indicator

account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a

quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data

on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian

puts a somewhat larger emphasis on this indicator in comparison to other league tables

which use similar variables

A very different take on this idea is present in the Washington Monthly which released its

first set of College Rankings in September 2005 With the declared aim of using an

institutionrsquos commitment to social mobility as a measure of quality it uses the percentage

of students from low-income backgrounds as an indicator (with percentage of students

receiving need-based government (Pell) grants used as a proxy)

Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy

statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the

percentage of the student population who are graduate students (arguably this is a

research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the

number of graduate students auditing classes is used as an indicator the assumption is

presumably that if people are auditing then the classes must be very attractive The Italian

La Repubblica ranks an institution according to the number of part-time students it has

contrary to prevailing North American views on the undesirability of part-time study the

Italian rankings see higher numbers of part-time students in a positive light as it is

evidence that an institution is becoming less rigid in its timetabling and

5

permitting students to juggle both work and study something which was nearly

impossible in that country just a few years ago

The Washington Monthly also has a final category of indicators which reflect studentsrsquo

beginning characteristics namely their likelihood of performing community service as

measured by the percentage of students in the US Peace Corps and Reserve Officer

Training Corps (ROTC) and the percentage of students involved in work-study in the

community3 Together these three indicators account for 33 of an institutionrsquos total

ranking

B Indicators of Learning InputsmdashStaff

Generally both the quantity and quality of staff are positively correlated with institutional

quality The problem of course is finding useful metrics for each of these factors

especially if one excludes as we have done here measures of research performance and

research intensity putting them in a separate category4

The simplest measure is simply the number of faculty unadjusted for things like size of

student body Most national league tables however prefer to use variations on the

concept of facultystudent ratio Others try to measure teaching intensity with measures

such as courses per teacher or hours spent in class per student (both in La Repubblica)

These kinds of measures usually account for between 2-5 of the final rankings although

in some cases (ie the Guardian) this figure can be as high as 20

Another important way of measuring how faculty resources are deployed is the measure

of average class size which is used only by Macleanrsquos and the US News and World Report

Ostensibly the reason for measuring class size is to account in some form for the degree

3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category

6

of attention devoted to actually teaching students Despite the fact that important research

(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level

the use of this indicator appears to be a spillover from the North American debates on

class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein

2000) Regardless of why the indicators are used they are extraordinarily important to

these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and

World Reportrsquos rankings respectively

A number of ranking systems try to look at staff qualifications such as the number of

PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the

Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further

than other surveys and actually looks at the proportion of classes taught by tenure-track

staff Others (ie THES) look at the number of foreign faculty based on the assumption

that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still

others (ie La Repubblica) look at the age structure of the faculty Another proxy for

institutional quality is the pay rates for tenured staff on the assumption that institutions

with higher rates of pay on average attract better faculty this measure has been used both

by the US News and World Report and Asiaweek

Finally a number of league tables rank faculty inputs on the basis of standardized third-

party evaluations 6

Education18 the Financial Times the Times the Guardian and the US

News and World Report league tables all use some sort of ranking criterion based at least in

part on this indicator or variations thereof

C Indicators of Learning InputsmdashResources

Resource inputsmdashcrudely the amount of current dollars equipment and books available

to students at an institutionmdashare widely considered an important measure of quality Yet

despite the apparent simplicity of counting dollars and measuring assets

5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each

university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future

7

the means by which institutional wealth is measured varies considerably between ranking

systems

There are a number of revenue-based measures of resources Macleanrsquos uses public funding

of institutional budgets as a factor in its analysis conversely the Financial Times uses the

private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of

total score) and the US News and World Report (5 of total score) also measure alumni

financial support as a measure of quality

For reasons that are not entirely clear league tables tend to favour measures of

expenditures rather than revenues The Guardian looks at total institutional expenditures

as an indicator Institutional expenditure on student services is used as a measure of

institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total

institutional scores respectively) Rzezspospolita does not measure student services

expenditures directly but does measure student services outputs such as number of

student athletes and number of study clubs which amounts to more or less the same thing

Macleanrsquos also gives out 433 of its total score based on institutional expenditures on

scholarships and bursaries

Various aspects of physical infrastructure are also used as measures of institutional

resources most directly in the case of La Repubblica which bases 317 of its total rank on

the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding

assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type

of physical infrastructure measured is available Internet bandwidth which was used by

Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth

roughly 3 of the total score

By some distance the infrastructure indicators most favoured by the compilers of league

tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on

this with 12 of the total quality mark being taken from various types of library

infrastructure measurements (including acquisitions per year total volumes average

number of volumes per student and yearly library expenditure outside of acquisitions)

Netbig and Education18 also use library volume holdings while Asiaweek the Financial

Times and the Times also use measures of library expenditures outside of acquisitions or

computerization of library resources as measures of institutional quality

8

One important factor to note is that most ranking systems do not normalize their resource

and infrastructure measures That is to say it is raw spending power or simple size of

assets that is usually measured rather than spending per studentprofessor or assets per

studentprofessor As a result a number of these rankings systems have inbuilt biases

towards larger institutions

D Indicators of Learning Outputs

Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge

learned over the course of a baccalaureate degreemdashshould be a basic indicator of

institutional quality Unfortunately good means of measuring these outputsmdashlike the

National Survey of Student Engagement (NSSE) and the College Learning Assessment

(CLA)mdashhave only recently become available and for the most part institutions are still

keeping their scores secret Outside of these measures only a few very crude indicators are

available which likely explains why learning outputs do not feature especially prominently

in most ranking schemes

The simplest types of measures of learning outputs are those linked to graduation and

retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong

and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7

the latter three also use rates of graduation from Masterrsquos programs as indicators In some

cases the weights on these measures can be very highmdashin the Guangdong rankings

graduation rates account for over 50 of the rankingmdashbut in most cases the weights are

10 or less Retention rates commonly meaning the progression rate of first-year students

into second year are accorded less importance The US News and World Report Macleanrsquos

the Melbourne Institute and La Repubblica all employ retention measures as indicators but

none of them are worth more than 4 of total weighting Two publications make specific

indicators for retention and graduation of international students Macleanrsquos (graduation

rates of international students) and the Melbourne Institute (retention rates of international

students) The Washington Monthly looks specifically at institutional retention rates

adjusted for the participation of lower-income students and gives higher scores to

institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores

and number of Pell Grant recipients the US

7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18

9

News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos

ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)

E Indicators of Final Outcomes

Final outcomes are indications of generalized outcomes for students after graduation

Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness

good citizenship) but given the somewhat utilitarian justifications for education that are

currently in fashion (see Wolf 2000) employment outcomes are the most commonly used

measure of final outcomes These are given particular emphasis by the Guardian (where

employment outcomes are worth 17 of the total score) but are also used by the Financial

Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the

Times are interestingly not concerned with employment per se but with ldquoemployment in

an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher

Education Statistics Agency (HESA) uses Standard Occupational Classifications to

measure the proportion of graduates in professional or white-collar jobs anyone not in

such a job is considered not to be working in an area related to their studies (it is unclear

what methodology is used by the Financial Times and the Times although we suspect their

methods are broadly similar)

The only other measure of final outcomes in use is percentage of graduates returning for

additional education which is an indicator used by both the Melbourne Institute and the

Financial Times This is a particularly important indicator for the latter as it is worth 21

of the final ranking

The lack of indicators concerning final outcomes is interesting since most government-

sponsored performance-indicator regimes around the world are very much concerned

with such measures especially with respect to employment Possibly this indicates that

ranking systems simply do not view education outcomes as relevant measures of

educational quality Alternatively it may be the case that they simply have not found a

reliable indicator of outcomes or that there are reliable indicators but that there is so little

variation between institutions that it makes no sense to rank based on the data

10

F Indicators of Research

Many of the league tables covered in this survey include sections and weightings related

to universitiesrsquo research efforts It is in this field of measurement that we see the greatest

diversity of indicators Presumably this is because research inputs and outputs lend

themselves much more easily to measurement and manipulation than other areas of

institutional activity

Three studies include research staff as part of their ranking scheme La Repubblica at 952

the Melbourne Institute (4) and Wuhan (078)

Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used

method of looking at research quality but it is not universally admired because different

disciplines use different means to communicate major advances in knowledge (leading

scientists invariably produce large numbers of journal articles leading social scientists

may produce fewer journal articles but instead have one or two long important

monographsmdashsee Hicks 2004) There is also some concern among non-English speaking

countries that they are penalized in international rankings because so many of the major

journals (notably Science and Nature) are printed in English However the one set of

rankings that uses separate indicators to monitor articles published in English and articles

published in another language (the Wuhan rankings) shows that the two indicators are

positively correlated institutions that have more Chinese publications are also likely to

have more English publications and vice versa

Several sets of league tables measure bibliometric citations in various publication indices

The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a

weight of 20 of the final total Guangdong also monitors other Chinese universities

specifically for citations in engineering publications and weights this at 29 Moreover it

tacks on an additional 1049 for citations in science-oriented indices such as the Science

Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the

same category of scientific citations while the Melbourne Institute rates science citations at

68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social

Science Citation Index which does not include the humanities) are noted in only two

league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne

Institute (32) Another way of measuring research

11

impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a

weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the

Melbourne Institute

The complement of citations is of course publications Listing the number of publications

an individual group department or whole university releases can act as a weak substitute

for citationsmdashweak because simply publishing a paper or monograph is no guarantee that

the general public or other researchers will even glance at the work Guangdong gives an

1179 weighting to publications in science-oriented indices such as the Science Citation

Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same

indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong

even has a separate category just for measuring publications in Science and Nature

although it accords it an almost derisory weighting of 06 Under publications in social

science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their

final totals For publications in other indices (where the subject indices are

undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for

Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of

rankings to include research monographs weighted at 033

In countries where there are specific third-party evaluations of research output academic

quality of research is sometimes used as a research indicator The Times puts a very large

30 weight on this indicator while the Financial Times puts it at 119

Research awards are another handy third-party measurement of quality as the number of

international and national awards won by faculty andor graduates is often considered a

useful measure of institutional success International research awardsmdash specifically the

number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator

by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as

particularly suspect in some quarters given that the points are based on where the

recipient went to school rather than on where they are or were on the

8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the

most prominent publication indices

9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences

12

faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead

for nearly a century Wuhan uses similar measures but only accords them a weight of 14

National research awards are more common as a quality indicator used by La Repubblica

(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)

Financial indicators of research are also very common Research budgets as a factor in the

overall assessment of research in universities are covered by the Financial Times (9)

Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for

total amount of research expenditure unfortunately it is unclear precisely what this

research expenditure represents or how it is determined although it is clearly indicated

that it does not represent the total number of grants or projects at a university Total

number of research-based grants and projects is weighted by Education18 at 15 and

Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science

and engineering and another 55 to those for social sciences and humanities Similarly

the Melbourne Institute gives 6 of its overall weight to public-source grants making no

distinction between areas of study

In a slightly different vein Netbig (46) and Wuhan (278) both list the number of

research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica

(095) all weigh research-basedaffiliated research institutions or centres for studies

Finally one can also measure research not simply in terms of the amount of money it

generates but also in terms of the amount of future income it will generate Both

Guangdong (245) and Wuhan (193) measure the number of patents issued to

universities as a quality indicator

A final way of measuring an institutionrsquos research intensity is to look at the range of its

course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of

doctoral and Masterrsquos programs offered as a proxy for research intensity

As with physical and financial resources few if any of the research indicators are

normalized to account for institutional size (either by student or faculty numbers) In the

world of rankings bigger almost always means better an institution with 100 faculty with

ten citations apiece will always look worse than an institution with 1001 faculty

13

with one citation each To the extent that the raw production of knowledge matters this

form of measurement is acceptable To the extent that rankings are meant to show how

well institutions are doing on a like-to-like basis or to show the efficiency of universities it

is plainly inadequate This should be of particular concern to Chinese policy-makers

whose ranking systems are especially reliant on research-based indicators

G Indicators of Reputation

The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo

Those rankings systems which use the results of reputation surveys as an indicator do so as

an indirect measure of quality based on the assumption that the employers academics and

academic administrators surveyed have opinions of institutional quality that are informed

up-to-date and impartial While these assumptions are clearly open to debate they

nevertheless form an important basis for many ranking systems Another reason for using

reputation measures is the paucity of other data availablemdashsome countries have few

independent measures of teaching effectiveness university resources or output and

reputation can thus act as a useful surrogate Reputation rankings are often criticized as

simply quantifying the common ignorance of the people being surveyed However to the

extent that the people being surveyed hold positions which have the potential to affect

large numbers of young people and whose positions actually require some knowledge of

institutional quality (ie officials in charge of graduate admissions corporate recruiters

etc) then reputation rankings make sense because they provide useful information for

students about the perceived value of the degrees that they could obtain from various

universities

The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and

the Times which both accord reputation a weighting of 50 in their overall ranking

scheme Education18 assigns it almost as much significance at 40 The US News and World

Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just

below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)

and Netbig (15) The only other study to include reputation is Wuhan (117)

14

1 Please see Appendix B for a glossary of the various bodies referenced in this document

Appendix A Detailed Listing of Indicators and their Sources

Beginning Characteristics Indicator

Used By Source1

Incoming grades Macleans University

Percentage with grades above a set limit

Macleans University

US News and World Report

University

Performance on national standardised tests or benchmarks

Asiaweek University

Education18 3rd-party JUPAS

Financial Times Government agency 3rd-party UCAS

Guardian University Guide

Government agency 3rd-party UCAS

Melbourne Institute Government agency 3rd-party DEST

Netbig National entrance examination board

Times Good University Guide

Government agency 3rd-party UCAS

US News University

Wuhan Unknown presumed government 3rd-party

Student status La Repubblica Government agency 3rd-party MIUR

Admittance selectivity general

Asiaweek University

Admittance number of applications to places

Asiaweek University

Financial Times Government agency 3rd-party UCAS

La Repubblica Government agency 3rd-party MIUR

US News University

15

Beginning Characteristics Indicator

Used By Source1

Out-of-locality student percentage

Macleans University

International student percentages

Financial Times Government agency 3rd-party HESA

Macleans University

Shanghai Institute of Educational Science

Times World University

Wuhan Unknown presumed university

Undergraduate students among all students percentages

Netbig

Wuhan Unknown presumed university

Ethnic diversity in student body

Guardian University

Learning Inputs ndash Staff Indicator

Used By Source

Facultystudent ratio Asiaweek University

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Financial Times Government agency 3rd-party HESA

La Repubblica Government agency 3rd-party MIUR

Times Good University Guide

Government agency 3rd-party HESA

Times World University

US News University

Wuhan Unknown presumed university

16

Learning Inputs ndash Staff Indicator

Used By Source

Social science faculty student ratio

Melbourne Government agency 3rd-party DEST

Science faculty student ratio

Melbourne Government agency 3rd-party DEST

Administrative staff student ratio

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Staff student ratio (regardless of division)

Guardian Government agency 3rd-party HESA

Netbig University

Course per teacher La Repubblica Government agency 3rd-party MIUR

Per-teacher university spending

Asiaweek University

Faculty pay rates fortenured staff

Asiaweek University

US News University

Number of full-time part-time faculty

Netbig University

US News University

Wuhan Unknown presumed university

Faculty with researchprojects

Wuhan Unknown presumed university

Class size differentiation Macleans University

US News University

Classes taught by tenured faculty

Macleans University

Exchange programmes hosted

La Repubblica Government agency 3rd-party AgNaSoc

Number of classes lsquoactually taughtrsquo

La Repubblica Government agency 3rd-party MIUR

of international faculty (v faculty as a whole)

Times World University

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

10INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

ReacuteputationUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

enquecircte meneacutee aupregraves des eacutetudiants outcome NSSE Melbourneenquecircte meneacutee aupregraves des diplocircmeacutes outcomeenquecircte meneacutee aupregraves des pairs outcome USNews Asiaweek THESenquecircte meneacutee aupregraves des employeurs outcome Macleans Melbourneenquecircte meneacutee aupregraves des stakeholders outcome Macleans Melbourneenquecircte meneacutee aupregraves du monde des affaires outcome Macleans Melbournecouverture meacutediatique outcome AT

RP gouvernance - 2008-06-23

11INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Planification et gestion strateacutegiquesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Pouvoirs des corps universitaire processus un Canliens dynamiques entre Conseil Assembleacutee universitaire commission des eacutetudes processus un Canprocessus formel de planification strateacutegique processus un Canproduction de donneacutees et indicateurs processus un Canprocessus deacutevaluation des faculteacutes deacutepartements et services processus un Canrevue systeacutematique des responsabiliteacutes et efficaciteacute des comiteacutes processus un Candeacutepenses de ladministration centrale deacutepenses de fonctionnement processus un Can

RP gouvernance - 2008-06-23

12INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Eacuteducation permanenteUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre de programmes offerts input un Cannombre deacutetudiants inscrits input un CanRevenus et deacutepenses du service input un Can

RP gouvernance - 2008-06-23

13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes

REacuteFEacuteRENCES

Aperccedilu densemble des principaux indicateurs

GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost

Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes

MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf

Indicateurs utiliseacutes dans les classements internationaux

Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml

RP gouvernance - 2008-06-23

ANNEXES

Quelques exemples de listes drsquoindicateurs

1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)

Voir page suivante

Preacutepareacute par Roland Proulx

Consultant en planification institutionnelle et veille strateacutegique

Montreacuteal mai 2008

Performance Indicators

Discussion Paper

May 2004 (Prepared by the Office of Planning amp Analysis)

Table of Contents

Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25

2

Table 1 ndash Number of Indictors by Type of Data Source

Table 1 shows that surveys are the least frequently used source of data for indicators

Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a

plurality of indicators come from this source This measure somewhat underestimates the

importance of surveys however as it does not account for the weighting given to each

indicator in each study In the THES World Rankings for instance there is only a single

survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos

Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one

indicator is worth 20 of the final score

Raw

indIcator count

Surve

y data

Third parties

Universities

AsiaweekmdashAsias Best Universities 18 - - 18

Daily Telegraph (2003) 1 - 1 -

Education18com 9 3 4 2

Excelencia 2001 71 - 71 -

Financial Times (2003) 17 - 17 -

Guangdong Institute of Management Science 17 - 14 3

GuardianmdashUniversity Guide 2005 7 - 2 5

La Repubblica 23 2 21 -

Macleans University Rankings 24 1 5 18

Melbourne Institutemdash International Standing of Australian Universities

26 3 23 -

Netbig 2004 18 1 10 7

Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15

Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities

6 - 5 1

The TimesmdashGood University Guide 2005 9 - 9 -

Times Higher Education SupplementmdashWorld University Rankings

5 1 1 3

US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11

Washington MonthlymdashCollege Rankings 2005 8 - 1 7

Wuhan University Centre for Science Evaluation 45 2 22 21

3

A Indicators of Beginning Characteristics

ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of

students at the time they begin their studies Fourteen of the 18 rankings examined in this

study use one or more indicators of the beginning characteristics of students to arrive at

their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on

these factors with 33 of the total ranking coming from this class of indicators but the

Guardian Education18 Asiaweek and the two other North American surveys also place

considerable emphasis on this category

There are six main indicators used to determine which institutions have students with

positive ldquobeginning characteristicsrdquo

The most common measure of beginning characteristics is performance on national

standardized tests with nine surveys using this as a measure Education18 and the

Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used

by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)

Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this

data is collected and standardized by national bodies it has the benefit of being seen as a

relatively impartial method of determining the relative ldquostrengthrdquo of the students entering

each institution Institutionsrsquo results can be scored by showing either averages or the

percentage of entering students meeting a particular standard

Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses

secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This

is a second-best solution made necessary by the absence of any national standardized test

in Canada (or indeed of any provincial standardized tests at the end of secondary school

in provinces other than Alberta) The lack of national standardization makes this an

undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is

truly equivalent to an ldquoArdquo in another jurisdiction

Another measure of the strength of the student body is the percentage of incoming

students receiving (third-party) scholarships which is worth 11 of the score in the

Wuhan survey One can also approach the issue by measuring institutional selectivity In

effect this method infers the strength of the student body by the proportion of

4

applicants rejected the theory being that the higher the number of rejected applicants the

stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)

Normally this measure is expressed as a straight ratio of acceptances to applications but it

can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the

greatest weight on this measure) as a ratio of enrolments to applications Within the US

there is some dispute as to what constitutes an offer of admission and whether or not late

admissions are included as noted by Ganeshananthan (2003)

Student bodies are often considered to be strong if the school is able to attract a large

number of international or out-of-district students or if they contain people from diverse

ethnic backgrounds A number of league tables use the international student indicator

(which like the selectivity indicator is arguably as much an indicator of prestige and

reputation as it is of student characteristics) although in no case does this indicator

account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a

quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data

on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian

puts a somewhat larger emphasis on this indicator in comparison to other league tables

which use similar variables

A very different take on this idea is present in the Washington Monthly which released its

first set of College Rankings in September 2005 With the declared aim of using an

institutionrsquos commitment to social mobility as a measure of quality it uses the percentage

of students from low-income backgrounds as an indicator (with percentage of students

receiving need-based government (Pell) grants used as a proxy)

Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy

statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the

percentage of the student population who are graduate students (arguably this is a

research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the

number of graduate students auditing classes is used as an indicator the assumption is

presumably that if people are auditing then the classes must be very attractive The Italian

La Repubblica ranks an institution according to the number of part-time students it has

contrary to prevailing North American views on the undesirability of part-time study the

Italian rankings see higher numbers of part-time students in a positive light as it is

evidence that an institution is becoming less rigid in its timetabling and

5

permitting students to juggle both work and study something which was nearly

impossible in that country just a few years ago

The Washington Monthly also has a final category of indicators which reflect studentsrsquo

beginning characteristics namely their likelihood of performing community service as

measured by the percentage of students in the US Peace Corps and Reserve Officer

Training Corps (ROTC) and the percentage of students involved in work-study in the

community3 Together these three indicators account for 33 of an institutionrsquos total

ranking

B Indicators of Learning InputsmdashStaff

Generally both the quantity and quality of staff are positively correlated with institutional

quality The problem of course is finding useful metrics for each of these factors

especially if one excludes as we have done here measures of research performance and

research intensity putting them in a separate category4

The simplest measure is simply the number of faculty unadjusted for things like size of

student body Most national league tables however prefer to use variations on the

concept of facultystudent ratio Others try to measure teaching intensity with measures

such as courses per teacher or hours spent in class per student (both in La Repubblica)

These kinds of measures usually account for between 2-5 of the final rankings although

in some cases (ie the Guardian) this figure can be as high as 20

Another important way of measuring how faculty resources are deployed is the measure

of average class size which is used only by Macleanrsquos and the US News and World Report

Ostensibly the reason for measuring class size is to account in some form for the degree

3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category

6

of attention devoted to actually teaching students Despite the fact that important research

(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level

the use of this indicator appears to be a spillover from the North American debates on

class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein

2000) Regardless of why the indicators are used they are extraordinarily important to

these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and

World Reportrsquos rankings respectively

A number of ranking systems try to look at staff qualifications such as the number of

PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the

Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further

than other surveys and actually looks at the proportion of classes taught by tenure-track

staff Others (ie THES) look at the number of foreign faculty based on the assumption

that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still

others (ie La Repubblica) look at the age structure of the faculty Another proxy for

institutional quality is the pay rates for tenured staff on the assumption that institutions

with higher rates of pay on average attract better faculty this measure has been used both

by the US News and World Report and Asiaweek

Finally a number of league tables rank faculty inputs on the basis of standardized third-

party evaluations 6

Education18 the Financial Times the Times the Guardian and the US

News and World Report league tables all use some sort of ranking criterion based at least in

part on this indicator or variations thereof

C Indicators of Learning InputsmdashResources

Resource inputsmdashcrudely the amount of current dollars equipment and books available

to students at an institutionmdashare widely considered an important measure of quality Yet

despite the apparent simplicity of counting dollars and measuring assets

5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each

university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future

7

the means by which institutional wealth is measured varies considerably between ranking

systems

There are a number of revenue-based measures of resources Macleanrsquos uses public funding

of institutional budgets as a factor in its analysis conversely the Financial Times uses the

private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of

total score) and the US News and World Report (5 of total score) also measure alumni

financial support as a measure of quality

For reasons that are not entirely clear league tables tend to favour measures of

expenditures rather than revenues The Guardian looks at total institutional expenditures

as an indicator Institutional expenditure on student services is used as a measure of

institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total

institutional scores respectively) Rzezspospolita does not measure student services

expenditures directly but does measure student services outputs such as number of

student athletes and number of study clubs which amounts to more or less the same thing

Macleanrsquos also gives out 433 of its total score based on institutional expenditures on

scholarships and bursaries

Various aspects of physical infrastructure are also used as measures of institutional

resources most directly in the case of La Repubblica which bases 317 of its total rank on

the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding

assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type

of physical infrastructure measured is available Internet bandwidth which was used by

Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth

roughly 3 of the total score

By some distance the infrastructure indicators most favoured by the compilers of league

tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on

this with 12 of the total quality mark being taken from various types of library

infrastructure measurements (including acquisitions per year total volumes average

number of volumes per student and yearly library expenditure outside of acquisitions)

Netbig and Education18 also use library volume holdings while Asiaweek the Financial

Times and the Times also use measures of library expenditures outside of acquisitions or

computerization of library resources as measures of institutional quality

8

One important factor to note is that most ranking systems do not normalize their resource

and infrastructure measures That is to say it is raw spending power or simple size of

assets that is usually measured rather than spending per studentprofessor or assets per

studentprofessor As a result a number of these rankings systems have inbuilt biases

towards larger institutions

D Indicators of Learning Outputs

Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge

learned over the course of a baccalaureate degreemdashshould be a basic indicator of

institutional quality Unfortunately good means of measuring these outputsmdashlike the

National Survey of Student Engagement (NSSE) and the College Learning Assessment

(CLA)mdashhave only recently become available and for the most part institutions are still

keeping their scores secret Outside of these measures only a few very crude indicators are

available which likely explains why learning outputs do not feature especially prominently

in most ranking schemes

The simplest types of measures of learning outputs are those linked to graduation and

retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong

and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7

the latter three also use rates of graduation from Masterrsquos programs as indicators In some

cases the weights on these measures can be very highmdashin the Guangdong rankings

graduation rates account for over 50 of the rankingmdashbut in most cases the weights are

10 or less Retention rates commonly meaning the progression rate of first-year students

into second year are accorded less importance The US News and World Report Macleanrsquos

the Melbourne Institute and La Repubblica all employ retention measures as indicators but

none of them are worth more than 4 of total weighting Two publications make specific

indicators for retention and graduation of international students Macleanrsquos (graduation

rates of international students) and the Melbourne Institute (retention rates of international

students) The Washington Monthly looks specifically at institutional retention rates

adjusted for the participation of lower-income students and gives higher scores to

institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores

and number of Pell Grant recipients the US

7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18

9

News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos

ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)

E Indicators of Final Outcomes

Final outcomes are indications of generalized outcomes for students after graduation

Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness

good citizenship) but given the somewhat utilitarian justifications for education that are

currently in fashion (see Wolf 2000) employment outcomes are the most commonly used

measure of final outcomes These are given particular emphasis by the Guardian (where

employment outcomes are worth 17 of the total score) but are also used by the Financial

Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the

Times are interestingly not concerned with employment per se but with ldquoemployment in

an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher

Education Statistics Agency (HESA) uses Standard Occupational Classifications to

measure the proportion of graduates in professional or white-collar jobs anyone not in

such a job is considered not to be working in an area related to their studies (it is unclear

what methodology is used by the Financial Times and the Times although we suspect their

methods are broadly similar)

The only other measure of final outcomes in use is percentage of graduates returning for

additional education which is an indicator used by both the Melbourne Institute and the

Financial Times This is a particularly important indicator for the latter as it is worth 21

of the final ranking

The lack of indicators concerning final outcomes is interesting since most government-

sponsored performance-indicator regimes around the world are very much concerned

with such measures especially with respect to employment Possibly this indicates that

ranking systems simply do not view education outcomes as relevant measures of

educational quality Alternatively it may be the case that they simply have not found a

reliable indicator of outcomes or that there are reliable indicators but that there is so little

variation between institutions that it makes no sense to rank based on the data

10

F Indicators of Research

Many of the league tables covered in this survey include sections and weightings related

to universitiesrsquo research efforts It is in this field of measurement that we see the greatest

diversity of indicators Presumably this is because research inputs and outputs lend

themselves much more easily to measurement and manipulation than other areas of

institutional activity

Three studies include research staff as part of their ranking scheme La Repubblica at 952

the Melbourne Institute (4) and Wuhan (078)

Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used

method of looking at research quality but it is not universally admired because different

disciplines use different means to communicate major advances in knowledge (leading

scientists invariably produce large numbers of journal articles leading social scientists

may produce fewer journal articles but instead have one or two long important

monographsmdashsee Hicks 2004) There is also some concern among non-English speaking

countries that they are penalized in international rankings because so many of the major

journals (notably Science and Nature) are printed in English However the one set of

rankings that uses separate indicators to monitor articles published in English and articles

published in another language (the Wuhan rankings) shows that the two indicators are

positively correlated institutions that have more Chinese publications are also likely to

have more English publications and vice versa

Several sets of league tables measure bibliometric citations in various publication indices

The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a

weight of 20 of the final total Guangdong also monitors other Chinese universities

specifically for citations in engineering publications and weights this at 29 Moreover it

tacks on an additional 1049 for citations in science-oriented indices such as the Science

Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the

same category of scientific citations while the Melbourne Institute rates science citations at

68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social

Science Citation Index which does not include the humanities) are noted in only two

league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne

Institute (32) Another way of measuring research

11

impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a

weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the

Melbourne Institute

The complement of citations is of course publications Listing the number of publications

an individual group department or whole university releases can act as a weak substitute

for citationsmdashweak because simply publishing a paper or monograph is no guarantee that

the general public or other researchers will even glance at the work Guangdong gives an

1179 weighting to publications in science-oriented indices such as the Science Citation

Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same

indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong

even has a separate category just for measuring publications in Science and Nature

although it accords it an almost derisory weighting of 06 Under publications in social

science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their

final totals For publications in other indices (where the subject indices are

undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for

Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of

rankings to include research monographs weighted at 033

In countries where there are specific third-party evaluations of research output academic

quality of research is sometimes used as a research indicator The Times puts a very large

30 weight on this indicator while the Financial Times puts it at 119

Research awards are another handy third-party measurement of quality as the number of

international and national awards won by faculty andor graduates is often considered a

useful measure of institutional success International research awardsmdash specifically the

number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator

by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as

particularly suspect in some quarters given that the points are based on where the

recipient went to school rather than on where they are or were on the

8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the

most prominent publication indices

9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences

12

faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead

for nearly a century Wuhan uses similar measures but only accords them a weight of 14

National research awards are more common as a quality indicator used by La Repubblica

(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)

Financial indicators of research are also very common Research budgets as a factor in the

overall assessment of research in universities are covered by the Financial Times (9)

Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for

total amount of research expenditure unfortunately it is unclear precisely what this

research expenditure represents or how it is determined although it is clearly indicated

that it does not represent the total number of grants or projects at a university Total

number of research-based grants and projects is weighted by Education18 at 15 and

Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science

and engineering and another 55 to those for social sciences and humanities Similarly

the Melbourne Institute gives 6 of its overall weight to public-source grants making no

distinction between areas of study

In a slightly different vein Netbig (46) and Wuhan (278) both list the number of

research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica

(095) all weigh research-basedaffiliated research institutions or centres for studies

Finally one can also measure research not simply in terms of the amount of money it

generates but also in terms of the amount of future income it will generate Both

Guangdong (245) and Wuhan (193) measure the number of patents issued to

universities as a quality indicator

A final way of measuring an institutionrsquos research intensity is to look at the range of its

course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of

doctoral and Masterrsquos programs offered as a proxy for research intensity

As with physical and financial resources few if any of the research indicators are

normalized to account for institutional size (either by student or faculty numbers) In the

world of rankings bigger almost always means better an institution with 100 faculty with

ten citations apiece will always look worse than an institution with 1001 faculty

13

with one citation each To the extent that the raw production of knowledge matters this

form of measurement is acceptable To the extent that rankings are meant to show how

well institutions are doing on a like-to-like basis or to show the efficiency of universities it

is plainly inadequate This should be of particular concern to Chinese policy-makers

whose ranking systems are especially reliant on research-based indicators

G Indicators of Reputation

The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo

Those rankings systems which use the results of reputation surveys as an indicator do so as

an indirect measure of quality based on the assumption that the employers academics and

academic administrators surveyed have opinions of institutional quality that are informed

up-to-date and impartial While these assumptions are clearly open to debate they

nevertheless form an important basis for many ranking systems Another reason for using

reputation measures is the paucity of other data availablemdashsome countries have few

independent measures of teaching effectiveness university resources or output and

reputation can thus act as a useful surrogate Reputation rankings are often criticized as

simply quantifying the common ignorance of the people being surveyed However to the

extent that the people being surveyed hold positions which have the potential to affect

large numbers of young people and whose positions actually require some knowledge of

institutional quality (ie officials in charge of graduate admissions corporate recruiters

etc) then reputation rankings make sense because they provide useful information for

students about the perceived value of the degrees that they could obtain from various

universities

The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and

the Times which both accord reputation a weighting of 50 in their overall ranking

scheme Education18 assigns it almost as much significance at 40 The US News and World

Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just

below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)

and Netbig (15) The only other study to include reputation is Wuhan (117)

14

1 Please see Appendix B for a glossary of the various bodies referenced in this document

Appendix A Detailed Listing of Indicators and their Sources

Beginning Characteristics Indicator

Used By Source1

Incoming grades Macleans University

Percentage with grades above a set limit

Macleans University

US News and World Report

University

Performance on national standardised tests or benchmarks

Asiaweek University

Education18 3rd-party JUPAS

Financial Times Government agency 3rd-party UCAS

Guardian University Guide

Government agency 3rd-party UCAS

Melbourne Institute Government agency 3rd-party DEST

Netbig National entrance examination board

Times Good University Guide

Government agency 3rd-party UCAS

US News University

Wuhan Unknown presumed government 3rd-party

Student status La Repubblica Government agency 3rd-party MIUR

Admittance selectivity general

Asiaweek University

Admittance number of applications to places

Asiaweek University

Financial Times Government agency 3rd-party UCAS

La Repubblica Government agency 3rd-party MIUR

US News University

15

Beginning Characteristics Indicator

Used By Source1

Out-of-locality student percentage

Macleans University

International student percentages

Financial Times Government agency 3rd-party HESA

Macleans University

Shanghai Institute of Educational Science

Times World University

Wuhan Unknown presumed university

Undergraduate students among all students percentages

Netbig

Wuhan Unknown presumed university

Ethnic diversity in student body

Guardian University

Learning Inputs ndash Staff Indicator

Used By Source

Facultystudent ratio Asiaweek University

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Financial Times Government agency 3rd-party HESA

La Repubblica Government agency 3rd-party MIUR

Times Good University Guide

Government agency 3rd-party HESA

Times World University

US News University

Wuhan Unknown presumed university

16

Learning Inputs ndash Staff Indicator

Used By Source

Social science faculty student ratio

Melbourne Government agency 3rd-party DEST

Science faculty student ratio

Melbourne Government agency 3rd-party DEST

Administrative staff student ratio

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Staff student ratio (regardless of division)

Guardian Government agency 3rd-party HESA

Netbig University

Course per teacher La Repubblica Government agency 3rd-party MIUR

Per-teacher university spending

Asiaweek University

Faculty pay rates fortenured staff

Asiaweek University

US News University

Number of full-time part-time faculty

Netbig University

US News University

Wuhan Unknown presumed university

Faculty with researchprojects

Wuhan Unknown presumed university

Class size differentiation Macleans University

US News University

Classes taught by tenured faculty

Macleans University

Exchange programmes hosted

La Repubblica Government agency 3rd-party AgNaSoc

Number of classes lsquoactually taughtrsquo

La Repubblica Government agency 3rd-party MIUR

of international faculty (v faculty as a whole)

Times World University

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

11INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Planification et gestion strateacutegiquesUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

Pouvoirs des corps universitaire processus un Canliens dynamiques entre Conseil Assembleacutee universitaire commission des eacutetudes processus un Canprocessus formel de planification strateacutegique processus un Canproduction de donneacutees et indicateurs processus un Canprocessus deacutevaluation des faculteacutes deacutepartements et services processus un Canrevue systeacutematique des responsabiliteacutes et efficaciteacute des comiteacutes processus un Candeacutepenses de ladministration centrale deacutepenses de fonctionnement processus un Can

RP gouvernance - 2008-06-23

12INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Eacuteducation permanenteUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre de programmes offerts input un Cannombre deacutetudiants inscrits input un CanRevenus et deacutepenses du service input un Can

RP gouvernance - 2008-06-23

13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes

REacuteFEacuteRENCES

Aperccedilu densemble des principaux indicateurs

GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost

Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes

MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf

Indicateurs utiliseacutes dans les classements internationaux

Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml

RP gouvernance - 2008-06-23

ANNEXES

Quelques exemples de listes drsquoindicateurs

1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)

Voir page suivante

Preacutepareacute par Roland Proulx

Consultant en planification institutionnelle et veille strateacutegique

Montreacuteal mai 2008

Performance Indicators

Discussion Paper

May 2004 (Prepared by the Office of Planning amp Analysis)

Table of Contents

Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25

2

Table 1 ndash Number of Indictors by Type of Data Source

Table 1 shows that surveys are the least frequently used source of data for indicators

Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a

plurality of indicators come from this source This measure somewhat underestimates the

importance of surveys however as it does not account for the weighting given to each

indicator in each study In the THES World Rankings for instance there is only a single

survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos

Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one

indicator is worth 20 of the final score

Raw

indIcator count

Surve

y data

Third parties

Universities

AsiaweekmdashAsias Best Universities 18 - - 18

Daily Telegraph (2003) 1 - 1 -

Education18com 9 3 4 2

Excelencia 2001 71 - 71 -

Financial Times (2003) 17 - 17 -

Guangdong Institute of Management Science 17 - 14 3

GuardianmdashUniversity Guide 2005 7 - 2 5

La Repubblica 23 2 21 -

Macleans University Rankings 24 1 5 18

Melbourne Institutemdash International Standing of Australian Universities

26 3 23 -

Netbig 2004 18 1 10 7

Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15

Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities

6 - 5 1

The TimesmdashGood University Guide 2005 9 - 9 -

Times Higher Education SupplementmdashWorld University Rankings

5 1 1 3

US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11

Washington MonthlymdashCollege Rankings 2005 8 - 1 7

Wuhan University Centre for Science Evaluation 45 2 22 21

3

A Indicators of Beginning Characteristics

ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of

students at the time they begin their studies Fourteen of the 18 rankings examined in this

study use one or more indicators of the beginning characteristics of students to arrive at

their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on

these factors with 33 of the total ranking coming from this class of indicators but the

Guardian Education18 Asiaweek and the two other North American surveys also place

considerable emphasis on this category

There are six main indicators used to determine which institutions have students with

positive ldquobeginning characteristicsrdquo

The most common measure of beginning characteristics is performance on national

standardized tests with nine surveys using this as a measure Education18 and the

Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used

by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)

Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this

data is collected and standardized by national bodies it has the benefit of being seen as a

relatively impartial method of determining the relative ldquostrengthrdquo of the students entering

each institution Institutionsrsquo results can be scored by showing either averages or the

percentage of entering students meeting a particular standard

Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses

secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This

is a second-best solution made necessary by the absence of any national standardized test

in Canada (or indeed of any provincial standardized tests at the end of secondary school

in provinces other than Alberta) The lack of national standardization makes this an

undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is

truly equivalent to an ldquoArdquo in another jurisdiction

Another measure of the strength of the student body is the percentage of incoming

students receiving (third-party) scholarships which is worth 11 of the score in the

Wuhan survey One can also approach the issue by measuring institutional selectivity In

effect this method infers the strength of the student body by the proportion of

4

applicants rejected the theory being that the higher the number of rejected applicants the

stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)

Normally this measure is expressed as a straight ratio of acceptances to applications but it

can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the

greatest weight on this measure) as a ratio of enrolments to applications Within the US

there is some dispute as to what constitutes an offer of admission and whether or not late

admissions are included as noted by Ganeshananthan (2003)

Student bodies are often considered to be strong if the school is able to attract a large

number of international or out-of-district students or if they contain people from diverse

ethnic backgrounds A number of league tables use the international student indicator

(which like the selectivity indicator is arguably as much an indicator of prestige and

reputation as it is of student characteristics) although in no case does this indicator

account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a

quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data

on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian

puts a somewhat larger emphasis on this indicator in comparison to other league tables

which use similar variables

A very different take on this idea is present in the Washington Monthly which released its

first set of College Rankings in September 2005 With the declared aim of using an

institutionrsquos commitment to social mobility as a measure of quality it uses the percentage

of students from low-income backgrounds as an indicator (with percentage of students

receiving need-based government (Pell) grants used as a proxy)

Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy

statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the

percentage of the student population who are graduate students (arguably this is a

research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the

number of graduate students auditing classes is used as an indicator the assumption is

presumably that if people are auditing then the classes must be very attractive The Italian

La Repubblica ranks an institution according to the number of part-time students it has

contrary to prevailing North American views on the undesirability of part-time study the

Italian rankings see higher numbers of part-time students in a positive light as it is

evidence that an institution is becoming less rigid in its timetabling and

5

permitting students to juggle both work and study something which was nearly

impossible in that country just a few years ago

The Washington Monthly also has a final category of indicators which reflect studentsrsquo

beginning characteristics namely their likelihood of performing community service as

measured by the percentage of students in the US Peace Corps and Reserve Officer

Training Corps (ROTC) and the percentage of students involved in work-study in the

community3 Together these three indicators account for 33 of an institutionrsquos total

ranking

B Indicators of Learning InputsmdashStaff

Generally both the quantity and quality of staff are positively correlated with institutional

quality The problem of course is finding useful metrics for each of these factors

especially if one excludes as we have done here measures of research performance and

research intensity putting them in a separate category4

The simplest measure is simply the number of faculty unadjusted for things like size of

student body Most national league tables however prefer to use variations on the

concept of facultystudent ratio Others try to measure teaching intensity with measures

such as courses per teacher or hours spent in class per student (both in La Repubblica)

These kinds of measures usually account for between 2-5 of the final rankings although

in some cases (ie the Guardian) this figure can be as high as 20

Another important way of measuring how faculty resources are deployed is the measure

of average class size which is used only by Macleanrsquos and the US News and World Report

Ostensibly the reason for measuring class size is to account in some form for the degree

3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category

6

of attention devoted to actually teaching students Despite the fact that important research

(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level

the use of this indicator appears to be a spillover from the North American debates on

class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein

2000) Regardless of why the indicators are used they are extraordinarily important to

these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and

World Reportrsquos rankings respectively

A number of ranking systems try to look at staff qualifications such as the number of

PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the

Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further

than other surveys and actually looks at the proportion of classes taught by tenure-track

staff Others (ie THES) look at the number of foreign faculty based on the assumption

that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still

others (ie La Repubblica) look at the age structure of the faculty Another proxy for

institutional quality is the pay rates for tenured staff on the assumption that institutions

with higher rates of pay on average attract better faculty this measure has been used both

by the US News and World Report and Asiaweek

Finally a number of league tables rank faculty inputs on the basis of standardized third-

party evaluations 6

Education18 the Financial Times the Times the Guardian and the US

News and World Report league tables all use some sort of ranking criterion based at least in

part on this indicator or variations thereof

C Indicators of Learning InputsmdashResources

Resource inputsmdashcrudely the amount of current dollars equipment and books available

to students at an institutionmdashare widely considered an important measure of quality Yet

despite the apparent simplicity of counting dollars and measuring assets

5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each

university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future

7

the means by which institutional wealth is measured varies considerably between ranking

systems

There are a number of revenue-based measures of resources Macleanrsquos uses public funding

of institutional budgets as a factor in its analysis conversely the Financial Times uses the

private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of

total score) and the US News and World Report (5 of total score) also measure alumni

financial support as a measure of quality

For reasons that are not entirely clear league tables tend to favour measures of

expenditures rather than revenues The Guardian looks at total institutional expenditures

as an indicator Institutional expenditure on student services is used as a measure of

institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total

institutional scores respectively) Rzezspospolita does not measure student services

expenditures directly but does measure student services outputs such as number of

student athletes and number of study clubs which amounts to more or less the same thing

Macleanrsquos also gives out 433 of its total score based on institutional expenditures on

scholarships and bursaries

Various aspects of physical infrastructure are also used as measures of institutional

resources most directly in the case of La Repubblica which bases 317 of its total rank on

the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding

assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type

of physical infrastructure measured is available Internet bandwidth which was used by

Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth

roughly 3 of the total score

By some distance the infrastructure indicators most favoured by the compilers of league

tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on

this with 12 of the total quality mark being taken from various types of library

infrastructure measurements (including acquisitions per year total volumes average

number of volumes per student and yearly library expenditure outside of acquisitions)

Netbig and Education18 also use library volume holdings while Asiaweek the Financial

Times and the Times also use measures of library expenditures outside of acquisitions or

computerization of library resources as measures of institutional quality

8

One important factor to note is that most ranking systems do not normalize their resource

and infrastructure measures That is to say it is raw spending power or simple size of

assets that is usually measured rather than spending per studentprofessor or assets per

studentprofessor As a result a number of these rankings systems have inbuilt biases

towards larger institutions

D Indicators of Learning Outputs

Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge

learned over the course of a baccalaureate degreemdashshould be a basic indicator of

institutional quality Unfortunately good means of measuring these outputsmdashlike the

National Survey of Student Engagement (NSSE) and the College Learning Assessment

(CLA)mdashhave only recently become available and for the most part institutions are still

keeping their scores secret Outside of these measures only a few very crude indicators are

available which likely explains why learning outputs do not feature especially prominently

in most ranking schemes

The simplest types of measures of learning outputs are those linked to graduation and

retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong

and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7

the latter three also use rates of graduation from Masterrsquos programs as indicators In some

cases the weights on these measures can be very highmdashin the Guangdong rankings

graduation rates account for over 50 of the rankingmdashbut in most cases the weights are

10 or less Retention rates commonly meaning the progression rate of first-year students

into second year are accorded less importance The US News and World Report Macleanrsquos

the Melbourne Institute and La Repubblica all employ retention measures as indicators but

none of them are worth more than 4 of total weighting Two publications make specific

indicators for retention and graduation of international students Macleanrsquos (graduation

rates of international students) and the Melbourne Institute (retention rates of international

students) The Washington Monthly looks specifically at institutional retention rates

adjusted for the participation of lower-income students and gives higher scores to

institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores

and number of Pell Grant recipients the US

7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18

9

News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos

ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)

E Indicators of Final Outcomes

Final outcomes are indications of generalized outcomes for students after graduation

Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness

good citizenship) but given the somewhat utilitarian justifications for education that are

currently in fashion (see Wolf 2000) employment outcomes are the most commonly used

measure of final outcomes These are given particular emphasis by the Guardian (where

employment outcomes are worth 17 of the total score) but are also used by the Financial

Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the

Times are interestingly not concerned with employment per se but with ldquoemployment in

an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher

Education Statistics Agency (HESA) uses Standard Occupational Classifications to

measure the proportion of graduates in professional or white-collar jobs anyone not in

such a job is considered not to be working in an area related to their studies (it is unclear

what methodology is used by the Financial Times and the Times although we suspect their

methods are broadly similar)

The only other measure of final outcomes in use is percentage of graduates returning for

additional education which is an indicator used by both the Melbourne Institute and the

Financial Times This is a particularly important indicator for the latter as it is worth 21

of the final ranking

The lack of indicators concerning final outcomes is interesting since most government-

sponsored performance-indicator regimes around the world are very much concerned

with such measures especially with respect to employment Possibly this indicates that

ranking systems simply do not view education outcomes as relevant measures of

educational quality Alternatively it may be the case that they simply have not found a

reliable indicator of outcomes or that there are reliable indicators but that there is so little

variation between institutions that it makes no sense to rank based on the data

10

F Indicators of Research

Many of the league tables covered in this survey include sections and weightings related

to universitiesrsquo research efforts It is in this field of measurement that we see the greatest

diversity of indicators Presumably this is because research inputs and outputs lend

themselves much more easily to measurement and manipulation than other areas of

institutional activity

Three studies include research staff as part of their ranking scheme La Repubblica at 952

the Melbourne Institute (4) and Wuhan (078)

Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used

method of looking at research quality but it is not universally admired because different

disciplines use different means to communicate major advances in knowledge (leading

scientists invariably produce large numbers of journal articles leading social scientists

may produce fewer journal articles but instead have one or two long important

monographsmdashsee Hicks 2004) There is also some concern among non-English speaking

countries that they are penalized in international rankings because so many of the major

journals (notably Science and Nature) are printed in English However the one set of

rankings that uses separate indicators to monitor articles published in English and articles

published in another language (the Wuhan rankings) shows that the two indicators are

positively correlated institutions that have more Chinese publications are also likely to

have more English publications and vice versa

Several sets of league tables measure bibliometric citations in various publication indices

The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a

weight of 20 of the final total Guangdong also monitors other Chinese universities

specifically for citations in engineering publications and weights this at 29 Moreover it

tacks on an additional 1049 for citations in science-oriented indices such as the Science

Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the

same category of scientific citations while the Melbourne Institute rates science citations at

68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social

Science Citation Index which does not include the humanities) are noted in only two

league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne

Institute (32) Another way of measuring research

11

impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a

weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the

Melbourne Institute

The complement of citations is of course publications Listing the number of publications

an individual group department or whole university releases can act as a weak substitute

for citationsmdashweak because simply publishing a paper or monograph is no guarantee that

the general public or other researchers will even glance at the work Guangdong gives an

1179 weighting to publications in science-oriented indices such as the Science Citation

Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same

indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong

even has a separate category just for measuring publications in Science and Nature

although it accords it an almost derisory weighting of 06 Under publications in social

science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their

final totals For publications in other indices (where the subject indices are

undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for

Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of

rankings to include research monographs weighted at 033

In countries where there are specific third-party evaluations of research output academic

quality of research is sometimes used as a research indicator The Times puts a very large

30 weight on this indicator while the Financial Times puts it at 119

Research awards are another handy third-party measurement of quality as the number of

international and national awards won by faculty andor graduates is often considered a

useful measure of institutional success International research awardsmdash specifically the

number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator

by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as

particularly suspect in some quarters given that the points are based on where the

recipient went to school rather than on where they are or were on the

8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the

most prominent publication indices

9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences

12

faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead

for nearly a century Wuhan uses similar measures but only accords them a weight of 14

National research awards are more common as a quality indicator used by La Repubblica

(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)

Financial indicators of research are also very common Research budgets as a factor in the

overall assessment of research in universities are covered by the Financial Times (9)

Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for

total amount of research expenditure unfortunately it is unclear precisely what this

research expenditure represents or how it is determined although it is clearly indicated

that it does not represent the total number of grants or projects at a university Total

number of research-based grants and projects is weighted by Education18 at 15 and

Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science

and engineering and another 55 to those for social sciences and humanities Similarly

the Melbourne Institute gives 6 of its overall weight to public-source grants making no

distinction between areas of study

In a slightly different vein Netbig (46) and Wuhan (278) both list the number of

research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica

(095) all weigh research-basedaffiliated research institutions or centres for studies

Finally one can also measure research not simply in terms of the amount of money it

generates but also in terms of the amount of future income it will generate Both

Guangdong (245) and Wuhan (193) measure the number of patents issued to

universities as a quality indicator

A final way of measuring an institutionrsquos research intensity is to look at the range of its

course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of

doctoral and Masterrsquos programs offered as a proxy for research intensity

As with physical and financial resources few if any of the research indicators are

normalized to account for institutional size (either by student or faculty numbers) In the

world of rankings bigger almost always means better an institution with 100 faculty with

ten citations apiece will always look worse than an institution with 1001 faculty

13

with one citation each To the extent that the raw production of knowledge matters this

form of measurement is acceptable To the extent that rankings are meant to show how

well institutions are doing on a like-to-like basis or to show the efficiency of universities it

is plainly inadequate This should be of particular concern to Chinese policy-makers

whose ranking systems are especially reliant on research-based indicators

G Indicators of Reputation

The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo

Those rankings systems which use the results of reputation surveys as an indicator do so as

an indirect measure of quality based on the assumption that the employers academics and

academic administrators surveyed have opinions of institutional quality that are informed

up-to-date and impartial While these assumptions are clearly open to debate they

nevertheless form an important basis for many ranking systems Another reason for using

reputation measures is the paucity of other data availablemdashsome countries have few

independent measures of teaching effectiveness university resources or output and

reputation can thus act as a useful surrogate Reputation rankings are often criticized as

simply quantifying the common ignorance of the people being surveyed However to the

extent that the people being surveyed hold positions which have the potential to affect

large numbers of young people and whose positions actually require some knowledge of

institutional quality (ie officials in charge of graduate admissions corporate recruiters

etc) then reputation rankings make sense because they provide useful information for

students about the perceived value of the degrees that they could obtain from various

universities

The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and

the Times which both accord reputation a weighting of 50 in their overall ranking

scheme Education18 assigns it almost as much significance at 40 The US News and World

Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just

below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)

and Netbig (15) The only other study to include reputation is Wuhan (117)

14

1 Please see Appendix B for a glossary of the various bodies referenced in this document

Appendix A Detailed Listing of Indicators and their Sources

Beginning Characteristics Indicator

Used By Source1

Incoming grades Macleans University

Percentage with grades above a set limit

Macleans University

US News and World Report

University

Performance on national standardised tests or benchmarks

Asiaweek University

Education18 3rd-party JUPAS

Financial Times Government agency 3rd-party UCAS

Guardian University Guide

Government agency 3rd-party UCAS

Melbourne Institute Government agency 3rd-party DEST

Netbig National entrance examination board

Times Good University Guide

Government agency 3rd-party UCAS

US News University

Wuhan Unknown presumed government 3rd-party

Student status La Repubblica Government agency 3rd-party MIUR

Admittance selectivity general

Asiaweek University

Admittance number of applications to places

Asiaweek University

Financial Times Government agency 3rd-party UCAS

La Repubblica Government agency 3rd-party MIUR

US News University

15

Beginning Characteristics Indicator

Used By Source1

Out-of-locality student percentage

Macleans University

International student percentages

Financial Times Government agency 3rd-party HESA

Macleans University

Shanghai Institute of Educational Science

Times World University

Wuhan Unknown presumed university

Undergraduate students among all students percentages

Netbig

Wuhan Unknown presumed university

Ethnic diversity in student body

Guardian University

Learning Inputs ndash Staff Indicator

Used By Source

Facultystudent ratio Asiaweek University

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Financial Times Government agency 3rd-party HESA

La Repubblica Government agency 3rd-party MIUR

Times Good University Guide

Government agency 3rd-party HESA

Times World University

US News University

Wuhan Unknown presumed university

16

Learning Inputs ndash Staff Indicator

Used By Source

Social science faculty student ratio

Melbourne Government agency 3rd-party DEST

Science faculty student ratio

Melbourne Government agency 3rd-party DEST

Administrative staff student ratio

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Staff student ratio (regardless of division)

Guardian Government agency 3rd-party HESA

Netbig University

Course per teacher La Repubblica Government agency 3rd-party MIUR

Per-teacher university spending

Asiaweek University

Faculty pay rates fortenured staff

Asiaweek University

US News University

Number of full-time part-time faculty

Netbig University

US News University

Wuhan Unknown presumed university

Faculty with researchprojects

Wuhan Unknown presumed university

Class size differentiation Macleans University

US News University

Classes taught by tenured faculty

Macleans University

Exchange programmes hosted

La Repubblica Government agency 3rd-party AgNaSoc

Number of classes lsquoactually taughtrsquo

La Repubblica Government agency 3rd-party MIUR

of international faculty (v faculty as a whole)

Times World University

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

12INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

INDICATEURS type universiteacutes

canadiennes et MEQ

classements nationaux classements internationaux

Eacuteducation permanenteUSNews and Report ampThe

CenterMacleans CHEDAAD GUARDIAN ASIAWEEK

DETYA AUSTRALIE

MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN

WEB

nombre de programmes offerts input un Cannombre deacutetudiants inscrits input un CanRevenus et deacutepenses du service input un Can

RP gouvernance - 2008-06-23

13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes

REacuteFEacuteRENCES

Aperccedilu densemble des principaux indicateurs

GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost

Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes

MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf

Indicateurs utiliseacutes dans les classements internationaux

Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml

RP gouvernance - 2008-06-23

ANNEXES

Quelques exemples de listes drsquoindicateurs

1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)

Voir page suivante

Preacutepareacute par Roland Proulx

Consultant en planification institutionnelle et veille strateacutegique

Montreacuteal mai 2008

Performance Indicators

Discussion Paper

May 2004 (Prepared by the Office of Planning amp Analysis)

Table of Contents

Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25

2

Table 1 ndash Number of Indictors by Type of Data Source

Table 1 shows that surveys are the least frequently used source of data for indicators

Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a

plurality of indicators come from this source This measure somewhat underestimates the

importance of surveys however as it does not account for the weighting given to each

indicator in each study In the THES World Rankings for instance there is only a single

survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos

Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one

indicator is worth 20 of the final score

Raw

indIcator count

Surve

y data

Third parties

Universities

AsiaweekmdashAsias Best Universities 18 - - 18

Daily Telegraph (2003) 1 - 1 -

Education18com 9 3 4 2

Excelencia 2001 71 - 71 -

Financial Times (2003) 17 - 17 -

Guangdong Institute of Management Science 17 - 14 3

GuardianmdashUniversity Guide 2005 7 - 2 5

La Repubblica 23 2 21 -

Macleans University Rankings 24 1 5 18

Melbourne Institutemdash International Standing of Australian Universities

26 3 23 -

Netbig 2004 18 1 10 7

Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15

Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities

6 - 5 1

The TimesmdashGood University Guide 2005 9 - 9 -

Times Higher Education SupplementmdashWorld University Rankings

5 1 1 3

US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11

Washington MonthlymdashCollege Rankings 2005 8 - 1 7

Wuhan University Centre for Science Evaluation 45 2 22 21

3

A Indicators of Beginning Characteristics

ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of

students at the time they begin their studies Fourteen of the 18 rankings examined in this

study use one or more indicators of the beginning characteristics of students to arrive at

their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on

these factors with 33 of the total ranking coming from this class of indicators but the

Guardian Education18 Asiaweek and the two other North American surveys also place

considerable emphasis on this category

There are six main indicators used to determine which institutions have students with

positive ldquobeginning characteristicsrdquo

The most common measure of beginning characteristics is performance on national

standardized tests with nine surveys using this as a measure Education18 and the

Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used

by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)

Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this

data is collected and standardized by national bodies it has the benefit of being seen as a

relatively impartial method of determining the relative ldquostrengthrdquo of the students entering

each institution Institutionsrsquo results can be scored by showing either averages or the

percentage of entering students meeting a particular standard

Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses

secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This

is a second-best solution made necessary by the absence of any national standardized test

in Canada (or indeed of any provincial standardized tests at the end of secondary school

in provinces other than Alberta) The lack of national standardization makes this an

undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is

truly equivalent to an ldquoArdquo in another jurisdiction

Another measure of the strength of the student body is the percentage of incoming

students receiving (third-party) scholarships which is worth 11 of the score in the

Wuhan survey One can also approach the issue by measuring institutional selectivity In

effect this method infers the strength of the student body by the proportion of

4

applicants rejected the theory being that the higher the number of rejected applicants the

stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)

Normally this measure is expressed as a straight ratio of acceptances to applications but it

can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the

greatest weight on this measure) as a ratio of enrolments to applications Within the US

there is some dispute as to what constitutes an offer of admission and whether or not late

admissions are included as noted by Ganeshananthan (2003)

Student bodies are often considered to be strong if the school is able to attract a large

number of international or out-of-district students or if they contain people from diverse

ethnic backgrounds A number of league tables use the international student indicator

(which like the selectivity indicator is arguably as much an indicator of prestige and

reputation as it is of student characteristics) although in no case does this indicator

account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a

quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data

on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian

puts a somewhat larger emphasis on this indicator in comparison to other league tables

which use similar variables

A very different take on this idea is present in the Washington Monthly which released its

first set of College Rankings in September 2005 With the declared aim of using an

institutionrsquos commitment to social mobility as a measure of quality it uses the percentage

of students from low-income backgrounds as an indicator (with percentage of students

receiving need-based government (Pell) grants used as a proxy)

Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy

statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the

percentage of the student population who are graduate students (arguably this is a

research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the

number of graduate students auditing classes is used as an indicator the assumption is

presumably that if people are auditing then the classes must be very attractive The Italian

La Repubblica ranks an institution according to the number of part-time students it has

contrary to prevailing North American views on the undesirability of part-time study the

Italian rankings see higher numbers of part-time students in a positive light as it is

evidence that an institution is becoming less rigid in its timetabling and

5

permitting students to juggle both work and study something which was nearly

impossible in that country just a few years ago

The Washington Monthly also has a final category of indicators which reflect studentsrsquo

beginning characteristics namely their likelihood of performing community service as

measured by the percentage of students in the US Peace Corps and Reserve Officer

Training Corps (ROTC) and the percentage of students involved in work-study in the

community3 Together these three indicators account for 33 of an institutionrsquos total

ranking

B Indicators of Learning InputsmdashStaff

Generally both the quantity and quality of staff are positively correlated with institutional

quality The problem of course is finding useful metrics for each of these factors

especially if one excludes as we have done here measures of research performance and

research intensity putting them in a separate category4

The simplest measure is simply the number of faculty unadjusted for things like size of

student body Most national league tables however prefer to use variations on the

concept of facultystudent ratio Others try to measure teaching intensity with measures

such as courses per teacher or hours spent in class per student (both in La Repubblica)

These kinds of measures usually account for between 2-5 of the final rankings although

in some cases (ie the Guardian) this figure can be as high as 20

Another important way of measuring how faculty resources are deployed is the measure

of average class size which is used only by Macleanrsquos and the US News and World Report

Ostensibly the reason for measuring class size is to account in some form for the degree

3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category

6

of attention devoted to actually teaching students Despite the fact that important research

(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level

the use of this indicator appears to be a spillover from the North American debates on

class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein

2000) Regardless of why the indicators are used they are extraordinarily important to

these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and

World Reportrsquos rankings respectively

A number of ranking systems try to look at staff qualifications such as the number of

PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the

Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further

than other surveys and actually looks at the proportion of classes taught by tenure-track

staff Others (ie THES) look at the number of foreign faculty based on the assumption

that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still

others (ie La Repubblica) look at the age structure of the faculty Another proxy for

institutional quality is the pay rates for tenured staff on the assumption that institutions

with higher rates of pay on average attract better faculty this measure has been used both

by the US News and World Report and Asiaweek

Finally a number of league tables rank faculty inputs on the basis of standardized third-

party evaluations 6

Education18 the Financial Times the Times the Guardian and the US

News and World Report league tables all use some sort of ranking criterion based at least in

part on this indicator or variations thereof

C Indicators of Learning InputsmdashResources

Resource inputsmdashcrudely the amount of current dollars equipment and books available

to students at an institutionmdashare widely considered an important measure of quality Yet

despite the apparent simplicity of counting dollars and measuring assets

5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each

university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future

7

the means by which institutional wealth is measured varies considerably between ranking

systems

There are a number of revenue-based measures of resources Macleanrsquos uses public funding

of institutional budgets as a factor in its analysis conversely the Financial Times uses the

private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of

total score) and the US News and World Report (5 of total score) also measure alumni

financial support as a measure of quality

For reasons that are not entirely clear league tables tend to favour measures of

expenditures rather than revenues The Guardian looks at total institutional expenditures

as an indicator Institutional expenditure on student services is used as a measure of

institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total

institutional scores respectively) Rzezspospolita does not measure student services

expenditures directly but does measure student services outputs such as number of

student athletes and number of study clubs which amounts to more or less the same thing

Macleanrsquos also gives out 433 of its total score based on institutional expenditures on

scholarships and bursaries

Various aspects of physical infrastructure are also used as measures of institutional

resources most directly in the case of La Repubblica which bases 317 of its total rank on

the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding

assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type

of physical infrastructure measured is available Internet bandwidth which was used by

Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth

roughly 3 of the total score

By some distance the infrastructure indicators most favoured by the compilers of league

tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on

this with 12 of the total quality mark being taken from various types of library

infrastructure measurements (including acquisitions per year total volumes average

number of volumes per student and yearly library expenditure outside of acquisitions)

Netbig and Education18 also use library volume holdings while Asiaweek the Financial

Times and the Times also use measures of library expenditures outside of acquisitions or

computerization of library resources as measures of institutional quality

8

One important factor to note is that most ranking systems do not normalize their resource

and infrastructure measures That is to say it is raw spending power or simple size of

assets that is usually measured rather than spending per studentprofessor or assets per

studentprofessor As a result a number of these rankings systems have inbuilt biases

towards larger institutions

D Indicators of Learning Outputs

Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge

learned over the course of a baccalaureate degreemdashshould be a basic indicator of

institutional quality Unfortunately good means of measuring these outputsmdashlike the

National Survey of Student Engagement (NSSE) and the College Learning Assessment

(CLA)mdashhave only recently become available and for the most part institutions are still

keeping their scores secret Outside of these measures only a few very crude indicators are

available which likely explains why learning outputs do not feature especially prominently

in most ranking schemes

The simplest types of measures of learning outputs are those linked to graduation and

retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong

and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7

the latter three also use rates of graduation from Masterrsquos programs as indicators In some

cases the weights on these measures can be very highmdashin the Guangdong rankings

graduation rates account for over 50 of the rankingmdashbut in most cases the weights are

10 or less Retention rates commonly meaning the progression rate of first-year students

into second year are accorded less importance The US News and World Report Macleanrsquos

the Melbourne Institute and La Repubblica all employ retention measures as indicators but

none of them are worth more than 4 of total weighting Two publications make specific

indicators for retention and graduation of international students Macleanrsquos (graduation

rates of international students) and the Melbourne Institute (retention rates of international

students) The Washington Monthly looks specifically at institutional retention rates

adjusted for the participation of lower-income students and gives higher scores to

institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores

and number of Pell Grant recipients the US

7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18

9

News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos

ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)

E Indicators of Final Outcomes

Final outcomes are indications of generalized outcomes for students after graduation

Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness

good citizenship) but given the somewhat utilitarian justifications for education that are

currently in fashion (see Wolf 2000) employment outcomes are the most commonly used

measure of final outcomes These are given particular emphasis by the Guardian (where

employment outcomes are worth 17 of the total score) but are also used by the Financial

Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the

Times are interestingly not concerned with employment per se but with ldquoemployment in

an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher

Education Statistics Agency (HESA) uses Standard Occupational Classifications to

measure the proportion of graduates in professional or white-collar jobs anyone not in

such a job is considered not to be working in an area related to their studies (it is unclear

what methodology is used by the Financial Times and the Times although we suspect their

methods are broadly similar)

The only other measure of final outcomes in use is percentage of graduates returning for

additional education which is an indicator used by both the Melbourne Institute and the

Financial Times This is a particularly important indicator for the latter as it is worth 21

of the final ranking

The lack of indicators concerning final outcomes is interesting since most government-

sponsored performance-indicator regimes around the world are very much concerned

with such measures especially with respect to employment Possibly this indicates that

ranking systems simply do not view education outcomes as relevant measures of

educational quality Alternatively it may be the case that they simply have not found a

reliable indicator of outcomes or that there are reliable indicators but that there is so little

variation between institutions that it makes no sense to rank based on the data

10

F Indicators of Research

Many of the league tables covered in this survey include sections and weightings related

to universitiesrsquo research efforts It is in this field of measurement that we see the greatest

diversity of indicators Presumably this is because research inputs and outputs lend

themselves much more easily to measurement and manipulation than other areas of

institutional activity

Three studies include research staff as part of their ranking scheme La Repubblica at 952

the Melbourne Institute (4) and Wuhan (078)

Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used

method of looking at research quality but it is not universally admired because different

disciplines use different means to communicate major advances in knowledge (leading

scientists invariably produce large numbers of journal articles leading social scientists

may produce fewer journal articles but instead have one or two long important

monographsmdashsee Hicks 2004) There is also some concern among non-English speaking

countries that they are penalized in international rankings because so many of the major

journals (notably Science and Nature) are printed in English However the one set of

rankings that uses separate indicators to monitor articles published in English and articles

published in another language (the Wuhan rankings) shows that the two indicators are

positively correlated institutions that have more Chinese publications are also likely to

have more English publications and vice versa

Several sets of league tables measure bibliometric citations in various publication indices

The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a

weight of 20 of the final total Guangdong also monitors other Chinese universities

specifically for citations in engineering publications and weights this at 29 Moreover it

tacks on an additional 1049 for citations in science-oriented indices such as the Science

Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the

same category of scientific citations while the Melbourne Institute rates science citations at

68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social

Science Citation Index which does not include the humanities) are noted in only two

league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne

Institute (32) Another way of measuring research

11

impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a

weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the

Melbourne Institute

The complement of citations is of course publications Listing the number of publications

an individual group department or whole university releases can act as a weak substitute

for citationsmdashweak because simply publishing a paper or monograph is no guarantee that

the general public or other researchers will even glance at the work Guangdong gives an

1179 weighting to publications in science-oriented indices such as the Science Citation

Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same

indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong

even has a separate category just for measuring publications in Science and Nature

although it accords it an almost derisory weighting of 06 Under publications in social

science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their

final totals For publications in other indices (where the subject indices are

undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for

Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of

rankings to include research monographs weighted at 033

In countries where there are specific third-party evaluations of research output academic

quality of research is sometimes used as a research indicator The Times puts a very large

30 weight on this indicator while the Financial Times puts it at 119

Research awards are another handy third-party measurement of quality as the number of

international and national awards won by faculty andor graduates is often considered a

useful measure of institutional success International research awardsmdash specifically the

number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator

by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as

particularly suspect in some quarters given that the points are based on where the

recipient went to school rather than on where they are or were on the

8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the

most prominent publication indices

9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences

12

faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead

for nearly a century Wuhan uses similar measures but only accords them a weight of 14

National research awards are more common as a quality indicator used by La Repubblica

(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)

Financial indicators of research are also very common Research budgets as a factor in the

overall assessment of research in universities are covered by the Financial Times (9)

Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for

total amount of research expenditure unfortunately it is unclear precisely what this

research expenditure represents or how it is determined although it is clearly indicated

that it does not represent the total number of grants or projects at a university Total

number of research-based grants and projects is weighted by Education18 at 15 and

Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science

and engineering and another 55 to those for social sciences and humanities Similarly

the Melbourne Institute gives 6 of its overall weight to public-source grants making no

distinction between areas of study

In a slightly different vein Netbig (46) and Wuhan (278) both list the number of

research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica

(095) all weigh research-basedaffiliated research institutions or centres for studies

Finally one can also measure research not simply in terms of the amount of money it

generates but also in terms of the amount of future income it will generate Both

Guangdong (245) and Wuhan (193) measure the number of patents issued to

universities as a quality indicator

A final way of measuring an institutionrsquos research intensity is to look at the range of its

course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of

doctoral and Masterrsquos programs offered as a proxy for research intensity

As with physical and financial resources few if any of the research indicators are

normalized to account for institutional size (either by student or faculty numbers) In the

world of rankings bigger almost always means better an institution with 100 faculty with

ten citations apiece will always look worse than an institution with 1001 faculty

13

with one citation each To the extent that the raw production of knowledge matters this

form of measurement is acceptable To the extent that rankings are meant to show how

well institutions are doing on a like-to-like basis or to show the efficiency of universities it

is plainly inadequate This should be of particular concern to Chinese policy-makers

whose ranking systems are especially reliant on research-based indicators

G Indicators of Reputation

The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo

Those rankings systems which use the results of reputation surveys as an indicator do so as

an indirect measure of quality based on the assumption that the employers academics and

academic administrators surveyed have opinions of institutional quality that are informed

up-to-date and impartial While these assumptions are clearly open to debate they

nevertheless form an important basis for many ranking systems Another reason for using

reputation measures is the paucity of other data availablemdashsome countries have few

independent measures of teaching effectiveness university resources or output and

reputation can thus act as a useful surrogate Reputation rankings are often criticized as

simply quantifying the common ignorance of the people being surveyed However to the

extent that the people being surveyed hold positions which have the potential to affect

large numbers of young people and whose positions actually require some knowledge of

institutional quality (ie officials in charge of graduate admissions corporate recruiters

etc) then reputation rankings make sense because they provide useful information for

students about the perceived value of the degrees that they could obtain from various

universities

The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and

the Times which both accord reputation a weighting of 50 in their overall ranking

scheme Education18 assigns it almost as much significance at 40 The US News and World

Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just

below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)

and Netbig (15) The only other study to include reputation is Wuhan (117)

14

1 Please see Appendix B for a glossary of the various bodies referenced in this document

Appendix A Detailed Listing of Indicators and their Sources

Beginning Characteristics Indicator

Used By Source1

Incoming grades Macleans University

Percentage with grades above a set limit

Macleans University

US News and World Report

University

Performance on national standardised tests or benchmarks

Asiaweek University

Education18 3rd-party JUPAS

Financial Times Government agency 3rd-party UCAS

Guardian University Guide

Government agency 3rd-party UCAS

Melbourne Institute Government agency 3rd-party DEST

Netbig National entrance examination board

Times Good University Guide

Government agency 3rd-party UCAS

US News University

Wuhan Unknown presumed government 3rd-party

Student status La Repubblica Government agency 3rd-party MIUR

Admittance selectivity general

Asiaweek University

Admittance number of applications to places

Asiaweek University

Financial Times Government agency 3rd-party UCAS

La Repubblica Government agency 3rd-party MIUR

US News University

15

Beginning Characteristics Indicator

Used By Source1

Out-of-locality student percentage

Macleans University

International student percentages

Financial Times Government agency 3rd-party HESA

Macleans University

Shanghai Institute of Educational Science

Times World University

Wuhan Unknown presumed university

Undergraduate students among all students percentages

Netbig

Wuhan Unknown presumed university

Ethnic diversity in student body

Guardian University

Learning Inputs ndash Staff Indicator

Used By Source

Facultystudent ratio Asiaweek University

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Financial Times Government agency 3rd-party HESA

La Repubblica Government agency 3rd-party MIUR

Times Good University Guide

Government agency 3rd-party HESA

Times World University

US News University

Wuhan Unknown presumed university

16

Learning Inputs ndash Staff Indicator

Used By Source

Social science faculty student ratio

Melbourne Government agency 3rd-party DEST

Science faculty student ratio

Melbourne Government agency 3rd-party DEST

Administrative staff student ratio

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Staff student ratio (regardless of division)

Guardian Government agency 3rd-party HESA

Netbig University

Course per teacher La Repubblica Government agency 3rd-party MIUR

Per-teacher university spending

Asiaweek University

Faculty pay rates fortenured staff

Asiaweek University

US News University

Number of full-time part-time faculty

Netbig University

US News University

Wuhan Unknown presumed university

Faculty with researchprojects

Wuhan Unknown presumed university

Class size differentiation Macleans University

US News University

Classes taught by tenured faculty

Macleans University

Exchange programmes hosted

La Repubblica Government agency 3rd-party AgNaSoc

Number of classes lsquoactually taughtrsquo

La Repubblica Government agency 3rd-party MIUR

of international faculty (v faculty as a whole)

Times World University

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES

A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes

REacuteFEacuteRENCES

Aperccedilu densemble des principaux indicateurs

GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost

Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes

MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf

Indicateurs utiliseacutes dans les classements internationaux

Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml

RP gouvernance - 2008-06-23

ANNEXES

Quelques exemples de listes drsquoindicateurs

1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)

Voir page suivante

Preacutepareacute par Roland Proulx

Consultant en planification institutionnelle et veille strateacutegique

Montreacuteal mai 2008

Performance Indicators

Discussion Paper

May 2004 (Prepared by the Office of Planning amp Analysis)

Table of Contents

Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25

2

Table 1 ndash Number of Indictors by Type of Data Source

Table 1 shows that surveys are the least frequently used source of data for indicators

Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a

plurality of indicators come from this source This measure somewhat underestimates the

importance of surveys however as it does not account for the weighting given to each

indicator in each study In the THES World Rankings for instance there is only a single

survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos

Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one

indicator is worth 20 of the final score

Raw

indIcator count

Surve

y data

Third parties

Universities

AsiaweekmdashAsias Best Universities 18 - - 18

Daily Telegraph (2003) 1 - 1 -

Education18com 9 3 4 2

Excelencia 2001 71 - 71 -

Financial Times (2003) 17 - 17 -

Guangdong Institute of Management Science 17 - 14 3

GuardianmdashUniversity Guide 2005 7 - 2 5

La Repubblica 23 2 21 -

Macleans University Rankings 24 1 5 18

Melbourne Institutemdash International Standing of Australian Universities

26 3 23 -

Netbig 2004 18 1 10 7

Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15

Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities

6 - 5 1

The TimesmdashGood University Guide 2005 9 - 9 -

Times Higher Education SupplementmdashWorld University Rankings

5 1 1 3

US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11

Washington MonthlymdashCollege Rankings 2005 8 - 1 7

Wuhan University Centre for Science Evaluation 45 2 22 21

3

A Indicators of Beginning Characteristics

ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of

students at the time they begin their studies Fourteen of the 18 rankings examined in this

study use one or more indicators of the beginning characteristics of students to arrive at

their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on

these factors with 33 of the total ranking coming from this class of indicators but the

Guardian Education18 Asiaweek and the two other North American surveys also place

considerable emphasis on this category

There are six main indicators used to determine which institutions have students with

positive ldquobeginning characteristicsrdquo

The most common measure of beginning characteristics is performance on national

standardized tests with nine surveys using this as a measure Education18 and the

Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used

by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)

Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this

data is collected and standardized by national bodies it has the benefit of being seen as a

relatively impartial method of determining the relative ldquostrengthrdquo of the students entering

each institution Institutionsrsquo results can be scored by showing either averages or the

percentage of entering students meeting a particular standard

Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses

secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This

is a second-best solution made necessary by the absence of any national standardized test

in Canada (or indeed of any provincial standardized tests at the end of secondary school

in provinces other than Alberta) The lack of national standardization makes this an

undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is

truly equivalent to an ldquoArdquo in another jurisdiction

Another measure of the strength of the student body is the percentage of incoming

students receiving (third-party) scholarships which is worth 11 of the score in the

Wuhan survey One can also approach the issue by measuring institutional selectivity In

effect this method infers the strength of the student body by the proportion of

4

applicants rejected the theory being that the higher the number of rejected applicants the

stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)

Normally this measure is expressed as a straight ratio of acceptances to applications but it

can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the

greatest weight on this measure) as a ratio of enrolments to applications Within the US

there is some dispute as to what constitutes an offer of admission and whether or not late

admissions are included as noted by Ganeshananthan (2003)

Student bodies are often considered to be strong if the school is able to attract a large

number of international or out-of-district students or if they contain people from diverse

ethnic backgrounds A number of league tables use the international student indicator

(which like the selectivity indicator is arguably as much an indicator of prestige and

reputation as it is of student characteristics) although in no case does this indicator

account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a

quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data

on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian

puts a somewhat larger emphasis on this indicator in comparison to other league tables

which use similar variables

A very different take on this idea is present in the Washington Monthly which released its

first set of College Rankings in September 2005 With the declared aim of using an

institutionrsquos commitment to social mobility as a measure of quality it uses the percentage

of students from low-income backgrounds as an indicator (with percentage of students

receiving need-based government (Pell) grants used as a proxy)

Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy

statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the

percentage of the student population who are graduate students (arguably this is a

research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the

number of graduate students auditing classes is used as an indicator the assumption is

presumably that if people are auditing then the classes must be very attractive The Italian

La Repubblica ranks an institution according to the number of part-time students it has

contrary to prevailing North American views on the undesirability of part-time study the

Italian rankings see higher numbers of part-time students in a positive light as it is

evidence that an institution is becoming less rigid in its timetabling and

5

permitting students to juggle both work and study something which was nearly

impossible in that country just a few years ago

The Washington Monthly also has a final category of indicators which reflect studentsrsquo

beginning characteristics namely their likelihood of performing community service as

measured by the percentage of students in the US Peace Corps and Reserve Officer

Training Corps (ROTC) and the percentage of students involved in work-study in the

community3 Together these three indicators account for 33 of an institutionrsquos total

ranking

B Indicators of Learning InputsmdashStaff

Generally both the quantity and quality of staff are positively correlated with institutional

quality The problem of course is finding useful metrics for each of these factors

especially if one excludes as we have done here measures of research performance and

research intensity putting them in a separate category4

The simplest measure is simply the number of faculty unadjusted for things like size of

student body Most national league tables however prefer to use variations on the

concept of facultystudent ratio Others try to measure teaching intensity with measures

such as courses per teacher or hours spent in class per student (both in La Repubblica)

These kinds of measures usually account for between 2-5 of the final rankings although

in some cases (ie the Guardian) this figure can be as high as 20

Another important way of measuring how faculty resources are deployed is the measure

of average class size which is used only by Macleanrsquos and the US News and World Report

Ostensibly the reason for measuring class size is to account in some form for the degree

3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category

6

of attention devoted to actually teaching students Despite the fact that important research

(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level

the use of this indicator appears to be a spillover from the North American debates on

class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein

2000) Regardless of why the indicators are used they are extraordinarily important to

these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and

World Reportrsquos rankings respectively

A number of ranking systems try to look at staff qualifications such as the number of

PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the

Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further

than other surveys and actually looks at the proportion of classes taught by tenure-track

staff Others (ie THES) look at the number of foreign faculty based on the assumption

that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still

others (ie La Repubblica) look at the age structure of the faculty Another proxy for

institutional quality is the pay rates for tenured staff on the assumption that institutions

with higher rates of pay on average attract better faculty this measure has been used both

by the US News and World Report and Asiaweek

Finally a number of league tables rank faculty inputs on the basis of standardized third-

party evaluations 6

Education18 the Financial Times the Times the Guardian and the US

News and World Report league tables all use some sort of ranking criterion based at least in

part on this indicator or variations thereof

C Indicators of Learning InputsmdashResources

Resource inputsmdashcrudely the amount of current dollars equipment and books available

to students at an institutionmdashare widely considered an important measure of quality Yet

despite the apparent simplicity of counting dollars and measuring assets

5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each

university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future

7

the means by which institutional wealth is measured varies considerably between ranking

systems

There are a number of revenue-based measures of resources Macleanrsquos uses public funding

of institutional budgets as a factor in its analysis conversely the Financial Times uses the

private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of

total score) and the US News and World Report (5 of total score) also measure alumni

financial support as a measure of quality

For reasons that are not entirely clear league tables tend to favour measures of

expenditures rather than revenues The Guardian looks at total institutional expenditures

as an indicator Institutional expenditure on student services is used as a measure of

institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total

institutional scores respectively) Rzezspospolita does not measure student services

expenditures directly but does measure student services outputs such as number of

student athletes and number of study clubs which amounts to more or less the same thing

Macleanrsquos also gives out 433 of its total score based on institutional expenditures on

scholarships and bursaries

Various aspects of physical infrastructure are also used as measures of institutional

resources most directly in the case of La Repubblica which bases 317 of its total rank on

the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding

assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type

of physical infrastructure measured is available Internet bandwidth which was used by

Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth

roughly 3 of the total score

By some distance the infrastructure indicators most favoured by the compilers of league

tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on

this with 12 of the total quality mark being taken from various types of library

infrastructure measurements (including acquisitions per year total volumes average

number of volumes per student and yearly library expenditure outside of acquisitions)

Netbig and Education18 also use library volume holdings while Asiaweek the Financial

Times and the Times also use measures of library expenditures outside of acquisitions or

computerization of library resources as measures of institutional quality

8

One important factor to note is that most ranking systems do not normalize their resource

and infrastructure measures That is to say it is raw spending power or simple size of

assets that is usually measured rather than spending per studentprofessor or assets per

studentprofessor As a result a number of these rankings systems have inbuilt biases

towards larger institutions

D Indicators of Learning Outputs

Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge

learned over the course of a baccalaureate degreemdashshould be a basic indicator of

institutional quality Unfortunately good means of measuring these outputsmdashlike the

National Survey of Student Engagement (NSSE) and the College Learning Assessment

(CLA)mdashhave only recently become available and for the most part institutions are still

keeping their scores secret Outside of these measures only a few very crude indicators are

available which likely explains why learning outputs do not feature especially prominently

in most ranking schemes

The simplest types of measures of learning outputs are those linked to graduation and

retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong

and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7

the latter three also use rates of graduation from Masterrsquos programs as indicators In some

cases the weights on these measures can be very highmdashin the Guangdong rankings

graduation rates account for over 50 of the rankingmdashbut in most cases the weights are

10 or less Retention rates commonly meaning the progression rate of first-year students

into second year are accorded less importance The US News and World Report Macleanrsquos

the Melbourne Institute and La Repubblica all employ retention measures as indicators but

none of them are worth more than 4 of total weighting Two publications make specific

indicators for retention and graduation of international students Macleanrsquos (graduation

rates of international students) and the Melbourne Institute (retention rates of international

students) The Washington Monthly looks specifically at institutional retention rates

adjusted for the participation of lower-income students and gives higher scores to

institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores

and number of Pell Grant recipients the US

7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18

9

News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos

ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)

E Indicators of Final Outcomes

Final outcomes are indications of generalized outcomes for students after graduation

Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness

good citizenship) but given the somewhat utilitarian justifications for education that are

currently in fashion (see Wolf 2000) employment outcomes are the most commonly used

measure of final outcomes These are given particular emphasis by the Guardian (where

employment outcomes are worth 17 of the total score) but are also used by the Financial

Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the

Times are interestingly not concerned with employment per se but with ldquoemployment in

an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher

Education Statistics Agency (HESA) uses Standard Occupational Classifications to

measure the proportion of graduates in professional or white-collar jobs anyone not in

such a job is considered not to be working in an area related to their studies (it is unclear

what methodology is used by the Financial Times and the Times although we suspect their

methods are broadly similar)

The only other measure of final outcomes in use is percentage of graduates returning for

additional education which is an indicator used by both the Melbourne Institute and the

Financial Times This is a particularly important indicator for the latter as it is worth 21

of the final ranking

The lack of indicators concerning final outcomes is interesting since most government-

sponsored performance-indicator regimes around the world are very much concerned

with such measures especially with respect to employment Possibly this indicates that

ranking systems simply do not view education outcomes as relevant measures of

educational quality Alternatively it may be the case that they simply have not found a

reliable indicator of outcomes or that there are reliable indicators but that there is so little

variation between institutions that it makes no sense to rank based on the data

10

F Indicators of Research

Many of the league tables covered in this survey include sections and weightings related

to universitiesrsquo research efforts It is in this field of measurement that we see the greatest

diversity of indicators Presumably this is because research inputs and outputs lend

themselves much more easily to measurement and manipulation than other areas of

institutional activity

Three studies include research staff as part of their ranking scheme La Repubblica at 952

the Melbourne Institute (4) and Wuhan (078)

Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used

method of looking at research quality but it is not universally admired because different

disciplines use different means to communicate major advances in knowledge (leading

scientists invariably produce large numbers of journal articles leading social scientists

may produce fewer journal articles but instead have one or two long important

monographsmdashsee Hicks 2004) There is also some concern among non-English speaking

countries that they are penalized in international rankings because so many of the major

journals (notably Science and Nature) are printed in English However the one set of

rankings that uses separate indicators to monitor articles published in English and articles

published in another language (the Wuhan rankings) shows that the two indicators are

positively correlated institutions that have more Chinese publications are also likely to

have more English publications and vice versa

Several sets of league tables measure bibliometric citations in various publication indices

The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a

weight of 20 of the final total Guangdong also monitors other Chinese universities

specifically for citations in engineering publications and weights this at 29 Moreover it

tacks on an additional 1049 for citations in science-oriented indices such as the Science

Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the

same category of scientific citations while the Melbourne Institute rates science citations at

68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social

Science Citation Index which does not include the humanities) are noted in only two

league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne

Institute (32) Another way of measuring research

11

impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a

weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the

Melbourne Institute

The complement of citations is of course publications Listing the number of publications

an individual group department or whole university releases can act as a weak substitute

for citationsmdashweak because simply publishing a paper or monograph is no guarantee that

the general public or other researchers will even glance at the work Guangdong gives an

1179 weighting to publications in science-oriented indices such as the Science Citation

Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same

indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong

even has a separate category just for measuring publications in Science and Nature

although it accords it an almost derisory weighting of 06 Under publications in social

science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their

final totals For publications in other indices (where the subject indices are

undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for

Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of

rankings to include research monographs weighted at 033

In countries where there are specific third-party evaluations of research output academic

quality of research is sometimes used as a research indicator The Times puts a very large

30 weight on this indicator while the Financial Times puts it at 119

Research awards are another handy third-party measurement of quality as the number of

international and national awards won by faculty andor graduates is often considered a

useful measure of institutional success International research awardsmdash specifically the

number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator

by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as

particularly suspect in some quarters given that the points are based on where the

recipient went to school rather than on where they are or were on the

8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the

most prominent publication indices

9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences

12

faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead

for nearly a century Wuhan uses similar measures but only accords them a weight of 14

National research awards are more common as a quality indicator used by La Repubblica

(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)

Financial indicators of research are also very common Research budgets as a factor in the

overall assessment of research in universities are covered by the Financial Times (9)

Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for

total amount of research expenditure unfortunately it is unclear precisely what this

research expenditure represents or how it is determined although it is clearly indicated

that it does not represent the total number of grants or projects at a university Total

number of research-based grants and projects is weighted by Education18 at 15 and

Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science

and engineering and another 55 to those for social sciences and humanities Similarly

the Melbourne Institute gives 6 of its overall weight to public-source grants making no

distinction between areas of study

In a slightly different vein Netbig (46) and Wuhan (278) both list the number of

research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica

(095) all weigh research-basedaffiliated research institutions or centres for studies

Finally one can also measure research not simply in terms of the amount of money it

generates but also in terms of the amount of future income it will generate Both

Guangdong (245) and Wuhan (193) measure the number of patents issued to

universities as a quality indicator

A final way of measuring an institutionrsquos research intensity is to look at the range of its

course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of

doctoral and Masterrsquos programs offered as a proxy for research intensity

As with physical and financial resources few if any of the research indicators are

normalized to account for institutional size (either by student or faculty numbers) In the

world of rankings bigger almost always means better an institution with 100 faculty with

ten citations apiece will always look worse than an institution with 1001 faculty

13

with one citation each To the extent that the raw production of knowledge matters this

form of measurement is acceptable To the extent that rankings are meant to show how

well institutions are doing on a like-to-like basis or to show the efficiency of universities it

is plainly inadequate This should be of particular concern to Chinese policy-makers

whose ranking systems are especially reliant on research-based indicators

G Indicators of Reputation

The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo

Those rankings systems which use the results of reputation surveys as an indicator do so as

an indirect measure of quality based on the assumption that the employers academics and

academic administrators surveyed have opinions of institutional quality that are informed

up-to-date and impartial While these assumptions are clearly open to debate they

nevertheless form an important basis for many ranking systems Another reason for using

reputation measures is the paucity of other data availablemdashsome countries have few

independent measures of teaching effectiveness university resources or output and

reputation can thus act as a useful surrogate Reputation rankings are often criticized as

simply quantifying the common ignorance of the people being surveyed However to the

extent that the people being surveyed hold positions which have the potential to affect

large numbers of young people and whose positions actually require some knowledge of

institutional quality (ie officials in charge of graduate admissions corporate recruiters

etc) then reputation rankings make sense because they provide useful information for

students about the perceived value of the degrees that they could obtain from various

universities

The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and

the Times which both accord reputation a weighting of 50 in their overall ranking

scheme Education18 assigns it almost as much significance at 40 The US News and World

Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just

below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)

and Netbig (15) The only other study to include reputation is Wuhan (117)

14

1 Please see Appendix B for a glossary of the various bodies referenced in this document

Appendix A Detailed Listing of Indicators and their Sources

Beginning Characteristics Indicator

Used By Source1

Incoming grades Macleans University

Percentage with grades above a set limit

Macleans University

US News and World Report

University

Performance on national standardised tests or benchmarks

Asiaweek University

Education18 3rd-party JUPAS

Financial Times Government agency 3rd-party UCAS

Guardian University Guide

Government agency 3rd-party UCAS

Melbourne Institute Government agency 3rd-party DEST

Netbig National entrance examination board

Times Good University Guide

Government agency 3rd-party UCAS

US News University

Wuhan Unknown presumed government 3rd-party

Student status La Repubblica Government agency 3rd-party MIUR

Admittance selectivity general

Asiaweek University

Admittance number of applications to places

Asiaweek University

Financial Times Government agency 3rd-party UCAS

La Repubblica Government agency 3rd-party MIUR

US News University

15

Beginning Characteristics Indicator

Used By Source1

Out-of-locality student percentage

Macleans University

International student percentages

Financial Times Government agency 3rd-party HESA

Macleans University

Shanghai Institute of Educational Science

Times World University

Wuhan Unknown presumed university

Undergraduate students among all students percentages

Netbig

Wuhan Unknown presumed university

Ethnic diversity in student body

Guardian University

Learning Inputs ndash Staff Indicator

Used By Source

Facultystudent ratio Asiaweek University

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Financial Times Government agency 3rd-party HESA

La Repubblica Government agency 3rd-party MIUR

Times Good University Guide

Government agency 3rd-party HESA

Times World University

US News University

Wuhan Unknown presumed university

16

Learning Inputs ndash Staff Indicator

Used By Source

Social science faculty student ratio

Melbourne Government agency 3rd-party DEST

Science faculty student ratio

Melbourne Government agency 3rd-party DEST

Administrative staff student ratio

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Staff student ratio (regardless of division)

Guardian Government agency 3rd-party HESA

Netbig University

Course per teacher La Repubblica Government agency 3rd-party MIUR

Per-teacher university spending

Asiaweek University

Faculty pay rates fortenured staff

Asiaweek University

US News University

Number of full-time part-time faculty

Netbig University

US News University

Wuhan Unknown presumed university

Faculty with researchprojects

Wuhan Unknown presumed university

Class size differentiation Macleans University

US News University

Classes taught by tenured faculty

Macleans University

Exchange programmes hosted

La Repubblica Government agency 3rd-party AgNaSoc

Number of classes lsquoactually taughtrsquo

La Repubblica Government agency 3rd-party MIUR

of international faculty (v faculty as a whole)

Times World University

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

ANNEXES

Quelques exemples de listes drsquoindicateurs

1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)

Voir page suivante

Preacutepareacute par Roland Proulx

Consultant en planification institutionnelle et veille strateacutegique

Montreacuteal mai 2008

Performance Indicators

Discussion Paper

May 2004 (Prepared by the Office of Planning amp Analysis)

Table of Contents

Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25

2

Table 1 ndash Number of Indictors by Type of Data Source

Table 1 shows that surveys are the least frequently used source of data for indicators

Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a

plurality of indicators come from this source This measure somewhat underestimates the

importance of surveys however as it does not account for the weighting given to each

indicator in each study In the THES World Rankings for instance there is only a single

survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos

Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one

indicator is worth 20 of the final score

Raw

indIcator count

Surve

y data

Third parties

Universities

AsiaweekmdashAsias Best Universities 18 - - 18

Daily Telegraph (2003) 1 - 1 -

Education18com 9 3 4 2

Excelencia 2001 71 - 71 -

Financial Times (2003) 17 - 17 -

Guangdong Institute of Management Science 17 - 14 3

GuardianmdashUniversity Guide 2005 7 - 2 5

La Repubblica 23 2 21 -

Macleans University Rankings 24 1 5 18

Melbourne Institutemdash International Standing of Australian Universities

26 3 23 -

Netbig 2004 18 1 10 7

Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15

Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities

6 - 5 1

The TimesmdashGood University Guide 2005 9 - 9 -

Times Higher Education SupplementmdashWorld University Rankings

5 1 1 3

US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11

Washington MonthlymdashCollege Rankings 2005 8 - 1 7

Wuhan University Centre for Science Evaluation 45 2 22 21

3

A Indicators of Beginning Characteristics

ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of

students at the time they begin their studies Fourteen of the 18 rankings examined in this

study use one or more indicators of the beginning characteristics of students to arrive at

their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on

these factors with 33 of the total ranking coming from this class of indicators but the

Guardian Education18 Asiaweek and the two other North American surveys also place

considerable emphasis on this category

There are six main indicators used to determine which institutions have students with

positive ldquobeginning characteristicsrdquo

The most common measure of beginning characteristics is performance on national

standardized tests with nine surveys using this as a measure Education18 and the

Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used

by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)

Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this

data is collected and standardized by national bodies it has the benefit of being seen as a

relatively impartial method of determining the relative ldquostrengthrdquo of the students entering

each institution Institutionsrsquo results can be scored by showing either averages or the

percentage of entering students meeting a particular standard

Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses

secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This

is a second-best solution made necessary by the absence of any national standardized test

in Canada (or indeed of any provincial standardized tests at the end of secondary school

in provinces other than Alberta) The lack of national standardization makes this an

undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is

truly equivalent to an ldquoArdquo in another jurisdiction

Another measure of the strength of the student body is the percentage of incoming

students receiving (third-party) scholarships which is worth 11 of the score in the

Wuhan survey One can also approach the issue by measuring institutional selectivity In

effect this method infers the strength of the student body by the proportion of

4

applicants rejected the theory being that the higher the number of rejected applicants the

stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)

Normally this measure is expressed as a straight ratio of acceptances to applications but it

can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the

greatest weight on this measure) as a ratio of enrolments to applications Within the US

there is some dispute as to what constitutes an offer of admission and whether or not late

admissions are included as noted by Ganeshananthan (2003)

Student bodies are often considered to be strong if the school is able to attract a large

number of international or out-of-district students or if they contain people from diverse

ethnic backgrounds A number of league tables use the international student indicator

(which like the selectivity indicator is arguably as much an indicator of prestige and

reputation as it is of student characteristics) although in no case does this indicator

account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a

quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data

on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian

puts a somewhat larger emphasis on this indicator in comparison to other league tables

which use similar variables

A very different take on this idea is present in the Washington Monthly which released its

first set of College Rankings in September 2005 With the declared aim of using an

institutionrsquos commitment to social mobility as a measure of quality it uses the percentage

of students from low-income backgrounds as an indicator (with percentage of students

receiving need-based government (Pell) grants used as a proxy)

Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy

statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the

percentage of the student population who are graduate students (arguably this is a

research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the

number of graduate students auditing classes is used as an indicator the assumption is

presumably that if people are auditing then the classes must be very attractive The Italian

La Repubblica ranks an institution according to the number of part-time students it has

contrary to prevailing North American views on the undesirability of part-time study the

Italian rankings see higher numbers of part-time students in a positive light as it is

evidence that an institution is becoming less rigid in its timetabling and

5

permitting students to juggle both work and study something which was nearly

impossible in that country just a few years ago

The Washington Monthly also has a final category of indicators which reflect studentsrsquo

beginning characteristics namely their likelihood of performing community service as

measured by the percentage of students in the US Peace Corps and Reserve Officer

Training Corps (ROTC) and the percentage of students involved in work-study in the

community3 Together these three indicators account for 33 of an institutionrsquos total

ranking

B Indicators of Learning InputsmdashStaff

Generally both the quantity and quality of staff are positively correlated with institutional

quality The problem of course is finding useful metrics for each of these factors

especially if one excludes as we have done here measures of research performance and

research intensity putting them in a separate category4

The simplest measure is simply the number of faculty unadjusted for things like size of

student body Most national league tables however prefer to use variations on the

concept of facultystudent ratio Others try to measure teaching intensity with measures

such as courses per teacher or hours spent in class per student (both in La Repubblica)

These kinds of measures usually account for between 2-5 of the final rankings although

in some cases (ie the Guardian) this figure can be as high as 20

Another important way of measuring how faculty resources are deployed is the measure

of average class size which is used only by Macleanrsquos and the US News and World Report

Ostensibly the reason for measuring class size is to account in some form for the degree

3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category

6

of attention devoted to actually teaching students Despite the fact that important research

(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level

the use of this indicator appears to be a spillover from the North American debates on

class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein

2000) Regardless of why the indicators are used they are extraordinarily important to

these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and

World Reportrsquos rankings respectively

A number of ranking systems try to look at staff qualifications such as the number of

PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the

Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further

than other surveys and actually looks at the proportion of classes taught by tenure-track

staff Others (ie THES) look at the number of foreign faculty based on the assumption

that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still

others (ie La Repubblica) look at the age structure of the faculty Another proxy for

institutional quality is the pay rates for tenured staff on the assumption that institutions

with higher rates of pay on average attract better faculty this measure has been used both

by the US News and World Report and Asiaweek

Finally a number of league tables rank faculty inputs on the basis of standardized third-

party evaluations 6

Education18 the Financial Times the Times the Guardian and the US

News and World Report league tables all use some sort of ranking criterion based at least in

part on this indicator or variations thereof

C Indicators of Learning InputsmdashResources

Resource inputsmdashcrudely the amount of current dollars equipment and books available

to students at an institutionmdashare widely considered an important measure of quality Yet

despite the apparent simplicity of counting dollars and measuring assets

5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each

university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future

7

the means by which institutional wealth is measured varies considerably between ranking

systems

There are a number of revenue-based measures of resources Macleanrsquos uses public funding

of institutional budgets as a factor in its analysis conversely the Financial Times uses the

private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of

total score) and the US News and World Report (5 of total score) also measure alumni

financial support as a measure of quality

For reasons that are not entirely clear league tables tend to favour measures of

expenditures rather than revenues The Guardian looks at total institutional expenditures

as an indicator Institutional expenditure on student services is used as a measure of

institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total

institutional scores respectively) Rzezspospolita does not measure student services

expenditures directly but does measure student services outputs such as number of

student athletes and number of study clubs which amounts to more or less the same thing

Macleanrsquos also gives out 433 of its total score based on institutional expenditures on

scholarships and bursaries

Various aspects of physical infrastructure are also used as measures of institutional

resources most directly in the case of La Repubblica which bases 317 of its total rank on

the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding

assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type

of physical infrastructure measured is available Internet bandwidth which was used by

Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth

roughly 3 of the total score

By some distance the infrastructure indicators most favoured by the compilers of league

tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on

this with 12 of the total quality mark being taken from various types of library

infrastructure measurements (including acquisitions per year total volumes average

number of volumes per student and yearly library expenditure outside of acquisitions)

Netbig and Education18 also use library volume holdings while Asiaweek the Financial

Times and the Times also use measures of library expenditures outside of acquisitions or

computerization of library resources as measures of institutional quality

8

One important factor to note is that most ranking systems do not normalize their resource

and infrastructure measures That is to say it is raw spending power or simple size of

assets that is usually measured rather than spending per studentprofessor or assets per

studentprofessor As a result a number of these rankings systems have inbuilt biases

towards larger institutions

D Indicators of Learning Outputs

Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge

learned over the course of a baccalaureate degreemdashshould be a basic indicator of

institutional quality Unfortunately good means of measuring these outputsmdashlike the

National Survey of Student Engagement (NSSE) and the College Learning Assessment

(CLA)mdashhave only recently become available and for the most part institutions are still

keeping their scores secret Outside of these measures only a few very crude indicators are

available which likely explains why learning outputs do not feature especially prominently

in most ranking schemes

The simplest types of measures of learning outputs are those linked to graduation and

retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong

and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7

the latter three also use rates of graduation from Masterrsquos programs as indicators In some

cases the weights on these measures can be very highmdashin the Guangdong rankings

graduation rates account for over 50 of the rankingmdashbut in most cases the weights are

10 or less Retention rates commonly meaning the progression rate of first-year students

into second year are accorded less importance The US News and World Report Macleanrsquos

the Melbourne Institute and La Repubblica all employ retention measures as indicators but

none of them are worth more than 4 of total weighting Two publications make specific

indicators for retention and graduation of international students Macleanrsquos (graduation

rates of international students) and the Melbourne Institute (retention rates of international

students) The Washington Monthly looks specifically at institutional retention rates

adjusted for the participation of lower-income students and gives higher scores to

institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores

and number of Pell Grant recipients the US

7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18

9

News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos

ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)

E Indicators of Final Outcomes

Final outcomes are indications of generalized outcomes for students after graduation

Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness

good citizenship) but given the somewhat utilitarian justifications for education that are

currently in fashion (see Wolf 2000) employment outcomes are the most commonly used

measure of final outcomes These are given particular emphasis by the Guardian (where

employment outcomes are worth 17 of the total score) but are also used by the Financial

Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the

Times are interestingly not concerned with employment per se but with ldquoemployment in

an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher

Education Statistics Agency (HESA) uses Standard Occupational Classifications to

measure the proportion of graduates in professional or white-collar jobs anyone not in

such a job is considered not to be working in an area related to their studies (it is unclear

what methodology is used by the Financial Times and the Times although we suspect their

methods are broadly similar)

The only other measure of final outcomes in use is percentage of graduates returning for

additional education which is an indicator used by both the Melbourne Institute and the

Financial Times This is a particularly important indicator for the latter as it is worth 21

of the final ranking

The lack of indicators concerning final outcomes is interesting since most government-

sponsored performance-indicator regimes around the world are very much concerned

with such measures especially with respect to employment Possibly this indicates that

ranking systems simply do not view education outcomes as relevant measures of

educational quality Alternatively it may be the case that they simply have not found a

reliable indicator of outcomes or that there are reliable indicators but that there is so little

variation between institutions that it makes no sense to rank based on the data

10

F Indicators of Research

Many of the league tables covered in this survey include sections and weightings related

to universitiesrsquo research efforts It is in this field of measurement that we see the greatest

diversity of indicators Presumably this is because research inputs and outputs lend

themselves much more easily to measurement and manipulation than other areas of

institutional activity

Three studies include research staff as part of their ranking scheme La Repubblica at 952

the Melbourne Institute (4) and Wuhan (078)

Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used

method of looking at research quality but it is not universally admired because different

disciplines use different means to communicate major advances in knowledge (leading

scientists invariably produce large numbers of journal articles leading social scientists

may produce fewer journal articles but instead have one or two long important

monographsmdashsee Hicks 2004) There is also some concern among non-English speaking

countries that they are penalized in international rankings because so many of the major

journals (notably Science and Nature) are printed in English However the one set of

rankings that uses separate indicators to monitor articles published in English and articles

published in another language (the Wuhan rankings) shows that the two indicators are

positively correlated institutions that have more Chinese publications are also likely to

have more English publications and vice versa

Several sets of league tables measure bibliometric citations in various publication indices

The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a

weight of 20 of the final total Guangdong also monitors other Chinese universities

specifically for citations in engineering publications and weights this at 29 Moreover it

tacks on an additional 1049 for citations in science-oriented indices such as the Science

Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the

same category of scientific citations while the Melbourne Institute rates science citations at

68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social

Science Citation Index which does not include the humanities) are noted in only two

league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne

Institute (32) Another way of measuring research

11

impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a

weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the

Melbourne Institute

The complement of citations is of course publications Listing the number of publications

an individual group department or whole university releases can act as a weak substitute

for citationsmdashweak because simply publishing a paper or monograph is no guarantee that

the general public or other researchers will even glance at the work Guangdong gives an

1179 weighting to publications in science-oriented indices such as the Science Citation

Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same

indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong

even has a separate category just for measuring publications in Science and Nature

although it accords it an almost derisory weighting of 06 Under publications in social

science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their

final totals For publications in other indices (where the subject indices are

undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for

Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of

rankings to include research monographs weighted at 033

In countries where there are specific third-party evaluations of research output academic

quality of research is sometimes used as a research indicator The Times puts a very large

30 weight on this indicator while the Financial Times puts it at 119

Research awards are another handy third-party measurement of quality as the number of

international and national awards won by faculty andor graduates is often considered a

useful measure of institutional success International research awardsmdash specifically the

number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator

by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as

particularly suspect in some quarters given that the points are based on where the

recipient went to school rather than on where they are or were on the

8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the

most prominent publication indices

9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences

12

faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead

for nearly a century Wuhan uses similar measures but only accords them a weight of 14

National research awards are more common as a quality indicator used by La Repubblica

(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)

Financial indicators of research are also very common Research budgets as a factor in the

overall assessment of research in universities are covered by the Financial Times (9)

Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for

total amount of research expenditure unfortunately it is unclear precisely what this

research expenditure represents or how it is determined although it is clearly indicated

that it does not represent the total number of grants or projects at a university Total

number of research-based grants and projects is weighted by Education18 at 15 and

Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science

and engineering and another 55 to those for social sciences and humanities Similarly

the Melbourne Institute gives 6 of its overall weight to public-source grants making no

distinction between areas of study

In a slightly different vein Netbig (46) and Wuhan (278) both list the number of

research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica

(095) all weigh research-basedaffiliated research institutions or centres for studies

Finally one can also measure research not simply in terms of the amount of money it

generates but also in terms of the amount of future income it will generate Both

Guangdong (245) and Wuhan (193) measure the number of patents issued to

universities as a quality indicator

A final way of measuring an institutionrsquos research intensity is to look at the range of its

course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of

doctoral and Masterrsquos programs offered as a proxy for research intensity

As with physical and financial resources few if any of the research indicators are

normalized to account for institutional size (either by student or faculty numbers) In the

world of rankings bigger almost always means better an institution with 100 faculty with

ten citations apiece will always look worse than an institution with 1001 faculty

13

with one citation each To the extent that the raw production of knowledge matters this

form of measurement is acceptable To the extent that rankings are meant to show how

well institutions are doing on a like-to-like basis or to show the efficiency of universities it

is plainly inadequate This should be of particular concern to Chinese policy-makers

whose ranking systems are especially reliant on research-based indicators

G Indicators of Reputation

The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo

Those rankings systems which use the results of reputation surveys as an indicator do so as

an indirect measure of quality based on the assumption that the employers academics and

academic administrators surveyed have opinions of institutional quality that are informed

up-to-date and impartial While these assumptions are clearly open to debate they

nevertheless form an important basis for many ranking systems Another reason for using

reputation measures is the paucity of other data availablemdashsome countries have few

independent measures of teaching effectiveness university resources or output and

reputation can thus act as a useful surrogate Reputation rankings are often criticized as

simply quantifying the common ignorance of the people being surveyed However to the

extent that the people being surveyed hold positions which have the potential to affect

large numbers of young people and whose positions actually require some knowledge of

institutional quality (ie officials in charge of graduate admissions corporate recruiters

etc) then reputation rankings make sense because they provide useful information for

students about the perceived value of the degrees that they could obtain from various

universities

The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and

the Times which both accord reputation a weighting of 50 in their overall ranking

scheme Education18 assigns it almost as much significance at 40 The US News and World

Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just

below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)

and Netbig (15) The only other study to include reputation is Wuhan (117)

14

1 Please see Appendix B for a glossary of the various bodies referenced in this document

Appendix A Detailed Listing of Indicators and their Sources

Beginning Characteristics Indicator

Used By Source1

Incoming grades Macleans University

Percentage with grades above a set limit

Macleans University

US News and World Report

University

Performance on national standardised tests or benchmarks

Asiaweek University

Education18 3rd-party JUPAS

Financial Times Government agency 3rd-party UCAS

Guardian University Guide

Government agency 3rd-party UCAS

Melbourne Institute Government agency 3rd-party DEST

Netbig National entrance examination board

Times Good University Guide

Government agency 3rd-party UCAS

US News University

Wuhan Unknown presumed government 3rd-party

Student status La Repubblica Government agency 3rd-party MIUR

Admittance selectivity general

Asiaweek University

Admittance number of applications to places

Asiaweek University

Financial Times Government agency 3rd-party UCAS

La Repubblica Government agency 3rd-party MIUR

US News University

15

Beginning Characteristics Indicator

Used By Source1

Out-of-locality student percentage

Macleans University

International student percentages

Financial Times Government agency 3rd-party HESA

Macleans University

Shanghai Institute of Educational Science

Times World University

Wuhan Unknown presumed university

Undergraduate students among all students percentages

Netbig

Wuhan Unknown presumed university

Ethnic diversity in student body

Guardian University

Learning Inputs ndash Staff Indicator

Used By Source

Facultystudent ratio Asiaweek University

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Financial Times Government agency 3rd-party HESA

La Repubblica Government agency 3rd-party MIUR

Times Good University Guide

Government agency 3rd-party HESA

Times World University

US News University

Wuhan Unknown presumed university

16

Learning Inputs ndash Staff Indicator

Used By Source

Social science faculty student ratio

Melbourne Government agency 3rd-party DEST

Science faculty student ratio

Melbourne Government agency 3rd-party DEST

Administrative staff student ratio

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Staff student ratio (regardless of division)

Guardian Government agency 3rd-party HESA

Netbig University

Course per teacher La Repubblica Government agency 3rd-party MIUR

Per-teacher university spending

Asiaweek University

Faculty pay rates fortenured staff

Asiaweek University

US News University

Number of full-time part-time faculty

Netbig University

US News University

Wuhan Unknown presumed university

Faculty with researchprojects

Wuhan Unknown presumed university

Class size differentiation Macleans University

US News University

Classes taught by tenured faculty

Macleans University

Exchange programmes hosted

La Repubblica Government agency 3rd-party AgNaSoc

Number of classes lsquoactually taughtrsquo

La Repubblica Government agency 3rd-party MIUR

of international faculty (v faculty as a whole)

Times World University

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

Performance Indicators

Discussion Paper

May 2004 (Prepared by the Office of Planning amp Analysis)

Table of Contents

Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25

2

Table 1 ndash Number of Indictors by Type of Data Source

Table 1 shows that surveys are the least frequently used source of data for indicators

Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a

plurality of indicators come from this source This measure somewhat underestimates the

importance of surveys however as it does not account for the weighting given to each

indicator in each study In the THES World Rankings for instance there is only a single

survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos

Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one

indicator is worth 20 of the final score

Raw

indIcator count

Surve

y data

Third parties

Universities

AsiaweekmdashAsias Best Universities 18 - - 18

Daily Telegraph (2003) 1 - 1 -

Education18com 9 3 4 2

Excelencia 2001 71 - 71 -

Financial Times (2003) 17 - 17 -

Guangdong Institute of Management Science 17 - 14 3

GuardianmdashUniversity Guide 2005 7 - 2 5

La Repubblica 23 2 21 -

Macleans University Rankings 24 1 5 18

Melbourne Institutemdash International Standing of Australian Universities

26 3 23 -

Netbig 2004 18 1 10 7

Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15

Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities

6 - 5 1

The TimesmdashGood University Guide 2005 9 - 9 -

Times Higher Education SupplementmdashWorld University Rankings

5 1 1 3

US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11

Washington MonthlymdashCollege Rankings 2005 8 - 1 7

Wuhan University Centre for Science Evaluation 45 2 22 21

3

A Indicators of Beginning Characteristics

ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of

students at the time they begin their studies Fourteen of the 18 rankings examined in this

study use one or more indicators of the beginning characteristics of students to arrive at

their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on

these factors with 33 of the total ranking coming from this class of indicators but the

Guardian Education18 Asiaweek and the two other North American surveys also place

considerable emphasis on this category

There are six main indicators used to determine which institutions have students with

positive ldquobeginning characteristicsrdquo

The most common measure of beginning characteristics is performance on national

standardized tests with nine surveys using this as a measure Education18 and the

Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used

by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)

Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this

data is collected and standardized by national bodies it has the benefit of being seen as a

relatively impartial method of determining the relative ldquostrengthrdquo of the students entering

each institution Institutionsrsquo results can be scored by showing either averages or the

percentage of entering students meeting a particular standard

Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses

secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This

is a second-best solution made necessary by the absence of any national standardized test

in Canada (or indeed of any provincial standardized tests at the end of secondary school

in provinces other than Alberta) The lack of national standardization makes this an

undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is

truly equivalent to an ldquoArdquo in another jurisdiction

Another measure of the strength of the student body is the percentage of incoming

students receiving (third-party) scholarships which is worth 11 of the score in the

Wuhan survey One can also approach the issue by measuring institutional selectivity In

effect this method infers the strength of the student body by the proportion of

4

applicants rejected the theory being that the higher the number of rejected applicants the

stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)

Normally this measure is expressed as a straight ratio of acceptances to applications but it

can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the

greatest weight on this measure) as a ratio of enrolments to applications Within the US

there is some dispute as to what constitutes an offer of admission and whether or not late

admissions are included as noted by Ganeshananthan (2003)

Student bodies are often considered to be strong if the school is able to attract a large

number of international or out-of-district students or if they contain people from diverse

ethnic backgrounds A number of league tables use the international student indicator

(which like the selectivity indicator is arguably as much an indicator of prestige and

reputation as it is of student characteristics) although in no case does this indicator

account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a

quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data

on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian

puts a somewhat larger emphasis on this indicator in comparison to other league tables

which use similar variables

A very different take on this idea is present in the Washington Monthly which released its

first set of College Rankings in September 2005 With the declared aim of using an

institutionrsquos commitment to social mobility as a measure of quality it uses the percentage

of students from low-income backgrounds as an indicator (with percentage of students

receiving need-based government (Pell) grants used as a proxy)

Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy

statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the

percentage of the student population who are graduate students (arguably this is a

research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the

number of graduate students auditing classes is used as an indicator the assumption is

presumably that if people are auditing then the classes must be very attractive The Italian

La Repubblica ranks an institution according to the number of part-time students it has

contrary to prevailing North American views on the undesirability of part-time study the

Italian rankings see higher numbers of part-time students in a positive light as it is

evidence that an institution is becoming less rigid in its timetabling and

5

permitting students to juggle both work and study something which was nearly

impossible in that country just a few years ago

The Washington Monthly also has a final category of indicators which reflect studentsrsquo

beginning characteristics namely their likelihood of performing community service as

measured by the percentage of students in the US Peace Corps and Reserve Officer

Training Corps (ROTC) and the percentage of students involved in work-study in the

community3 Together these three indicators account for 33 of an institutionrsquos total

ranking

B Indicators of Learning InputsmdashStaff

Generally both the quantity and quality of staff are positively correlated with institutional

quality The problem of course is finding useful metrics for each of these factors

especially if one excludes as we have done here measures of research performance and

research intensity putting them in a separate category4

The simplest measure is simply the number of faculty unadjusted for things like size of

student body Most national league tables however prefer to use variations on the

concept of facultystudent ratio Others try to measure teaching intensity with measures

such as courses per teacher or hours spent in class per student (both in La Repubblica)

These kinds of measures usually account for between 2-5 of the final rankings although

in some cases (ie the Guardian) this figure can be as high as 20

Another important way of measuring how faculty resources are deployed is the measure

of average class size which is used only by Macleanrsquos and the US News and World Report

Ostensibly the reason for measuring class size is to account in some form for the degree

3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category

6

of attention devoted to actually teaching students Despite the fact that important research

(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level

the use of this indicator appears to be a spillover from the North American debates on

class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein

2000) Regardless of why the indicators are used they are extraordinarily important to

these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and

World Reportrsquos rankings respectively

A number of ranking systems try to look at staff qualifications such as the number of

PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the

Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further

than other surveys and actually looks at the proportion of classes taught by tenure-track

staff Others (ie THES) look at the number of foreign faculty based on the assumption

that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still

others (ie La Repubblica) look at the age structure of the faculty Another proxy for

institutional quality is the pay rates for tenured staff on the assumption that institutions

with higher rates of pay on average attract better faculty this measure has been used both

by the US News and World Report and Asiaweek

Finally a number of league tables rank faculty inputs on the basis of standardized third-

party evaluations 6

Education18 the Financial Times the Times the Guardian and the US

News and World Report league tables all use some sort of ranking criterion based at least in

part on this indicator or variations thereof

C Indicators of Learning InputsmdashResources

Resource inputsmdashcrudely the amount of current dollars equipment and books available

to students at an institutionmdashare widely considered an important measure of quality Yet

despite the apparent simplicity of counting dollars and measuring assets

5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each

university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future

7

the means by which institutional wealth is measured varies considerably between ranking

systems

There are a number of revenue-based measures of resources Macleanrsquos uses public funding

of institutional budgets as a factor in its analysis conversely the Financial Times uses the

private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of

total score) and the US News and World Report (5 of total score) also measure alumni

financial support as a measure of quality

For reasons that are not entirely clear league tables tend to favour measures of

expenditures rather than revenues The Guardian looks at total institutional expenditures

as an indicator Institutional expenditure on student services is used as a measure of

institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total

institutional scores respectively) Rzezspospolita does not measure student services

expenditures directly but does measure student services outputs such as number of

student athletes and number of study clubs which amounts to more or less the same thing

Macleanrsquos also gives out 433 of its total score based on institutional expenditures on

scholarships and bursaries

Various aspects of physical infrastructure are also used as measures of institutional

resources most directly in the case of La Repubblica which bases 317 of its total rank on

the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding

assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type

of physical infrastructure measured is available Internet bandwidth which was used by

Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth

roughly 3 of the total score

By some distance the infrastructure indicators most favoured by the compilers of league

tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on

this with 12 of the total quality mark being taken from various types of library

infrastructure measurements (including acquisitions per year total volumes average

number of volumes per student and yearly library expenditure outside of acquisitions)

Netbig and Education18 also use library volume holdings while Asiaweek the Financial

Times and the Times also use measures of library expenditures outside of acquisitions or

computerization of library resources as measures of institutional quality

8

One important factor to note is that most ranking systems do not normalize their resource

and infrastructure measures That is to say it is raw spending power or simple size of

assets that is usually measured rather than spending per studentprofessor or assets per

studentprofessor As a result a number of these rankings systems have inbuilt biases

towards larger institutions

D Indicators of Learning Outputs

Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge

learned over the course of a baccalaureate degreemdashshould be a basic indicator of

institutional quality Unfortunately good means of measuring these outputsmdashlike the

National Survey of Student Engagement (NSSE) and the College Learning Assessment

(CLA)mdashhave only recently become available and for the most part institutions are still

keeping their scores secret Outside of these measures only a few very crude indicators are

available which likely explains why learning outputs do not feature especially prominently

in most ranking schemes

The simplest types of measures of learning outputs are those linked to graduation and

retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong

and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7

the latter three also use rates of graduation from Masterrsquos programs as indicators In some

cases the weights on these measures can be very highmdashin the Guangdong rankings

graduation rates account for over 50 of the rankingmdashbut in most cases the weights are

10 or less Retention rates commonly meaning the progression rate of first-year students

into second year are accorded less importance The US News and World Report Macleanrsquos

the Melbourne Institute and La Repubblica all employ retention measures as indicators but

none of them are worth more than 4 of total weighting Two publications make specific

indicators for retention and graduation of international students Macleanrsquos (graduation

rates of international students) and the Melbourne Institute (retention rates of international

students) The Washington Monthly looks specifically at institutional retention rates

adjusted for the participation of lower-income students and gives higher scores to

institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores

and number of Pell Grant recipients the US

7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18

9

News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos

ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)

E Indicators of Final Outcomes

Final outcomes are indications of generalized outcomes for students after graduation

Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness

good citizenship) but given the somewhat utilitarian justifications for education that are

currently in fashion (see Wolf 2000) employment outcomes are the most commonly used

measure of final outcomes These are given particular emphasis by the Guardian (where

employment outcomes are worth 17 of the total score) but are also used by the Financial

Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the

Times are interestingly not concerned with employment per se but with ldquoemployment in

an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher

Education Statistics Agency (HESA) uses Standard Occupational Classifications to

measure the proportion of graduates in professional or white-collar jobs anyone not in

such a job is considered not to be working in an area related to their studies (it is unclear

what methodology is used by the Financial Times and the Times although we suspect their

methods are broadly similar)

The only other measure of final outcomes in use is percentage of graduates returning for

additional education which is an indicator used by both the Melbourne Institute and the

Financial Times This is a particularly important indicator for the latter as it is worth 21

of the final ranking

The lack of indicators concerning final outcomes is interesting since most government-

sponsored performance-indicator regimes around the world are very much concerned

with such measures especially with respect to employment Possibly this indicates that

ranking systems simply do not view education outcomes as relevant measures of

educational quality Alternatively it may be the case that they simply have not found a

reliable indicator of outcomes or that there are reliable indicators but that there is so little

variation between institutions that it makes no sense to rank based on the data

10

F Indicators of Research

Many of the league tables covered in this survey include sections and weightings related

to universitiesrsquo research efforts It is in this field of measurement that we see the greatest

diversity of indicators Presumably this is because research inputs and outputs lend

themselves much more easily to measurement and manipulation than other areas of

institutional activity

Three studies include research staff as part of their ranking scheme La Repubblica at 952

the Melbourne Institute (4) and Wuhan (078)

Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used

method of looking at research quality but it is not universally admired because different

disciplines use different means to communicate major advances in knowledge (leading

scientists invariably produce large numbers of journal articles leading social scientists

may produce fewer journal articles but instead have one or two long important

monographsmdashsee Hicks 2004) There is also some concern among non-English speaking

countries that they are penalized in international rankings because so many of the major

journals (notably Science and Nature) are printed in English However the one set of

rankings that uses separate indicators to monitor articles published in English and articles

published in another language (the Wuhan rankings) shows that the two indicators are

positively correlated institutions that have more Chinese publications are also likely to

have more English publications and vice versa

Several sets of league tables measure bibliometric citations in various publication indices

The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a

weight of 20 of the final total Guangdong also monitors other Chinese universities

specifically for citations in engineering publications and weights this at 29 Moreover it

tacks on an additional 1049 for citations in science-oriented indices such as the Science

Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the

same category of scientific citations while the Melbourne Institute rates science citations at

68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social

Science Citation Index which does not include the humanities) are noted in only two

league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne

Institute (32) Another way of measuring research

11

impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a

weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the

Melbourne Institute

The complement of citations is of course publications Listing the number of publications

an individual group department or whole university releases can act as a weak substitute

for citationsmdashweak because simply publishing a paper or monograph is no guarantee that

the general public or other researchers will even glance at the work Guangdong gives an

1179 weighting to publications in science-oriented indices such as the Science Citation

Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same

indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong

even has a separate category just for measuring publications in Science and Nature

although it accords it an almost derisory weighting of 06 Under publications in social

science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their

final totals For publications in other indices (where the subject indices are

undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for

Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of

rankings to include research monographs weighted at 033

In countries where there are specific third-party evaluations of research output academic

quality of research is sometimes used as a research indicator The Times puts a very large

30 weight on this indicator while the Financial Times puts it at 119

Research awards are another handy third-party measurement of quality as the number of

international and national awards won by faculty andor graduates is often considered a

useful measure of institutional success International research awardsmdash specifically the

number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator

by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as

particularly suspect in some quarters given that the points are based on where the

recipient went to school rather than on where they are or were on the

8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the

most prominent publication indices

9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences

12

faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead

for nearly a century Wuhan uses similar measures but only accords them a weight of 14

National research awards are more common as a quality indicator used by La Repubblica

(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)

Financial indicators of research are also very common Research budgets as a factor in the

overall assessment of research in universities are covered by the Financial Times (9)

Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for

total amount of research expenditure unfortunately it is unclear precisely what this

research expenditure represents or how it is determined although it is clearly indicated

that it does not represent the total number of grants or projects at a university Total

number of research-based grants and projects is weighted by Education18 at 15 and

Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science

and engineering and another 55 to those for social sciences and humanities Similarly

the Melbourne Institute gives 6 of its overall weight to public-source grants making no

distinction between areas of study

In a slightly different vein Netbig (46) and Wuhan (278) both list the number of

research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica

(095) all weigh research-basedaffiliated research institutions or centres for studies

Finally one can also measure research not simply in terms of the amount of money it

generates but also in terms of the amount of future income it will generate Both

Guangdong (245) and Wuhan (193) measure the number of patents issued to

universities as a quality indicator

A final way of measuring an institutionrsquos research intensity is to look at the range of its

course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of

doctoral and Masterrsquos programs offered as a proxy for research intensity

As with physical and financial resources few if any of the research indicators are

normalized to account for institutional size (either by student or faculty numbers) In the

world of rankings bigger almost always means better an institution with 100 faculty with

ten citations apiece will always look worse than an institution with 1001 faculty

13

with one citation each To the extent that the raw production of knowledge matters this

form of measurement is acceptable To the extent that rankings are meant to show how

well institutions are doing on a like-to-like basis or to show the efficiency of universities it

is plainly inadequate This should be of particular concern to Chinese policy-makers

whose ranking systems are especially reliant on research-based indicators

G Indicators of Reputation

The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo

Those rankings systems which use the results of reputation surveys as an indicator do so as

an indirect measure of quality based on the assumption that the employers academics and

academic administrators surveyed have opinions of institutional quality that are informed

up-to-date and impartial While these assumptions are clearly open to debate they

nevertheless form an important basis for many ranking systems Another reason for using

reputation measures is the paucity of other data availablemdashsome countries have few

independent measures of teaching effectiveness university resources or output and

reputation can thus act as a useful surrogate Reputation rankings are often criticized as

simply quantifying the common ignorance of the people being surveyed However to the

extent that the people being surveyed hold positions which have the potential to affect

large numbers of young people and whose positions actually require some knowledge of

institutional quality (ie officials in charge of graduate admissions corporate recruiters

etc) then reputation rankings make sense because they provide useful information for

students about the perceived value of the degrees that they could obtain from various

universities

The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and

the Times which both accord reputation a weighting of 50 in their overall ranking

scheme Education18 assigns it almost as much significance at 40 The US News and World

Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just

below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)

and Netbig (15) The only other study to include reputation is Wuhan (117)

14

1 Please see Appendix B for a glossary of the various bodies referenced in this document

Appendix A Detailed Listing of Indicators and their Sources

Beginning Characteristics Indicator

Used By Source1

Incoming grades Macleans University

Percentage with grades above a set limit

Macleans University

US News and World Report

University

Performance on national standardised tests or benchmarks

Asiaweek University

Education18 3rd-party JUPAS

Financial Times Government agency 3rd-party UCAS

Guardian University Guide

Government agency 3rd-party UCAS

Melbourne Institute Government agency 3rd-party DEST

Netbig National entrance examination board

Times Good University Guide

Government agency 3rd-party UCAS

US News University

Wuhan Unknown presumed government 3rd-party

Student status La Repubblica Government agency 3rd-party MIUR

Admittance selectivity general

Asiaweek University

Admittance number of applications to places

Asiaweek University

Financial Times Government agency 3rd-party UCAS

La Repubblica Government agency 3rd-party MIUR

US News University

15

Beginning Characteristics Indicator

Used By Source1

Out-of-locality student percentage

Macleans University

International student percentages

Financial Times Government agency 3rd-party HESA

Macleans University

Shanghai Institute of Educational Science

Times World University

Wuhan Unknown presumed university

Undergraduate students among all students percentages

Netbig

Wuhan Unknown presumed university

Ethnic diversity in student body

Guardian University

Learning Inputs ndash Staff Indicator

Used By Source

Facultystudent ratio Asiaweek University

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Financial Times Government agency 3rd-party HESA

La Repubblica Government agency 3rd-party MIUR

Times Good University Guide

Government agency 3rd-party HESA

Times World University

US News University

Wuhan Unknown presumed university

16

Learning Inputs ndash Staff Indicator

Used By Source

Social science faculty student ratio

Melbourne Government agency 3rd-party DEST

Science faculty student ratio

Melbourne Government agency 3rd-party DEST

Administrative staff student ratio

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Staff student ratio (regardless of division)

Guardian Government agency 3rd-party HESA

Netbig University

Course per teacher La Repubblica Government agency 3rd-party MIUR

Per-teacher university spending

Asiaweek University

Faculty pay rates fortenured staff

Asiaweek University

US News University

Number of full-time part-time faculty

Netbig University

US News University

Wuhan Unknown presumed university

Faculty with researchprojects

Wuhan Unknown presumed university

Class size differentiation Macleans University

US News University

Classes taught by tenured faculty

Macleans University

Exchange programmes hosted

La Repubblica Government agency 3rd-party AgNaSoc

Number of classes lsquoactually taughtrsquo

La Repubblica Government agency 3rd-party MIUR

of international faculty (v faculty as a whole)

Times World University

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

Table of Contents

Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25

2

Table 1 ndash Number of Indictors by Type of Data Source

Table 1 shows that surveys are the least frequently used source of data for indicators

Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a

plurality of indicators come from this source This measure somewhat underestimates the

importance of surveys however as it does not account for the weighting given to each

indicator in each study In the THES World Rankings for instance there is only a single

survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos

Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one

indicator is worth 20 of the final score

Raw

indIcator count

Surve

y data

Third parties

Universities

AsiaweekmdashAsias Best Universities 18 - - 18

Daily Telegraph (2003) 1 - 1 -

Education18com 9 3 4 2

Excelencia 2001 71 - 71 -

Financial Times (2003) 17 - 17 -

Guangdong Institute of Management Science 17 - 14 3

GuardianmdashUniversity Guide 2005 7 - 2 5

La Repubblica 23 2 21 -

Macleans University Rankings 24 1 5 18

Melbourne Institutemdash International Standing of Australian Universities

26 3 23 -

Netbig 2004 18 1 10 7

Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15

Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities

6 - 5 1

The TimesmdashGood University Guide 2005 9 - 9 -

Times Higher Education SupplementmdashWorld University Rankings

5 1 1 3

US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11

Washington MonthlymdashCollege Rankings 2005 8 - 1 7

Wuhan University Centre for Science Evaluation 45 2 22 21

3

A Indicators of Beginning Characteristics

ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of

students at the time they begin their studies Fourteen of the 18 rankings examined in this

study use one or more indicators of the beginning characteristics of students to arrive at

their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on

these factors with 33 of the total ranking coming from this class of indicators but the

Guardian Education18 Asiaweek and the two other North American surveys also place

considerable emphasis on this category

There are six main indicators used to determine which institutions have students with

positive ldquobeginning characteristicsrdquo

The most common measure of beginning characteristics is performance on national

standardized tests with nine surveys using this as a measure Education18 and the

Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used

by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)

Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this

data is collected and standardized by national bodies it has the benefit of being seen as a

relatively impartial method of determining the relative ldquostrengthrdquo of the students entering

each institution Institutionsrsquo results can be scored by showing either averages or the

percentage of entering students meeting a particular standard

Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses

secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This

is a second-best solution made necessary by the absence of any national standardized test

in Canada (or indeed of any provincial standardized tests at the end of secondary school

in provinces other than Alberta) The lack of national standardization makes this an

undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is

truly equivalent to an ldquoArdquo in another jurisdiction

Another measure of the strength of the student body is the percentage of incoming

students receiving (third-party) scholarships which is worth 11 of the score in the

Wuhan survey One can also approach the issue by measuring institutional selectivity In

effect this method infers the strength of the student body by the proportion of

4

applicants rejected the theory being that the higher the number of rejected applicants the

stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)

Normally this measure is expressed as a straight ratio of acceptances to applications but it

can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the

greatest weight on this measure) as a ratio of enrolments to applications Within the US

there is some dispute as to what constitutes an offer of admission and whether or not late

admissions are included as noted by Ganeshananthan (2003)

Student bodies are often considered to be strong if the school is able to attract a large

number of international or out-of-district students or if they contain people from diverse

ethnic backgrounds A number of league tables use the international student indicator

(which like the selectivity indicator is arguably as much an indicator of prestige and

reputation as it is of student characteristics) although in no case does this indicator

account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a

quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data

on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian

puts a somewhat larger emphasis on this indicator in comparison to other league tables

which use similar variables

A very different take on this idea is present in the Washington Monthly which released its

first set of College Rankings in September 2005 With the declared aim of using an

institutionrsquos commitment to social mobility as a measure of quality it uses the percentage

of students from low-income backgrounds as an indicator (with percentage of students

receiving need-based government (Pell) grants used as a proxy)

Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy

statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the

percentage of the student population who are graduate students (arguably this is a

research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the

number of graduate students auditing classes is used as an indicator the assumption is

presumably that if people are auditing then the classes must be very attractive The Italian

La Repubblica ranks an institution according to the number of part-time students it has

contrary to prevailing North American views on the undesirability of part-time study the

Italian rankings see higher numbers of part-time students in a positive light as it is

evidence that an institution is becoming less rigid in its timetabling and

5

permitting students to juggle both work and study something which was nearly

impossible in that country just a few years ago

The Washington Monthly also has a final category of indicators which reflect studentsrsquo

beginning characteristics namely their likelihood of performing community service as

measured by the percentage of students in the US Peace Corps and Reserve Officer

Training Corps (ROTC) and the percentage of students involved in work-study in the

community3 Together these three indicators account for 33 of an institutionrsquos total

ranking

B Indicators of Learning InputsmdashStaff

Generally both the quantity and quality of staff are positively correlated with institutional

quality The problem of course is finding useful metrics for each of these factors

especially if one excludes as we have done here measures of research performance and

research intensity putting them in a separate category4

The simplest measure is simply the number of faculty unadjusted for things like size of

student body Most national league tables however prefer to use variations on the

concept of facultystudent ratio Others try to measure teaching intensity with measures

such as courses per teacher or hours spent in class per student (both in La Repubblica)

These kinds of measures usually account for between 2-5 of the final rankings although

in some cases (ie the Guardian) this figure can be as high as 20

Another important way of measuring how faculty resources are deployed is the measure

of average class size which is used only by Macleanrsquos and the US News and World Report

Ostensibly the reason for measuring class size is to account in some form for the degree

3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category

6

of attention devoted to actually teaching students Despite the fact that important research

(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level

the use of this indicator appears to be a spillover from the North American debates on

class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein

2000) Regardless of why the indicators are used they are extraordinarily important to

these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and

World Reportrsquos rankings respectively

A number of ranking systems try to look at staff qualifications such as the number of

PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the

Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further

than other surveys and actually looks at the proportion of classes taught by tenure-track

staff Others (ie THES) look at the number of foreign faculty based on the assumption

that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still

others (ie La Repubblica) look at the age structure of the faculty Another proxy for

institutional quality is the pay rates for tenured staff on the assumption that institutions

with higher rates of pay on average attract better faculty this measure has been used both

by the US News and World Report and Asiaweek

Finally a number of league tables rank faculty inputs on the basis of standardized third-

party evaluations 6

Education18 the Financial Times the Times the Guardian and the US

News and World Report league tables all use some sort of ranking criterion based at least in

part on this indicator or variations thereof

C Indicators of Learning InputsmdashResources

Resource inputsmdashcrudely the amount of current dollars equipment and books available

to students at an institutionmdashare widely considered an important measure of quality Yet

despite the apparent simplicity of counting dollars and measuring assets

5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each

university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future

7

the means by which institutional wealth is measured varies considerably between ranking

systems

There are a number of revenue-based measures of resources Macleanrsquos uses public funding

of institutional budgets as a factor in its analysis conversely the Financial Times uses the

private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of

total score) and the US News and World Report (5 of total score) also measure alumni

financial support as a measure of quality

For reasons that are not entirely clear league tables tend to favour measures of

expenditures rather than revenues The Guardian looks at total institutional expenditures

as an indicator Institutional expenditure on student services is used as a measure of

institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total

institutional scores respectively) Rzezspospolita does not measure student services

expenditures directly but does measure student services outputs such as number of

student athletes and number of study clubs which amounts to more or less the same thing

Macleanrsquos also gives out 433 of its total score based on institutional expenditures on

scholarships and bursaries

Various aspects of physical infrastructure are also used as measures of institutional

resources most directly in the case of La Repubblica which bases 317 of its total rank on

the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding

assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type

of physical infrastructure measured is available Internet bandwidth which was used by

Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth

roughly 3 of the total score

By some distance the infrastructure indicators most favoured by the compilers of league

tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on

this with 12 of the total quality mark being taken from various types of library

infrastructure measurements (including acquisitions per year total volumes average

number of volumes per student and yearly library expenditure outside of acquisitions)

Netbig and Education18 also use library volume holdings while Asiaweek the Financial

Times and the Times also use measures of library expenditures outside of acquisitions or

computerization of library resources as measures of institutional quality

8

One important factor to note is that most ranking systems do not normalize their resource

and infrastructure measures That is to say it is raw spending power or simple size of

assets that is usually measured rather than spending per studentprofessor or assets per

studentprofessor As a result a number of these rankings systems have inbuilt biases

towards larger institutions

D Indicators of Learning Outputs

Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge

learned over the course of a baccalaureate degreemdashshould be a basic indicator of

institutional quality Unfortunately good means of measuring these outputsmdashlike the

National Survey of Student Engagement (NSSE) and the College Learning Assessment

(CLA)mdashhave only recently become available and for the most part institutions are still

keeping their scores secret Outside of these measures only a few very crude indicators are

available which likely explains why learning outputs do not feature especially prominently

in most ranking schemes

The simplest types of measures of learning outputs are those linked to graduation and

retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong

and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7

the latter three also use rates of graduation from Masterrsquos programs as indicators In some

cases the weights on these measures can be very highmdashin the Guangdong rankings

graduation rates account for over 50 of the rankingmdashbut in most cases the weights are

10 or less Retention rates commonly meaning the progression rate of first-year students

into second year are accorded less importance The US News and World Report Macleanrsquos

the Melbourne Institute and La Repubblica all employ retention measures as indicators but

none of them are worth more than 4 of total weighting Two publications make specific

indicators for retention and graduation of international students Macleanrsquos (graduation

rates of international students) and the Melbourne Institute (retention rates of international

students) The Washington Monthly looks specifically at institutional retention rates

adjusted for the participation of lower-income students and gives higher scores to

institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores

and number of Pell Grant recipients the US

7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18

9

News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos

ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)

E Indicators of Final Outcomes

Final outcomes are indications of generalized outcomes for students after graduation

Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness

good citizenship) but given the somewhat utilitarian justifications for education that are

currently in fashion (see Wolf 2000) employment outcomes are the most commonly used

measure of final outcomes These are given particular emphasis by the Guardian (where

employment outcomes are worth 17 of the total score) but are also used by the Financial

Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the

Times are interestingly not concerned with employment per se but with ldquoemployment in

an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher

Education Statistics Agency (HESA) uses Standard Occupational Classifications to

measure the proportion of graduates in professional or white-collar jobs anyone not in

such a job is considered not to be working in an area related to their studies (it is unclear

what methodology is used by the Financial Times and the Times although we suspect their

methods are broadly similar)

The only other measure of final outcomes in use is percentage of graduates returning for

additional education which is an indicator used by both the Melbourne Institute and the

Financial Times This is a particularly important indicator for the latter as it is worth 21

of the final ranking

The lack of indicators concerning final outcomes is interesting since most government-

sponsored performance-indicator regimes around the world are very much concerned

with such measures especially with respect to employment Possibly this indicates that

ranking systems simply do not view education outcomes as relevant measures of

educational quality Alternatively it may be the case that they simply have not found a

reliable indicator of outcomes or that there are reliable indicators but that there is so little

variation between institutions that it makes no sense to rank based on the data

10

F Indicators of Research

Many of the league tables covered in this survey include sections and weightings related

to universitiesrsquo research efforts It is in this field of measurement that we see the greatest

diversity of indicators Presumably this is because research inputs and outputs lend

themselves much more easily to measurement and manipulation than other areas of

institutional activity

Three studies include research staff as part of their ranking scheme La Repubblica at 952

the Melbourne Institute (4) and Wuhan (078)

Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used

method of looking at research quality but it is not universally admired because different

disciplines use different means to communicate major advances in knowledge (leading

scientists invariably produce large numbers of journal articles leading social scientists

may produce fewer journal articles but instead have one or two long important

monographsmdashsee Hicks 2004) There is also some concern among non-English speaking

countries that they are penalized in international rankings because so many of the major

journals (notably Science and Nature) are printed in English However the one set of

rankings that uses separate indicators to monitor articles published in English and articles

published in another language (the Wuhan rankings) shows that the two indicators are

positively correlated institutions that have more Chinese publications are also likely to

have more English publications and vice versa

Several sets of league tables measure bibliometric citations in various publication indices

The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a

weight of 20 of the final total Guangdong also monitors other Chinese universities

specifically for citations in engineering publications and weights this at 29 Moreover it

tacks on an additional 1049 for citations in science-oriented indices such as the Science

Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the

same category of scientific citations while the Melbourne Institute rates science citations at

68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social

Science Citation Index which does not include the humanities) are noted in only two

league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne

Institute (32) Another way of measuring research

11

impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a

weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the

Melbourne Institute

The complement of citations is of course publications Listing the number of publications

an individual group department or whole university releases can act as a weak substitute

for citationsmdashweak because simply publishing a paper or monograph is no guarantee that

the general public or other researchers will even glance at the work Guangdong gives an

1179 weighting to publications in science-oriented indices such as the Science Citation

Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same

indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong

even has a separate category just for measuring publications in Science and Nature

although it accords it an almost derisory weighting of 06 Under publications in social

science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their

final totals For publications in other indices (where the subject indices are

undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for

Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of

rankings to include research monographs weighted at 033

In countries where there are specific third-party evaluations of research output academic

quality of research is sometimes used as a research indicator The Times puts a very large

30 weight on this indicator while the Financial Times puts it at 119

Research awards are another handy third-party measurement of quality as the number of

international and national awards won by faculty andor graduates is often considered a

useful measure of institutional success International research awardsmdash specifically the

number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator

by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as

particularly suspect in some quarters given that the points are based on where the

recipient went to school rather than on where they are or were on the

8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the

most prominent publication indices

9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences

12

faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead

for nearly a century Wuhan uses similar measures but only accords them a weight of 14

National research awards are more common as a quality indicator used by La Repubblica

(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)

Financial indicators of research are also very common Research budgets as a factor in the

overall assessment of research in universities are covered by the Financial Times (9)

Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for

total amount of research expenditure unfortunately it is unclear precisely what this

research expenditure represents or how it is determined although it is clearly indicated

that it does not represent the total number of grants or projects at a university Total

number of research-based grants and projects is weighted by Education18 at 15 and

Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science

and engineering and another 55 to those for social sciences and humanities Similarly

the Melbourne Institute gives 6 of its overall weight to public-source grants making no

distinction between areas of study

In a slightly different vein Netbig (46) and Wuhan (278) both list the number of

research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica

(095) all weigh research-basedaffiliated research institutions or centres for studies

Finally one can also measure research not simply in terms of the amount of money it

generates but also in terms of the amount of future income it will generate Both

Guangdong (245) and Wuhan (193) measure the number of patents issued to

universities as a quality indicator

A final way of measuring an institutionrsquos research intensity is to look at the range of its

course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of

doctoral and Masterrsquos programs offered as a proxy for research intensity

As with physical and financial resources few if any of the research indicators are

normalized to account for institutional size (either by student or faculty numbers) In the

world of rankings bigger almost always means better an institution with 100 faculty with

ten citations apiece will always look worse than an institution with 1001 faculty

13

with one citation each To the extent that the raw production of knowledge matters this

form of measurement is acceptable To the extent that rankings are meant to show how

well institutions are doing on a like-to-like basis or to show the efficiency of universities it

is plainly inadequate This should be of particular concern to Chinese policy-makers

whose ranking systems are especially reliant on research-based indicators

G Indicators of Reputation

The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo

Those rankings systems which use the results of reputation surveys as an indicator do so as

an indirect measure of quality based on the assumption that the employers academics and

academic administrators surveyed have opinions of institutional quality that are informed

up-to-date and impartial While these assumptions are clearly open to debate they

nevertheless form an important basis for many ranking systems Another reason for using

reputation measures is the paucity of other data availablemdashsome countries have few

independent measures of teaching effectiveness university resources or output and

reputation can thus act as a useful surrogate Reputation rankings are often criticized as

simply quantifying the common ignorance of the people being surveyed However to the

extent that the people being surveyed hold positions which have the potential to affect

large numbers of young people and whose positions actually require some knowledge of

institutional quality (ie officials in charge of graduate admissions corporate recruiters

etc) then reputation rankings make sense because they provide useful information for

students about the perceived value of the degrees that they could obtain from various

universities

The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and

the Times which both accord reputation a weighting of 50 in their overall ranking

scheme Education18 assigns it almost as much significance at 40 The US News and World

Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just

below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)

and Netbig (15) The only other study to include reputation is Wuhan (117)

14

1 Please see Appendix B for a glossary of the various bodies referenced in this document

Appendix A Detailed Listing of Indicators and their Sources

Beginning Characteristics Indicator

Used By Source1

Incoming grades Macleans University

Percentage with grades above a set limit

Macleans University

US News and World Report

University

Performance on national standardised tests or benchmarks

Asiaweek University

Education18 3rd-party JUPAS

Financial Times Government agency 3rd-party UCAS

Guardian University Guide

Government agency 3rd-party UCAS

Melbourne Institute Government agency 3rd-party DEST

Netbig National entrance examination board

Times Good University Guide

Government agency 3rd-party UCAS

US News University

Wuhan Unknown presumed government 3rd-party

Student status La Repubblica Government agency 3rd-party MIUR

Admittance selectivity general

Asiaweek University

Admittance number of applications to places

Asiaweek University

Financial Times Government agency 3rd-party UCAS

La Repubblica Government agency 3rd-party MIUR

US News University

15

Beginning Characteristics Indicator

Used By Source1

Out-of-locality student percentage

Macleans University

International student percentages

Financial Times Government agency 3rd-party HESA

Macleans University

Shanghai Institute of Educational Science

Times World University

Wuhan Unknown presumed university

Undergraduate students among all students percentages

Netbig

Wuhan Unknown presumed university

Ethnic diversity in student body

Guardian University

Learning Inputs ndash Staff Indicator

Used By Source

Facultystudent ratio Asiaweek University

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Financial Times Government agency 3rd-party HESA

La Repubblica Government agency 3rd-party MIUR

Times Good University Guide

Government agency 3rd-party HESA

Times World University

US News University

Wuhan Unknown presumed university

16

Learning Inputs ndash Staff Indicator

Used By Source

Social science faculty student ratio

Melbourne Government agency 3rd-party DEST

Science faculty student ratio

Melbourne Government agency 3rd-party DEST

Administrative staff student ratio

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Staff student ratio (regardless of division)

Guardian Government agency 3rd-party HESA

Netbig University

Course per teacher La Repubblica Government agency 3rd-party MIUR

Per-teacher university spending

Asiaweek University

Faculty pay rates fortenured staff

Asiaweek University

US News University

Number of full-time part-time faculty

Netbig University

US News University

Wuhan Unknown presumed university

Faculty with researchprojects

Wuhan Unknown presumed university

Class size differentiation Macleans University

US News University

Classes taught by tenured faculty

Macleans University

Exchange programmes hosted

La Repubblica Government agency 3rd-party AgNaSoc

Number of classes lsquoactually taughtrsquo

La Repubblica Government agency 3rd-party MIUR

of international faculty (v faculty as a whole)

Times World University

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

2

Table 1 ndash Number of Indictors by Type of Data Source

Table 1 shows that surveys are the least frequently used source of data for indicators

Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a

plurality of indicators come from this source This measure somewhat underestimates the

importance of surveys however as it does not account for the weighting given to each

indicator in each study In the THES World Rankings for instance there is only a single

survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos

Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one

indicator is worth 20 of the final score

Raw

indIcator count

Surve

y data

Third parties

Universities

AsiaweekmdashAsias Best Universities 18 - - 18

Daily Telegraph (2003) 1 - 1 -

Education18com 9 3 4 2

Excelencia 2001 71 - 71 -

Financial Times (2003) 17 - 17 -

Guangdong Institute of Management Science 17 - 14 3

GuardianmdashUniversity Guide 2005 7 - 2 5

La Repubblica 23 2 21 -

Macleans University Rankings 24 1 5 18

Melbourne Institutemdash International Standing of Australian Universities

26 3 23 -

Netbig 2004 18 1 10 7

Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15

Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities

6 - 5 1

The TimesmdashGood University Guide 2005 9 - 9 -

Times Higher Education SupplementmdashWorld University Rankings

5 1 1 3

US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11

Washington MonthlymdashCollege Rankings 2005 8 - 1 7

Wuhan University Centre for Science Evaluation 45 2 22 21

3

A Indicators of Beginning Characteristics

ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of

students at the time they begin their studies Fourteen of the 18 rankings examined in this

study use one or more indicators of the beginning characteristics of students to arrive at

their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on

these factors with 33 of the total ranking coming from this class of indicators but the

Guardian Education18 Asiaweek and the two other North American surveys also place

considerable emphasis on this category

There are six main indicators used to determine which institutions have students with

positive ldquobeginning characteristicsrdquo

The most common measure of beginning characteristics is performance on national

standardized tests with nine surveys using this as a measure Education18 and the

Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used

by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)

Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this

data is collected and standardized by national bodies it has the benefit of being seen as a

relatively impartial method of determining the relative ldquostrengthrdquo of the students entering

each institution Institutionsrsquo results can be scored by showing either averages or the

percentage of entering students meeting a particular standard

Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses

secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This

is a second-best solution made necessary by the absence of any national standardized test

in Canada (or indeed of any provincial standardized tests at the end of secondary school

in provinces other than Alberta) The lack of national standardization makes this an

undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is

truly equivalent to an ldquoArdquo in another jurisdiction

Another measure of the strength of the student body is the percentage of incoming

students receiving (third-party) scholarships which is worth 11 of the score in the

Wuhan survey One can also approach the issue by measuring institutional selectivity In

effect this method infers the strength of the student body by the proportion of

4

applicants rejected the theory being that the higher the number of rejected applicants the

stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)

Normally this measure is expressed as a straight ratio of acceptances to applications but it

can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the

greatest weight on this measure) as a ratio of enrolments to applications Within the US

there is some dispute as to what constitutes an offer of admission and whether or not late

admissions are included as noted by Ganeshananthan (2003)

Student bodies are often considered to be strong if the school is able to attract a large

number of international or out-of-district students or if they contain people from diverse

ethnic backgrounds A number of league tables use the international student indicator

(which like the selectivity indicator is arguably as much an indicator of prestige and

reputation as it is of student characteristics) although in no case does this indicator

account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a

quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data

on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian

puts a somewhat larger emphasis on this indicator in comparison to other league tables

which use similar variables

A very different take on this idea is present in the Washington Monthly which released its

first set of College Rankings in September 2005 With the declared aim of using an

institutionrsquos commitment to social mobility as a measure of quality it uses the percentage

of students from low-income backgrounds as an indicator (with percentage of students

receiving need-based government (Pell) grants used as a proxy)

Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy

statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the

percentage of the student population who are graduate students (arguably this is a

research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the

number of graduate students auditing classes is used as an indicator the assumption is

presumably that if people are auditing then the classes must be very attractive The Italian

La Repubblica ranks an institution according to the number of part-time students it has

contrary to prevailing North American views on the undesirability of part-time study the

Italian rankings see higher numbers of part-time students in a positive light as it is

evidence that an institution is becoming less rigid in its timetabling and

5

permitting students to juggle both work and study something which was nearly

impossible in that country just a few years ago

The Washington Monthly also has a final category of indicators which reflect studentsrsquo

beginning characteristics namely their likelihood of performing community service as

measured by the percentage of students in the US Peace Corps and Reserve Officer

Training Corps (ROTC) and the percentage of students involved in work-study in the

community3 Together these three indicators account for 33 of an institutionrsquos total

ranking

B Indicators of Learning InputsmdashStaff

Generally both the quantity and quality of staff are positively correlated with institutional

quality The problem of course is finding useful metrics for each of these factors

especially if one excludes as we have done here measures of research performance and

research intensity putting them in a separate category4

The simplest measure is simply the number of faculty unadjusted for things like size of

student body Most national league tables however prefer to use variations on the

concept of facultystudent ratio Others try to measure teaching intensity with measures

such as courses per teacher or hours spent in class per student (both in La Repubblica)

These kinds of measures usually account for between 2-5 of the final rankings although

in some cases (ie the Guardian) this figure can be as high as 20

Another important way of measuring how faculty resources are deployed is the measure

of average class size which is used only by Macleanrsquos and the US News and World Report

Ostensibly the reason for measuring class size is to account in some form for the degree

3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category

6

of attention devoted to actually teaching students Despite the fact that important research

(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level

the use of this indicator appears to be a spillover from the North American debates on

class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein

2000) Regardless of why the indicators are used they are extraordinarily important to

these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and

World Reportrsquos rankings respectively

A number of ranking systems try to look at staff qualifications such as the number of

PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the

Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further

than other surveys and actually looks at the proportion of classes taught by tenure-track

staff Others (ie THES) look at the number of foreign faculty based on the assumption

that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still

others (ie La Repubblica) look at the age structure of the faculty Another proxy for

institutional quality is the pay rates for tenured staff on the assumption that institutions

with higher rates of pay on average attract better faculty this measure has been used both

by the US News and World Report and Asiaweek

Finally a number of league tables rank faculty inputs on the basis of standardized third-

party evaluations 6

Education18 the Financial Times the Times the Guardian and the US

News and World Report league tables all use some sort of ranking criterion based at least in

part on this indicator or variations thereof

C Indicators of Learning InputsmdashResources

Resource inputsmdashcrudely the amount of current dollars equipment and books available

to students at an institutionmdashare widely considered an important measure of quality Yet

despite the apparent simplicity of counting dollars and measuring assets

5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each

university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future

7

the means by which institutional wealth is measured varies considerably between ranking

systems

There are a number of revenue-based measures of resources Macleanrsquos uses public funding

of institutional budgets as a factor in its analysis conversely the Financial Times uses the

private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of

total score) and the US News and World Report (5 of total score) also measure alumni

financial support as a measure of quality

For reasons that are not entirely clear league tables tend to favour measures of

expenditures rather than revenues The Guardian looks at total institutional expenditures

as an indicator Institutional expenditure on student services is used as a measure of

institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total

institutional scores respectively) Rzezspospolita does not measure student services

expenditures directly but does measure student services outputs such as number of

student athletes and number of study clubs which amounts to more or less the same thing

Macleanrsquos also gives out 433 of its total score based on institutional expenditures on

scholarships and bursaries

Various aspects of physical infrastructure are also used as measures of institutional

resources most directly in the case of La Repubblica which bases 317 of its total rank on

the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding

assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type

of physical infrastructure measured is available Internet bandwidth which was used by

Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth

roughly 3 of the total score

By some distance the infrastructure indicators most favoured by the compilers of league

tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on

this with 12 of the total quality mark being taken from various types of library

infrastructure measurements (including acquisitions per year total volumes average

number of volumes per student and yearly library expenditure outside of acquisitions)

Netbig and Education18 also use library volume holdings while Asiaweek the Financial

Times and the Times also use measures of library expenditures outside of acquisitions or

computerization of library resources as measures of institutional quality

8

One important factor to note is that most ranking systems do not normalize their resource

and infrastructure measures That is to say it is raw spending power or simple size of

assets that is usually measured rather than spending per studentprofessor or assets per

studentprofessor As a result a number of these rankings systems have inbuilt biases

towards larger institutions

D Indicators of Learning Outputs

Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge

learned over the course of a baccalaureate degreemdashshould be a basic indicator of

institutional quality Unfortunately good means of measuring these outputsmdashlike the

National Survey of Student Engagement (NSSE) and the College Learning Assessment

(CLA)mdashhave only recently become available and for the most part institutions are still

keeping their scores secret Outside of these measures only a few very crude indicators are

available which likely explains why learning outputs do not feature especially prominently

in most ranking schemes

The simplest types of measures of learning outputs are those linked to graduation and

retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong

and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7

the latter three also use rates of graduation from Masterrsquos programs as indicators In some

cases the weights on these measures can be very highmdashin the Guangdong rankings

graduation rates account for over 50 of the rankingmdashbut in most cases the weights are

10 or less Retention rates commonly meaning the progression rate of first-year students

into second year are accorded less importance The US News and World Report Macleanrsquos

the Melbourne Institute and La Repubblica all employ retention measures as indicators but

none of them are worth more than 4 of total weighting Two publications make specific

indicators for retention and graduation of international students Macleanrsquos (graduation

rates of international students) and the Melbourne Institute (retention rates of international

students) The Washington Monthly looks specifically at institutional retention rates

adjusted for the participation of lower-income students and gives higher scores to

institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores

and number of Pell Grant recipients the US

7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18

9

News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos

ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)

E Indicators of Final Outcomes

Final outcomes are indications of generalized outcomes for students after graduation

Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness

good citizenship) but given the somewhat utilitarian justifications for education that are

currently in fashion (see Wolf 2000) employment outcomes are the most commonly used

measure of final outcomes These are given particular emphasis by the Guardian (where

employment outcomes are worth 17 of the total score) but are also used by the Financial

Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the

Times are interestingly not concerned with employment per se but with ldquoemployment in

an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher

Education Statistics Agency (HESA) uses Standard Occupational Classifications to

measure the proportion of graduates in professional or white-collar jobs anyone not in

such a job is considered not to be working in an area related to their studies (it is unclear

what methodology is used by the Financial Times and the Times although we suspect their

methods are broadly similar)

The only other measure of final outcomes in use is percentage of graduates returning for

additional education which is an indicator used by both the Melbourne Institute and the

Financial Times This is a particularly important indicator for the latter as it is worth 21

of the final ranking

The lack of indicators concerning final outcomes is interesting since most government-

sponsored performance-indicator regimes around the world are very much concerned

with such measures especially with respect to employment Possibly this indicates that

ranking systems simply do not view education outcomes as relevant measures of

educational quality Alternatively it may be the case that they simply have not found a

reliable indicator of outcomes or that there are reliable indicators but that there is so little

variation between institutions that it makes no sense to rank based on the data

10

F Indicators of Research

Many of the league tables covered in this survey include sections and weightings related

to universitiesrsquo research efforts It is in this field of measurement that we see the greatest

diversity of indicators Presumably this is because research inputs and outputs lend

themselves much more easily to measurement and manipulation than other areas of

institutional activity

Three studies include research staff as part of their ranking scheme La Repubblica at 952

the Melbourne Institute (4) and Wuhan (078)

Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used

method of looking at research quality but it is not universally admired because different

disciplines use different means to communicate major advances in knowledge (leading

scientists invariably produce large numbers of journal articles leading social scientists

may produce fewer journal articles but instead have one or two long important

monographsmdashsee Hicks 2004) There is also some concern among non-English speaking

countries that they are penalized in international rankings because so many of the major

journals (notably Science and Nature) are printed in English However the one set of

rankings that uses separate indicators to monitor articles published in English and articles

published in another language (the Wuhan rankings) shows that the two indicators are

positively correlated institutions that have more Chinese publications are also likely to

have more English publications and vice versa

Several sets of league tables measure bibliometric citations in various publication indices

The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a

weight of 20 of the final total Guangdong also monitors other Chinese universities

specifically for citations in engineering publications and weights this at 29 Moreover it

tacks on an additional 1049 for citations in science-oriented indices such as the Science

Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the

same category of scientific citations while the Melbourne Institute rates science citations at

68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social

Science Citation Index which does not include the humanities) are noted in only two

league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne

Institute (32) Another way of measuring research

11

impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a

weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the

Melbourne Institute

The complement of citations is of course publications Listing the number of publications

an individual group department or whole university releases can act as a weak substitute

for citationsmdashweak because simply publishing a paper or monograph is no guarantee that

the general public or other researchers will even glance at the work Guangdong gives an

1179 weighting to publications in science-oriented indices such as the Science Citation

Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same

indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong

even has a separate category just for measuring publications in Science and Nature

although it accords it an almost derisory weighting of 06 Under publications in social

science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their

final totals For publications in other indices (where the subject indices are

undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for

Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of

rankings to include research monographs weighted at 033

In countries where there are specific third-party evaluations of research output academic

quality of research is sometimes used as a research indicator The Times puts a very large

30 weight on this indicator while the Financial Times puts it at 119

Research awards are another handy third-party measurement of quality as the number of

international and national awards won by faculty andor graduates is often considered a

useful measure of institutional success International research awardsmdash specifically the

number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator

by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as

particularly suspect in some quarters given that the points are based on where the

recipient went to school rather than on where they are or were on the

8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the

most prominent publication indices

9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences

12

faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead

for nearly a century Wuhan uses similar measures but only accords them a weight of 14

National research awards are more common as a quality indicator used by La Repubblica

(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)

Financial indicators of research are also very common Research budgets as a factor in the

overall assessment of research in universities are covered by the Financial Times (9)

Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for

total amount of research expenditure unfortunately it is unclear precisely what this

research expenditure represents or how it is determined although it is clearly indicated

that it does not represent the total number of grants or projects at a university Total

number of research-based grants and projects is weighted by Education18 at 15 and

Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science

and engineering and another 55 to those for social sciences and humanities Similarly

the Melbourne Institute gives 6 of its overall weight to public-source grants making no

distinction between areas of study

In a slightly different vein Netbig (46) and Wuhan (278) both list the number of

research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica

(095) all weigh research-basedaffiliated research institutions or centres for studies

Finally one can also measure research not simply in terms of the amount of money it

generates but also in terms of the amount of future income it will generate Both

Guangdong (245) and Wuhan (193) measure the number of patents issued to

universities as a quality indicator

A final way of measuring an institutionrsquos research intensity is to look at the range of its

course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of

doctoral and Masterrsquos programs offered as a proxy for research intensity

As with physical and financial resources few if any of the research indicators are

normalized to account for institutional size (either by student or faculty numbers) In the

world of rankings bigger almost always means better an institution with 100 faculty with

ten citations apiece will always look worse than an institution with 1001 faculty

13

with one citation each To the extent that the raw production of knowledge matters this

form of measurement is acceptable To the extent that rankings are meant to show how

well institutions are doing on a like-to-like basis or to show the efficiency of universities it

is plainly inadequate This should be of particular concern to Chinese policy-makers

whose ranking systems are especially reliant on research-based indicators

G Indicators of Reputation

The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo

Those rankings systems which use the results of reputation surveys as an indicator do so as

an indirect measure of quality based on the assumption that the employers academics and

academic administrators surveyed have opinions of institutional quality that are informed

up-to-date and impartial While these assumptions are clearly open to debate they

nevertheless form an important basis for many ranking systems Another reason for using

reputation measures is the paucity of other data availablemdashsome countries have few

independent measures of teaching effectiveness university resources or output and

reputation can thus act as a useful surrogate Reputation rankings are often criticized as

simply quantifying the common ignorance of the people being surveyed However to the

extent that the people being surveyed hold positions which have the potential to affect

large numbers of young people and whose positions actually require some knowledge of

institutional quality (ie officials in charge of graduate admissions corporate recruiters

etc) then reputation rankings make sense because they provide useful information for

students about the perceived value of the degrees that they could obtain from various

universities

The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and

the Times which both accord reputation a weighting of 50 in their overall ranking

scheme Education18 assigns it almost as much significance at 40 The US News and World

Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just

below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)

and Netbig (15) The only other study to include reputation is Wuhan (117)

14

1 Please see Appendix B for a glossary of the various bodies referenced in this document

Appendix A Detailed Listing of Indicators and their Sources

Beginning Characteristics Indicator

Used By Source1

Incoming grades Macleans University

Percentage with grades above a set limit

Macleans University

US News and World Report

University

Performance on national standardised tests or benchmarks

Asiaweek University

Education18 3rd-party JUPAS

Financial Times Government agency 3rd-party UCAS

Guardian University Guide

Government agency 3rd-party UCAS

Melbourne Institute Government agency 3rd-party DEST

Netbig National entrance examination board

Times Good University Guide

Government agency 3rd-party UCAS

US News University

Wuhan Unknown presumed government 3rd-party

Student status La Repubblica Government agency 3rd-party MIUR

Admittance selectivity general

Asiaweek University

Admittance number of applications to places

Asiaweek University

Financial Times Government agency 3rd-party UCAS

La Repubblica Government agency 3rd-party MIUR

US News University

15

Beginning Characteristics Indicator

Used By Source1

Out-of-locality student percentage

Macleans University

International student percentages

Financial Times Government agency 3rd-party HESA

Macleans University

Shanghai Institute of Educational Science

Times World University

Wuhan Unknown presumed university

Undergraduate students among all students percentages

Netbig

Wuhan Unknown presumed university

Ethnic diversity in student body

Guardian University

Learning Inputs ndash Staff Indicator

Used By Source

Facultystudent ratio Asiaweek University

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Financial Times Government agency 3rd-party HESA

La Repubblica Government agency 3rd-party MIUR

Times Good University Guide

Government agency 3rd-party HESA

Times World University

US News University

Wuhan Unknown presumed university

16

Learning Inputs ndash Staff Indicator

Used By Source

Social science faculty student ratio

Melbourne Government agency 3rd-party DEST

Science faculty student ratio

Melbourne Government agency 3rd-party DEST

Administrative staff student ratio

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Staff student ratio (regardless of division)

Guardian Government agency 3rd-party HESA

Netbig University

Course per teacher La Repubblica Government agency 3rd-party MIUR

Per-teacher university spending

Asiaweek University

Faculty pay rates fortenured staff

Asiaweek University

US News University

Number of full-time part-time faculty

Netbig University

US News University

Wuhan Unknown presumed university

Faculty with researchprojects

Wuhan Unknown presumed university

Class size differentiation Macleans University

US News University

Classes taught by tenured faculty

Macleans University

Exchange programmes hosted

La Repubblica Government agency 3rd-party AgNaSoc

Number of classes lsquoactually taughtrsquo

La Repubblica Government agency 3rd-party MIUR

of international faculty (v faculty as a whole)

Times World University

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

3

A Indicators of Beginning Characteristics

ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of

students at the time they begin their studies Fourteen of the 18 rankings examined in this

study use one or more indicators of the beginning characteristics of students to arrive at

their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on

these factors with 33 of the total ranking coming from this class of indicators but the

Guardian Education18 Asiaweek and the two other North American surveys also place

considerable emphasis on this category

There are six main indicators used to determine which institutions have students with

positive ldquobeginning characteristicsrdquo

The most common measure of beginning characteristics is performance on national

standardized tests with nine surveys using this as a measure Education18 and the

Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used

by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)

Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this

data is collected and standardized by national bodies it has the benefit of being seen as a

relatively impartial method of determining the relative ldquostrengthrdquo of the students entering

each institution Institutionsrsquo results can be scored by showing either averages or the

percentage of entering students meeting a particular standard

Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses

secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This

is a second-best solution made necessary by the absence of any national standardized test

in Canada (or indeed of any provincial standardized tests at the end of secondary school

in provinces other than Alberta) The lack of national standardization makes this an

undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is

truly equivalent to an ldquoArdquo in another jurisdiction

Another measure of the strength of the student body is the percentage of incoming

students receiving (third-party) scholarships which is worth 11 of the score in the

Wuhan survey One can also approach the issue by measuring institutional selectivity In

effect this method infers the strength of the student body by the proportion of

4

applicants rejected the theory being that the higher the number of rejected applicants the

stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)

Normally this measure is expressed as a straight ratio of acceptances to applications but it

can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the

greatest weight on this measure) as a ratio of enrolments to applications Within the US

there is some dispute as to what constitutes an offer of admission and whether or not late

admissions are included as noted by Ganeshananthan (2003)

Student bodies are often considered to be strong if the school is able to attract a large

number of international or out-of-district students or if they contain people from diverse

ethnic backgrounds A number of league tables use the international student indicator

(which like the selectivity indicator is arguably as much an indicator of prestige and

reputation as it is of student characteristics) although in no case does this indicator

account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a

quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data

on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian

puts a somewhat larger emphasis on this indicator in comparison to other league tables

which use similar variables

A very different take on this idea is present in the Washington Monthly which released its

first set of College Rankings in September 2005 With the declared aim of using an

institutionrsquos commitment to social mobility as a measure of quality it uses the percentage

of students from low-income backgrounds as an indicator (with percentage of students

receiving need-based government (Pell) grants used as a proxy)

Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy

statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the

percentage of the student population who are graduate students (arguably this is a

research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the

number of graduate students auditing classes is used as an indicator the assumption is

presumably that if people are auditing then the classes must be very attractive The Italian

La Repubblica ranks an institution according to the number of part-time students it has

contrary to prevailing North American views on the undesirability of part-time study the

Italian rankings see higher numbers of part-time students in a positive light as it is

evidence that an institution is becoming less rigid in its timetabling and

5

permitting students to juggle both work and study something which was nearly

impossible in that country just a few years ago

The Washington Monthly also has a final category of indicators which reflect studentsrsquo

beginning characteristics namely their likelihood of performing community service as

measured by the percentage of students in the US Peace Corps and Reserve Officer

Training Corps (ROTC) and the percentage of students involved in work-study in the

community3 Together these three indicators account for 33 of an institutionrsquos total

ranking

B Indicators of Learning InputsmdashStaff

Generally both the quantity and quality of staff are positively correlated with institutional

quality The problem of course is finding useful metrics for each of these factors

especially if one excludes as we have done here measures of research performance and

research intensity putting them in a separate category4

The simplest measure is simply the number of faculty unadjusted for things like size of

student body Most national league tables however prefer to use variations on the

concept of facultystudent ratio Others try to measure teaching intensity with measures

such as courses per teacher or hours spent in class per student (both in La Repubblica)

These kinds of measures usually account for between 2-5 of the final rankings although

in some cases (ie the Guardian) this figure can be as high as 20

Another important way of measuring how faculty resources are deployed is the measure

of average class size which is used only by Macleanrsquos and the US News and World Report

Ostensibly the reason for measuring class size is to account in some form for the degree

3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category

6

of attention devoted to actually teaching students Despite the fact that important research

(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level

the use of this indicator appears to be a spillover from the North American debates on

class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein

2000) Regardless of why the indicators are used they are extraordinarily important to

these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and

World Reportrsquos rankings respectively

A number of ranking systems try to look at staff qualifications such as the number of

PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the

Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further

than other surveys and actually looks at the proportion of classes taught by tenure-track

staff Others (ie THES) look at the number of foreign faculty based on the assumption

that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still

others (ie La Repubblica) look at the age structure of the faculty Another proxy for

institutional quality is the pay rates for tenured staff on the assumption that institutions

with higher rates of pay on average attract better faculty this measure has been used both

by the US News and World Report and Asiaweek

Finally a number of league tables rank faculty inputs on the basis of standardized third-

party evaluations 6

Education18 the Financial Times the Times the Guardian and the US

News and World Report league tables all use some sort of ranking criterion based at least in

part on this indicator or variations thereof

C Indicators of Learning InputsmdashResources

Resource inputsmdashcrudely the amount of current dollars equipment and books available

to students at an institutionmdashare widely considered an important measure of quality Yet

despite the apparent simplicity of counting dollars and measuring assets

5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each

university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future

7

the means by which institutional wealth is measured varies considerably between ranking

systems

There are a number of revenue-based measures of resources Macleanrsquos uses public funding

of institutional budgets as a factor in its analysis conversely the Financial Times uses the

private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of

total score) and the US News and World Report (5 of total score) also measure alumni

financial support as a measure of quality

For reasons that are not entirely clear league tables tend to favour measures of

expenditures rather than revenues The Guardian looks at total institutional expenditures

as an indicator Institutional expenditure on student services is used as a measure of

institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total

institutional scores respectively) Rzezspospolita does not measure student services

expenditures directly but does measure student services outputs such as number of

student athletes and number of study clubs which amounts to more or less the same thing

Macleanrsquos also gives out 433 of its total score based on institutional expenditures on

scholarships and bursaries

Various aspects of physical infrastructure are also used as measures of institutional

resources most directly in the case of La Repubblica which bases 317 of its total rank on

the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding

assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type

of physical infrastructure measured is available Internet bandwidth which was used by

Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth

roughly 3 of the total score

By some distance the infrastructure indicators most favoured by the compilers of league

tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on

this with 12 of the total quality mark being taken from various types of library

infrastructure measurements (including acquisitions per year total volumes average

number of volumes per student and yearly library expenditure outside of acquisitions)

Netbig and Education18 also use library volume holdings while Asiaweek the Financial

Times and the Times also use measures of library expenditures outside of acquisitions or

computerization of library resources as measures of institutional quality

8

One important factor to note is that most ranking systems do not normalize their resource

and infrastructure measures That is to say it is raw spending power or simple size of

assets that is usually measured rather than spending per studentprofessor or assets per

studentprofessor As a result a number of these rankings systems have inbuilt biases

towards larger institutions

D Indicators of Learning Outputs

Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge

learned over the course of a baccalaureate degreemdashshould be a basic indicator of

institutional quality Unfortunately good means of measuring these outputsmdashlike the

National Survey of Student Engagement (NSSE) and the College Learning Assessment

(CLA)mdashhave only recently become available and for the most part institutions are still

keeping their scores secret Outside of these measures only a few very crude indicators are

available which likely explains why learning outputs do not feature especially prominently

in most ranking schemes

The simplest types of measures of learning outputs are those linked to graduation and

retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong

and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7

the latter three also use rates of graduation from Masterrsquos programs as indicators In some

cases the weights on these measures can be very highmdashin the Guangdong rankings

graduation rates account for over 50 of the rankingmdashbut in most cases the weights are

10 or less Retention rates commonly meaning the progression rate of first-year students

into second year are accorded less importance The US News and World Report Macleanrsquos

the Melbourne Institute and La Repubblica all employ retention measures as indicators but

none of them are worth more than 4 of total weighting Two publications make specific

indicators for retention and graduation of international students Macleanrsquos (graduation

rates of international students) and the Melbourne Institute (retention rates of international

students) The Washington Monthly looks specifically at institutional retention rates

adjusted for the participation of lower-income students and gives higher scores to

institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores

and number of Pell Grant recipients the US

7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18

9

News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos

ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)

E Indicators of Final Outcomes

Final outcomes are indications of generalized outcomes for students after graduation

Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness

good citizenship) but given the somewhat utilitarian justifications for education that are

currently in fashion (see Wolf 2000) employment outcomes are the most commonly used

measure of final outcomes These are given particular emphasis by the Guardian (where

employment outcomes are worth 17 of the total score) but are also used by the Financial

Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the

Times are interestingly not concerned with employment per se but with ldquoemployment in

an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher

Education Statistics Agency (HESA) uses Standard Occupational Classifications to

measure the proportion of graduates in professional or white-collar jobs anyone not in

such a job is considered not to be working in an area related to their studies (it is unclear

what methodology is used by the Financial Times and the Times although we suspect their

methods are broadly similar)

The only other measure of final outcomes in use is percentage of graduates returning for

additional education which is an indicator used by both the Melbourne Institute and the

Financial Times This is a particularly important indicator for the latter as it is worth 21

of the final ranking

The lack of indicators concerning final outcomes is interesting since most government-

sponsored performance-indicator regimes around the world are very much concerned

with such measures especially with respect to employment Possibly this indicates that

ranking systems simply do not view education outcomes as relevant measures of

educational quality Alternatively it may be the case that they simply have not found a

reliable indicator of outcomes or that there are reliable indicators but that there is so little

variation between institutions that it makes no sense to rank based on the data

10

F Indicators of Research

Many of the league tables covered in this survey include sections and weightings related

to universitiesrsquo research efforts It is in this field of measurement that we see the greatest

diversity of indicators Presumably this is because research inputs and outputs lend

themselves much more easily to measurement and manipulation than other areas of

institutional activity

Three studies include research staff as part of their ranking scheme La Repubblica at 952

the Melbourne Institute (4) and Wuhan (078)

Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used

method of looking at research quality but it is not universally admired because different

disciplines use different means to communicate major advances in knowledge (leading

scientists invariably produce large numbers of journal articles leading social scientists

may produce fewer journal articles but instead have one or two long important

monographsmdashsee Hicks 2004) There is also some concern among non-English speaking

countries that they are penalized in international rankings because so many of the major

journals (notably Science and Nature) are printed in English However the one set of

rankings that uses separate indicators to monitor articles published in English and articles

published in another language (the Wuhan rankings) shows that the two indicators are

positively correlated institutions that have more Chinese publications are also likely to

have more English publications and vice versa

Several sets of league tables measure bibliometric citations in various publication indices

The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a

weight of 20 of the final total Guangdong also monitors other Chinese universities

specifically for citations in engineering publications and weights this at 29 Moreover it

tacks on an additional 1049 for citations in science-oriented indices such as the Science

Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the

same category of scientific citations while the Melbourne Institute rates science citations at

68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social

Science Citation Index which does not include the humanities) are noted in only two

league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne

Institute (32) Another way of measuring research

11

impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a

weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the

Melbourne Institute

The complement of citations is of course publications Listing the number of publications

an individual group department or whole university releases can act as a weak substitute

for citationsmdashweak because simply publishing a paper or monograph is no guarantee that

the general public or other researchers will even glance at the work Guangdong gives an

1179 weighting to publications in science-oriented indices such as the Science Citation

Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same

indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong

even has a separate category just for measuring publications in Science and Nature

although it accords it an almost derisory weighting of 06 Under publications in social

science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their

final totals For publications in other indices (where the subject indices are

undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for

Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of

rankings to include research monographs weighted at 033

In countries where there are specific third-party evaluations of research output academic

quality of research is sometimes used as a research indicator The Times puts a very large

30 weight on this indicator while the Financial Times puts it at 119

Research awards are another handy third-party measurement of quality as the number of

international and national awards won by faculty andor graduates is often considered a

useful measure of institutional success International research awardsmdash specifically the

number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator

by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as

particularly suspect in some quarters given that the points are based on where the

recipient went to school rather than on where they are or were on the

8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the

most prominent publication indices

9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences

12

faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead

for nearly a century Wuhan uses similar measures but only accords them a weight of 14

National research awards are more common as a quality indicator used by La Repubblica

(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)

Financial indicators of research are also very common Research budgets as a factor in the

overall assessment of research in universities are covered by the Financial Times (9)

Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for

total amount of research expenditure unfortunately it is unclear precisely what this

research expenditure represents or how it is determined although it is clearly indicated

that it does not represent the total number of grants or projects at a university Total

number of research-based grants and projects is weighted by Education18 at 15 and

Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science

and engineering and another 55 to those for social sciences and humanities Similarly

the Melbourne Institute gives 6 of its overall weight to public-source grants making no

distinction between areas of study

In a slightly different vein Netbig (46) and Wuhan (278) both list the number of

research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica

(095) all weigh research-basedaffiliated research institutions or centres for studies

Finally one can also measure research not simply in terms of the amount of money it

generates but also in terms of the amount of future income it will generate Both

Guangdong (245) and Wuhan (193) measure the number of patents issued to

universities as a quality indicator

A final way of measuring an institutionrsquos research intensity is to look at the range of its

course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of

doctoral and Masterrsquos programs offered as a proxy for research intensity

As with physical and financial resources few if any of the research indicators are

normalized to account for institutional size (either by student or faculty numbers) In the

world of rankings bigger almost always means better an institution with 100 faculty with

ten citations apiece will always look worse than an institution with 1001 faculty

13

with one citation each To the extent that the raw production of knowledge matters this

form of measurement is acceptable To the extent that rankings are meant to show how

well institutions are doing on a like-to-like basis or to show the efficiency of universities it

is plainly inadequate This should be of particular concern to Chinese policy-makers

whose ranking systems are especially reliant on research-based indicators

G Indicators of Reputation

The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo

Those rankings systems which use the results of reputation surveys as an indicator do so as

an indirect measure of quality based on the assumption that the employers academics and

academic administrators surveyed have opinions of institutional quality that are informed

up-to-date and impartial While these assumptions are clearly open to debate they

nevertheless form an important basis for many ranking systems Another reason for using

reputation measures is the paucity of other data availablemdashsome countries have few

independent measures of teaching effectiveness university resources or output and

reputation can thus act as a useful surrogate Reputation rankings are often criticized as

simply quantifying the common ignorance of the people being surveyed However to the

extent that the people being surveyed hold positions which have the potential to affect

large numbers of young people and whose positions actually require some knowledge of

institutional quality (ie officials in charge of graduate admissions corporate recruiters

etc) then reputation rankings make sense because they provide useful information for

students about the perceived value of the degrees that they could obtain from various

universities

The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and

the Times which both accord reputation a weighting of 50 in their overall ranking

scheme Education18 assigns it almost as much significance at 40 The US News and World

Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just

below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)

and Netbig (15) The only other study to include reputation is Wuhan (117)

14

1 Please see Appendix B for a glossary of the various bodies referenced in this document

Appendix A Detailed Listing of Indicators and their Sources

Beginning Characteristics Indicator

Used By Source1

Incoming grades Macleans University

Percentage with grades above a set limit

Macleans University

US News and World Report

University

Performance on national standardised tests or benchmarks

Asiaweek University

Education18 3rd-party JUPAS

Financial Times Government agency 3rd-party UCAS

Guardian University Guide

Government agency 3rd-party UCAS

Melbourne Institute Government agency 3rd-party DEST

Netbig National entrance examination board

Times Good University Guide

Government agency 3rd-party UCAS

US News University

Wuhan Unknown presumed government 3rd-party

Student status La Repubblica Government agency 3rd-party MIUR

Admittance selectivity general

Asiaweek University

Admittance number of applications to places

Asiaweek University

Financial Times Government agency 3rd-party UCAS

La Repubblica Government agency 3rd-party MIUR

US News University

15

Beginning Characteristics Indicator

Used By Source1

Out-of-locality student percentage

Macleans University

International student percentages

Financial Times Government agency 3rd-party HESA

Macleans University

Shanghai Institute of Educational Science

Times World University

Wuhan Unknown presumed university

Undergraduate students among all students percentages

Netbig

Wuhan Unknown presumed university

Ethnic diversity in student body

Guardian University

Learning Inputs ndash Staff Indicator

Used By Source

Facultystudent ratio Asiaweek University

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Financial Times Government agency 3rd-party HESA

La Repubblica Government agency 3rd-party MIUR

Times Good University Guide

Government agency 3rd-party HESA

Times World University

US News University

Wuhan Unknown presumed university

16

Learning Inputs ndash Staff Indicator

Used By Source

Social science faculty student ratio

Melbourne Government agency 3rd-party DEST

Science faculty student ratio

Melbourne Government agency 3rd-party DEST

Administrative staff student ratio

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Staff student ratio (regardless of division)

Guardian Government agency 3rd-party HESA

Netbig University

Course per teacher La Repubblica Government agency 3rd-party MIUR

Per-teacher university spending

Asiaweek University

Faculty pay rates fortenured staff

Asiaweek University

US News University

Number of full-time part-time faculty

Netbig University

US News University

Wuhan Unknown presumed university

Faculty with researchprojects

Wuhan Unknown presumed university

Class size differentiation Macleans University

US News University

Classes taught by tenured faculty

Macleans University

Exchange programmes hosted

La Repubblica Government agency 3rd-party AgNaSoc

Number of classes lsquoactually taughtrsquo

La Repubblica Government agency 3rd-party MIUR

of international faculty (v faculty as a whole)

Times World University

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

4

applicants rejected the theory being that the higher the number of rejected applicants the

stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)

Normally this measure is expressed as a straight ratio of acceptances to applications but it

can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the

greatest weight on this measure) as a ratio of enrolments to applications Within the US

there is some dispute as to what constitutes an offer of admission and whether or not late

admissions are included as noted by Ganeshananthan (2003)

Student bodies are often considered to be strong if the school is able to attract a large

number of international or out-of-district students or if they contain people from diverse

ethnic backgrounds A number of league tables use the international student indicator

(which like the selectivity indicator is arguably as much an indicator of prestige and

reputation as it is of student characteristics) although in no case does this indicator

account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a

quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data

on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian

puts a somewhat larger emphasis on this indicator in comparison to other league tables

which use similar variables

A very different take on this idea is present in the Washington Monthly which released its

first set of College Rankings in September 2005 With the declared aim of using an

institutionrsquos commitment to social mobility as a measure of quality it uses the percentage

of students from low-income backgrounds as an indicator (with percentage of students

receiving need-based government (Pell) grants used as a proxy)

Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy

statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the

percentage of the student population who are graduate students (arguably this is a

research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the

number of graduate students auditing classes is used as an indicator the assumption is

presumably that if people are auditing then the classes must be very attractive The Italian

La Repubblica ranks an institution according to the number of part-time students it has

contrary to prevailing North American views on the undesirability of part-time study the

Italian rankings see higher numbers of part-time students in a positive light as it is

evidence that an institution is becoming less rigid in its timetabling and

5

permitting students to juggle both work and study something which was nearly

impossible in that country just a few years ago

The Washington Monthly also has a final category of indicators which reflect studentsrsquo

beginning characteristics namely their likelihood of performing community service as

measured by the percentage of students in the US Peace Corps and Reserve Officer

Training Corps (ROTC) and the percentage of students involved in work-study in the

community3 Together these three indicators account for 33 of an institutionrsquos total

ranking

B Indicators of Learning InputsmdashStaff

Generally both the quantity and quality of staff are positively correlated with institutional

quality The problem of course is finding useful metrics for each of these factors

especially if one excludes as we have done here measures of research performance and

research intensity putting them in a separate category4

The simplest measure is simply the number of faculty unadjusted for things like size of

student body Most national league tables however prefer to use variations on the

concept of facultystudent ratio Others try to measure teaching intensity with measures

such as courses per teacher or hours spent in class per student (both in La Repubblica)

These kinds of measures usually account for between 2-5 of the final rankings although

in some cases (ie the Guardian) this figure can be as high as 20

Another important way of measuring how faculty resources are deployed is the measure

of average class size which is used only by Macleanrsquos and the US News and World Report

Ostensibly the reason for measuring class size is to account in some form for the degree

3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category

6

of attention devoted to actually teaching students Despite the fact that important research

(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level

the use of this indicator appears to be a spillover from the North American debates on

class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein

2000) Regardless of why the indicators are used they are extraordinarily important to

these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and

World Reportrsquos rankings respectively

A number of ranking systems try to look at staff qualifications such as the number of

PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the

Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further

than other surveys and actually looks at the proportion of classes taught by tenure-track

staff Others (ie THES) look at the number of foreign faculty based on the assumption

that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still

others (ie La Repubblica) look at the age structure of the faculty Another proxy for

institutional quality is the pay rates for tenured staff on the assumption that institutions

with higher rates of pay on average attract better faculty this measure has been used both

by the US News and World Report and Asiaweek

Finally a number of league tables rank faculty inputs on the basis of standardized third-

party evaluations 6

Education18 the Financial Times the Times the Guardian and the US

News and World Report league tables all use some sort of ranking criterion based at least in

part on this indicator or variations thereof

C Indicators of Learning InputsmdashResources

Resource inputsmdashcrudely the amount of current dollars equipment and books available

to students at an institutionmdashare widely considered an important measure of quality Yet

despite the apparent simplicity of counting dollars and measuring assets

5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each

university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future

7

the means by which institutional wealth is measured varies considerably between ranking

systems

There are a number of revenue-based measures of resources Macleanrsquos uses public funding

of institutional budgets as a factor in its analysis conversely the Financial Times uses the

private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of

total score) and the US News and World Report (5 of total score) also measure alumni

financial support as a measure of quality

For reasons that are not entirely clear league tables tend to favour measures of

expenditures rather than revenues The Guardian looks at total institutional expenditures

as an indicator Institutional expenditure on student services is used as a measure of

institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total

institutional scores respectively) Rzezspospolita does not measure student services

expenditures directly but does measure student services outputs such as number of

student athletes and number of study clubs which amounts to more or less the same thing

Macleanrsquos also gives out 433 of its total score based on institutional expenditures on

scholarships and bursaries

Various aspects of physical infrastructure are also used as measures of institutional

resources most directly in the case of La Repubblica which bases 317 of its total rank on

the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding

assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type

of physical infrastructure measured is available Internet bandwidth which was used by

Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth

roughly 3 of the total score

By some distance the infrastructure indicators most favoured by the compilers of league

tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on

this with 12 of the total quality mark being taken from various types of library

infrastructure measurements (including acquisitions per year total volumes average

number of volumes per student and yearly library expenditure outside of acquisitions)

Netbig and Education18 also use library volume holdings while Asiaweek the Financial

Times and the Times also use measures of library expenditures outside of acquisitions or

computerization of library resources as measures of institutional quality

8

One important factor to note is that most ranking systems do not normalize their resource

and infrastructure measures That is to say it is raw spending power or simple size of

assets that is usually measured rather than spending per studentprofessor or assets per

studentprofessor As a result a number of these rankings systems have inbuilt biases

towards larger institutions

D Indicators of Learning Outputs

Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge

learned over the course of a baccalaureate degreemdashshould be a basic indicator of

institutional quality Unfortunately good means of measuring these outputsmdashlike the

National Survey of Student Engagement (NSSE) and the College Learning Assessment

(CLA)mdashhave only recently become available and for the most part institutions are still

keeping their scores secret Outside of these measures only a few very crude indicators are

available which likely explains why learning outputs do not feature especially prominently

in most ranking schemes

The simplest types of measures of learning outputs are those linked to graduation and

retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong

and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7

the latter three also use rates of graduation from Masterrsquos programs as indicators In some

cases the weights on these measures can be very highmdashin the Guangdong rankings

graduation rates account for over 50 of the rankingmdashbut in most cases the weights are

10 or less Retention rates commonly meaning the progression rate of first-year students

into second year are accorded less importance The US News and World Report Macleanrsquos

the Melbourne Institute and La Repubblica all employ retention measures as indicators but

none of them are worth more than 4 of total weighting Two publications make specific

indicators for retention and graduation of international students Macleanrsquos (graduation

rates of international students) and the Melbourne Institute (retention rates of international

students) The Washington Monthly looks specifically at institutional retention rates

adjusted for the participation of lower-income students and gives higher scores to

institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores

and number of Pell Grant recipients the US

7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18

9

News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos

ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)

E Indicators of Final Outcomes

Final outcomes are indications of generalized outcomes for students after graduation

Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness

good citizenship) but given the somewhat utilitarian justifications for education that are

currently in fashion (see Wolf 2000) employment outcomes are the most commonly used

measure of final outcomes These are given particular emphasis by the Guardian (where

employment outcomes are worth 17 of the total score) but are also used by the Financial

Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the

Times are interestingly not concerned with employment per se but with ldquoemployment in

an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher

Education Statistics Agency (HESA) uses Standard Occupational Classifications to

measure the proportion of graduates in professional or white-collar jobs anyone not in

such a job is considered not to be working in an area related to their studies (it is unclear

what methodology is used by the Financial Times and the Times although we suspect their

methods are broadly similar)

The only other measure of final outcomes in use is percentage of graduates returning for

additional education which is an indicator used by both the Melbourne Institute and the

Financial Times This is a particularly important indicator for the latter as it is worth 21

of the final ranking

The lack of indicators concerning final outcomes is interesting since most government-

sponsored performance-indicator regimes around the world are very much concerned

with such measures especially with respect to employment Possibly this indicates that

ranking systems simply do not view education outcomes as relevant measures of

educational quality Alternatively it may be the case that they simply have not found a

reliable indicator of outcomes or that there are reliable indicators but that there is so little

variation between institutions that it makes no sense to rank based on the data

10

F Indicators of Research

Many of the league tables covered in this survey include sections and weightings related

to universitiesrsquo research efforts It is in this field of measurement that we see the greatest

diversity of indicators Presumably this is because research inputs and outputs lend

themselves much more easily to measurement and manipulation than other areas of

institutional activity

Three studies include research staff as part of their ranking scheme La Repubblica at 952

the Melbourne Institute (4) and Wuhan (078)

Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used

method of looking at research quality but it is not universally admired because different

disciplines use different means to communicate major advances in knowledge (leading

scientists invariably produce large numbers of journal articles leading social scientists

may produce fewer journal articles but instead have one or two long important

monographsmdashsee Hicks 2004) There is also some concern among non-English speaking

countries that they are penalized in international rankings because so many of the major

journals (notably Science and Nature) are printed in English However the one set of

rankings that uses separate indicators to monitor articles published in English and articles

published in another language (the Wuhan rankings) shows that the two indicators are

positively correlated institutions that have more Chinese publications are also likely to

have more English publications and vice versa

Several sets of league tables measure bibliometric citations in various publication indices

The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a

weight of 20 of the final total Guangdong also monitors other Chinese universities

specifically for citations in engineering publications and weights this at 29 Moreover it

tacks on an additional 1049 for citations in science-oriented indices such as the Science

Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the

same category of scientific citations while the Melbourne Institute rates science citations at

68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social

Science Citation Index which does not include the humanities) are noted in only two

league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne

Institute (32) Another way of measuring research

11

impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a

weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the

Melbourne Institute

The complement of citations is of course publications Listing the number of publications

an individual group department or whole university releases can act as a weak substitute

for citationsmdashweak because simply publishing a paper or monograph is no guarantee that

the general public or other researchers will even glance at the work Guangdong gives an

1179 weighting to publications in science-oriented indices such as the Science Citation

Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same

indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong

even has a separate category just for measuring publications in Science and Nature

although it accords it an almost derisory weighting of 06 Under publications in social

science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their

final totals For publications in other indices (where the subject indices are

undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for

Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of

rankings to include research monographs weighted at 033

In countries where there are specific third-party evaluations of research output academic

quality of research is sometimes used as a research indicator The Times puts a very large

30 weight on this indicator while the Financial Times puts it at 119

Research awards are another handy third-party measurement of quality as the number of

international and national awards won by faculty andor graduates is often considered a

useful measure of institutional success International research awardsmdash specifically the

number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator

by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as

particularly suspect in some quarters given that the points are based on where the

recipient went to school rather than on where they are or were on the

8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the

most prominent publication indices

9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences

12

faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead

for nearly a century Wuhan uses similar measures but only accords them a weight of 14

National research awards are more common as a quality indicator used by La Repubblica

(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)

Financial indicators of research are also very common Research budgets as a factor in the

overall assessment of research in universities are covered by the Financial Times (9)

Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for

total amount of research expenditure unfortunately it is unclear precisely what this

research expenditure represents or how it is determined although it is clearly indicated

that it does not represent the total number of grants or projects at a university Total

number of research-based grants and projects is weighted by Education18 at 15 and

Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science

and engineering and another 55 to those for social sciences and humanities Similarly

the Melbourne Institute gives 6 of its overall weight to public-source grants making no

distinction between areas of study

In a slightly different vein Netbig (46) and Wuhan (278) both list the number of

research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica

(095) all weigh research-basedaffiliated research institutions or centres for studies

Finally one can also measure research not simply in terms of the amount of money it

generates but also in terms of the amount of future income it will generate Both

Guangdong (245) and Wuhan (193) measure the number of patents issued to

universities as a quality indicator

A final way of measuring an institutionrsquos research intensity is to look at the range of its

course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of

doctoral and Masterrsquos programs offered as a proxy for research intensity

As with physical and financial resources few if any of the research indicators are

normalized to account for institutional size (either by student or faculty numbers) In the

world of rankings bigger almost always means better an institution with 100 faculty with

ten citations apiece will always look worse than an institution with 1001 faculty

13

with one citation each To the extent that the raw production of knowledge matters this

form of measurement is acceptable To the extent that rankings are meant to show how

well institutions are doing on a like-to-like basis or to show the efficiency of universities it

is plainly inadequate This should be of particular concern to Chinese policy-makers

whose ranking systems are especially reliant on research-based indicators

G Indicators of Reputation

The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo

Those rankings systems which use the results of reputation surveys as an indicator do so as

an indirect measure of quality based on the assumption that the employers academics and

academic administrators surveyed have opinions of institutional quality that are informed

up-to-date and impartial While these assumptions are clearly open to debate they

nevertheless form an important basis for many ranking systems Another reason for using

reputation measures is the paucity of other data availablemdashsome countries have few

independent measures of teaching effectiveness university resources or output and

reputation can thus act as a useful surrogate Reputation rankings are often criticized as

simply quantifying the common ignorance of the people being surveyed However to the

extent that the people being surveyed hold positions which have the potential to affect

large numbers of young people and whose positions actually require some knowledge of

institutional quality (ie officials in charge of graduate admissions corporate recruiters

etc) then reputation rankings make sense because they provide useful information for

students about the perceived value of the degrees that they could obtain from various

universities

The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and

the Times which both accord reputation a weighting of 50 in their overall ranking

scheme Education18 assigns it almost as much significance at 40 The US News and World

Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just

below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)

and Netbig (15) The only other study to include reputation is Wuhan (117)

14

1 Please see Appendix B for a glossary of the various bodies referenced in this document

Appendix A Detailed Listing of Indicators and their Sources

Beginning Characteristics Indicator

Used By Source1

Incoming grades Macleans University

Percentage with grades above a set limit

Macleans University

US News and World Report

University

Performance on national standardised tests or benchmarks

Asiaweek University

Education18 3rd-party JUPAS

Financial Times Government agency 3rd-party UCAS

Guardian University Guide

Government agency 3rd-party UCAS

Melbourne Institute Government agency 3rd-party DEST

Netbig National entrance examination board

Times Good University Guide

Government agency 3rd-party UCAS

US News University

Wuhan Unknown presumed government 3rd-party

Student status La Repubblica Government agency 3rd-party MIUR

Admittance selectivity general

Asiaweek University

Admittance number of applications to places

Asiaweek University

Financial Times Government agency 3rd-party UCAS

La Repubblica Government agency 3rd-party MIUR

US News University

15

Beginning Characteristics Indicator

Used By Source1

Out-of-locality student percentage

Macleans University

International student percentages

Financial Times Government agency 3rd-party HESA

Macleans University

Shanghai Institute of Educational Science

Times World University

Wuhan Unknown presumed university

Undergraduate students among all students percentages

Netbig

Wuhan Unknown presumed university

Ethnic diversity in student body

Guardian University

Learning Inputs ndash Staff Indicator

Used By Source

Facultystudent ratio Asiaweek University

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Financial Times Government agency 3rd-party HESA

La Repubblica Government agency 3rd-party MIUR

Times Good University Guide

Government agency 3rd-party HESA

Times World University

US News University

Wuhan Unknown presumed university

16

Learning Inputs ndash Staff Indicator

Used By Source

Social science faculty student ratio

Melbourne Government agency 3rd-party DEST

Science faculty student ratio

Melbourne Government agency 3rd-party DEST

Administrative staff student ratio

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Staff student ratio (regardless of division)

Guardian Government agency 3rd-party HESA

Netbig University

Course per teacher La Repubblica Government agency 3rd-party MIUR

Per-teacher university spending

Asiaweek University

Faculty pay rates fortenured staff

Asiaweek University

US News University

Number of full-time part-time faculty

Netbig University

US News University

Wuhan Unknown presumed university

Faculty with researchprojects

Wuhan Unknown presumed university

Class size differentiation Macleans University

US News University

Classes taught by tenured faculty

Macleans University

Exchange programmes hosted

La Repubblica Government agency 3rd-party AgNaSoc

Number of classes lsquoactually taughtrsquo

La Repubblica Government agency 3rd-party MIUR

of international faculty (v faculty as a whole)

Times World University

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

5

permitting students to juggle both work and study something which was nearly

impossible in that country just a few years ago

The Washington Monthly also has a final category of indicators which reflect studentsrsquo

beginning characteristics namely their likelihood of performing community service as

measured by the percentage of students in the US Peace Corps and Reserve Officer

Training Corps (ROTC) and the percentage of students involved in work-study in the

community3 Together these three indicators account for 33 of an institutionrsquos total

ranking

B Indicators of Learning InputsmdashStaff

Generally both the quantity and quality of staff are positively correlated with institutional

quality The problem of course is finding useful metrics for each of these factors

especially if one excludes as we have done here measures of research performance and

research intensity putting them in a separate category4

The simplest measure is simply the number of faculty unadjusted for things like size of

student body Most national league tables however prefer to use variations on the

concept of facultystudent ratio Others try to measure teaching intensity with measures

such as courses per teacher or hours spent in class per student (both in La Repubblica)

These kinds of measures usually account for between 2-5 of the final rankings although

in some cases (ie the Guardian) this figure can be as high as 20

Another important way of measuring how faculty resources are deployed is the measure

of average class size which is used only by Macleanrsquos and the US News and World Report

Ostensibly the reason for measuring class size is to account in some form for the degree

3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category

6

of attention devoted to actually teaching students Despite the fact that important research

(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level

the use of this indicator appears to be a spillover from the North American debates on

class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein

2000) Regardless of why the indicators are used they are extraordinarily important to

these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and

World Reportrsquos rankings respectively

A number of ranking systems try to look at staff qualifications such as the number of

PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the

Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further

than other surveys and actually looks at the proportion of classes taught by tenure-track

staff Others (ie THES) look at the number of foreign faculty based on the assumption

that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still

others (ie La Repubblica) look at the age structure of the faculty Another proxy for

institutional quality is the pay rates for tenured staff on the assumption that institutions

with higher rates of pay on average attract better faculty this measure has been used both

by the US News and World Report and Asiaweek

Finally a number of league tables rank faculty inputs on the basis of standardized third-

party evaluations 6

Education18 the Financial Times the Times the Guardian and the US

News and World Report league tables all use some sort of ranking criterion based at least in

part on this indicator or variations thereof

C Indicators of Learning InputsmdashResources

Resource inputsmdashcrudely the amount of current dollars equipment and books available

to students at an institutionmdashare widely considered an important measure of quality Yet

despite the apparent simplicity of counting dollars and measuring assets

5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each

university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future

7

the means by which institutional wealth is measured varies considerably between ranking

systems

There are a number of revenue-based measures of resources Macleanrsquos uses public funding

of institutional budgets as a factor in its analysis conversely the Financial Times uses the

private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of

total score) and the US News and World Report (5 of total score) also measure alumni

financial support as a measure of quality

For reasons that are not entirely clear league tables tend to favour measures of

expenditures rather than revenues The Guardian looks at total institutional expenditures

as an indicator Institutional expenditure on student services is used as a measure of

institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total

institutional scores respectively) Rzezspospolita does not measure student services

expenditures directly but does measure student services outputs such as number of

student athletes and number of study clubs which amounts to more or less the same thing

Macleanrsquos also gives out 433 of its total score based on institutional expenditures on

scholarships and bursaries

Various aspects of physical infrastructure are also used as measures of institutional

resources most directly in the case of La Repubblica which bases 317 of its total rank on

the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding

assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type

of physical infrastructure measured is available Internet bandwidth which was used by

Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth

roughly 3 of the total score

By some distance the infrastructure indicators most favoured by the compilers of league

tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on

this with 12 of the total quality mark being taken from various types of library

infrastructure measurements (including acquisitions per year total volumes average

number of volumes per student and yearly library expenditure outside of acquisitions)

Netbig and Education18 also use library volume holdings while Asiaweek the Financial

Times and the Times also use measures of library expenditures outside of acquisitions or

computerization of library resources as measures of institutional quality

8

One important factor to note is that most ranking systems do not normalize their resource

and infrastructure measures That is to say it is raw spending power or simple size of

assets that is usually measured rather than spending per studentprofessor or assets per

studentprofessor As a result a number of these rankings systems have inbuilt biases

towards larger institutions

D Indicators of Learning Outputs

Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge

learned over the course of a baccalaureate degreemdashshould be a basic indicator of

institutional quality Unfortunately good means of measuring these outputsmdashlike the

National Survey of Student Engagement (NSSE) and the College Learning Assessment

(CLA)mdashhave only recently become available and for the most part institutions are still

keeping their scores secret Outside of these measures only a few very crude indicators are

available which likely explains why learning outputs do not feature especially prominently

in most ranking schemes

The simplest types of measures of learning outputs are those linked to graduation and

retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong

and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7

the latter three also use rates of graduation from Masterrsquos programs as indicators In some

cases the weights on these measures can be very highmdashin the Guangdong rankings

graduation rates account for over 50 of the rankingmdashbut in most cases the weights are

10 or less Retention rates commonly meaning the progression rate of first-year students

into second year are accorded less importance The US News and World Report Macleanrsquos

the Melbourne Institute and La Repubblica all employ retention measures as indicators but

none of them are worth more than 4 of total weighting Two publications make specific

indicators for retention and graduation of international students Macleanrsquos (graduation

rates of international students) and the Melbourne Institute (retention rates of international

students) The Washington Monthly looks specifically at institutional retention rates

adjusted for the participation of lower-income students and gives higher scores to

institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores

and number of Pell Grant recipients the US

7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18

9

News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos

ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)

E Indicators of Final Outcomes

Final outcomes are indications of generalized outcomes for students after graduation

Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness

good citizenship) but given the somewhat utilitarian justifications for education that are

currently in fashion (see Wolf 2000) employment outcomes are the most commonly used

measure of final outcomes These are given particular emphasis by the Guardian (where

employment outcomes are worth 17 of the total score) but are also used by the Financial

Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the

Times are interestingly not concerned with employment per se but with ldquoemployment in

an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher

Education Statistics Agency (HESA) uses Standard Occupational Classifications to

measure the proportion of graduates in professional or white-collar jobs anyone not in

such a job is considered not to be working in an area related to their studies (it is unclear

what methodology is used by the Financial Times and the Times although we suspect their

methods are broadly similar)

The only other measure of final outcomes in use is percentage of graduates returning for

additional education which is an indicator used by both the Melbourne Institute and the

Financial Times This is a particularly important indicator for the latter as it is worth 21

of the final ranking

The lack of indicators concerning final outcomes is interesting since most government-

sponsored performance-indicator regimes around the world are very much concerned

with such measures especially with respect to employment Possibly this indicates that

ranking systems simply do not view education outcomes as relevant measures of

educational quality Alternatively it may be the case that they simply have not found a

reliable indicator of outcomes or that there are reliable indicators but that there is so little

variation between institutions that it makes no sense to rank based on the data

10

F Indicators of Research

Many of the league tables covered in this survey include sections and weightings related

to universitiesrsquo research efforts It is in this field of measurement that we see the greatest

diversity of indicators Presumably this is because research inputs and outputs lend

themselves much more easily to measurement and manipulation than other areas of

institutional activity

Three studies include research staff as part of their ranking scheme La Repubblica at 952

the Melbourne Institute (4) and Wuhan (078)

Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used

method of looking at research quality but it is not universally admired because different

disciplines use different means to communicate major advances in knowledge (leading

scientists invariably produce large numbers of journal articles leading social scientists

may produce fewer journal articles but instead have one or two long important

monographsmdashsee Hicks 2004) There is also some concern among non-English speaking

countries that they are penalized in international rankings because so many of the major

journals (notably Science and Nature) are printed in English However the one set of

rankings that uses separate indicators to monitor articles published in English and articles

published in another language (the Wuhan rankings) shows that the two indicators are

positively correlated institutions that have more Chinese publications are also likely to

have more English publications and vice versa

Several sets of league tables measure bibliometric citations in various publication indices

The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a

weight of 20 of the final total Guangdong also monitors other Chinese universities

specifically for citations in engineering publications and weights this at 29 Moreover it

tacks on an additional 1049 for citations in science-oriented indices such as the Science

Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the

same category of scientific citations while the Melbourne Institute rates science citations at

68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social

Science Citation Index which does not include the humanities) are noted in only two

league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne

Institute (32) Another way of measuring research

11

impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a

weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the

Melbourne Institute

The complement of citations is of course publications Listing the number of publications

an individual group department or whole university releases can act as a weak substitute

for citationsmdashweak because simply publishing a paper or monograph is no guarantee that

the general public or other researchers will even glance at the work Guangdong gives an

1179 weighting to publications in science-oriented indices such as the Science Citation

Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same

indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong

even has a separate category just for measuring publications in Science and Nature

although it accords it an almost derisory weighting of 06 Under publications in social

science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their

final totals For publications in other indices (where the subject indices are

undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for

Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of

rankings to include research monographs weighted at 033

In countries where there are specific third-party evaluations of research output academic

quality of research is sometimes used as a research indicator The Times puts a very large

30 weight on this indicator while the Financial Times puts it at 119

Research awards are another handy third-party measurement of quality as the number of

international and national awards won by faculty andor graduates is often considered a

useful measure of institutional success International research awardsmdash specifically the

number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator

by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as

particularly suspect in some quarters given that the points are based on where the

recipient went to school rather than on where they are or were on the

8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the

most prominent publication indices

9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences

12

faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead

for nearly a century Wuhan uses similar measures but only accords them a weight of 14

National research awards are more common as a quality indicator used by La Repubblica

(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)

Financial indicators of research are also very common Research budgets as a factor in the

overall assessment of research in universities are covered by the Financial Times (9)

Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for

total amount of research expenditure unfortunately it is unclear precisely what this

research expenditure represents or how it is determined although it is clearly indicated

that it does not represent the total number of grants or projects at a university Total

number of research-based grants and projects is weighted by Education18 at 15 and

Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science

and engineering and another 55 to those for social sciences and humanities Similarly

the Melbourne Institute gives 6 of its overall weight to public-source grants making no

distinction between areas of study

In a slightly different vein Netbig (46) and Wuhan (278) both list the number of

research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica

(095) all weigh research-basedaffiliated research institutions or centres for studies

Finally one can also measure research not simply in terms of the amount of money it

generates but also in terms of the amount of future income it will generate Both

Guangdong (245) and Wuhan (193) measure the number of patents issued to

universities as a quality indicator

A final way of measuring an institutionrsquos research intensity is to look at the range of its

course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of

doctoral and Masterrsquos programs offered as a proxy for research intensity

As with physical and financial resources few if any of the research indicators are

normalized to account for institutional size (either by student or faculty numbers) In the

world of rankings bigger almost always means better an institution with 100 faculty with

ten citations apiece will always look worse than an institution with 1001 faculty

13

with one citation each To the extent that the raw production of knowledge matters this

form of measurement is acceptable To the extent that rankings are meant to show how

well institutions are doing on a like-to-like basis or to show the efficiency of universities it

is plainly inadequate This should be of particular concern to Chinese policy-makers

whose ranking systems are especially reliant on research-based indicators

G Indicators of Reputation

The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo

Those rankings systems which use the results of reputation surveys as an indicator do so as

an indirect measure of quality based on the assumption that the employers academics and

academic administrators surveyed have opinions of institutional quality that are informed

up-to-date and impartial While these assumptions are clearly open to debate they

nevertheless form an important basis for many ranking systems Another reason for using

reputation measures is the paucity of other data availablemdashsome countries have few

independent measures of teaching effectiveness university resources or output and

reputation can thus act as a useful surrogate Reputation rankings are often criticized as

simply quantifying the common ignorance of the people being surveyed However to the

extent that the people being surveyed hold positions which have the potential to affect

large numbers of young people and whose positions actually require some knowledge of

institutional quality (ie officials in charge of graduate admissions corporate recruiters

etc) then reputation rankings make sense because they provide useful information for

students about the perceived value of the degrees that they could obtain from various

universities

The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and

the Times which both accord reputation a weighting of 50 in their overall ranking

scheme Education18 assigns it almost as much significance at 40 The US News and World

Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just

below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)

and Netbig (15) The only other study to include reputation is Wuhan (117)

14

1 Please see Appendix B for a glossary of the various bodies referenced in this document

Appendix A Detailed Listing of Indicators and their Sources

Beginning Characteristics Indicator

Used By Source1

Incoming grades Macleans University

Percentage with grades above a set limit

Macleans University

US News and World Report

University

Performance on national standardised tests or benchmarks

Asiaweek University

Education18 3rd-party JUPAS

Financial Times Government agency 3rd-party UCAS

Guardian University Guide

Government agency 3rd-party UCAS

Melbourne Institute Government agency 3rd-party DEST

Netbig National entrance examination board

Times Good University Guide

Government agency 3rd-party UCAS

US News University

Wuhan Unknown presumed government 3rd-party

Student status La Repubblica Government agency 3rd-party MIUR

Admittance selectivity general

Asiaweek University

Admittance number of applications to places

Asiaweek University

Financial Times Government agency 3rd-party UCAS

La Repubblica Government agency 3rd-party MIUR

US News University

15

Beginning Characteristics Indicator

Used By Source1

Out-of-locality student percentage

Macleans University

International student percentages

Financial Times Government agency 3rd-party HESA

Macleans University

Shanghai Institute of Educational Science

Times World University

Wuhan Unknown presumed university

Undergraduate students among all students percentages

Netbig

Wuhan Unknown presumed university

Ethnic diversity in student body

Guardian University

Learning Inputs ndash Staff Indicator

Used By Source

Facultystudent ratio Asiaweek University

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Financial Times Government agency 3rd-party HESA

La Repubblica Government agency 3rd-party MIUR

Times Good University Guide

Government agency 3rd-party HESA

Times World University

US News University

Wuhan Unknown presumed university

16

Learning Inputs ndash Staff Indicator

Used By Source

Social science faculty student ratio

Melbourne Government agency 3rd-party DEST

Science faculty student ratio

Melbourne Government agency 3rd-party DEST

Administrative staff student ratio

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Staff student ratio (regardless of division)

Guardian Government agency 3rd-party HESA

Netbig University

Course per teacher La Repubblica Government agency 3rd-party MIUR

Per-teacher university spending

Asiaweek University

Faculty pay rates fortenured staff

Asiaweek University

US News University

Number of full-time part-time faculty

Netbig University

US News University

Wuhan Unknown presumed university

Faculty with researchprojects

Wuhan Unknown presumed university

Class size differentiation Macleans University

US News University

Classes taught by tenured faculty

Macleans University

Exchange programmes hosted

La Repubblica Government agency 3rd-party AgNaSoc

Number of classes lsquoactually taughtrsquo

La Repubblica Government agency 3rd-party MIUR

of international faculty (v faculty as a whole)

Times World University

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

6

of attention devoted to actually teaching students Despite the fact that important research

(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level

the use of this indicator appears to be a spillover from the North American debates on

class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein

2000) Regardless of why the indicators are used they are extraordinarily important to

these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and

World Reportrsquos rankings respectively

A number of ranking systems try to look at staff qualifications such as the number of

PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the

Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further

than other surveys and actually looks at the proportion of classes taught by tenure-track

staff Others (ie THES) look at the number of foreign faculty based on the assumption

that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still

others (ie La Repubblica) look at the age structure of the faculty Another proxy for

institutional quality is the pay rates for tenured staff on the assumption that institutions

with higher rates of pay on average attract better faculty this measure has been used both

by the US News and World Report and Asiaweek

Finally a number of league tables rank faculty inputs on the basis of standardized third-

party evaluations 6

Education18 the Financial Times the Times the Guardian and the US

News and World Report league tables all use some sort of ranking criterion based at least in

part on this indicator or variations thereof

C Indicators of Learning InputsmdashResources

Resource inputsmdashcrudely the amount of current dollars equipment and books available

to students at an institutionmdashare widely considered an important measure of quality Yet

despite the apparent simplicity of counting dollars and measuring assets

5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each

university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future

7

the means by which institutional wealth is measured varies considerably between ranking

systems

There are a number of revenue-based measures of resources Macleanrsquos uses public funding

of institutional budgets as a factor in its analysis conversely the Financial Times uses the

private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of

total score) and the US News and World Report (5 of total score) also measure alumni

financial support as a measure of quality

For reasons that are not entirely clear league tables tend to favour measures of

expenditures rather than revenues The Guardian looks at total institutional expenditures

as an indicator Institutional expenditure on student services is used as a measure of

institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total

institutional scores respectively) Rzezspospolita does not measure student services

expenditures directly but does measure student services outputs such as number of

student athletes and number of study clubs which amounts to more or less the same thing

Macleanrsquos also gives out 433 of its total score based on institutional expenditures on

scholarships and bursaries

Various aspects of physical infrastructure are also used as measures of institutional

resources most directly in the case of La Repubblica which bases 317 of its total rank on

the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding

assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type

of physical infrastructure measured is available Internet bandwidth which was used by

Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth

roughly 3 of the total score

By some distance the infrastructure indicators most favoured by the compilers of league

tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on

this with 12 of the total quality mark being taken from various types of library

infrastructure measurements (including acquisitions per year total volumes average

number of volumes per student and yearly library expenditure outside of acquisitions)

Netbig and Education18 also use library volume holdings while Asiaweek the Financial

Times and the Times also use measures of library expenditures outside of acquisitions or

computerization of library resources as measures of institutional quality

8

One important factor to note is that most ranking systems do not normalize their resource

and infrastructure measures That is to say it is raw spending power or simple size of

assets that is usually measured rather than spending per studentprofessor or assets per

studentprofessor As a result a number of these rankings systems have inbuilt biases

towards larger institutions

D Indicators of Learning Outputs

Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge

learned over the course of a baccalaureate degreemdashshould be a basic indicator of

institutional quality Unfortunately good means of measuring these outputsmdashlike the

National Survey of Student Engagement (NSSE) and the College Learning Assessment

(CLA)mdashhave only recently become available and for the most part institutions are still

keeping their scores secret Outside of these measures only a few very crude indicators are

available which likely explains why learning outputs do not feature especially prominently

in most ranking schemes

The simplest types of measures of learning outputs are those linked to graduation and

retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong

and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7

the latter three also use rates of graduation from Masterrsquos programs as indicators In some

cases the weights on these measures can be very highmdashin the Guangdong rankings

graduation rates account for over 50 of the rankingmdashbut in most cases the weights are

10 or less Retention rates commonly meaning the progression rate of first-year students

into second year are accorded less importance The US News and World Report Macleanrsquos

the Melbourne Institute and La Repubblica all employ retention measures as indicators but

none of them are worth more than 4 of total weighting Two publications make specific

indicators for retention and graduation of international students Macleanrsquos (graduation

rates of international students) and the Melbourne Institute (retention rates of international

students) The Washington Monthly looks specifically at institutional retention rates

adjusted for the participation of lower-income students and gives higher scores to

institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores

and number of Pell Grant recipients the US

7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18

9

News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos

ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)

E Indicators of Final Outcomes

Final outcomes are indications of generalized outcomes for students after graduation

Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness

good citizenship) but given the somewhat utilitarian justifications for education that are

currently in fashion (see Wolf 2000) employment outcomes are the most commonly used

measure of final outcomes These are given particular emphasis by the Guardian (where

employment outcomes are worth 17 of the total score) but are also used by the Financial

Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the

Times are interestingly not concerned with employment per se but with ldquoemployment in

an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher

Education Statistics Agency (HESA) uses Standard Occupational Classifications to

measure the proportion of graduates in professional or white-collar jobs anyone not in

such a job is considered not to be working in an area related to their studies (it is unclear

what methodology is used by the Financial Times and the Times although we suspect their

methods are broadly similar)

The only other measure of final outcomes in use is percentage of graduates returning for

additional education which is an indicator used by both the Melbourne Institute and the

Financial Times This is a particularly important indicator for the latter as it is worth 21

of the final ranking

The lack of indicators concerning final outcomes is interesting since most government-

sponsored performance-indicator regimes around the world are very much concerned

with such measures especially with respect to employment Possibly this indicates that

ranking systems simply do not view education outcomes as relevant measures of

educational quality Alternatively it may be the case that they simply have not found a

reliable indicator of outcomes or that there are reliable indicators but that there is so little

variation between institutions that it makes no sense to rank based on the data

10

F Indicators of Research

Many of the league tables covered in this survey include sections and weightings related

to universitiesrsquo research efforts It is in this field of measurement that we see the greatest

diversity of indicators Presumably this is because research inputs and outputs lend

themselves much more easily to measurement and manipulation than other areas of

institutional activity

Three studies include research staff as part of their ranking scheme La Repubblica at 952

the Melbourne Institute (4) and Wuhan (078)

Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used

method of looking at research quality but it is not universally admired because different

disciplines use different means to communicate major advances in knowledge (leading

scientists invariably produce large numbers of journal articles leading social scientists

may produce fewer journal articles but instead have one or two long important

monographsmdashsee Hicks 2004) There is also some concern among non-English speaking

countries that they are penalized in international rankings because so many of the major

journals (notably Science and Nature) are printed in English However the one set of

rankings that uses separate indicators to monitor articles published in English and articles

published in another language (the Wuhan rankings) shows that the two indicators are

positively correlated institutions that have more Chinese publications are also likely to

have more English publications and vice versa

Several sets of league tables measure bibliometric citations in various publication indices

The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a

weight of 20 of the final total Guangdong also monitors other Chinese universities

specifically for citations in engineering publications and weights this at 29 Moreover it

tacks on an additional 1049 for citations in science-oriented indices such as the Science

Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the

same category of scientific citations while the Melbourne Institute rates science citations at

68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social

Science Citation Index which does not include the humanities) are noted in only two

league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne

Institute (32) Another way of measuring research

11

impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a

weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the

Melbourne Institute

The complement of citations is of course publications Listing the number of publications

an individual group department or whole university releases can act as a weak substitute

for citationsmdashweak because simply publishing a paper or monograph is no guarantee that

the general public or other researchers will even glance at the work Guangdong gives an

1179 weighting to publications in science-oriented indices such as the Science Citation

Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same

indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong

even has a separate category just for measuring publications in Science and Nature

although it accords it an almost derisory weighting of 06 Under publications in social

science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their

final totals For publications in other indices (where the subject indices are

undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for

Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of

rankings to include research monographs weighted at 033

In countries where there are specific third-party evaluations of research output academic

quality of research is sometimes used as a research indicator The Times puts a very large

30 weight on this indicator while the Financial Times puts it at 119

Research awards are another handy third-party measurement of quality as the number of

international and national awards won by faculty andor graduates is often considered a

useful measure of institutional success International research awardsmdash specifically the

number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator

by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as

particularly suspect in some quarters given that the points are based on where the

recipient went to school rather than on where they are or were on the

8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the

most prominent publication indices

9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences

12

faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead

for nearly a century Wuhan uses similar measures but only accords them a weight of 14

National research awards are more common as a quality indicator used by La Repubblica

(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)

Financial indicators of research are also very common Research budgets as a factor in the

overall assessment of research in universities are covered by the Financial Times (9)

Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for

total amount of research expenditure unfortunately it is unclear precisely what this

research expenditure represents or how it is determined although it is clearly indicated

that it does not represent the total number of grants or projects at a university Total

number of research-based grants and projects is weighted by Education18 at 15 and

Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science

and engineering and another 55 to those for social sciences and humanities Similarly

the Melbourne Institute gives 6 of its overall weight to public-source grants making no

distinction between areas of study

In a slightly different vein Netbig (46) and Wuhan (278) both list the number of

research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica

(095) all weigh research-basedaffiliated research institutions or centres for studies

Finally one can also measure research not simply in terms of the amount of money it

generates but also in terms of the amount of future income it will generate Both

Guangdong (245) and Wuhan (193) measure the number of patents issued to

universities as a quality indicator

A final way of measuring an institutionrsquos research intensity is to look at the range of its

course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of

doctoral and Masterrsquos programs offered as a proxy for research intensity

As with physical and financial resources few if any of the research indicators are

normalized to account for institutional size (either by student or faculty numbers) In the

world of rankings bigger almost always means better an institution with 100 faculty with

ten citations apiece will always look worse than an institution with 1001 faculty

13

with one citation each To the extent that the raw production of knowledge matters this

form of measurement is acceptable To the extent that rankings are meant to show how

well institutions are doing on a like-to-like basis or to show the efficiency of universities it

is plainly inadequate This should be of particular concern to Chinese policy-makers

whose ranking systems are especially reliant on research-based indicators

G Indicators of Reputation

The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo

Those rankings systems which use the results of reputation surveys as an indicator do so as

an indirect measure of quality based on the assumption that the employers academics and

academic administrators surveyed have opinions of institutional quality that are informed

up-to-date and impartial While these assumptions are clearly open to debate they

nevertheless form an important basis for many ranking systems Another reason for using

reputation measures is the paucity of other data availablemdashsome countries have few

independent measures of teaching effectiveness university resources or output and

reputation can thus act as a useful surrogate Reputation rankings are often criticized as

simply quantifying the common ignorance of the people being surveyed However to the

extent that the people being surveyed hold positions which have the potential to affect

large numbers of young people and whose positions actually require some knowledge of

institutional quality (ie officials in charge of graduate admissions corporate recruiters

etc) then reputation rankings make sense because they provide useful information for

students about the perceived value of the degrees that they could obtain from various

universities

The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and

the Times which both accord reputation a weighting of 50 in their overall ranking

scheme Education18 assigns it almost as much significance at 40 The US News and World

Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just

below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)

and Netbig (15) The only other study to include reputation is Wuhan (117)

14

1 Please see Appendix B for a glossary of the various bodies referenced in this document

Appendix A Detailed Listing of Indicators and their Sources

Beginning Characteristics Indicator

Used By Source1

Incoming grades Macleans University

Percentage with grades above a set limit

Macleans University

US News and World Report

University

Performance on national standardised tests or benchmarks

Asiaweek University

Education18 3rd-party JUPAS

Financial Times Government agency 3rd-party UCAS

Guardian University Guide

Government agency 3rd-party UCAS

Melbourne Institute Government agency 3rd-party DEST

Netbig National entrance examination board

Times Good University Guide

Government agency 3rd-party UCAS

US News University

Wuhan Unknown presumed government 3rd-party

Student status La Repubblica Government agency 3rd-party MIUR

Admittance selectivity general

Asiaweek University

Admittance number of applications to places

Asiaweek University

Financial Times Government agency 3rd-party UCAS

La Repubblica Government agency 3rd-party MIUR

US News University

15

Beginning Characteristics Indicator

Used By Source1

Out-of-locality student percentage

Macleans University

International student percentages

Financial Times Government agency 3rd-party HESA

Macleans University

Shanghai Institute of Educational Science

Times World University

Wuhan Unknown presumed university

Undergraduate students among all students percentages

Netbig

Wuhan Unknown presumed university

Ethnic diversity in student body

Guardian University

Learning Inputs ndash Staff Indicator

Used By Source

Facultystudent ratio Asiaweek University

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Financial Times Government agency 3rd-party HESA

La Repubblica Government agency 3rd-party MIUR

Times Good University Guide

Government agency 3rd-party HESA

Times World University

US News University

Wuhan Unknown presumed university

16

Learning Inputs ndash Staff Indicator

Used By Source

Social science faculty student ratio

Melbourne Government agency 3rd-party DEST

Science faculty student ratio

Melbourne Government agency 3rd-party DEST

Administrative staff student ratio

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Staff student ratio (regardless of division)

Guardian Government agency 3rd-party HESA

Netbig University

Course per teacher La Repubblica Government agency 3rd-party MIUR

Per-teacher university spending

Asiaweek University

Faculty pay rates fortenured staff

Asiaweek University

US News University

Number of full-time part-time faculty

Netbig University

US News University

Wuhan Unknown presumed university

Faculty with researchprojects

Wuhan Unknown presumed university

Class size differentiation Macleans University

US News University

Classes taught by tenured faculty

Macleans University

Exchange programmes hosted

La Repubblica Government agency 3rd-party AgNaSoc

Number of classes lsquoactually taughtrsquo

La Repubblica Government agency 3rd-party MIUR

of international faculty (v faculty as a whole)

Times World University

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

7

the means by which institutional wealth is measured varies considerably between ranking

systems

There are a number of revenue-based measures of resources Macleanrsquos uses public funding

of institutional budgets as a factor in its analysis conversely the Financial Times uses the

private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of

total score) and the US News and World Report (5 of total score) also measure alumni

financial support as a measure of quality

For reasons that are not entirely clear league tables tend to favour measures of

expenditures rather than revenues The Guardian looks at total institutional expenditures

as an indicator Institutional expenditure on student services is used as a measure of

institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total

institutional scores respectively) Rzezspospolita does not measure student services

expenditures directly but does measure student services outputs such as number of

student athletes and number of study clubs which amounts to more or less the same thing

Macleanrsquos also gives out 433 of its total score based on institutional expenditures on

scholarships and bursaries

Various aspects of physical infrastructure are also used as measures of institutional

resources most directly in the case of La Repubblica which bases 317 of its total rank on

the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding

assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type

of physical infrastructure measured is available Internet bandwidth which was used by

Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth

roughly 3 of the total score

By some distance the infrastructure indicators most favoured by the compilers of league

tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on

this with 12 of the total quality mark being taken from various types of library

infrastructure measurements (including acquisitions per year total volumes average

number of volumes per student and yearly library expenditure outside of acquisitions)

Netbig and Education18 also use library volume holdings while Asiaweek the Financial

Times and the Times also use measures of library expenditures outside of acquisitions or

computerization of library resources as measures of institutional quality

8

One important factor to note is that most ranking systems do not normalize their resource

and infrastructure measures That is to say it is raw spending power or simple size of

assets that is usually measured rather than spending per studentprofessor or assets per

studentprofessor As a result a number of these rankings systems have inbuilt biases

towards larger institutions

D Indicators of Learning Outputs

Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge

learned over the course of a baccalaureate degreemdashshould be a basic indicator of

institutional quality Unfortunately good means of measuring these outputsmdashlike the

National Survey of Student Engagement (NSSE) and the College Learning Assessment

(CLA)mdashhave only recently become available and for the most part institutions are still

keeping their scores secret Outside of these measures only a few very crude indicators are

available which likely explains why learning outputs do not feature especially prominently

in most ranking schemes

The simplest types of measures of learning outputs are those linked to graduation and

retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong

and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7

the latter three also use rates of graduation from Masterrsquos programs as indicators In some

cases the weights on these measures can be very highmdashin the Guangdong rankings

graduation rates account for over 50 of the rankingmdashbut in most cases the weights are

10 or less Retention rates commonly meaning the progression rate of first-year students

into second year are accorded less importance The US News and World Report Macleanrsquos

the Melbourne Institute and La Repubblica all employ retention measures as indicators but

none of them are worth more than 4 of total weighting Two publications make specific

indicators for retention and graduation of international students Macleanrsquos (graduation

rates of international students) and the Melbourne Institute (retention rates of international

students) The Washington Monthly looks specifically at institutional retention rates

adjusted for the participation of lower-income students and gives higher scores to

institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores

and number of Pell Grant recipients the US

7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18

9

News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos

ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)

E Indicators of Final Outcomes

Final outcomes are indications of generalized outcomes for students after graduation

Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness

good citizenship) but given the somewhat utilitarian justifications for education that are

currently in fashion (see Wolf 2000) employment outcomes are the most commonly used

measure of final outcomes These are given particular emphasis by the Guardian (where

employment outcomes are worth 17 of the total score) but are also used by the Financial

Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the

Times are interestingly not concerned with employment per se but with ldquoemployment in

an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher

Education Statistics Agency (HESA) uses Standard Occupational Classifications to

measure the proportion of graduates in professional or white-collar jobs anyone not in

such a job is considered not to be working in an area related to their studies (it is unclear

what methodology is used by the Financial Times and the Times although we suspect their

methods are broadly similar)

The only other measure of final outcomes in use is percentage of graduates returning for

additional education which is an indicator used by both the Melbourne Institute and the

Financial Times This is a particularly important indicator for the latter as it is worth 21

of the final ranking

The lack of indicators concerning final outcomes is interesting since most government-

sponsored performance-indicator regimes around the world are very much concerned

with such measures especially with respect to employment Possibly this indicates that

ranking systems simply do not view education outcomes as relevant measures of

educational quality Alternatively it may be the case that they simply have not found a

reliable indicator of outcomes or that there are reliable indicators but that there is so little

variation between institutions that it makes no sense to rank based on the data

10

F Indicators of Research

Many of the league tables covered in this survey include sections and weightings related

to universitiesrsquo research efforts It is in this field of measurement that we see the greatest

diversity of indicators Presumably this is because research inputs and outputs lend

themselves much more easily to measurement and manipulation than other areas of

institutional activity

Three studies include research staff as part of their ranking scheme La Repubblica at 952

the Melbourne Institute (4) and Wuhan (078)

Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used

method of looking at research quality but it is not universally admired because different

disciplines use different means to communicate major advances in knowledge (leading

scientists invariably produce large numbers of journal articles leading social scientists

may produce fewer journal articles but instead have one or two long important

monographsmdashsee Hicks 2004) There is also some concern among non-English speaking

countries that they are penalized in international rankings because so many of the major

journals (notably Science and Nature) are printed in English However the one set of

rankings that uses separate indicators to monitor articles published in English and articles

published in another language (the Wuhan rankings) shows that the two indicators are

positively correlated institutions that have more Chinese publications are also likely to

have more English publications and vice versa

Several sets of league tables measure bibliometric citations in various publication indices

The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a

weight of 20 of the final total Guangdong also monitors other Chinese universities

specifically for citations in engineering publications and weights this at 29 Moreover it

tacks on an additional 1049 for citations in science-oriented indices such as the Science

Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the

same category of scientific citations while the Melbourne Institute rates science citations at

68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social

Science Citation Index which does not include the humanities) are noted in only two

league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne

Institute (32) Another way of measuring research

11

impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a

weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the

Melbourne Institute

The complement of citations is of course publications Listing the number of publications

an individual group department or whole university releases can act as a weak substitute

for citationsmdashweak because simply publishing a paper or monograph is no guarantee that

the general public or other researchers will even glance at the work Guangdong gives an

1179 weighting to publications in science-oriented indices such as the Science Citation

Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same

indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong

even has a separate category just for measuring publications in Science and Nature

although it accords it an almost derisory weighting of 06 Under publications in social

science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their

final totals For publications in other indices (where the subject indices are

undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for

Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of

rankings to include research monographs weighted at 033

In countries where there are specific third-party evaluations of research output academic

quality of research is sometimes used as a research indicator The Times puts a very large

30 weight on this indicator while the Financial Times puts it at 119

Research awards are another handy third-party measurement of quality as the number of

international and national awards won by faculty andor graduates is often considered a

useful measure of institutional success International research awardsmdash specifically the

number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator

by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as

particularly suspect in some quarters given that the points are based on where the

recipient went to school rather than on where they are or were on the

8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the

most prominent publication indices

9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences

12

faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead

for nearly a century Wuhan uses similar measures but only accords them a weight of 14

National research awards are more common as a quality indicator used by La Repubblica

(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)

Financial indicators of research are also very common Research budgets as a factor in the

overall assessment of research in universities are covered by the Financial Times (9)

Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for

total amount of research expenditure unfortunately it is unclear precisely what this

research expenditure represents or how it is determined although it is clearly indicated

that it does not represent the total number of grants or projects at a university Total

number of research-based grants and projects is weighted by Education18 at 15 and

Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science

and engineering and another 55 to those for social sciences and humanities Similarly

the Melbourne Institute gives 6 of its overall weight to public-source grants making no

distinction between areas of study

In a slightly different vein Netbig (46) and Wuhan (278) both list the number of

research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica

(095) all weigh research-basedaffiliated research institutions or centres for studies

Finally one can also measure research not simply in terms of the amount of money it

generates but also in terms of the amount of future income it will generate Both

Guangdong (245) and Wuhan (193) measure the number of patents issued to

universities as a quality indicator

A final way of measuring an institutionrsquos research intensity is to look at the range of its

course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of

doctoral and Masterrsquos programs offered as a proxy for research intensity

As with physical and financial resources few if any of the research indicators are

normalized to account for institutional size (either by student or faculty numbers) In the

world of rankings bigger almost always means better an institution with 100 faculty with

ten citations apiece will always look worse than an institution with 1001 faculty

13

with one citation each To the extent that the raw production of knowledge matters this

form of measurement is acceptable To the extent that rankings are meant to show how

well institutions are doing on a like-to-like basis or to show the efficiency of universities it

is plainly inadequate This should be of particular concern to Chinese policy-makers

whose ranking systems are especially reliant on research-based indicators

G Indicators of Reputation

The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo

Those rankings systems which use the results of reputation surveys as an indicator do so as

an indirect measure of quality based on the assumption that the employers academics and

academic administrators surveyed have opinions of institutional quality that are informed

up-to-date and impartial While these assumptions are clearly open to debate they

nevertheless form an important basis for many ranking systems Another reason for using

reputation measures is the paucity of other data availablemdashsome countries have few

independent measures of teaching effectiveness university resources or output and

reputation can thus act as a useful surrogate Reputation rankings are often criticized as

simply quantifying the common ignorance of the people being surveyed However to the

extent that the people being surveyed hold positions which have the potential to affect

large numbers of young people and whose positions actually require some knowledge of

institutional quality (ie officials in charge of graduate admissions corporate recruiters

etc) then reputation rankings make sense because they provide useful information for

students about the perceived value of the degrees that they could obtain from various

universities

The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and

the Times which both accord reputation a weighting of 50 in their overall ranking

scheme Education18 assigns it almost as much significance at 40 The US News and World

Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just

below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)

and Netbig (15) The only other study to include reputation is Wuhan (117)

14

1 Please see Appendix B for a glossary of the various bodies referenced in this document

Appendix A Detailed Listing of Indicators and their Sources

Beginning Characteristics Indicator

Used By Source1

Incoming grades Macleans University

Percentage with grades above a set limit

Macleans University

US News and World Report

University

Performance on national standardised tests or benchmarks

Asiaweek University

Education18 3rd-party JUPAS

Financial Times Government agency 3rd-party UCAS

Guardian University Guide

Government agency 3rd-party UCAS

Melbourne Institute Government agency 3rd-party DEST

Netbig National entrance examination board

Times Good University Guide

Government agency 3rd-party UCAS

US News University

Wuhan Unknown presumed government 3rd-party

Student status La Repubblica Government agency 3rd-party MIUR

Admittance selectivity general

Asiaweek University

Admittance number of applications to places

Asiaweek University

Financial Times Government agency 3rd-party UCAS

La Repubblica Government agency 3rd-party MIUR

US News University

15

Beginning Characteristics Indicator

Used By Source1

Out-of-locality student percentage

Macleans University

International student percentages

Financial Times Government agency 3rd-party HESA

Macleans University

Shanghai Institute of Educational Science

Times World University

Wuhan Unknown presumed university

Undergraduate students among all students percentages

Netbig

Wuhan Unknown presumed university

Ethnic diversity in student body

Guardian University

Learning Inputs ndash Staff Indicator

Used By Source

Facultystudent ratio Asiaweek University

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Financial Times Government agency 3rd-party HESA

La Repubblica Government agency 3rd-party MIUR

Times Good University Guide

Government agency 3rd-party HESA

Times World University

US News University

Wuhan Unknown presumed university

16

Learning Inputs ndash Staff Indicator

Used By Source

Social science faculty student ratio

Melbourne Government agency 3rd-party DEST

Science faculty student ratio

Melbourne Government agency 3rd-party DEST

Administrative staff student ratio

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Staff student ratio (regardless of division)

Guardian Government agency 3rd-party HESA

Netbig University

Course per teacher La Repubblica Government agency 3rd-party MIUR

Per-teacher university spending

Asiaweek University

Faculty pay rates fortenured staff

Asiaweek University

US News University

Number of full-time part-time faculty

Netbig University

US News University

Wuhan Unknown presumed university

Faculty with researchprojects

Wuhan Unknown presumed university

Class size differentiation Macleans University

US News University

Classes taught by tenured faculty

Macleans University

Exchange programmes hosted

La Repubblica Government agency 3rd-party AgNaSoc

Number of classes lsquoactually taughtrsquo

La Repubblica Government agency 3rd-party MIUR

of international faculty (v faculty as a whole)

Times World University

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

8

One important factor to note is that most ranking systems do not normalize their resource

and infrastructure measures That is to say it is raw spending power or simple size of

assets that is usually measured rather than spending per studentprofessor or assets per

studentprofessor As a result a number of these rankings systems have inbuilt biases

towards larger institutions

D Indicators of Learning Outputs

Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge

learned over the course of a baccalaureate degreemdashshould be a basic indicator of

institutional quality Unfortunately good means of measuring these outputsmdashlike the

National Survey of Student Engagement (NSSE) and the College Learning Assessment

(CLA)mdashhave only recently become available and for the most part institutions are still

keeping their scores secret Outside of these measures only a few very crude indicators are

available which likely explains why learning outputs do not feature especially prominently

in most ranking schemes

The simplest types of measures of learning outputs are those linked to graduation and

retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong

and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7

the latter three also use rates of graduation from Masterrsquos programs as indicators In some

cases the weights on these measures can be very highmdashin the Guangdong rankings

graduation rates account for over 50 of the rankingmdashbut in most cases the weights are

10 or less Retention rates commonly meaning the progression rate of first-year students

into second year are accorded less importance The US News and World Report Macleanrsquos

the Melbourne Institute and La Repubblica all employ retention measures as indicators but

none of them are worth more than 4 of total weighting Two publications make specific

indicators for retention and graduation of international students Macleanrsquos (graduation

rates of international students) and the Melbourne Institute (retention rates of international

students) The Washington Monthly looks specifically at institutional retention rates

adjusted for the participation of lower-income students and gives higher scores to

institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores

and number of Pell Grant recipients the US

7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18

9

News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos

ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)

E Indicators of Final Outcomes

Final outcomes are indications of generalized outcomes for students after graduation

Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness

good citizenship) but given the somewhat utilitarian justifications for education that are

currently in fashion (see Wolf 2000) employment outcomes are the most commonly used

measure of final outcomes These are given particular emphasis by the Guardian (where

employment outcomes are worth 17 of the total score) but are also used by the Financial

Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the

Times are interestingly not concerned with employment per se but with ldquoemployment in

an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher

Education Statistics Agency (HESA) uses Standard Occupational Classifications to

measure the proportion of graduates in professional or white-collar jobs anyone not in

such a job is considered not to be working in an area related to their studies (it is unclear

what methodology is used by the Financial Times and the Times although we suspect their

methods are broadly similar)

The only other measure of final outcomes in use is percentage of graduates returning for

additional education which is an indicator used by both the Melbourne Institute and the

Financial Times This is a particularly important indicator for the latter as it is worth 21

of the final ranking

The lack of indicators concerning final outcomes is interesting since most government-

sponsored performance-indicator regimes around the world are very much concerned

with such measures especially with respect to employment Possibly this indicates that

ranking systems simply do not view education outcomes as relevant measures of

educational quality Alternatively it may be the case that they simply have not found a

reliable indicator of outcomes or that there are reliable indicators but that there is so little

variation between institutions that it makes no sense to rank based on the data

10

F Indicators of Research

Many of the league tables covered in this survey include sections and weightings related

to universitiesrsquo research efforts It is in this field of measurement that we see the greatest

diversity of indicators Presumably this is because research inputs and outputs lend

themselves much more easily to measurement and manipulation than other areas of

institutional activity

Three studies include research staff as part of their ranking scheme La Repubblica at 952

the Melbourne Institute (4) and Wuhan (078)

Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used

method of looking at research quality but it is not universally admired because different

disciplines use different means to communicate major advances in knowledge (leading

scientists invariably produce large numbers of journal articles leading social scientists

may produce fewer journal articles but instead have one or two long important

monographsmdashsee Hicks 2004) There is also some concern among non-English speaking

countries that they are penalized in international rankings because so many of the major

journals (notably Science and Nature) are printed in English However the one set of

rankings that uses separate indicators to monitor articles published in English and articles

published in another language (the Wuhan rankings) shows that the two indicators are

positively correlated institutions that have more Chinese publications are also likely to

have more English publications and vice versa

Several sets of league tables measure bibliometric citations in various publication indices

The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a

weight of 20 of the final total Guangdong also monitors other Chinese universities

specifically for citations in engineering publications and weights this at 29 Moreover it

tacks on an additional 1049 for citations in science-oriented indices such as the Science

Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the

same category of scientific citations while the Melbourne Institute rates science citations at

68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social

Science Citation Index which does not include the humanities) are noted in only two

league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne

Institute (32) Another way of measuring research

11

impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a

weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the

Melbourne Institute

The complement of citations is of course publications Listing the number of publications

an individual group department or whole university releases can act as a weak substitute

for citationsmdashweak because simply publishing a paper or monograph is no guarantee that

the general public or other researchers will even glance at the work Guangdong gives an

1179 weighting to publications in science-oriented indices such as the Science Citation

Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same

indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong

even has a separate category just for measuring publications in Science and Nature

although it accords it an almost derisory weighting of 06 Under publications in social

science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their

final totals For publications in other indices (where the subject indices are

undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for

Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of

rankings to include research monographs weighted at 033

In countries where there are specific third-party evaluations of research output academic

quality of research is sometimes used as a research indicator The Times puts a very large

30 weight on this indicator while the Financial Times puts it at 119

Research awards are another handy third-party measurement of quality as the number of

international and national awards won by faculty andor graduates is often considered a

useful measure of institutional success International research awardsmdash specifically the

number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator

by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as

particularly suspect in some quarters given that the points are based on where the

recipient went to school rather than on where they are or were on the

8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the

most prominent publication indices

9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences

12

faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead

for nearly a century Wuhan uses similar measures but only accords them a weight of 14

National research awards are more common as a quality indicator used by La Repubblica

(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)

Financial indicators of research are also very common Research budgets as a factor in the

overall assessment of research in universities are covered by the Financial Times (9)

Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for

total amount of research expenditure unfortunately it is unclear precisely what this

research expenditure represents or how it is determined although it is clearly indicated

that it does not represent the total number of grants or projects at a university Total

number of research-based grants and projects is weighted by Education18 at 15 and

Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science

and engineering and another 55 to those for social sciences and humanities Similarly

the Melbourne Institute gives 6 of its overall weight to public-source grants making no

distinction between areas of study

In a slightly different vein Netbig (46) and Wuhan (278) both list the number of

research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica

(095) all weigh research-basedaffiliated research institutions or centres for studies

Finally one can also measure research not simply in terms of the amount of money it

generates but also in terms of the amount of future income it will generate Both

Guangdong (245) and Wuhan (193) measure the number of patents issued to

universities as a quality indicator

A final way of measuring an institutionrsquos research intensity is to look at the range of its

course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of

doctoral and Masterrsquos programs offered as a proxy for research intensity

As with physical and financial resources few if any of the research indicators are

normalized to account for institutional size (either by student or faculty numbers) In the

world of rankings bigger almost always means better an institution with 100 faculty with

ten citations apiece will always look worse than an institution with 1001 faculty

13

with one citation each To the extent that the raw production of knowledge matters this

form of measurement is acceptable To the extent that rankings are meant to show how

well institutions are doing on a like-to-like basis or to show the efficiency of universities it

is plainly inadequate This should be of particular concern to Chinese policy-makers

whose ranking systems are especially reliant on research-based indicators

G Indicators of Reputation

The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo

Those rankings systems which use the results of reputation surveys as an indicator do so as

an indirect measure of quality based on the assumption that the employers academics and

academic administrators surveyed have opinions of institutional quality that are informed

up-to-date and impartial While these assumptions are clearly open to debate they

nevertheless form an important basis for many ranking systems Another reason for using

reputation measures is the paucity of other data availablemdashsome countries have few

independent measures of teaching effectiveness university resources or output and

reputation can thus act as a useful surrogate Reputation rankings are often criticized as

simply quantifying the common ignorance of the people being surveyed However to the

extent that the people being surveyed hold positions which have the potential to affect

large numbers of young people and whose positions actually require some knowledge of

institutional quality (ie officials in charge of graduate admissions corporate recruiters

etc) then reputation rankings make sense because they provide useful information for

students about the perceived value of the degrees that they could obtain from various

universities

The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and

the Times which both accord reputation a weighting of 50 in their overall ranking

scheme Education18 assigns it almost as much significance at 40 The US News and World

Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just

below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)

and Netbig (15) The only other study to include reputation is Wuhan (117)

14

1 Please see Appendix B for a glossary of the various bodies referenced in this document

Appendix A Detailed Listing of Indicators and their Sources

Beginning Characteristics Indicator

Used By Source1

Incoming grades Macleans University

Percentage with grades above a set limit

Macleans University

US News and World Report

University

Performance on national standardised tests or benchmarks

Asiaweek University

Education18 3rd-party JUPAS

Financial Times Government agency 3rd-party UCAS

Guardian University Guide

Government agency 3rd-party UCAS

Melbourne Institute Government agency 3rd-party DEST

Netbig National entrance examination board

Times Good University Guide

Government agency 3rd-party UCAS

US News University

Wuhan Unknown presumed government 3rd-party

Student status La Repubblica Government agency 3rd-party MIUR

Admittance selectivity general

Asiaweek University

Admittance number of applications to places

Asiaweek University

Financial Times Government agency 3rd-party UCAS

La Repubblica Government agency 3rd-party MIUR

US News University

15

Beginning Characteristics Indicator

Used By Source1

Out-of-locality student percentage

Macleans University

International student percentages

Financial Times Government agency 3rd-party HESA

Macleans University

Shanghai Institute of Educational Science

Times World University

Wuhan Unknown presumed university

Undergraduate students among all students percentages

Netbig

Wuhan Unknown presumed university

Ethnic diversity in student body

Guardian University

Learning Inputs ndash Staff Indicator

Used By Source

Facultystudent ratio Asiaweek University

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Financial Times Government agency 3rd-party HESA

La Repubblica Government agency 3rd-party MIUR

Times Good University Guide

Government agency 3rd-party HESA

Times World University

US News University

Wuhan Unknown presumed university

16

Learning Inputs ndash Staff Indicator

Used By Source

Social science faculty student ratio

Melbourne Government agency 3rd-party DEST

Science faculty student ratio

Melbourne Government agency 3rd-party DEST

Administrative staff student ratio

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Staff student ratio (regardless of division)

Guardian Government agency 3rd-party HESA

Netbig University

Course per teacher La Repubblica Government agency 3rd-party MIUR

Per-teacher university spending

Asiaweek University

Faculty pay rates fortenured staff

Asiaweek University

US News University

Number of full-time part-time faculty

Netbig University

US News University

Wuhan Unknown presumed university

Faculty with researchprojects

Wuhan Unknown presumed university

Class size differentiation Macleans University

US News University

Classes taught by tenured faculty

Macleans University

Exchange programmes hosted

La Repubblica Government agency 3rd-party AgNaSoc

Number of classes lsquoactually taughtrsquo

La Repubblica Government agency 3rd-party MIUR

of international faculty (v faculty as a whole)

Times World University

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

9

News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos

ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)

E Indicators of Final Outcomes

Final outcomes are indications of generalized outcomes for students after graduation

Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness

good citizenship) but given the somewhat utilitarian justifications for education that are

currently in fashion (see Wolf 2000) employment outcomes are the most commonly used

measure of final outcomes These are given particular emphasis by the Guardian (where

employment outcomes are worth 17 of the total score) but are also used by the Financial

Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the

Times are interestingly not concerned with employment per se but with ldquoemployment in

an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher

Education Statistics Agency (HESA) uses Standard Occupational Classifications to

measure the proportion of graduates in professional or white-collar jobs anyone not in

such a job is considered not to be working in an area related to their studies (it is unclear

what methodology is used by the Financial Times and the Times although we suspect their

methods are broadly similar)

The only other measure of final outcomes in use is percentage of graduates returning for

additional education which is an indicator used by both the Melbourne Institute and the

Financial Times This is a particularly important indicator for the latter as it is worth 21

of the final ranking

The lack of indicators concerning final outcomes is interesting since most government-

sponsored performance-indicator regimes around the world are very much concerned

with such measures especially with respect to employment Possibly this indicates that

ranking systems simply do not view education outcomes as relevant measures of

educational quality Alternatively it may be the case that they simply have not found a

reliable indicator of outcomes or that there are reliable indicators but that there is so little

variation between institutions that it makes no sense to rank based on the data

10

F Indicators of Research

Many of the league tables covered in this survey include sections and weightings related

to universitiesrsquo research efforts It is in this field of measurement that we see the greatest

diversity of indicators Presumably this is because research inputs and outputs lend

themselves much more easily to measurement and manipulation than other areas of

institutional activity

Three studies include research staff as part of their ranking scheme La Repubblica at 952

the Melbourne Institute (4) and Wuhan (078)

Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used

method of looking at research quality but it is not universally admired because different

disciplines use different means to communicate major advances in knowledge (leading

scientists invariably produce large numbers of journal articles leading social scientists

may produce fewer journal articles but instead have one or two long important

monographsmdashsee Hicks 2004) There is also some concern among non-English speaking

countries that they are penalized in international rankings because so many of the major

journals (notably Science and Nature) are printed in English However the one set of

rankings that uses separate indicators to monitor articles published in English and articles

published in another language (the Wuhan rankings) shows that the two indicators are

positively correlated institutions that have more Chinese publications are also likely to

have more English publications and vice versa

Several sets of league tables measure bibliometric citations in various publication indices

The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a

weight of 20 of the final total Guangdong also monitors other Chinese universities

specifically for citations in engineering publications and weights this at 29 Moreover it

tacks on an additional 1049 for citations in science-oriented indices such as the Science

Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the

same category of scientific citations while the Melbourne Institute rates science citations at

68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social

Science Citation Index which does not include the humanities) are noted in only two

league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne

Institute (32) Another way of measuring research

11

impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a

weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the

Melbourne Institute

The complement of citations is of course publications Listing the number of publications

an individual group department or whole university releases can act as a weak substitute

for citationsmdashweak because simply publishing a paper or monograph is no guarantee that

the general public or other researchers will even glance at the work Guangdong gives an

1179 weighting to publications in science-oriented indices such as the Science Citation

Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same

indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong

even has a separate category just for measuring publications in Science and Nature

although it accords it an almost derisory weighting of 06 Under publications in social

science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their

final totals For publications in other indices (where the subject indices are

undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for

Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of

rankings to include research monographs weighted at 033

In countries where there are specific third-party evaluations of research output academic

quality of research is sometimes used as a research indicator The Times puts a very large

30 weight on this indicator while the Financial Times puts it at 119

Research awards are another handy third-party measurement of quality as the number of

international and national awards won by faculty andor graduates is often considered a

useful measure of institutional success International research awardsmdash specifically the

number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator

by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as

particularly suspect in some quarters given that the points are based on where the

recipient went to school rather than on where they are or were on the

8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the

most prominent publication indices

9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences

12

faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead

for nearly a century Wuhan uses similar measures but only accords them a weight of 14

National research awards are more common as a quality indicator used by La Repubblica

(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)

Financial indicators of research are also very common Research budgets as a factor in the

overall assessment of research in universities are covered by the Financial Times (9)

Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for

total amount of research expenditure unfortunately it is unclear precisely what this

research expenditure represents or how it is determined although it is clearly indicated

that it does not represent the total number of grants or projects at a university Total

number of research-based grants and projects is weighted by Education18 at 15 and

Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science

and engineering and another 55 to those for social sciences and humanities Similarly

the Melbourne Institute gives 6 of its overall weight to public-source grants making no

distinction between areas of study

In a slightly different vein Netbig (46) and Wuhan (278) both list the number of

research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica

(095) all weigh research-basedaffiliated research institutions or centres for studies

Finally one can also measure research not simply in terms of the amount of money it

generates but also in terms of the amount of future income it will generate Both

Guangdong (245) and Wuhan (193) measure the number of patents issued to

universities as a quality indicator

A final way of measuring an institutionrsquos research intensity is to look at the range of its

course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of

doctoral and Masterrsquos programs offered as a proxy for research intensity

As with physical and financial resources few if any of the research indicators are

normalized to account for institutional size (either by student or faculty numbers) In the

world of rankings bigger almost always means better an institution with 100 faculty with

ten citations apiece will always look worse than an institution with 1001 faculty

13

with one citation each To the extent that the raw production of knowledge matters this

form of measurement is acceptable To the extent that rankings are meant to show how

well institutions are doing on a like-to-like basis or to show the efficiency of universities it

is plainly inadequate This should be of particular concern to Chinese policy-makers

whose ranking systems are especially reliant on research-based indicators

G Indicators of Reputation

The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo

Those rankings systems which use the results of reputation surveys as an indicator do so as

an indirect measure of quality based on the assumption that the employers academics and

academic administrators surveyed have opinions of institutional quality that are informed

up-to-date and impartial While these assumptions are clearly open to debate they

nevertheless form an important basis for many ranking systems Another reason for using

reputation measures is the paucity of other data availablemdashsome countries have few

independent measures of teaching effectiveness university resources or output and

reputation can thus act as a useful surrogate Reputation rankings are often criticized as

simply quantifying the common ignorance of the people being surveyed However to the

extent that the people being surveyed hold positions which have the potential to affect

large numbers of young people and whose positions actually require some knowledge of

institutional quality (ie officials in charge of graduate admissions corporate recruiters

etc) then reputation rankings make sense because they provide useful information for

students about the perceived value of the degrees that they could obtain from various

universities

The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and

the Times which both accord reputation a weighting of 50 in their overall ranking

scheme Education18 assigns it almost as much significance at 40 The US News and World

Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just

below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)

and Netbig (15) The only other study to include reputation is Wuhan (117)

14

1 Please see Appendix B for a glossary of the various bodies referenced in this document

Appendix A Detailed Listing of Indicators and their Sources

Beginning Characteristics Indicator

Used By Source1

Incoming grades Macleans University

Percentage with grades above a set limit

Macleans University

US News and World Report

University

Performance on national standardised tests or benchmarks

Asiaweek University

Education18 3rd-party JUPAS

Financial Times Government agency 3rd-party UCAS

Guardian University Guide

Government agency 3rd-party UCAS

Melbourne Institute Government agency 3rd-party DEST

Netbig National entrance examination board

Times Good University Guide

Government agency 3rd-party UCAS

US News University

Wuhan Unknown presumed government 3rd-party

Student status La Repubblica Government agency 3rd-party MIUR

Admittance selectivity general

Asiaweek University

Admittance number of applications to places

Asiaweek University

Financial Times Government agency 3rd-party UCAS

La Repubblica Government agency 3rd-party MIUR

US News University

15

Beginning Characteristics Indicator

Used By Source1

Out-of-locality student percentage

Macleans University

International student percentages

Financial Times Government agency 3rd-party HESA

Macleans University

Shanghai Institute of Educational Science

Times World University

Wuhan Unknown presumed university

Undergraduate students among all students percentages

Netbig

Wuhan Unknown presumed university

Ethnic diversity in student body

Guardian University

Learning Inputs ndash Staff Indicator

Used By Source

Facultystudent ratio Asiaweek University

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Financial Times Government agency 3rd-party HESA

La Repubblica Government agency 3rd-party MIUR

Times Good University Guide

Government agency 3rd-party HESA

Times World University

US News University

Wuhan Unknown presumed university

16

Learning Inputs ndash Staff Indicator

Used By Source

Social science faculty student ratio

Melbourne Government agency 3rd-party DEST

Science faculty student ratio

Melbourne Government agency 3rd-party DEST

Administrative staff student ratio

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Staff student ratio (regardless of division)

Guardian Government agency 3rd-party HESA

Netbig University

Course per teacher La Repubblica Government agency 3rd-party MIUR

Per-teacher university spending

Asiaweek University

Faculty pay rates fortenured staff

Asiaweek University

US News University

Number of full-time part-time faculty

Netbig University

US News University

Wuhan Unknown presumed university

Faculty with researchprojects

Wuhan Unknown presumed university

Class size differentiation Macleans University

US News University

Classes taught by tenured faculty

Macleans University

Exchange programmes hosted

La Repubblica Government agency 3rd-party AgNaSoc

Number of classes lsquoactually taughtrsquo

La Repubblica Government agency 3rd-party MIUR

of international faculty (v faculty as a whole)

Times World University

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

10

F Indicators of Research

Many of the league tables covered in this survey include sections and weightings related

to universitiesrsquo research efforts It is in this field of measurement that we see the greatest

diversity of indicators Presumably this is because research inputs and outputs lend

themselves much more easily to measurement and manipulation than other areas of

institutional activity

Three studies include research staff as part of their ranking scheme La Repubblica at 952

the Melbourne Institute (4) and Wuhan (078)

Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used

method of looking at research quality but it is not universally admired because different

disciplines use different means to communicate major advances in knowledge (leading

scientists invariably produce large numbers of journal articles leading social scientists

may produce fewer journal articles but instead have one or two long important

monographsmdashsee Hicks 2004) There is also some concern among non-English speaking

countries that they are penalized in international rankings because so many of the major

journals (notably Science and Nature) are printed in English However the one set of

rankings that uses separate indicators to monitor articles published in English and articles

published in another language (the Wuhan rankings) shows that the two indicators are

positively correlated institutions that have more Chinese publications are also likely to

have more English publications and vice versa

Several sets of league tables measure bibliometric citations in various publication indices

The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a

weight of 20 of the final total Guangdong also monitors other Chinese universities

specifically for citations in engineering publications and weights this at 29 Moreover it

tacks on an additional 1049 for citations in science-oriented indices such as the Science

Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the

same category of scientific citations while the Melbourne Institute rates science citations at

68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social

Science Citation Index which does not include the humanities) are noted in only two

league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne

Institute (32) Another way of measuring research

11

impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a

weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the

Melbourne Institute

The complement of citations is of course publications Listing the number of publications

an individual group department or whole university releases can act as a weak substitute

for citationsmdashweak because simply publishing a paper or monograph is no guarantee that

the general public or other researchers will even glance at the work Guangdong gives an

1179 weighting to publications in science-oriented indices such as the Science Citation

Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same

indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong

even has a separate category just for measuring publications in Science and Nature

although it accords it an almost derisory weighting of 06 Under publications in social

science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their

final totals For publications in other indices (where the subject indices are

undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for

Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of

rankings to include research monographs weighted at 033

In countries where there are specific third-party evaluations of research output academic

quality of research is sometimes used as a research indicator The Times puts a very large

30 weight on this indicator while the Financial Times puts it at 119

Research awards are another handy third-party measurement of quality as the number of

international and national awards won by faculty andor graduates is often considered a

useful measure of institutional success International research awardsmdash specifically the

number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator

by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as

particularly suspect in some quarters given that the points are based on where the

recipient went to school rather than on where they are or were on the

8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the

most prominent publication indices

9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences

12

faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead

for nearly a century Wuhan uses similar measures but only accords them a weight of 14

National research awards are more common as a quality indicator used by La Repubblica

(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)

Financial indicators of research are also very common Research budgets as a factor in the

overall assessment of research in universities are covered by the Financial Times (9)

Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for

total amount of research expenditure unfortunately it is unclear precisely what this

research expenditure represents or how it is determined although it is clearly indicated

that it does not represent the total number of grants or projects at a university Total

number of research-based grants and projects is weighted by Education18 at 15 and

Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science

and engineering and another 55 to those for social sciences and humanities Similarly

the Melbourne Institute gives 6 of its overall weight to public-source grants making no

distinction between areas of study

In a slightly different vein Netbig (46) and Wuhan (278) both list the number of

research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica

(095) all weigh research-basedaffiliated research institutions or centres for studies

Finally one can also measure research not simply in terms of the amount of money it

generates but also in terms of the amount of future income it will generate Both

Guangdong (245) and Wuhan (193) measure the number of patents issued to

universities as a quality indicator

A final way of measuring an institutionrsquos research intensity is to look at the range of its

course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of

doctoral and Masterrsquos programs offered as a proxy for research intensity

As with physical and financial resources few if any of the research indicators are

normalized to account for institutional size (either by student or faculty numbers) In the

world of rankings bigger almost always means better an institution with 100 faculty with

ten citations apiece will always look worse than an institution with 1001 faculty

13

with one citation each To the extent that the raw production of knowledge matters this

form of measurement is acceptable To the extent that rankings are meant to show how

well institutions are doing on a like-to-like basis or to show the efficiency of universities it

is plainly inadequate This should be of particular concern to Chinese policy-makers

whose ranking systems are especially reliant on research-based indicators

G Indicators of Reputation

The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo

Those rankings systems which use the results of reputation surveys as an indicator do so as

an indirect measure of quality based on the assumption that the employers academics and

academic administrators surveyed have opinions of institutional quality that are informed

up-to-date and impartial While these assumptions are clearly open to debate they

nevertheless form an important basis for many ranking systems Another reason for using

reputation measures is the paucity of other data availablemdashsome countries have few

independent measures of teaching effectiveness university resources or output and

reputation can thus act as a useful surrogate Reputation rankings are often criticized as

simply quantifying the common ignorance of the people being surveyed However to the

extent that the people being surveyed hold positions which have the potential to affect

large numbers of young people and whose positions actually require some knowledge of

institutional quality (ie officials in charge of graduate admissions corporate recruiters

etc) then reputation rankings make sense because they provide useful information for

students about the perceived value of the degrees that they could obtain from various

universities

The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and

the Times which both accord reputation a weighting of 50 in their overall ranking

scheme Education18 assigns it almost as much significance at 40 The US News and World

Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just

below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)

and Netbig (15) The only other study to include reputation is Wuhan (117)

14

1 Please see Appendix B for a glossary of the various bodies referenced in this document

Appendix A Detailed Listing of Indicators and their Sources

Beginning Characteristics Indicator

Used By Source1

Incoming grades Macleans University

Percentage with grades above a set limit

Macleans University

US News and World Report

University

Performance on national standardised tests or benchmarks

Asiaweek University

Education18 3rd-party JUPAS

Financial Times Government agency 3rd-party UCAS

Guardian University Guide

Government agency 3rd-party UCAS

Melbourne Institute Government agency 3rd-party DEST

Netbig National entrance examination board

Times Good University Guide

Government agency 3rd-party UCAS

US News University

Wuhan Unknown presumed government 3rd-party

Student status La Repubblica Government agency 3rd-party MIUR

Admittance selectivity general

Asiaweek University

Admittance number of applications to places

Asiaweek University

Financial Times Government agency 3rd-party UCAS

La Repubblica Government agency 3rd-party MIUR

US News University

15

Beginning Characteristics Indicator

Used By Source1

Out-of-locality student percentage

Macleans University

International student percentages

Financial Times Government agency 3rd-party HESA

Macleans University

Shanghai Institute of Educational Science

Times World University

Wuhan Unknown presumed university

Undergraduate students among all students percentages

Netbig

Wuhan Unknown presumed university

Ethnic diversity in student body

Guardian University

Learning Inputs ndash Staff Indicator

Used By Source

Facultystudent ratio Asiaweek University

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Financial Times Government agency 3rd-party HESA

La Repubblica Government agency 3rd-party MIUR

Times Good University Guide

Government agency 3rd-party HESA

Times World University

US News University

Wuhan Unknown presumed university

16

Learning Inputs ndash Staff Indicator

Used By Source

Social science faculty student ratio

Melbourne Government agency 3rd-party DEST

Science faculty student ratio

Melbourne Government agency 3rd-party DEST

Administrative staff student ratio

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Staff student ratio (regardless of division)

Guardian Government agency 3rd-party HESA

Netbig University

Course per teacher La Repubblica Government agency 3rd-party MIUR

Per-teacher university spending

Asiaweek University

Faculty pay rates fortenured staff

Asiaweek University

US News University

Number of full-time part-time faculty

Netbig University

US News University

Wuhan Unknown presumed university

Faculty with researchprojects

Wuhan Unknown presumed university

Class size differentiation Macleans University

US News University

Classes taught by tenured faculty

Macleans University

Exchange programmes hosted

La Repubblica Government agency 3rd-party AgNaSoc

Number of classes lsquoactually taughtrsquo

La Repubblica Government agency 3rd-party MIUR

of international faculty (v faculty as a whole)

Times World University

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

11

impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a

weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the

Melbourne Institute

The complement of citations is of course publications Listing the number of publications

an individual group department or whole university releases can act as a weak substitute

for citationsmdashweak because simply publishing a paper or monograph is no guarantee that

the general public or other researchers will even glance at the work Guangdong gives an

1179 weighting to publications in science-oriented indices such as the Science Citation

Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same

indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong

even has a separate category just for measuring publications in Science and Nature

although it accords it an almost derisory weighting of 06 Under publications in social

science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their

final totals For publications in other indices (where the subject indices are

undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for

Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of

rankings to include research monographs weighted at 033

In countries where there are specific third-party evaluations of research output academic

quality of research is sometimes used as a research indicator The Times puts a very large

30 weight on this indicator while the Financial Times puts it at 119

Research awards are another handy third-party measurement of quality as the number of

international and national awards won by faculty andor graduates is often considered a

useful measure of institutional success International research awardsmdash specifically the

number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator

by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as

particularly suspect in some quarters given that the points are based on where the

recipient went to school rather than on where they are or were on the

8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the

most prominent publication indices

9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences

12

faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead

for nearly a century Wuhan uses similar measures but only accords them a weight of 14

National research awards are more common as a quality indicator used by La Repubblica

(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)

Financial indicators of research are also very common Research budgets as a factor in the

overall assessment of research in universities are covered by the Financial Times (9)

Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for

total amount of research expenditure unfortunately it is unclear precisely what this

research expenditure represents or how it is determined although it is clearly indicated

that it does not represent the total number of grants or projects at a university Total

number of research-based grants and projects is weighted by Education18 at 15 and

Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science

and engineering and another 55 to those for social sciences and humanities Similarly

the Melbourne Institute gives 6 of its overall weight to public-source grants making no

distinction between areas of study

In a slightly different vein Netbig (46) and Wuhan (278) both list the number of

research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica

(095) all weigh research-basedaffiliated research institutions or centres for studies

Finally one can also measure research not simply in terms of the amount of money it

generates but also in terms of the amount of future income it will generate Both

Guangdong (245) and Wuhan (193) measure the number of patents issued to

universities as a quality indicator

A final way of measuring an institutionrsquos research intensity is to look at the range of its

course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of

doctoral and Masterrsquos programs offered as a proxy for research intensity

As with physical and financial resources few if any of the research indicators are

normalized to account for institutional size (either by student or faculty numbers) In the

world of rankings bigger almost always means better an institution with 100 faculty with

ten citations apiece will always look worse than an institution with 1001 faculty

13

with one citation each To the extent that the raw production of knowledge matters this

form of measurement is acceptable To the extent that rankings are meant to show how

well institutions are doing on a like-to-like basis or to show the efficiency of universities it

is plainly inadequate This should be of particular concern to Chinese policy-makers

whose ranking systems are especially reliant on research-based indicators

G Indicators of Reputation

The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo

Those rankings systems which use the results of reputation surveys as an indicator do so as

an indirect measure of quality based on the assumption that the employers academics and

academic administrators surveyed have opinions of institutional quality that are informed

up-to-date and impartial While these assumptions are clearly open to debate they

nevertheless form an important basis for many ranking systems Another reason for using

reputation measures is the paucity of other data availablemdashsome countries have few

independent measures of teaching effectiveness university resources or output and

reputation can thus act as a useful surrogate Reputation rankings are often criticized as

simply quantifying the common ignorance of the people being surveyed However to the

extent that the people being surveyed hold positions which have the potential to affect

large numbers of young people and whose positions actually require some knowledge of

institutional quality (ie officials in charge of graduate admissions corporate recruiters

etc) then reputation rankings make sense because they provide useful information for

students about the perceived value of the degrees that they could obtain from various

universities

The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and

the Times which both accord reputation a weighting of 50 in their overall ranking

scheme Education18 assigns it almost as much significance at 40 The US News and World

Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just

below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)

and Netbig (15) The only other study to include reputation is Wuhan (117)

14

1 Please see Appendix B for a glossary of the various bodies referenced in this document

Appendix A Detailed Listing of Indicators and their Sources

Beginning Characteristics Indicator

Used By Source1

Incoming grades Macleans University

Percentage with grades above a set limit

Macleans University

US News and World Report

University

Performance on national standardised tests or benchmarks

Asiaweek University

Education18 3rd-party JUPAS

Financial Times Government agency 3rd-party UCAS

Guardian University Guide

Government agency 3rd-party UCAS

Melbourne Institute Government agency 3rd-party DEST

Netbig National entrance examination board

Times Good University Guide

Government agency 3rd-party UCAS

US News University

Wuhan Unknown presumed government 3rd-party

Student status La Repubblica Government agency 3rd-party MIUR

Admittance selectivity general

Asiaweek University

Admittance number of applications to places

Asiaweek University

Financial Times Government agency 3rd-party UCAS

La Repubblica Government agency 3rd-party MIUR

US News University

15

Beginning Characteristics Indicator

Used By Source1

Out-of-locality student percentage

Macleans University

International student percentages

Financial Times Government agency 3rd-party HESA

Macleans University

Shanghai Institute of Educational Science

Times World University

Wuhan Unknown presumed university

Undergraduate students among all students percentages

Netbig

Wuhan Unknown presumed university

Ethnic diversity in student body

Guardian University

Learning Inputs ndash Staff Indicator

Used By Source

Facultystudent ratio Asiaweek University

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Financial Times Government agency 3rd-party HESA

La Repubblica Government agency 3rd-party MIUR

Times Good University Guide

Government agency 3rd-party HESA

Times World University

US News University

Wuhan Unknown presumed university

16

Learning Inputs ndash Staff Indicator

Used By Source

Social science faculty student ratio

Melbourne Government agency 3rd-party DEST

Science faculty student ratio

Melbourne Government agency 3rd-party DEST

Administrative staff student ratio

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Staff student ratio (regardless of division)

Guardian Government agency 3rd-party HESA

Netbig University

Course per teacher La Repubblica Government agency 3rd-party MIUR

Per-teacher university spending

Asiaweek University

Faculty pay rates fortenured staff

Asiaweek University

US News University

Number of full-time part-time faculty

Netbig University

US News University

Wuhan Unknown presumed university

Faculty with researchprojects

Wuhan Unknown presumed university

Class size differentiation Macleans University

US News University

Classes taught by tenured faculty

Macleans University

Exchange programmes hosted

La Repubblica Government agency 3rd-party AgNaSoc

Number of classes lsquoactually taughtrsquo

La Repubblica Government agency 3rd-party MIUR

of international faculty (v faculty as a whole)

Times World University

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

12

faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead

for nearly a century Wuhan uses similar measures but only accords them a weight of 14

National research awards are more common as a quality indicator used by La Repubblica

(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)

Financial indicators of research are also very common Research budgets as a factor in the

overall assessment of research in universities are covered by the Financial Times (9)

Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for

total amount of research expenditure unfortunately it is unclear precisely what this

research expenditure represents or how it is determined although it is clearly indicated

that it does not represent the total number of grants or projects at a university Total

number of research-based grants and projects is weighted by Education18 at 15 and

Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science

and engineering and another 55 to those for social sciences and humanities Similarly

the Melbourne Institute gives 6 of its overall weight to public-source grants making no

distinction between areas of study

In a slightly different vein Netbig (46) and Wuhan (278) both list the number of

research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica

(095) all weigh research-basedaffiliated research institutions or centres for studies

Finally one can also measure research not simply in terms of the amount of money it

generates but also in terms of the amount of future income it will generate Both

Guangdong (245) and Wuhan (193) measure the number of patents issued to

universities as a quality indicator

A final way of measuring an institutionrsquos research intensity is to look at the range of its

course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of

doctoral and Masterrsquos programs offered as a proxy for research intensity

As with physical and financial resources few if any of the research indicators are

normalized to account for institutional size (either by student or faculty numbers) In the

world of rankings bigger almost always means better an institution with 100 faculty with

ten citations apiece will always look worse than an institution with 1001 faculty

13

with one citation each To the extent that the raw production of knowledge matters this

form of measurement is acceptable To the extent that rankings are meant to show how

well institutions are doing on a like-to-like basis or to show the efficiency of universities it

is plainly inadequate This should be of particular concern to Chinese policy-makers

whose ranking systems are especially reliant on research-based indicators

G Indicators of Reputation

The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo

Those rankings systems which use the results of reputation surveys as an indicator do so as

an indirect measure of quality based on the assumption that the employers academics and

academic administrators surveyed have opinions of institutional quality that are informed

up-to-date and impartial While these assumptions are clearly open to debate they

nevertheless form an important basis for many ranking systems Another reason for using

reputation measures is the paucity of other data availablemdashsome countries have few

independent measures of teaching effectiveness university resources or output and

reputation can thus act as a useful surrogate Reputation rankings are often criticized as

simply quantifying the common ignorance of the people being surveyed However to the

extent that the people being surveyed hold positions which have the potential to affect

large numbers of young people and whose positions actually require some knowledge of

institutional quality (ie officials in charge of graduate admissions corporate recruiters

etc) then reputation rankings make sense because they provide useful information for

students about the perceived value of the degrees that they could obtain from various

universities

The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and

the Times which both accord reputation a weighting of 50 in their overall ranking

scheme Education18 assigns it almost as much significance at 40 The US News and World

Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just

below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)

and Netbig (15) The only other study to include reputation is Wuhan (117)

14

1 Please see Appendix B for a glossary of the various bodies referenced in this document

Appendix A Detailed Listing of Indicators and their Sources

Beginning Characteristics Indicator

Used By Source1

Incoming grades Macleans University

Percentage with grades above a set limit

Macleans University

US News and World Report

University

Performance on national standardised tests or benchmarks

Asiaweek University

Education18 3rd-party JUPAS

Financial Times Government agency 3rd-party UCAS

Guardian University Guide

Government agency 3rd-party UCAS

Melbourne Institute Government agency 3rd-party DEST

Netbig National entrance examination board

Times Good University Guide

Government agency 3rd-party UCAS

US News University

Wuhan Unknown presumed government 3rd-party

Student status La Repubblica Government agency 3rd-party MIUR

Admittance selectivity general

Asiaweek University

Admittance number of applications to places

Asiaweek University

Financial Times Government agency 3rd-party UCAS

La Repubblica Government agency 3rd-party MIUR

US News University

15

Beginning Characteristics Indicator

Used By Source1

Out-of-locality student percentage

Macleans University

International student percentages

Financial Times Government agency 3rd-party HESA

Macleans University

Shanghai Institute of Educational Science

Times World University

Wuhan Unknown presumed university

Undergraduate students among all students percentages

Netbig

Wuhan Unknown presumed university

Ethnic diversity in student body

Guardian University

Learning Inputs ndash Staff Indicator

Used By Source

Facultystudent ratio Asiaweek University

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Financial Times Government agency 3rd-party HESA

La Repubblica Government agency 3rd-party MIUR

Times Good University Guide

Government agency 3rd-party HESA

Times World University

US News University

Wuhan Unknown presumed university

16

Learning Inputs ndash Staff Indicator

Used By Source

Social science faculty student ratio

Melbourne Government agency 3rd-party DEST

Science faculty student ratio

Melbourne Government agency 3rd-party DEST

Administrative staff student ratio

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Staff student ratio (regardless of division)

Guardian Government agency 3rd-party HESA

Netbig University

Course per teacher La Repubblica Government agency 3rd-party MIUR

Per-teacher university spending

Asiaweek University

Faculty pay rates fortenured staff

Asiaweek University

US News University

Number of full-time part-time faculty

Netbig University

US News University

Wuhan Unknown presumed university

Faculty with researchprojects

Wuhan Unknown presumed university

Class size differentiation Macleans University

US News University

Classes taught by tenured faculty

Macleans University

Exchange programmes hosted

La Repubblica Government agency 3rd-party AgNaSoc

Number of classes lsquoactually taughtrsquo

La Repubblica Government agency 3rd-party MIUR

of international faculty (v faculty as a whole)

Times World University

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

13

with one citation each To the extent that the raw production of knowledge matters this

form of measurement is acceptable To the extent that rankings are meant to show how

well institutions are doing on a like-to-like basis or to show the efficiency of universities it

is plainly inadequate This should be of particular concern to Chinese policy-makers

whose ranking systems are especially reliant on research-based indicators

G Indicators of Reputation

The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo

Those rankings systems which use the results of reputation surveys as an indicator do so as

an indirect measure of quality based on the assumption that the employers academics and

academic administrators surveyed have opinions of institutional quality that are informed

up-to-date and impartial While these assumptions are clearly open to debate they

nevertheless form an important basis for many ranking systems Another reason for using

reputation measures is the paucity of other data availablemdashsome countries have few

independent measures of teaching effectiveness university resources or output and

reputation can thus act as a useful surrogate Reputation rankings are often criticized as

simply quantifying the common ignorance of the people being surveyed However to the

extent that the people being surveyed hold positions which have the potential to affect

large numbers of young people and whose positions actually require some knowledge of

institutional quality (ie officials in charge of graduate admissions corporate recruiters

etc) then reputation rankings make sense because they provide useful information for

students about the perceived value of the degrees that they could obtain from various

universities

The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and

the Times which both accord reputation a weighting of 50 in their overall ranking

scheme Education18 assigns it almost as much significance at 40 The US News and World

Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just

below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)

and Netbig (15) The only other study to include reputation is Wuhan (117)

14

1 Please see Appendix B for a glossary of the various bodies referenced in this document

Appendix A Detailed Listing of Indicators and their Sources

Beginning Characteristics Indicator

Used By Source1

Incoming grades Macleans University

Percentage with grades above a set limit

Macleans University

US News and World Report

University

Performance on national standardised tests or benchmarks

Asiaweek University

Education18 3rd-party JUPAS

Financial Times Government agency 3rd-party UCAS

Guardian University Guide

Government agency 3rd-party UCAS

Melbourne Institute Government agency 3rd-party DEST

Netbig National entrance examination board

Times Good University Guide

Government agency 3rd-party UCAS

US News University

Wuhan Unknown presumed government 3rd-party

Student status La Repubblica Government agency 3rd-party MIUR

Admittance selectivity general

Asiaweek University

Admittance number of applications to places

Asiaweek University

Financial Times Government agency 3rd-party UCAS

La Repubblica Government agency 3rd-party MIUR

US News University

15

Beginning Characteristics Indicator

Used By Source1

Out-of-locality student percentage

Macleans University

International student percentages

Financial Times Government agency 3rd-party HESA

Macleans University

Shanghai Institute of Educational Science

Times World University

Wuhan Unknown presumed university

Undergraduate students among all students percentages

Netbig

Wuhan Unknown presumed university

Ethnic diversity in student body

Guardian University

Learning Inputs ndash Staff Indicator

Used By Source

Facultystudent ratio Asiaweek University

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Financial Times Government agency 3rd-party HESA

La Repubblica Government agency 3rd-party MIUR

Times Good University Guide

Government agency 3rd-party HESA

Times World University

US News University

Wuhan Unknown presumed university

16

Learning Inputs ndash Staff Indicator

Used By Source

Social science faculty student ratio

Melbourne Government agency 3rd-party DEST

Science faculty student ratio

Melbourne Government agency 3rd-party DEST

Administrative staff student ratio

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Staff student ratio (regardless of division)

Guardian Government agency 3rd-party HESA

Netbig University

Course per teacher La Repubblica Government agency 3rd-party MIUR

Per-teacher university spending

Asiaweek University

Faculty pay rates fortenured staff

Asiaweek University

US News University

Number of full-time part-time faculty

Netbig University

US News University

Wuhan Unknown presumed university

Faculty with researchprojects

Wuhan Unknown presumed university

Class size differentiation Macleans University

US News University

Classes taught by tenured faculty

Macleans University

Exchange programmes hosted

La Repubblica Government agency 3rd-party AgNaSoc

Number of classes lsquoactually taughtrsquo

La Repubblica Government agency 3rd-party MIUR

of international faculty (v faculty as a whole)

Times World University

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

14

1 Please see Appendix B for a glossary of the various bodies referenced in this document

Appendix A Detailed Listing of Indicators and their Sources

Beginning Characteristics Indicator

Used By Source1

Incoming grades Macleans University

Percentage with grades above a set limit

Macleans University

US News and World Report

University

Performance on national standardised tests or benchmarks

Asiaweek University

Education18 3rd-party JUPAS

Financial Times Government agency 3rd-party UCAS

Guardian University Guide

Government agency 3rd-party UCAS

Melbourne Institute Government agency 3rd-party DEST

Netbig National entrance examination board

Times Good University Guide

Government agency 3rd-party UCAS

US News University

Wuhan Unknown presumed government 3rd-party

Student status La Repubblica Government agency 3rd-party MIUR

Admittance selectivity general

Asiaweek University

Admittance number of applications to places

Asiaweek University

Financial Times Government agency 3rd-party UCAS

La Repubblica Government agency 3rd-party MIUR

US News University

15

Beginning Characteristics Indicator

Used By Source1

Out-of-locality student percentage

Macleans University

International student percentages

Financial Times Government agency 3rd-party HESA

Macleans University

Shanghai Institute of Educational Science

Times World University

Wuhan Unknown presumed university

Undergraduate students among all students percentages

Netbig

Wuhan Unknown presumed university

Ethnic diversity in student body

Guardian University

Learning Inputs ndash Staff Indicator

Used By Source

Facultystudent ratio Asiaweek University

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Financial Times Government agency 3rd-party HESA

La Repubblica Government agency 3rd-party MIUR

Times Good University Guide

Government agency 3rd-party HESA

Times World University

US News University

Wuhan Unknown presumed university

16

Learning Inputs ndash Staff Indicator

Used By Source

Social science faculty student ratio

Melbourne Government agency 3rd-party DEST

Science faculty student ratio

Melbourne Government agency 3rd-party DEST

Administrative staff student ratio

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Staff student ratio (regardless of division)

Guardian Government agency 3rd-party HESA

Netbig University

Course per teacher La Repubblica Government agency 3rd-party MIUR

Per-teacher university spending

Asiaweek University

Faculty pay rates fortenured staff

Asiaweek University

US News University

Number of full-time part-time faculty

Netbig University

US News University

Wuhan Unknown presumed university

Faculty with researchprojects

Wuhan Unknown presumed university

Class size differentiation Macleans University

US News University

Classes taught by tenured faculty

Macleans University

Exchange programmes hosted

La Repubblica Government agency 3rd-party AgNaSoc

Number of classes lsquoactually taughtrsquo

La Repubblica Government agency 3rd-party MIUR

of international faculty (v faculty as a whole)

Times World University

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

15

Beginning Characteristics Indicator

Used By Source1

Out-of-locality student percentage

Macleans University

International student percentages

Financial Times Government agency 3rd-party HESA

Macleans University

Shanghai Institute of Educational Science

Times World University

Wuhan Unknown presumed university

Undergraduate students among all students percentages

Netbig

Wuhan Unknown presumed university

Ethnic diversity in student body

Guardian University

Learning Inputs ndash Staff Indicator

Used By Source

Facultystudent ratio Asiaweek University

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Financial Times Government agency 3rd-party HESA

La Repubblica Government agency 3rd-party MIUR

Times Good University Guide

Government agency 3rd-party HESA

Times World University

US News University

Wuhan Unknown presumed university

16

Learning Inputs ndash Staff Indicator

Used By Source

Social science faculty student ratio

Melbourne Government agency 3rd-party DEST

Science faculty student ratio

Melbourne Government agency 3rd-party DEST

Administrative staff student ratio

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Staff student ratio (regardless of division)

Guardian Government agency 3rd-party HESA

Netbig University

Course per teacher La Repubblica Government agency 3rd-party MIUR

Per-teacher university spending

Asiaweek University

Faculty pay rates fortenured staff

Asiaweek University

US News University

Number of full-time part-time faculty

Netbig University

US News University

Wuhan Unknown presumed university

Faculty with researchprojects

Wuhan Unknown presumed university

Class size differentiation Macleans University

US News University

Classes taught by tenured faculty

Macleans University

Exchange programmes hosted

La Repubblica Government agency 3rd-party AgNaSoc

Number of classes lsquoactually taughtrsquo

La Repubblica Government agency 3rd-party MIUR

of international faculty (v faculty as a whole)

Times World University

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

16

Learning Inputs ndash Staff Indicator

Used By Source

Social science faculty student ratio

Melbourne Government agency 3rd-party DEST

Science faculty student ratio

Melbourne Government agency 3rd-party DEST

Administrative staff student ratio

Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas

Staff student ratio (regardless of division)

Guardian Government agency 3rd-party HESA

Netbig University

Course per teacher La Repubblica Government agency 3rd-party MIUR

Per-teacher university spending

Asiaweek University

Faculty pay rates fortenured staff

Asiaweek University

US News University

Number of full-time part-time faculty

Netbig University

US News University

Wuhan Unknown presumed university

Faculty with researchprojects

Wuhan Unknown presumed university

Class size differentiation Macleans University

US News University

Classes taught by tenured faculty

Macleans University

Exchange programmes hosted

La Repubblica Government agency 3rd-party AgNaSoc

Number of classes lsquoactually taughtrsquo

La Repubblica Government agency 3rd-party MIUR

of international faculty (v faculty as a whole)

Times World University

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

17

Learning Inputs ndash Staff Indicator

Used By Source

Aging and staffreplacement churn issues

La Repubblica Government agency 3rd-party MIUR

Teaching quality Faculty performance on standardised 3rd-party tests if given

Education18 3rd-party TLQPR

Financial Times Government agency 3rd-party QAA HESA

Times Good University Guide

Government agency 3rd-party QAA HESA

US News University

Teaching quality Performance on own metrics

Guardian Survey (cobbled together from QAA scores)

Teaching quality Qualifications for teaching positions (PhDs Masters etc)

Asiaweek University

Education18 University

Macleans University

Netbig University

US News University

Number of doctoral and Masters programmes

Asiaweek University

Netbig Unknown presumed university

Wuhan Unknown presumed university

Student efforts Hours spent in class per student

La Repubblica Government agency 3rd-party CNVSU

Student efforts student participation in exchange projects

La Repubblica Government agency 3rd-party AgNaSoc

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

18

Learning Inputs ndash Resources Indicator

Used By Source

Physical infrastructure Number of lecture spaces

La Repubblica Government agency 3rd-party MIUR

Physical infrastructure Library Acquisitions per year

Macleans University

Physical infrastructure Library total volumes

Education18 University

Macleans University

Netbig Unknown presumed university

Physical infrastructure Library volumes per student

Macleans University

Physical infrastructure Library Yearly expenditures outside of acquisitions

Asiaweek University

Financial Times Government agency 3rd-party HESA

Macleans University

Physical infrastructure Internet bandwidth

Asiaweek University

Physical infrastructure Computerisation of library resources

Asiaweek University

Financial Times Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Public funding total of institutional budget

Macleans University

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

19

Learning Inputs ndash Resources Indicator

Used By Source

Funding and financial resources Private funding total (including supporting foundations and charitable organisations)

Financial Times Government agency 3rd-party HESA

Funding and financial resources Alumni support

Macleans University

US News University

Funding and financial resources Student services

Macleans University

Times Good University Guide Government agency 3rd-party HESA

Funding and financial resources Science grants

Macleans University

Funding and financial resources Social sciences and humanities grants

Macleans University

Funding and financial resources Expenditure

Guardian Government agency 3rd-party HESA

Shanghai Institute of Educational Science

Funding and financial resources Bursaries and scholarships disbursed by public private bodies

La Repubblica Government agency 3rd-party MIUR

Macleans University

Shanghai Institute of Educational Science

Wuhan Unknown presumed university or government agency 3rdparty

Funding and financial resources

La Repubblica Government agency 3rd-party AgNaSoc

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

20

Learning Inputs ndash Resources Indicator

Used By Source

Awards (not research awards) subsidised or unsubsidised

Macleans University

Learning Outputs Indicator

Used By Source

Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)

Shanghai Jiao Tong University

Times Good University Guide Government agency 3rd-party HESA

US News University

Graduation rate Undergraduates only

Guangdong Institute of Management Science

Unknown

La Repubblica

Macleans University

Melbourne Institute Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Masters only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

Graduation rate Doctoral students only

Guangdong Unknown

Melbourne Government agency 3rd-party DEST

Wuhan Unknown presumed university

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

21

Learning Inputs ndash Resources Indicator

Used By Source

Graduation rate International students

Macleans University

Type of degree obtained Financial Times Government agency 3rd-party HESA

Retention 1st to 2nd year

La Repubblica Government agency 3rd-party CNVSU

Macleans University

Melbourne Institute Government agency 3rd-party DEST

US News University

Final Outcomes Indicator Used By Source

Work status Financial Times Government agency 3rd-party HESA

Guardian Government agency 3rd-party HESA

Times Good University Guide Government agency 3rd-party HESA

Wuhan Unknown presumed survey or government agency 3rd-party

Further professional education

Financial Times Government agency 3rd-party HESA

Melbourne Survey government agency 3rd-party DEST

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

22

Research Indicator Used By Source

Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)

La Repubblica

Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST

Wuhan Unknown presumed government agency 3rd-party

Academic quality of research

CUAA Unknown

Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC

Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates

Times Good University Guide

Awards International Shanghai Jiao Tong

Wuhan Unknown presumed government agency 3rd-party

Awards National Guangdong Unknown presumed government agency 3rd-party

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed government agency 3rd-party

Awards Regional (ie stateprovincial or within national borders)

Guangdong Unknown presumed government agency 3rd-party

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

23

Research Indicator Used By Source

Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)

Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party SCI CSTPC

Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong 3rd-party

Citations Highly cited (as determined by Thomson-ISI)

Melbourne 3rd-party Non-lab ESI

Shanghai Jiao Tong

Wuhan 3rd-party ISI-related indices

Citations Other Asiaweek 3rd-party

Shanghai Jiao Tong 3rd-party

Times World 3rd-party

Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI

Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)

Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science

Publications Published papers in science-oriented indices (ie the Science Citation Index)

Guangdong Unknown

Melbourne 3rd-party Lab ESI

Netbig 3rd-party SCI Engineering Index

Wuhan 3rd-party CSTPC SCI

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

24

Research Indicator

Used By

Source

Publications Published in social science-oriented indices (ie the Social Science Citation Index)

M elbourne

3rd-party Non-lab ESI

Netbig

3rd-party SSCI

Publications Published papers in other indices ndash

Asiaweek

3rd-party

Education18

3rd-party RGC

Guangdong

Unknown

Wuhan 3rd-party AHCI and others not described fully

Publications Books (other)

Asiaweek 3rd-party

Research budget including grants

Asiaweek University

Financial Times

Government agency 3rd-party RAE 2001

Research budget Expenditure (undefined)

Wuhan

Unknown presumed survey or university

Research budget Total number of grants and projects

Education18

3rd-party RGC

Wuhan

Government agency 3rd-party NSF(c) and NSSF(c)

Patents

Guangdong

Unknown

Wuhan

Unknown presumed government agency 3rd-party

Number of research-based chairs per institution

Netbig

Government agency 3rd-party

Wuhan

Unknown presumed government agency 3rd-party

Number of research-basedaffiliated research institutions centres for studies etc

La Repubblica

Netbig Government agency 3rd-party

Wuhan Unknown presumed university

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

25

Research Indicator

Used By

Source

Other output

Guangdong

Unknown

Wuhan

Unknown

Reputation Indicator Used By Source

Among studentsgraduates

Melbourne Survey

Among academics Asiaweek Survey

Education18 Survey

Netbig Survey

Times World Survey

US News Survey

Wuhan Survey

Among general society business sector others outside direct connection to university

Education18 Survey

Macleans Survey

Melbourne Survey

Wuhan Survey

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

The Real World of PERFORMANCE INDICATORS

A review of their use in selected Commonwealth countries

Prepared by Dorothy Davis Director International Students Division

IDP Education Australia

For CHEMS Commonwealth Higher Education

Management Service 36 Gordon Square

London WC 1H0PF

March 1996

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

DEUXIEgraveME PARTIE

CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS

RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES

utilisation des ressources

qualiteacute de lenseignement

activiteacutes de la recherche

PREPARE PAR ROLAND PROULX

CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE

MONTREAL MAI 2008

wwwigopporg 2

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

wwwigopporg 3

Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de

comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de

nombreux deacutefis parmi lesquels figurent les suivants

Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel

Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes

Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)

Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international

Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute

La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles

Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables

Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de

reacuteunir des indicateurs refleacutetant ces exigences

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

1

Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Utilisation des ressources (16 indicateurs)

des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national

subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national

salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national

Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national

Deacutepenses totales consacreacutees aux TIC input universiteacute national

Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain

budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip

eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national

RP gouvernance-reddition de comptes2008-07-08

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Qualiteacute de lenseignement (29 indicateurs)

Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international

dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international

des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ

perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national

Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational

Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international

nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain

nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national

Education permanente nombre de programmes offerts input universiteacute

taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome

Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international

RP gouvernance-reddition de comptes2008-07-08

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition

3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes

trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche

INDICATEURS TYPE SOURCE REPEacuteRAGE

INTERNE EXTERNE

Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii

de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national

Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational

Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national

Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national

nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national

des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national

classements internationaux outcome THES Shanghai Taiwan nationalinternational

RP gouvernance-reddition de comptes2008-07-08

  • 1Premiegravere partie-Rapport sur les indicateurs de performance
    • Preacutepareacute par Roland Proulx
    • Consultant en planification institutionnelle et veille strateacutegique
      • 2Inventaire des indicateurs
        • inventaire
          • 3ANNEXES
          • 4Performance Indicators
          • 5a Page couverture-A world of difference
          • 5bTexte-AWorld-of-Difference
          • 5ctableau0a word
          • 6The Real World of
          • 7gouvernance3
          • 8gouvernance_reddition
            • reddition