Handling Chinese-Language Bibliographic Data A North American Perspective Honk Kong Library Association, August 22 nd, 2003 Clément Arsenault, assistant

Handling Chinese-Language Bibliographic Data

A North American Perspective

Honk Kong Library Association, August 22nd, 2003Clément Arsenault, assistant professor

École de bibliothéconomie et des sciences de l’informationUniversité de Montréal

©2003, Clément Arsenault 2

Overview

Multilingual / Multiscript Information Systems

Transliteration, Transcription and Romanization

Romanization Systems for Chinese

Transliteration in Bibliographic Records

Word Division

Parsing Chinese Text

Word Division for Bibliographic Control

An Retrieval Experiment


Multilingual / Multiscript Info. Systems

Integrate several languages and/or Several scripts

10 major scripts, to write ~95% of all languages

Japanese– Hiragana あいうえお…– Katakana アイウエオ…

Korean 가각갂갃간…Chinese 甲乙丙丁…

Romanabcdeéèêœ …

Greek αβγδε …Cyrillic авгдеж …Hebrew אבגדה …Arabic ث ت ا ب …

ح جIndic (11) …अआइईउऊThai …กขฃคฅฆ



System contains records representing items in more than one language

System contains records that are, in total or in part, in more than one language

The system interface is in more than one language• System prompts• Command / query language

The system is able to display text in more than one script The system allows the end user to build queries in more

than one script



Non-Roman data in North American OPACs

Stored? Displayed?

yes

no

yes

no

Indexed?yes

no

Romanization Vernacular

Cata

log

uin

gR

etr

ieval


Chinese Language: Some Facts

Number of characters• 9,353 in 1st century C.E.• 47,043 in 1716 ( 康熙字典 )• ~60,000 in 1990 ( 漢語大字典 )

Occurrence• 1,000 characters 90%• 2,400 characters 99%• 3,800 characters 99.9%• 6,600 characters 99.999%



What is Transliteration?• Script conversion

– Transliteration:– Transcription:

script scriptsound script

What is Romanization?• Converting a script to the Roman script

Romanizing Chinese script• Only transcription is possible


What sounds?• Vast number of regionalects / dialects• Standard is Mandarin (based on Beijing)

cha — Northern

zo — Suzhou

dzo — Wenzhou

te — Xiamen (Amoy)

tssa — Guangzhou (Canton)

茶Romanization Systems for Chinese


What sounds?• Then how to render it… ?

春


chunch’untchuntchwuntchountchounne...


Historical overview• Fanqie method (early 1st millennium)

• 烃 = 土 + 丁 (tu + ding)

• Matteo Ricci & Father Nicolas Trigault (17th cent.)• Hundreds of schemes developed since then

• Mostly developed by Westerners• Wade (English) Wade-Giles (English/American)• EFEO (French)• Yale (American)• Lessing-Othmer (German)• …



Historical overview• Systems developed by Chinese

• Gwoyeu Romatzyh Pinin Faashyh (1928)• Beifangxua Latinxua Sin Wenz (1931)• Hanyu pinyin fang’an (1956)

Romanizing Chinese for bibliographic Control in North America

• Wade-Giles (through October 2000)• Pinyin (After October 2000)



Wade-Giles vs Pinyin

Example: 唐宋全诗 Wade-Giles: T‘ang2 Sung4 ch‘üan2 shih1

Pinyin: Táng Sòng quán shī


Wade-Giles Used mostly in English-speaking

countries Was used until 2000 at LC (and

mainly in NA libraries) Rarely used in teaching anymore Heavy use of punctuation and

diacritics

Pinyin Used internationally

Used for many years in libraries in Europe and Australia

Used for teaching Minimal use of punctuation and

diacritics



Is transliteration necessary / useful? Necessary for oral and written communications

All 川崎 models come fully equipped.



Is transliteration necessary / useful? Necessary for oral and written communications

All 川崎 models come fully equipped. All Kawasaki models come fully equipped.



Is there a need for Romanized fields in bibliographic records?

• In printed records?• In electronic records?

A special case for Chinese• Three major obstacles

• Filing: difficult to browse Chinese characters• Data entry: users need to Romanize anyway • 25% of sources in Roman only (Anderson 1972)



Filing Chinese characters• Number of strokes• Semantic roots

• Then, number of strokes

• Based on shape• 4-corners method

• By sound• Romanization… (A–Z)

• Simplest and fastest method


Data entry• Keyboards (more than 700 methods)

• Special keyboards• QWERTY or AZERTY keyboards

– orthographic-based methods– phonetic-based methods

• Special devices• OCR• Pressure sensitive tablets• Voice recognition…



Word Division

Chinese is written without word delimiters

多接近大自然總是不錯的，因為人是從大自然而來的。

But Romanized Chinese could/should be…Duo jie jin da zi ran zong shi bu cuo de, yin wei ren shi cong da zi ran er lai de.

Duo jiejin daziran zongshi bucuo de, yinwei ren shi cong daziran erlai de.


Reasons for delimiting Romanized Chinese• Syllabic structure is too simple for efficient retrieval

• ~1300 single syllables (~400 base syllables)• “mā, má, mǎ, mà” indexed as “ma”

• Single syllables• High level of ambiguity (homophones)• Ambiguous 8 times out of 9• Readability is almost nil

• Joined syllables• Resolves ~95% of ambiguity cases (King 1983)• Greatly improves readability

Word Division


But, no consistent rules… 中國話

• Zhong guo hua

• Zhong-guo hua

• Zhongguo hua

• Zhongguohua

Word Division


Parsing Text

What is a word?• Visual word• Semantic/syntactic word...

Often based on conventions Not always consistent (in Google, 4 Aug. 2003)

– earring (461,000) ear ring (18,200)– shoemaker (465,000) shoe maker (21,100)– bottleneck (419,000) bottle neck (32,800)– firefighter (687,000) fire fighter (121,000)– flowerpot (42,100) flower pot (54,300)


Word Division for Bibliographic Control

1997: LC announces change to Pinyin Use monosyllabic or polysyllabic transcription?

Monosyllabic division Polysyllabic division

• Consistent • Increases recall • Lowers precision • Easier to convert from

existing Wade-Giles • Easier to generate

Romanization from a string of Chinese characters

• Difficult to be consistent

• Lack of established standard • The proper format according to Hanyu

pinyin fang’an (the PRC pinyin standard) • Represents the nature of the language • Improves readablity when browsing • Improves precision • More effective in voice recognition / text-

to-speech implementations


A Retrieval Experiment Experiment designed to test the

effect of syllable aggregation on retrieval

Part of a Doctoral Thesis at University of Toronto


Statement of the Problem

Conversion to pinyin (1st Oct. 2000–1st Oct. 2001) No inclusion of tones Text division (syllable aggregation)

• Monosyllabic for common words (e.g., 东西 dong xi)• Polysyllabic for proper words (e.g., 上海 Shanghai)

Consequences: Two different methods used together

• Confusing!!! Only ~400 index “terms” available for all common

words• Too few!!!


Statement of the Problem

Conversion from Wade-Giles to Pinyin• Convert to monosyllabic?• Convert to polysyllabic?

Potential impact on retrieval / browsing Measure impact on retrieval

• Effectiveness (success in finding records)• Efficiency (effort spent to find them, i.e., time)


Research Questions

Determine if using polysyllabic pinyin entries, over monosyllabic pinyin entries, in bibliographic records improves retrieval effectiveness and efficiency in known-item exact-title searches.

Determine if using polysyllabic pinyin entries, over monosyllabic pinyin entries, in bibliographic records improves retrieval effectiveness and efficiency in known-item keywords-in-title searches.


Research Questions

In other words What is the effect of aggregation patterns on…

Six variables were defined Six hypotheses

Exact-title Keywords

Efficiency Q1 Q3

Effectiveness Q2 Q4


Research Questions

Definitions Exact-title search mode (with implied truncation)

Request for “Gone with the wind”

QUERY: “gone with the”

Keyword search modeRequest for “Gone with the wind”

QUERY: “wind” AND “gone”


Hypotheses

Effect of using polysyllabic transcription over monosyllabic Predictions

Efficiency Effectiveness Phrase Keywords Phrase Keywords


Methodology

Retrieval task:• Search 2 lists of 20 titles (in Chinese characters) using:

– Wade-Giles Romanization (WG)– Pinyin-monosyllabic Romanization (mPY)– Pinyin-polysyllabic Romanization (pPY)

• Replicate using two search modes:– Exact-title searching (phrase matching)– Keyword searching

• Measure:– Time to complete task (efficiency)– Number of items/records found (effectiveness)


Methodology: sampling

Purposive sample of 30 students• Graduate students• Native speakers of Chinese• Good working knowledge of Romanization

Each participant was given $20 CAN 30 participants × 2 tasks = 60 trials


Methodology: design and procedures

My main statistical design was a 2 × 3 randomized factorial design with unbalanced proportional data. Participants were replicated over factor A.

BA WG mPY pPY

X-title 6 12 12KW 6 12 12

BA WG mPY pPY

X-title µ11 µ12 µ13

KW µ21 µ22 µ23


Methodology: apparatus

20 titles × 2 lists = 40 titles 3 databases of ca. 50K records (RLIN db)

• WG / mPY / pPY Databases running on Microsoft Access Interface in HTML format accessed with Web

browser ASP links interface to database and records

transaction logs


Methodology: apparatus

Titles to be searchedID-

number

1. 颤栗 / 蒋伯潜 ____ — ____

2. 盐山新志：河北省 / 汪美瑞 ____ — ____

3. 生死场 / 顾宝民 ____ — ____4. 西藏那曲地区土地资源 / 施其明 ____ — ____


Transaction Log Analysis (TLA)

Components of TLA

Database

Logging Program

Inte

rfac

e

Logs

End-user

Methodology: data collection


Interaction with external software components

Internet

Internet Information ServerASP

Scripts

Win NT

SQL Server

HTML Files

OD

B C

ADO

ASPWWW



Internet

Internet Information ServerASP

Scripts

Win NT

SQL Server

HTML Files

OD

BC

ADO

ASPWWW

DatabaseLogging Program

Inte

rfa

ce

Logs

End-user

TLA

ASP



Generated logs



Statistical analysis

Exact-title KeywordsWG /mPY

WG /pPY

mPY /pPY

WG /mPY

WG /pPY

mPY /pPY

Completion time — —Time/item found — Expected Search Length — — — — Number of queries — — — —Success rate — — — —Success rate per query — — — — — —

Effi

cie

ncy

Eff

ecti

ven

ess


Results Aggregation improves search efficiency for title

searches Keyword search is especially influenced by

aggregation

Keyword search is especially important for Chinese titles since it is not unusual that the pronunciation of one of the first characters in the title is unknown

x-title Keywords

Efficiency Q1 Q3

Effectiveness

Q2 Q4


Results

Using mono- and polysyllabic aggregation concurrently is a great source of confusion to end-users

Retrieval with Romanization works relatively well

Success rates for know-item searches vary between 72% and 91% depending on Romanization system used


Results

However, for a non-negligible proportion of end users, Romanization-based retrieval poses real problems

Around 25% of the participants made between 50 and 80 Romanization errors during the retrieval task

0

1

2

3

4

5

6

7

0–4 10–14 20–24 30–34 40–44 50–54 60–64 70–74 80–84

Number of Errors

Num

ber

of P

arti

cipan

ts


Results Cause of errors

Aggregation• A1: Two unlinked units were linked

(e.g., dong xi / dongxi)

• A2: One linked unit was unlinked(e.g., Shanghai / Shang hai)

Romanization• R1: Character was misread

(e.g., 粟 su / 栗 li)

• R2: Romanization was misspelled(e.g., 林 was written ling instead of lin)


Results

Aggregation and Romanization errors

0,20,30,40,50,60,70,80,9

WG mPY pPY

(av

era

ge

pe

r q

ue

ry) Aggregation errors

Romanization errors


Results Wade-Giles notation, more “forgiving”

Wade-Giles Pinyin

chen / ch’en chen / zhen

chi / ch’i ji / qi

chu / ch’u / chü / ch’ü zhu / chu / ju / qu

… …


Results Analysis of Romanization errors

Romanization• R1: character misread (e.g., 粟 su /

栗 li)

• R2: Romanization misspelled• chen / cheng (dental nasal vs. velar nasal)• cu / zu (voiced fricatives vs. unvoiced fricatives)• hu / fu (glottal fricative vs. labiodental fricative)• la / na (alveolar lateral vs. alveolar nasal)


Results

Phonetic confusion

0

0,05

0,1

0,15

0,2

Fricatives Nasals Other

(avera

ge p

er

qu

ery

)


Conclusions

1. In KW mode, polysyllabic entries help improve precision

2. More aggregation errors in polysyllabic, but overall not overwhelming

3. Dual aggregation format is confusing to end-users (important source of error)

4. Still relatively high proportion of errors caused by confusion in Romanization


Further research Project #1

Using vernacular script for retrieval• What model???

Query DataInput

Romanization

Other input method

Romanization

汉字

Romanization

汉字


Further research Project #2

Using XML to encode non-Roman bibliographic data

• What is the viability of XML as a conversion format for bibliographic records containing non-Roman data?

• How can we use existing conversion schema, for instance those developed at LC?

• Does XML offer the required flexibility for publishing non-Roman on the Web with enhanced retrieval capabilities?


Further research


Further research Benefits

Integration of resources created under a decentralized environment

Creation of specialized retrieval tools adapted to the specific nature of the data

Increased visibility for resources in non-Roman alphabets


Questions Thank you!

Documents

Handling Chinese-Language Bibliographic Data A North American Perspective Honk Kong Library Association, August 22 nd, 2003 Clément Arsenault, assistant