Assembling a Balanced Corpus from the Internet
Johan Dewe, Jussi Karlgren, Ivan Bretan · DSpace repository (University of Tartu) · 1998
re s e a rc h it is c ru c ia l to h a v e m a te ria ls a v a ila b le fo r e x tra c tio n o f s ta tis tic s, tra in in g p ro b a b ilis tic a lg o rith m s, a n d te stin g h y p o th e se s a b o u t la n g u a g e a n d la n g u a g e p ro c e s s in g in g en era l.In re c e n t y e a rs, th e a w a re n e ss th a t te x t is n o t j u s t te x t, b u t th a t te x ts c o m e s in se v e ra l fo rm s, h a s sp re a d fro m m o re th e o re tic a l a n d lite ra ry su b fie ld s o f lin g u istic s to th e m o re p ra c tic a lly o rie n te d in fo rm a tio n re trie v a l a n d n a tu ra l la n g u a g e p ro c e s sin g fields.A s a c o n se q u e n c e , s e v e ra l te s t c o lle c tio n s a v a ila b le fo r re s e a rc h e x p lic itly a tte m p t to c o v e r m a n y o r m o s t w e ll-e sta b lish e d te x tu a l g e n r e s , o r fu n c tio n a l s ty le s in w e ll-b a la n c e d p ro p o rtio n s (Fi a n c is a n d K u c e ra , 1982; K ä llg re n , 1990).T h e c re a tio n o f s u c h a c o lle c tio n is a c o m p le x m a tte r in se v e ra l re sp e c ts. O u r re se a rc h a re a is to b u ild re trie v a l to o ls fo r th e In te rn e t, a n d th u s, fo r o u r p u rp o se s, th e c h o ic e o f g en res to in c lu d e is o n e o f th e m o re c e n tra l p ro b le m s : th e re is n o w e ll-e sta b lish e d g e n re p a le tte fo r In te rn e t m a te ria ls.T o fin d m a te ria ls to e x p e rim e n t w ith , w e n e e d to c re a te th e m in a fo rm su ita b le fo r o u r p u rp o s e s .T h is is a d o u b le e d g e d p ro b le m , in v o lv in g b o th v a g u e ly e x p re s s e d u s e r e x p e c ta tio n s a n d e sta b lis h in g c a te g o rie s u s in g la rg e n u m b e rs o f fe a tu re s w h ic h ta k e n sin g ly h a v e lo w p re d ic tiv e a n d e x p la n a to ry p o w e r.T h is p a p e r g iv e s a n o u tlin e o f th e m e th o d o lo g y w e u s e fo r d e te rm in in g w h ic h g e n re s to in c lu d e .Stylistic Variation and Genre T e x ts e x h ib it c o n s id e ra b le v a ria tio n .W h ile th e v a ria tio n in to p ic o r c o n te n t is q u ite o b v io u s a n d th e b a s is fo r m o s t c a te g o riz a tio n e n te rp rise s in in fo rm a tio n re trie v a l re se a rc h v a ria tio n in sty le is a s n o tic e a b le , a n d fo rm s a s e c o n d b a s is fo r c a te g o riz a tio n : p o e try , p ro s e , n o n -fic tio n , re fe re n c e m a te ria ls , a n d s o fo rth a re a ll s ty listic c a te g o rie s o r g en res.S ty lis tic v a ria tio n s h o w s th ro u g h s ty lis tic item s: o b s e rv a b le c h o ic e s o f lin g u istic ite m s. S ty lis tic ite m s c a n b e o b se rv e d o n a n y le v e l o f lin g u istic a b s tra c tio n : le x ic a l, fo r th e c h o ic e b e tw e e n w o rd s o f sim ila r m e a n in g b u t d iffe re n t c o n n o ta tio n s; sy n ta c tic , f o r th e c h o ic e b e tw e e n e q u iv a le n t c o n stru c tio n s w ith d iffe re n t c o m m u n ic a tiv e im p o rt; te x tu a l, fo r d e c isio n s o f te x tu a l o rg a n iz a tio n .E a c h sty listic ite m is o f little im p o rt, b u t ta k e n to g e th e r th e y a re in d ic a tiv e o f sy ste m a tic d iffe re n c e s.A s e t o f d o c u m e n ts \v ith a p e rc e iv e d c o n s is te n t te n d e n c y to m a k e th e sa m e s ty listic 100