Performance of NC-ISC on Ohsumed

Ohsumed is a well known dataset consisting of 34389 documents classified among 23 classes (each document can belong to several classes). The documents use a vocabulary of 30689 words. Dataset split is a random 80/20%. I’ve used the dataset provided at the Alessandro Moschitti’s corpora webpage and more exactly the preprocessed data available on the Rate Adapting Poisson model site. Please note that I have not managed to reproduce the results (for LSI and TF-IDF) from The Rate Adapting Poisson (RAP) model for Information Retrieval and Object Recognition – Peter V. Gehler, Alex D. Holub and Max Welling, ICML 2006. A short email discussion with P. Gehler did not help much to understand the source of the difference. Anyway, the most interesting stuff reside in the difference between the raw data (TF-IDF) and the processed data.

Here are my results :

ohsumed

If you look at the results of the RAP paper, you’ll see that precision of TF-IDF tops at 0.84, and goes down to 0.66, while mine tops at 0.88 and goes down to 0.59. My methodology for counting an « answer » good or false is this : good if at least one class of the candidate is also a class of the query document, bad otherwise. It is either one or zero, no in-between. Comments about this are really welcome.

My other neighbors

Ok, I’ve got a lot of things to show you, but unfortunately, not muchtime to prepare the data. Here is another comparison between LSA and my method (NC-ISC), for the neighbors of the word « 200″. I know it’s not really a word, but at least it is easier to sort out the good neighbors from the bad ones with this kind of word, right ?
So here is the list, on the left the neighbors found by my method, on the right those found by LSI (or more precisely my faster implementation of LSI). I’ve highlighted the undisputable (I’ve been pretty liberal though) numbers/quantities in green, and the undisputable unrelated words in red. Related but unsure words where left unhighlighted. You can dispute me on any word however if you think I was wrong, my judgment is far worst than my system’s.

For LSI, the first completely unrelated word appears at 165, versus 427 for NC-ISC. In the last 100 neighbors, 11 are good candidates for LSI (some of them are really borderline,you judge) , versus 32 for NC-ISC.

These neighbors were obtained with the exact same method than in my previous post. No internal knowledge of the words composition was used (that would be cheating for such a problem).

By the way, thanks to my method, I’ve learned a new word : umpteen. It seems that LSI didn’t find it as a potential neighbor for « 200″.

NC-ISC LSI (ISC)
1 200 200
2 500 300
3 300 500
4 250 400
5 600 250
6 400 600
7 100 100
8 150 150
9 700 800
10 350 700
11 800 1000
12 120 350
13 1000 120
14 900 5000
15 60 900
16 450 450
17 140 1500
18 80 0
19 5000 750
20 70 140
21 50 4000
22 240 3000
23 850 650
24 750 160
25 40 850
26 550 7000
27 0 550
28 160 170
29 hundred 2500
30 170 50
31 180 180
32 thousand 240
33 130 1300
34 4000 sq
35 1500 80
36 230 thousand
37 fifty 130
38 30 1200
39 3000 hundred
40 720 60
41 7000 1400
42 1200 8000
43 125 3500
44 280 70
45 8000 million
46 2500 hundreds
47 560 125
48 90 40
49 75 560
50 35 110
51 220 2200
52 110 260
53 hundreds 90
54 forty 1600
55 fifteen 2400
56 680 280
57 65 thousands
58 seventy 680
59 270 230
60 ten 75
61 million millions
62 sixty 220
63 tens 10000
64 twenty 210
65 20 175
66 thirty cubic
67 50 460
68 420 480
69 225 2300
70 1300 billions
71 480 100 000
72 260 30
73 eighty 370
74 3500 950
75 960 360
76 100 000 275
77 360 380
78 570 1700
79 seven 1150
80 370 270
81 175 billion
82 330 225
83 275 330
84 eight 490
85 twelve fifty
86 45 50
87 five 320
88 460 820
89 210 960
90 10000 580
91 nine 390
92 1100 530
93 190 430
94 six 1100
95 sixteen 20
96 430 520
97 thirteen 135
98 950 620
99 650 6000
100 fourteen 190
101 ninety 770
102 millions trillion
103 375 310
104 440 860
105 15 440
106 660
107 1400 pmol
108 eleven 2000
109 48 9000
110 640 11m
111 224 tens
112 thousands 340
113 Fifteen 760
114 114 ½
115 10 µmol
116 dozens 730
117 880 per
118 Six
119 36 928
120 Thirty 625
121 620 980
122 530
123 156 315
124 930
125 770 880
126 Sixteen 25
127 Ten 265
128 980 610
129 72 ng
130 Five 640
131 2400 475
132 820 dozens
133 135 10
134 185 65
135 eighteen 670
136 172 690
137 340 375
138 seventeen 920
139 Fourteen 1800
140 Fifty forty
141 64 cu
142 540 40m
143 290 UNKNOWN
144 25 ¼
145 four 10m
146 470 2100
147 490 710
148 42 45
149 320 720
150 Seventeen
151 35
152 132 95
153 Thirteen Million
154 38 420
155 47 500m
156 66 Hundred
157 Four 325
158 115 twenty
159 55 675
160 660 1250
161 890
162 Three 425
163 63 60m
164 690 145
165 144 Tsu
166 Sixty 630
167 780 thirty
168 Eleven 185
169 Eighteen 250m
170 108 395
171 Eight pg
172 165
173 3 000 3 000
174 24 195
175 dozen 1m
176 Two 156
177 625 seventy
178 105 470
179 323 830
180 580 236
181 255 410
182 85 IU
183 18 sixty
184 134 kilo
185 Twenty 990
186 325 15m
187 155 ten
188 syllabic
189 three 478
190 106 µg
191 sq 165
192 186 570
193 422 212
194 265 numbers
195 825 970
196 154 725
197 112 510
198 cubic frac13
199 12
200 Twelve 290
201 37 pesetas
202 145 Macoumba
203 33 Fundacio
204 14 Alessandria
205 Nine 840
206 34 429
207 84 fifteen
208 58 5m
209 168 GREENPEACE
210 Forty 875
211 46 24
212 102 100000
213 121 Ludovit
214 740 FRINGE
215 630 bakings
216 Fewer 6m
217 13 MILLION
218 billions 132
219 trillion 647
220 57 yuan
221 380 Pilmé
222 39 Getachew
223 16 Deployment
224 410 780
225 22 114
226 32 mg
227 327 25m
228 174 eighty
229 umpteen francs
230 710 Neally
231 365 FEARFUL
232 28 burble
233 333 115
234 billion Varsha
235 920 Scotches
236 432 McClanahan
237 670 reducibility
238 195 Familiale
239 1250 Accounts…
240 273 combo
241 number 450g
242 41 roack
243 118 objectifications
244 couple kukri
245 208 355
246 Seven Habara
247 131 foody
248 2200 Aeroengines
249 77 Hatu
250 62 Elicitation
251 128 Arigoni
252 17 silicite
253 990 Rodent
254 760 16 March 2005
255 610 174
256 78 164
257 171 schoolchilden
258 52 neuropsychologically
259 369 Mrs Hall
260 349 scorch
261 two Worpington
262 754 Electroacupuncture
263 ½ Checkers
264 Several Kanezo
265 297 Feakle
266 520 Curtlingside
267 196 623
268 54 343
269 123 15
270 136 PURPOSEFUL
271 88 Kjellstrand
272 100000 ILPEC
273 545 17m
274 few mmol
275 453 540
276 116 DRS
277 334 dollars
278 157 3bn
279 326 heah
280 68 bodyplans
281 488 Aggressive
282 940 ‘legislative
283 146 Judicature
284 266 Delight…
285 204 CFRS
286 830 benserazide
287 73 2m
288 970 reconaissance
289 95 MASSINGBERD
290 fewer Azinger
291 56 lizzies
292 930 Hooraydom
293 ¾ bovril
294 395 549
295 860 Theognidea
296 890 Monstrosities
297 67 Democratically
298 Ninety Per
299 295 MPhys
300 cu l919
301 Eighty 12VDC
302 53 actio
303 256 SHE…
304 390 PUERTO
305 381 ‘Restructuring
306 104 WILLNER
307 51 unrebellious
308 74 Ciriaco
309 Hundred steaming…
310 178 DEPC
311 425 CUTE
312 324 niobium
313 several Eicken
314 465 DEPLOYMENT
315 268 Poulsen
316 810 NED
317 875 farme
318 549 Fanelli
319 122 contemporaneo
320 478 litres
321 614 Sutker
322 2300 reconfirmation
323 310 DDSI
324 385 105
325 43 Fewer
326 124 Kulturmuhle
327 192 compo
328 69 Aberration
329 19 353
330 82 Trivulziana
331 26 Oultram
332 227 Hurdles
333 475 NIPPV
334 164 Kalmu
335 many fighting for
336 317 Djer
337 575 slctd
338 138 pbk
339 21 HCo
340 27 GNCTU
341 61 copyrights
342 336 pp69
343 259 overrepresentation
344 numbers ingrafted
345 76 ¾
346 per Quand
347 44 nonhegemon
348 512 Ings
349 732 Caesar…
350 142 WeeKly
351 86 vexes
352 1600 swingy
353 126 Novgorodsev
354 999 heteromorphy
355 169 Bouffant
356 handful Wyre
357 113 Twentysomething
358 2 Mondi
359 445 Fastnets
360 49 Skadar
361 485 seacliffs
362 590 QualityTeam
363 79 Maystadt
364 184 FOZZ
365 525 forminant
366 139 ‘Trust
367 107 productivity…
368 11 Emptor
369 Many Bergstrand
370 31 organum
371 103 Fortunes
372 456 21sst
373 372 614
374 148 newletter
375 29 happiness…
376 pesetas 430905
377 198 Elongated
378 ¼ Charolais…
379 153 Carolana
380 167 WCM
381 243 poireau
382 frac13 HRBMOA
383 nineteen arabicum
384 443 synnoetics
385 840 heatedly
386 215 Davros
387 235 Bundesnachrichtendienst
388 345 300m
389 355 wicca
390 87 Schiemann
391 267 Assante
392 1150 Panitch
393 152 imparters
394 Seventy Hueber
395 133 Staart
396 147 Ponce
397 grams BUTLIN
398 257 45753
399 Few urophthalmus
400 231 LIMAVADY
401 188 Crina
402 Some Diprotodon
403 461 Clune
404 474 Caradog
405 870 WONKY
406 287 posessing
407 288 horseshoes
408 quarter Thornett
409 137 decolonisation
410 tenths 506bn
411 All 131
412 81 RAIS
413 399 Parozzo
414 247 Camesi
415 thirds ASLEF
416 161 Stockmayer
417 countless opera…
418 1m ALton
419 462 MONSTROUS
420 127 archduke
421 9000 40m²
422 0 Munding
423 11m Troup
424 59 hatinhensis
425 313 Brunmayr
426 101 Shiban
427 evacuate Masheke
428 557 discographer
429 366 1340s
430 358 unsold
431 477 Marleigh
432 283 IRONIC
433 143 depletes
434 109 warthog
435 99 Phillies
436 francs 142g
437 294 Wodemed
438 149 Quix
439 244 l906
440 458 quadrathon
441 SEVEN Eddings
442 6m discorporate
443 441 Mr Yáñez
444 285 mesenteric
445 98 Lumley…
446 83 businesswise
447 23 RSPP
448 counting Defreyn
449 5 37318
450 among Patches
451 94 the new
452 472 MultiWorks
453 111 institutionalization
454 8 cretaceous
455 675 Dosedel
456 183 Bornfreund
457 209 Azeri
458 548 ‘bridge’
459 amongst Scoope
460 119 sacrementality
461 majority Kibble
462 307 megawatts
463 141 740
464 5m 940
465 pmol neurobiology
466 205 Killermont
467 60m Jehu
468 75p vookodlaks
469 129 Codmans
470 159 climsy
471 unarmed JS
472 228 877
473 412 Versus
474 FOUR Medlicot
475 202 MARGATE
476 218 Informalism
477 318 EUR 307
478 162 Esbjerg
479 189 vis
480 666 Yetminster
481 795 Periglacial
482 181 criminal…
483 500m Tamaris
484 269 activi
485 kilo Ache
486 261 zig
487 Most HP15
488 9 Contravention
489 8m 7368
490 495 pigdog
491 315 Munchkin
492 467 HyperStream
493 633 327
494 316 586
495 468 WES
496 301 sailskiffs
497 underpaid Daaad
498 455 55mm
499 429 Gudent
500 277 GenBank
501 262 CONTRIBUTES
502 241 Upadhyay
503 216 Safranski
504 389 CAPITALS
505 499 Carbofuran
506 359 apostasy
507 335 198741
508 222 ‘undertake
509 308 432
510 plus Nescafe
511 remaining Jaen
512 cents Dacorum
513 298 COMPOTE
514 Those Stegs
515 Nineteen Petunia
516 litres gergasites
517 585 Poetry…
518 647 Discussants
519 309 795
520 92 towndwellers
521 877 safetly
522 2100 Bodenham
523 217 p64
524 centimetres ligate
525 211 IR81
526 442 Lasts
527 FIVE egoistical
528 89 Callosum
529 245 Willkommen
530 other Sesto
531 254 Frederikstraat
532 312 Satcoms
533 Other Khubthi
534 435 Engwirda
535 pence Wierdsma
536 353 KEF
537 364 deathshead
538 recruit managerialism
539 203 Lamarckenis
540 24 23220
541 registered Wilcocks
542 unsold proabortion
543 538 bussed
544 236 pregnancies…
545 6000 Partan
546 446 viewcam
547 487 TANNETT
548 282 PREZIOSE
549 925 M67
550 212 hymophylus
551 fewest Claro
552 253 1 000 amendments
553 innumerable palmettes
554 436 enchantments…
555 605 aqualungs
556 96 Litchet
557 45p Hobor
558 296 822937
559 1800 GUMS
560 623 DISCO
561 3m Clobbered
562 displaced gamey
563 THREE Stimson
564 individual Molua
565 6 LIDBA
566 626 Swingland
567 278 superspy
568 7 forwardness
569 15p pence
570 consecutive 999
571 449 Taraki
572 2m sovkhoz
573 237 despairingly…
574 187 grams
575 367 Wallacean
576 guineas sellability
577 272 Lymphomatous
578 176 impersonate
579 7m vrroom
580 sending Nekemte
581 extra Egyptologists
582 173 punishingly
583 pg metaborate
584 219 Arnolds
585 proportion Won
586 1700 Perikli
587 331 overmorphinisation
588 928 Buitoni
589 plight metric
590 229 Urchin
591 numerous Daiichi
592 705 Benumbed
593 Among Maravilla
594 Individual hakweed
595 574 Aloisia
596 158 Thoc
597 exodus Schönborn
598 91 Borringdon
599 Whadcoat riot…
600 half premonitionary
601 shillings Llanfairpwll
602 393 Schantz
603 213 Sagna
604 536 brittling
605 454 RLL
606 those UNFPA
607 3bn brare
608 221 Anghiari
609 531 1666mph
610 TWO RNAs
611 CFA Modulyo
612 Numerous Irreverent
613 510 ABORTIVE
614 179 countless
615 milligrams Turbocharging
616 influx steins
617 343 NEUDESK
618 massacred 5
619 421 Rearmament
620 civilian faoil
621 millionth 16H
622 337 Spinnakered
623 repatriation Joyrider
624 382 636
625 7p Rps40
626 995 peintre
627 166 dramatherapy
628 all Strandhill
629 151 Bahriye
630 246 52085
631 MgCl weightiness
632 metric UNRG
633 60p Orace
634 hiring Currency
635 163 Narses
636 555 Melanocytes
637 730 autorepression
638 older 1470
639 pounds 295
640 slaughtering scatterbrained
641 319 naturalisation
642 selected Incise
643 117 chromomere
644 outnumbered SEDISI
645 562 Morua
646 419 maximands
647 311 Abbs
648 414 Clanwilliam
649 assorted Chickens’
650 Generally Ayra
651 DM1 propitiated
652 304 Esdec
653 457 Dabo
654 354 shamateur
655 4 Panathanaikos
656 attract chirp
657 shilling Keetmanshoep
658 715 Huttwil
659 405 delevopment
660 258 semidarkness
661 426 piffle
662 sell Dourif
663 kilowatt Nemerov
664 314 Mr Mejdahl
665 289 proprietorships
666 276 Jonglei
667 586 Characterized
668 scores 1550
669 MILLION Sisterson
670 glut Rowlinson
671 recruiting Kennaugh
672 274 Radically
673 417 enameller
674 14m Emmons
675 assembled Worksong
676 cent UNDCP
677 206 Coproduction
678 ordinary Bradypus
679 disabled lolly
680 248 Glase’s
681 outnumber Cinncinatti
682 388 Bayerischer
683 451 Slovaks
684 ferrying thoroughfares
685 517 Lothiah
686 µmol concious
687 699 Beltelecom
688 these Discordiale
689 293 Attanasio
690 x antisepsis
691 398 Luttman
692 surviving legitimising
693 339 36650
694 kg ‘standards’
695 shortlisted Secrétaire
696 97 Forgandenny
697 4m Ailsi
698 marshalling throughways
699 775 Pilce
700 444 kagoul
701 These Parasite
702 322 Blucher
703 250m 5kyr
704 send 538
705 kilograms Sylvio
706 EIGHT Pics
707 601 Albuhera
708 252 Kaufmann’s
709 199 ductile
710 71 Compañia
711 40m Bancario
712 376 Vigils
713 Older Chromex
714 338 Casely
715 424 RENNIE
716 shortage inclusivism
717 µg ESAG
718 418 salbutamol
719 starving price’
720 freeing polychromism
721 735 Angeleno
722 264 nematomorphs
723 344 Mr Santoro
724 232 Elandre
725 shelters Belau
726 448 Sagep
727 Interested ILLEGAL
728 employing ‘way
729 391 Woodeson
730 mistreatment biquaternions
731 0d 71000
732 533 7m
733 25p Spotless
734 347 laserballs
735 279 anteater
736 286 Senneh
737 expelling competitive – the
738 5 840555
739 413 Sartilly
740 fellow Lochassynt
741 lots 9094
742 involving Rande
743 Thousand LU1
744 8bn enuretics
745 617 delighteth
746 2p crick
747 725 colourations
748 263 499
749 xx viceless
750 attracted Gyrene
751 368 crutchless
752 paid Papple
753 15m Issachar
754 3 Ginevra
755 ranking Favart
756 smaller Enough…
757 protect Beechey
758 foreign 365
759 Kosovar SPP
760 299 Packed
761 Ugandan gesetzlicher
762 weighs cp16
763 409 Klinck
764 529 Burnout
765 40s 130pc
766 194 Seahorse
767 197 folastre
768 1450 floristics
769 576 tempeh
770 30m Shucard
771 hire Hicolour
772 separate woodfiller
773 disaffected TODD
774 selling Hellesdon
775 Younger ADPRT
776 allowing Tickel
777 attracting ripon
778 insuring Raarsa
779 25m Voorhies
780 various Duosome
781 accommodate DOONEY
782 685 zonkered
783 193 Dol
784 201 4PR
785 fifths languages’
786 equipping garda
787 massacre ALUM
788 2 RODDY
789 3 McNair
790 284 exception – and
791 bogus arcmin
792 supply 215
793 242 simpatico
794 forcing Luckhaus
795 378 400m
796 migrant indicción
797 371 Beschantnikh
798 374 190E
799 dollars Spen
800 thousandth saxed
801 TEN Kabilia
802 561 Viccei
803 Both georgoi
804 489 Eurom
805 Previous Sach
806 2000 resussitated
807 300m Atrypa
808 carrying Weighbridge
809 624 Glenann
810 devalued Disengagement
811 some MexTech
812 9m Ghandour
813 342 dehumanises
814 seamen Stetchworth
815 4d Laxolt
816 505 autoflight
817 75m 35m
818 successive Qila
819 Per Previati
820 636 class – the
821 volunteer Kutubu
822 allow frölich
823 enable CASCADE
824 637 Retainer
825 768 aid…
826 mostly 14282
827 239 ravage
828 403 Quadraphony
829 Experienced Emiliani
830 generation adpressed
831 pou Oxygene
832 363 Outlaws
833 µ greyscales
834 km scheme’s
835 counterfeit francaise
836 249 coreceptors
837 placing take’
838 508 Rhie
839 SIX Pentathlon
840 urging unlatch
841 35m oozy
842 50p Bilancetta
843 nought RS
844 handicapped square
845 luring freedwomen
846 ng Entertain
847 397 Chairmanships
848 buying rupees
849 226 Teachta
850 303 multinucleate
851 433 720
852 fours TEXTB
853 177 INR1
854 including bladdered
855 educate Supercard
856 fide spirtu
857 628 aîné
858 FEW subdeacon
859 refit iubebunt
860 30p 2567
861 needy Wordmarc
862 preponderance Radomiro
863 Chinese Mynours
864 Sending Ferrovie
865 between DG Enterprise
866 rupees 8931977
867 younger STUN
868 escorting JURA
869 expulsion Arrigo
870 476 Thinkabout
871 overseas Sobers
872 enables Petrakis
873 eligible beruhrt
874 slaughter Directive 89
875 Smaller Astakakalij
876 911 Ahs
877 sold patronize
878 widows mulattas
879 approximately Go1
880 stealing ‘stealth
881 348 Vanbutchel
882 DM CONCUSSION
883 10m 75 million
884 498 20m
885 list Narrations
886 includes Forehead
887 defenceless Ayissi
888 incoming Uhlig
889 452 Hospices
890 281 disipal
891 93 Weenie
892 alternate Hibiya
893 0 clothworkers
894 ohms 5×3
895 buys rearfoot
896 748 PNS
897 quarters chloridel
898 fives unaspiring
899 352 r7A
900 unemployed King Gyanendra
901 harassing Honiton
902 stateless meeker
903 millimetres instantiates
904 local encamping
905 both tryst
906 battleground Phromnia
907 missing Jacopi
908 retrain 60
909 mega spiffingly
910 GBP Pleated
911 Sinhalese Girod
912 milliseconds fireproof
913 416 Stoelting
914 NINE Matsuura
915 minority ignoring…
916 351 disparage
917 taxpayers’ Konalyck
918 imprisoning fretwire…
919 428 39071
920 394 Thurlby
921 range Manjil
922 available Huskey
923 532 5Mbps
924 whereby Kosovo – and
925 destitute Béatrix
926 271 Alave
927 excluding unled
928 untrained NgendabanyikwaPublic
929 411 7F
930 swap Schneidau
931 423 Haemorrhoids
932 indigenous Chenal
933 employ Truffe
934 sample anticlericalism
935 832
936 generations 37936
937 BEd km
938 series rights – physical
939 fined DISORDERS
940 Wolds Antapologia
941 214 Urania
942 transporting Polymethylmethacrylate
943 comprising Häider
944 bringing reimbursements
945 10p Manzù
946 292 donfirm
947 urged migrating
948 putting tosea
949 Currency Kettner
950 British Intaglio
951 524 Saez
952 homeless Ladyland
953 unoccupied Handcrafts
954 Certain C40103
955 intimidate Byne
956 404 archways
957 Potential Moorby
958 Asian kindergartener
959 400m Behaved
960 wealthier premelting
961 troublesome JULIO
962 493 brownings
963 attracts Hooded
964 377 yomp
965 24m Sherleys
966 sells esn
967 charging Juvenus
968 329 GARRY
969 sized Refolding
970 luckier Hrachya
971 4p Extensification
972 lists Tramezaïgues
973 Hooded Sahlin
974 615 enslave
975 innocent COLOURCODE
976 inundated shearography
977 recalcitrant Parabroteas
978 scattering Melocomic
979 penny SDR87
980 collection Kuttner
981 631 erinacea
982 additional fewer
983 402 NIMCO
984 hauling Minister Zenawi
985 visiting biggish
986 priced 3000lb
987 closing 4m
988 790 512
989 group mistreatment
990 young 1680
991 costing Nobber
992 387 Wishbone
993 567 reformlet
994 listing DNASTAR
995 bribed Balcombe
996 638 saccharine
997 471 Monteviot
998 sledges aulacogens
999 enabling 259p
1000 units superpoint
1001 686 manganiferous
1002 lire bovis
1003 Croats Pak
1004 buy productson
1005 intercepting oxidised
1006 Leading Koki
1007 sixes 9mJy
1008 729 Ovocný
1009 Separate Affane
1010 multiple abdicates
1011 17m Risi
1012 trained MARTON
1013 687 Knappertsbusch
1014 collected Concord
1015 707 carbocations
1016 Sahrawi mastics
1017 Larger Curit
1018 504 alleluias
1019 release Swankers
1020 pound DAYLIGHTS
1021 deter Coupons
1022 patrolling potsh
1023 persecuting Nedbank
1024 191 Bimbim