nc-isc

I made a big mistake in my last post about our results improvements, the precision/recall curves of our experiments were in “gain” mode, that is, the curve tend toward zero as it approach the precision of a random selection. As a result, instead of finishing at about 5% precision (for a 20 classes problem with balanced classes, that’s what one should obtain). So our actual result are even slightly better than what you saw in my previous post.

More importantly, I’ve included here another variant of semantic knowledge injection, this time using bi and trigrams in addition to normal words (for instance, “mother board” is now considered as a unique vocabulary item) – the bi and trigrams are automatically selected.

The vocabulary size climbs from about 60K to 95K, with many grammatical constructs (“but if” and such), and the results are quite amazing : an improvement of precision of 4-5% on the left, and about +10% precision toward the 0.1 recall mark.

This is probably a more important result than what we’ve achieved with class knowledge injection, since class knowledge is rarely available in real world problems (in the case of product recommendation, you don’t know the class of your users, for instance).

The fact that our method, using only inner knowledge, can outperform, up to the 2% recall mark, the Deep Autoencoders using class knowledge, is probably more important than any result I could have only using class information.

)

Ohsumed is a well known dataset consisting of 34389 documents classified among 23 classes (each document can belong to several classes). The documents use a vocabulary of 30689 words. Dataset split is a random 80/20%. I’ve used the dataset provided at the Alessandro Moschitti’s corpora webpage and more exactly the preprocessed data available on the Rate Adapting Poisson model site. Please note that I have not managed to reproduce the results (for LSI and TF-IDF) from The Rate Adapting Poisson (RAP) model for Information Retrieval and Object Recognition – Peter V. Gehler, Alex D. Holub and Max Welling, ICML 2006. A short email discussion with P. Gehler did not help much to understand the source of the difference. Anyway, the most interesting stuff reside in the difference between the raw data (TF-IDF) and the processed data.

Here are my results :

If you look at the results of the RAP paper, you’ll see that precision of TF-IDF tops at 0.84, and goes down to 0.66, while mine tops at 0.88 and goes down to 0.59. My methodology for counting an “answer” good or false is this : good if at least one class of the candidate is also a class of the query document, bad otherwise. It is either one or zero, no in-between. Comments about this are really welcome.

Ok, I’ve got a lot of things to show you, but unfortunately, not muchtime to prepare the data. Here is another comparison between LSA and my method (NC-ISC), for the neighbors of the word “200”. I know it’s not really a word, but at least it is easier to sort out the good neighbors from the bad ones with this kind of word, right ?
So here is the list, on the left the neighbors found by my method, on the right those found by LSI (or more precisely my faster implementation of LSI). I’ve highlighted the undisputable (I’ve been pretty liberal though) numbers/quantities in green, and the undisputable unrelated words in red. Related but unsure words where left unhighlighted. You can dispute me on any word however if you think I was wrong, my judgment is far worst than my system’s.

For LSI, the first completely unrelated word appears at 165, versus 427 for NC-ISC. In the last 100 neighbors, 11 are good candidates for LSI (some of them are really borderline,you judge) , versus 32 for NC-ISC.

These neighbors were obtained with the exact same method than in my previous post. No internal knowledge of the words composition was used (that would be cheating for such a problem).

By the way, thanks to my method, I’ve learned a new word : umpteen. It seems that LSI didn’t find it as a potential neighbor for “200”.

In this post I show you a few (sort of) neighbours of “yellow”. I compare the neighbourhood of the word vectors obtained from LSI (or more precisely my fast variant of LSI) and using my method “NC-ISC”. The original cooccurrence matrix has been obtained from a merge of Europarl corpus and the BNC. No preprocessing have been applied to the corpus, not even lowercasing.

The vocabulary size is about 520000, the matrix has been transformed with positive pointwise mutual information.

The cooccurrence relation is simply “the next word”. Other relations could be used (including classical word windows), but I have chosen to show you this relation because it seems to be the most basic possible.

To sum up, the 520000*520000 cooccurrence matrix containing about 25M entries has been transformed into a 192 features-words dictionary.

Here are the (long, sorry) lists obtained

• on the left, using my method (NC-ISC)
• on the right, using the equivalent of a simple LSI (i.e. a thin SVD)

Things are getting a bit weird with LSI from the 100th row, approximately. On the other hands, NC-ISC keeps listing colors, aspects (visual then physical), for quite a long time. I’m not a native English speaker, so if you want to list the “acceptable” neighbours from the bad ones, you’re welcome !