Extract the vocabulary from the Inuktitut Corpus

[GoogleCodeExporter]

Purpose of addition of this task:
We need a list of (probable words) for any task dealing with Inuktitut. To get 
the list we plan on using Inuktitut Magazine (since it is representative 
literature). We would like to extract all the words from the magazine, along 
with their frequency information which can give a clue as to their function vs 
content word status, and their likelyhood when spell checking a document or 
doing any predictive task. This word list will be further processed to find 
roots for the Inuktitut Search, and to find morphemes for the Inuktitut spell 
checker.

After the review, please add a Inuktitut Magazine section to Inuktitut Corpus 
wiki page.
*detailing the number of words per language in the corpus
*detailing the average number of words per file
*maybe some statistics about how dense the vocabulary is per file (ie dense or 
diverse vocab)
*Discussing the quality and drawbacks of the corpus

After the review, the expected next step is to:
*Generate jape grammars to highlight words in context 
*Work on morphological parsing of the vocab

Original issue reported on code.google.com by [email protected] on 9 Nov 2011 at 10:34

• Blocked on: Write a script to extract text from Inuktitut Magazine #13

[GoogleCodeExporter]

Original comment by [email protected] on 10 Nov 2011 at 12:32

• Changed state: Started

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

The wordlist from InuktitutMagazine vol 77-104 
https://github.com/cesine/CorpusInuktitut/blob/master/gen/InuktitutMagzine_word_
frequencies_matrix_by_document.csv

The script that was used to create the matrix:
http://code.google.com/p/ilanguagelab/source/browse/src/com/fieldlinguist/groovy
/ExtractTermFrequenciesMatrix.groovy

The script that was used to create the word frequency files
http://code.google.com/p/ilanguagelab/source/browse/src/com/fieldlinguist/groovy
/ExtractWordsOrderBySuffix.groovy

Original comment by [email protected] on 25 Nov 2011 at 4:48

• Changed state: Fixed

[GoogleCodeExporter]

Original comment by [email protected] on 25 Nov 2011 at 10:25

• Added labels: Type-Implementation