Practice of Natural Language Processing

Objectives

• implementing an example code about natural language processing for C++
• only for Korean language
• practicing chapter 6 of the book descripted on https://github.com/freelec/ai-math-book

Needs

• extracting morphemes
• making bag-of-words
• removing stop-words
• caculating TF-IDF
• MSE or cross-entropy loss function
• making 'Category Classifier'
• cross-validation
• ...

Achieved

• extracting morphemes using mecab-ko
  • https://github.com/Pusnow/mecab-ko-msvc
• making bag-of-words
• removing stop-words
• caculating TF-IDF
• ...

Running Order

1. test/morpheme-extraction-test
2. test/bag-of-words-test
3. test/stop-words-removal-test
4. test/tf-idf-test
5. ...