https://web.stanford.edu/class/cs276/pa/pa2.pdf
語(yǔ)料：
lm corpus： 99,904 documents
query corpus： 819,722 編輯距離最多為1
Levenshtein automaton
比較清晰的ppt：
http://web.stanford.edu/class/cs276/handouts/spell_correction.pdf
目前的做法：
image.png
提升的方法：
image.png
額外的加分項(xiàng)：
- 考慮編輯距離在1以上的情況
- 除了斯坦福網(wǎng)站的語(yǔ)料朴沿，嘗試其他語(yǔ)料
- 訓(xùn)練語(yǔ)言模型的時(shí)候撬腾，考慮其他的平滑方式，例如 Kneser-Ney smoothing
- K-gram index
- Levenshtein Automata：uses a finite state automata for fuzzy matching of words
git：https://gist.github.com/Arachnid/491973
blog：http://blog.notdot.net/2010/07/Damn-Cool-Algorithms-Levenshtein-Automata
https://github.com/aitounejjar/pa2-Spell-Corrector
https://github.com/pangolulu/spelling-corrector
Moore 文章語(yǔ)音+拼寫model
- Toutanova K, Moore R C. Pronunciation modeling for improved spelling correction[C]//Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics, 2002: 144-151.
Moore 之前的文章：拼寫model
- Brill E, Moore R C. An improved error model for noisy channel spelling correction[C]//Proceedings of the 38th Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics, 2000: 286-293.
引用了Moore的文章
- Martins B, Silva M J. Spelling correction for search engine queries[M]//Advances in Natural Language Processing. Springer, Berlin, Heidelberg, 2004: 372-383.
- Sun X, Gao J, Micol D, et al. Learning phrase-based spelling error models from clickthrough data[C]//Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2010: 266-274.
-
Multi-level feature extraction for spelling correction
- Wilcox-O’Hearn A, Hirst G, Budanitsky A. Real-word spelling correction with trigrams: A reconsideration of the Mays, Damerau, and Mercer model[C]//International conference on intelligent text processing and computational linguistics. Springer, Berlin, Heidelberg, 2008: 605-616.
Gao J, Li X, Micol D, et al. A large scale ranker-based system for search query spelling correction[C]//Proceedings of the 23rd International Conference on Computational Linguistics. Association for Computational Linguistics, 2010: 358-366.
