本篇文章是舊版本的訓(xùn)練教程氛驮，適應(yīng)于新版本的訓(xùn)練流程請(qǐng)查看：http://www.reibang.com/p/7a2c40dd6560
題庫(kù)特殊字符語(yǔ)言包訓(xùn)練流程
題庫(kù)中會(huì)出現(xiàn)很多Tesseract無(wú)法識(shí)別的特殊字符腕柜，例如：≤ ≥ ×等等济似。本教程旨在訓(xùn)練新的語(yǔ)言包改善這些特殊字符的識(shí)別效果。訓(xùn)練思路為盏缤，利用Tesseract認(rèn)識(shí)的字符替代這些特殊字符(與特殊字符外形相近砰蠢，比如，‘,’替代‘唉铜，’台舱，或者題庫(kù)中沒(méi)有的字符替換原有字符)。
準(zhǔn)備訓(xùn)練依賴項(xiàng)
git clone https://github.com/tesseract-ocr/tesseract.git
git clone https://github.com/tesseract-ocr/langdata.git
git clone https://github.com/tesseract-ocr/tessdata.git
cp ./tessdata/eng.traineddata ./tesseract/tessdata
cp ./tessdata/chi_sim.traineddata ./tesseract/tessdata
下載tesseract包后需要本地安裝潭流，可看前面發(fā)布的安裝教程：Tess4.0 windows編譯與使用竞惋，tesseract linux 安裝與編譯。
添加新的字體
打開(kāi)./training/language-specific.sh文件灰嫉，在CHI_SIM_FONTS欄目下添加Times New Roman字體內(nèi)容拆宛，添加后內(nèi)容如下：
CHI_SIM_FONTS=( \
"AR PL UKai CN" \
"AR PL UMing Patched Light"
"AR PL UKai TW" \
"AR PL UMing TW MBE Light" \
"AR PL UKai Patched" \
"Arial Unicode MS" \
"Arial Unicode MS Bold" \
"WenQuanYi Zen Hei Medium" \
"Times New Roman, Bold" \
"Times New Roman, Bold Italic" \
"Times New Roman, Italic" \
"Times New Roman," \
)
外形相近的字符替換
應(yīng)用Times New Roman字體中現(xiàn)有的字符替換外形相近的字符，見(jiàn)以下是替換代碼：
text_line.replace('（','(').replace('）',')').replace('﹣','-').replace('．','.')\
.replace('：', ':')
替換其他特殊字符
訓(xùn)練流程
第一步
將文件training/tesstrain.sh復(fù)制備份training/tesstrain0.sh讼撒，并在tesstrain0.sh中將54行以下的內(nèi)容添加注釋浑厚，注釋后的代碼為:
source "$(dirname $0)/tesstrain_utils.sh"
ARGV=("$@")
parse_flags
echo $WORKSPACE_DIR
mkdir -p ${TRAINING_DIR}
# tlog "\n=== Starting training for language '${LANG_CODE}'"
#
# source "$(dirname $0)/language-specific.sh"
# set_lang_specific_parameters ${LANG_CODE}
#
# initialize_fontconfig
#
# phase_I_generate_image 8
# phase_UP_generate_unicharset
# phase_D_generate_dawg
# if ((LINEDATA)); then
# phase_E_extract_features "lstm.train" 8 "lstmf"
# make__lstmdata
# else
# phase_E_extract_features "box.train" 8 "tr"
# phase_C_cluster_prototypes "${TRAINING_DIR}/${LANG_CODE}.normproto"
# if [[ "${ENABLE_SHAPE_CLUSTERING}" == "y" ]]; then
# phase_S_cluster_shapes
# fi
# phase_M_cluster_microfeatures
# phase_B_generate_ambiguities
# make__traineddata
# fi
#
# tlog "\nCompleted training for language '${LANG_CODE}'\n"
執(zhí)行以下命令：
training/tesstrain0.sh --fonts_dir /usr/share/fonts \
--training_text ../training_data/input_data0.txt \
--langdata_dir ../langdata --tessdata_dir ./tessdata \
--lang chi_sim --linedata_only --noextract_font_properties \
--exposures "0" --fontlist "SIMSUN" \
--output_dir ~/tesstutorial/chitest
第二步
按照以下步驟對(duì)shell文件進(jìn)行修改：
- 查看系統(tǒng)/tmp目錄下的文件夾股耽，找到格式為'tmp.*'的文件夾，并將此文件夾名進(jìn)行復(fù)制钳幅。
- 將文件training/tesstrain_utils.sh復(fù)制備份training/tesstrain_utils0.sh物蝙，并將tesstrain_utils0.sh中的WORKSPACE_DIR變量名賦值為'tmp.*'(/tmp目錄下相應(yīng)的文件夾名稱)
- 將training/tesstrain0.sh文件中source "$(dirname $0)/tesstrain_utils.sh"語(yǔ)句改成source "$(dirname $0)/tesstrain_utils0.sh"
- 將training/tesstrain0.sh文件的注釋內(nèi)容進(jìn)行更改，更改后的內(nèi)容如下：
source "$(dirname $0)/tesstrain_utils0.sh"
ARGV=("$@")
parse_flags
echo $WORKSPACE_DIR
mkdir -p ${TRAINING_DIR}
tlog "\n=== Starting training for language '${LANG_CODE}'"
source "$(dirname $0)/language-specific.sh"
set_lang_specific_parameters ${LANG_CODE}
initialize_fontconfig
phase_I_generate_image 8
# phase_UP_generate_unicharset
# phase_D_generate_dawg
# if ((LINEDATA)); then
# phase_E_extract_features "lstm.train" 8 "lstmf"
# make__lstmdata
# else
# phase_E_extract_features "box.train" 8 "tr"
# phase_C_cluster_prototypes "${TRAINING_DIR}/${LANG_CODE}.normproto"
# if [[ "${ENABLE_SHAPE_CLUSTERING}" == "y" ]]; then
# phase_S_cluster_shapes
# fi
# phase_M_cluster_microfeatures
# phase_B_generate_ambiguities
# make__traineddata
# fi
#
# tlog "\nCompleted training for language '${LANG_CODE}'\n"
執(zhí)行以下命令：
training/tesstrain0.sh --fonts_dir /usr/share/fonts \
--training_text ../training_data/input_data0.txt \
--langdata_dir ../langdata --tessdata_dir ./tessdata \
--lang chi_sim --linedata_only --noextract_font_properties \
--exposures "0" --fontlist "SIMSUN" \
--output_dir ~/tesstutorial/chitest
第三步
在系統(tǒng)桌面進(jìn)入"/tmp/tmp.J8JYbdYYrv/chi_sim"目錄敢艰，修改后綴名為.box文件诬乞，將此文件中的特殊字符修改為想要替換的其他字符，注意：每個(gè)字符的替換需要保持一致哦钠导。
例如：本文例將所有的'≤'符號(hào)替換成"小"丽惭。
第四步
修改training/tesstrain0.sh文件中的注釋內(nèi)容，修改后如下：
source "$(dirname $0)/tesstrain_utils0.sh"
ARGV=("$@")
parse_flags
# mkdir -p ${TRAINING_DIR}
# tlog "\n=== Starting training for language '${LANG_CODE}'"
#
# source "$(dirname $0)/language-specific.sh"
# set_lang_specific_parameters ${LANG_CODE}
#
# initialize_fontconfig
#
# phase_I_generate_image 8
echo $TRAINING_DIR
phase_UP_generate_unicharset
phase_D_generate_dawg
if ((LINEDATA)); then
phase_E_extract_features "lstm.train" 8 "lstmf"
make__lstmdata
else
phase_E_extract_features "box.train" 8 "tr"
phase_C_cluster_prototypes "${TRAINING_DIR}/${LANG_CODE}.normproto"
if [[ "${ENABLE_SHAPE_CLUSTERING}" == "y" ]]; then
phase_S_cluster_shapes
fi
phase_M_cluster_microfeatures
phase_B_generate_ambiguities
make__traineddata
fi
tlog "\nCompleted training for language '${LANG_CODE}'\n"
執(zhí)行以下命令：
training/tesstrain0.sh --fonts_dir /usr/share/fonts \
--training_text ../training_data/input_data0.txt \
--langdata_dir ../langdata --tessdata_dir ./tessdata \
--lang chi_sim --linedata_only --noextract_font_properties \
--exposures "0" --fontlist "SIMSUN" \
--output_dir ~/tesstutorial/chitest
第五步
按步驟執(zhí)行以下命令:
- mkdir -p ~/tesstutorial/chituned_from_chisim
- combine_tessdata -e ../tessdata/chi_sim.traineddata ~/tesstutorial/chituned_from_chisim/chi_sim.lstm
- lstmtraining --model_output ~/tesstutorial/chituned_from_chisim/chituned --continue_from ~/tesstutorial/chituned_from_chisim/chi_sim.lstm --train_listfile ~/tesstutorial/chitest/chi_sim.training_files.txt --eval_listfile ~/tesstutorial/chitest/chi_sim.training_files.txt --target_error_rate 0.01
- lstmtraining --model_output ~/tesstutorial/chituned_from_chisim/chituned.lstm
--continue_from ~/tesstutorial/chituned_from_chisim/chituned_checkpoint
--stop_training - combine_tessdata -o ./tessdata/chi_sim.traineddata
~/tesstutorial/chituned_from_chisim/chituned.lstm
~/tesstutorial/chitest/chi_sim.lstm-number-dawg
~/tesstutorial/chitest/chi_sim.lstm-punc-dawg
~/tesstutorial/chitest/chi_sim.lstm-word-dawg
新生成的traineddata在tesseract/tessdata目錄下辈双。
