1實(shí)例說明
實(shí)現(xiàn)對(duì) 鋼鐵是怎樣練成的.txt 進(jìn)行全文索引记劝，以及和普通檢索方式 （java.lang.String的 equals）進(jìn)行效率對(duì)比
開發(fā)過程
1.將大文檔切分成多個(gè)小文檔
這一步 并非 是必須的，為了更好的展示 Lucene的一些功能盖矫，將文檔切分為多個(gè)較小的文檔，并給每個(gè)文檔一個(gè)唯一的ID（文件名稱）
2.eclipse 編寫代碼
3.效果測(cè)試
通過Lucene的檢索 和 java字符串檢索击奶，進(jìn)行性能上的比較辈双，得出結(jié)論
項(xiàng)目準(zhǔn)備
eclipse 導(dǎo)入這兩個(gè)jar包 （由于是入門實(shí)例 jar包為老版本，太新會(huì)報(bào)錯(cuò)）
1.文檔預(yù)處理類 FilePreprocess
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
public class FilePreprocess {
//預(yù)處理
public static void preprocess(File file,String outputDir){
try{
//拆分成小文件
splitToSmallFiles(file,outputDir);
}catch(Exception e ){
e.printStackTrace();
}
}
public static void splitToSmallFiles(File file,String outputpath)throws IOException{
int filePointer = 0; //文件命名 自增序列
int MAX_SIZE = 10240; //設(shè)置文件大小
BufferedWriter writer = null;
BufferedReader reader = new BufferedReader(new FileReader(file));
StringBuffer buffer = new StringBuffer();
String line = reader.readLine();
while(line != null){
buffer.append(line).append("\r\n");
if(buffer.toString().getBytes().length >= MAX_SIZE){ //如果超出限制大小 new file
writer = new BufferedWriter(new FileWriter(outputpath + "output" + filePointer+".txt"));
writer.write(buffer.toString());
writer.close();
filePointer ++;
buffer = new StringBuffer();
}
line = reader.readLine();
}
writer = new BufferedWriter(new FileWriter(outputpath + "output" + filePointer + ".txt"));
writer.write(buffer.toString());
writer.close();
}
//入口
public static void main(String [] args){
String inputFile = "f:\\book.txt"; //讀取文件
String outputf = "f:\\outputFolder\\"; //文件預(yù)處理后輸出目錄
if(!new File(outputf).exists()){
new File(outputf).mkdirs();
}
FilePreprocess f = new FilePreprocess();
f.preprocess(new File(inputFile), outputf); //分塊處理
}
}```
>> 運(yùn)行后 輸出路徑 則會(huì)生成 多個(gè)小文檔
####2.創(chuàng)建處理文檔的索引類 IndexProcessor####
```package lucene;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.index.IndexWriter;
import jeasy.analysis.MMAnalyzer;
public class IndexProcessor {
//索引存儲(chǔ)目錄
private String INDEX_STORE_PATH = "f:\\index";
//創(chuàng)建索引
public void createIndex(String inputDir){
try {
//利用分詞工具創(chuàng)建 IndexWriter
IndexWriter writer = new IndexWriter(INDEX_STORE_PATH,new MMAnalyzer(),true);
File filesDir = new File(inputDir);
//取得 要建立 索引的文件數(shù)組
File[] files = filesDir.listFiles();
for(int i = 0 ;i < files.length ; i++){
String fileName = files[i].getName();
if(fileName.substring(fileName.lastIndexOf(".")).equals(".txt")){
//創(chuàng)建新的Document
Document doc = new Document();
//為文件名創(chuàng)建一個(gè) Field
Field field = new Field("filename",files[i].getName(),Field.Store.YES,Field.Index.TOKENIZED);
doc.add(field);
field = new Field("content",loadFileToString(files[i]),Field.Store.NO,Field.Index.TOKENIZED);
doc.add(field);
//把Document加入 IndexWriter
writer.addDocument(doc);
}
}
writer.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
//加載文檔 生成字符串
public String loadFileToString(File f) {
try{
BufferedReader br = new BufferedReader(new FileReader(f));
StringBuffer sb = new StringBuffer();
String line = br.readLine();
while(line != null){
sb.append(line);
line = br.readLine();
}
br.close();
return sb.toString();
} catch (IOException e) {
e.printStackTrace();
return null;
}
}
public static void main(String[] args){
IndexProcessor pr = new IndexProcessor();
pr.createIndex("f://outputFolder");
}
}```
**索引目錄結(jié)構(gòu)生成**
####3.創(chuàng)建索引搜索類 Searchpackage lucene;####
```package lucene;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.util.Date;
import java.util.Iterator;
import java.util.LinkedHashMap;
import java.util.Map;
import org.apache.lucene.index.Term;
import org.apache.lucene.index.TermDocs;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.Query;
import org.apache.lucene.search.TermQuery;
public class Search {
private String INDEX_STORE_PATH = "f:\\index";
//利用lucene 索引 搜索
public void indexSearch(String searchType,String searchKey){
try{
System.out.println("##使用索引方式搜索##");
System.out.println("======================");
//根據(jù)索引位置簡(jiǎn)歷 IndexSearch
IndexSearcher searcher = new IndexSearcher(INDEX_STORE_PATH);
//建立搜索單元,searchType 代表要搜索的Field,searchKey代表關(guān)鍵字
Term t = new Term(searchType,searchKey);
//由Term產(chǎn)生 Query
Query q = new TermQuery(t);
//搜索開始時(shí)間
Date beginTime = new Date();
//獲取一個(gè) 的枚舉對(duì)象 TermDocs
TermDocs docs = searcher.getIndexReader().termDocs(t);
while(docs.next()){
System.out.print("find" + docs.freq() + "matches in");
System.out.println(searcher.getIndexReader().document(docs.doc()).getField("filename").stringValue());
}
Date endTime = new Date();
long timeofsearch = endTime.getTime() - beginTime.getTime();
System.out.println("總耗時(shí):" + timeofsearch);
}
catch(Exception e){
e.printStackTrace();
}
}
public void stringSearch(String keyword,String searchDir){
System.out.println("##使用字符串方式搜索##");
System.out.println("======================");
File filesDir = new File(searchDir);
//返回目錄文件夾所有文件數(shù)組
File[] files = filesDir.listFiles();
//HM 保存文件名和匹配次數(shù)對(duì)
Map rs = new LinkedHashMap();
//搜索開始時(shí)間
Date beginTime = new Date();
for(int i = 0 ;i < files.length ; i ++){
int hits = 0;
try{
BufferedReader br = new BufferedReader(new FileReader(files[i]));
StringBuffer sb = new StringBuffer();
String line = br.readLine();
while(line != null){
sb.append(line);
line = br.readLine();
}
br.close();
//將 stringBuffer 轉(zhuǎn)化成 String,以便于搜索
String stringToSearch = sb.toString();
//從 0 索引 查詢 -length + length = 0
int fromIndex = -keyword.length();
int len = stringToSearch.indexOf(keyword,fromIndex + keyword.length());
while((fromIndex = len)!= -1){
hits++;
}
//將文件名 和 匹配次數(shù) 加入 HashMap
rs.put(files[i].getName(), new Integer(hits));
}
catch(Exception e){
e.printStackTrace();
}
}
Iterator it = rs.keySet().iterator();
while(it.hasNext()){
String fileName = (String)it.next();
Integer hits = (Integer)rs.get(fileName);
System.out.println("find" + hits.intValue() + "matches in"+ fileName);
}
Date endTime = new Date();
long timeOfSearch = endTime.getTime() - beginTime.getTime();
System.out.println("使用字符串匹配方式總耗時(shí):" + timeOfSearch + "ms");
}
public static void main(String[] args) {
Search s = new Search();
s.indexSearch("content", "保爾");
System.out.println();
s.stringSearch("保爾", "f:\\outputFolder");
}
}```
>**相比于字符串查詢柜砾，發(fā)現(xiàn)索引方式效率更高**