本文是elasticsearch源碼分析系列文檔的第一篇,因為之前一直從事搜索廣告的工作颇象,也一直想寫一下關(guān)于lucene和es的文章,現(xiàn)在終于有機會了乳蛾。本篇作為開篇,簡單介紹了elasticsearch源碼在本機的編譯環(huán)境搭建,為后面源碼分析提供基礎(chǔ)肃叶。
用到的工具有:IntelliJ Idea
蹂随,JDK1.8
,gradle4.3
因惭,elasticsearch-6.0.1 的發(fā)行版
(es官網(wǎng)通常都是下載最新版岳锁,從這里可以下載需要的歷史版本 https://www.elastic.co/cn/downloads/past-releases#elasticsearch
下載源碼
elasticsearch的源碼是托管在github的,但是github下載很慢筛欢,所以選擇了從gitee下載浸锨,地址如下:https://gitee.com/mirrors/elasticsearch
下載elasticsearch的主干版本唇聘,在7.x的版本了版姑,目前項目中很少有用到這么新的版本的。通常都還停留在6.x甚至5.x迟郎。我們選擇6.0.1的版本剥险。
git checkout v6.0.1
因為elasticsearch在5.X版本后默認的編譯工具從Maven換到了Gradle。需要在本機配置好Gradle環(huán)境宪肖,如有必要可以切換國內(nèi)的源表制。本機使用Gradle4.3
和JDK1.8編譯elasticsearch。(因為elasticsearch 6.0.1的版本要求gradle的版本必須在4.3或以上控乾,否則編譯會報錯)
在elasticsearch中有個類記錄當(dāng)前的版本么介,即org.elasticsearch.Version
,如下:
public class Version implements Comparable<Version> {
/*
* The logic for ID is: XXYYZZAA, where XX is major version, YY is minor version, ZZ is revision, and AA is alpha/beta/rc indicator AA
* values below 25 are for alpha builder (since 5.0), and above 25 and below 50 are beta builds, and below 99 are RC builds, with 99
* indicating a release the (internal) format of the id is there so we can easily do after/before checks on the id
*/
public static final Version V_6_0_1 =
new Version(V_6_0_1_ID, org.apache.lucene.util.Version.LUCENE_7_0_1);
public static final Version CURRENT = V_6_0_1;
可以看到當(dāng)前es
的版本的6.0.1
蜕衡,使用的lucene
的版本是7.0.1
壤短。
Gradle編譯
我們使用idea查看源碼,但是直接導(dǎo)入idea會有很多包找不到慨仿,并且idea的加載的速度會非常慢久脯,所以我們采用gradle命令行的方式編譯。因為我們使用idea作為IDE镰吆,所以命令是gradle idea(如果IDE是eclipse帘撰,那命令就是gradle eclipse)。編譯的時間很長万皿,我在本機用了10min(macbook pro 2015)摧找,成功編譯的話會有提示build successful
,如下圖:
導(dǎo)入IDEA
源碼編譯成功后牢硅,啟動IDE(可以直接在當(dāng)前目前執(zhí)行idea .
)慰于,雖然已經(jīng)用gradle編譯過了,但是idea還是會進行很長時間的配置和加載唤衫,等待完全加載完畢后婆赠,查看項目如下圖,基本能看到項目的各個模塊:
嘗試啟動
elasticsearch的啟動類是org.elasticsearch.bootstrap.Elasticsearch
,該類的main方法是啟動入口休里,如下圖:
同樣蛆挫,在這個package下還有一個類
org.elasticsearch.bootstrap.Bootstrap
,很容易引起誤解這個才是啟動類妙黍°睬郑看過代碼之后發(fā)現(xiàn)Elasticsearch的啟動類會調(diào)用Bootstrap的方法做初始化,如下:啟動當(dāng)然需要一些配置拭嫁,不然是會啟動報錯的可免。
- 第一個報錯
ERROR: the system property [es.path.conf] must be set
es.path.conf是配置文件的路徑,如果路徑下沒有配置文件做粤,將使用默認配置浇借,可將elasticsearch.yml文件放在此路徑下。
解決方法: 我這里直接將該路徑指向了下載好的elasticsearch-6.0.1 的發(fā)行版路徑怕品,如下
-Des.path.conf=/xxx/soft/elasticsearch-6.0.1
- 第二個報錯
Exception in thread "main" java.lang.IllegalStateException: path.home is not configured
at org.elasticsearch.env.Environment.<init>(Environment.java:101)
at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:85)
at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:78)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:69)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134)
at org.elasticsearch.cli.Command.main(Command.java:90)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85)
這是es運行的home路徑妇垢。plugins,modules肉康,lib等都會在此路徑下相應(yīng)的子路徑加載闯估。如果不指定,默認的data吼和,log涨薪,config路徑也會創(chuàng)建在此home路徑下。但其實因為代碼的問題config路徑必須指定炫乓,否則會報錯刚夺。
解決方法:同樣將這個路徑指向了下載好的elasticsearch-6.0.1 的發(fā)行版路徑,如下:
-Des.path.home=/xxx/soft/elasticsearch-6.0.1
- 第三個報錯
Exception in thread "main" 2020-05-27 17:41:03,055 main ERROR No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'log4j2.debug' to show Log4j2 internal initialization logging.
2020-05-27 17:41:03,164 main ERROR Could not register mbeans java.security.AccessControlException: access denied ("javax.management.MBeanTrustPermission" "register")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:585)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.checkMBeanTrustPermission(DefaultMBeanServerInterceptor.java:1848)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:322)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.logging.log4j.core.jmx.Server.register(Server.java:389)
at org.apache.logging.log4j.core.jmx.Server.reregisterMBeansAfterReconfigure(Server.java:167)
at org.apache.logging.log4j.core.jmx.Server.reregisterMBeansAfterReconfigure(Server.java:140)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:556)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:617)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:634)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:229)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:242)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:174)
at org.apache.logging.log4j.LogManager.getLogger(LogManager.java:648)
at org.elasticsearch.common.logging.ESLoggerFactory.getLogger(ESLoggerFactory.java:54)
at org.elasticsearch.common.logging.ESLoggerFactory.getLogger(ESLoggerFactory.java:62)
at org.elasticsearch.common.logging.Loggers.getLogger(Loggers.java:101)
at org.elasticsearch.ExceptionsHelper.<clinit>(ExceptionsHelper.java:42)
at org.elasticsearch.ElasticsearchException.toString(ElasticsearchException.java:663)
at java.lang.String.valueOf(String.java:2994)
at java.io.PrintStream.println(PrintStream.java:821)
at java.lang.Throwable$WrappedPrintStream.println(Throwable.java:748)
at java.lang.Throwable.printStackTrace(Throwable.java:655)
at java.lang.Throwable.printStackTrace(Throwable.java:643)
at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1061)
at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1052)
at java.lang.Thread.dispatchUncaughtException(Thread.java:1959)
解決方法:添加啟動配置厢岂,-Dlog4j2.disable.jmx=true
光督。
由于對java security不太了解,這是網(wǎng)上找到的解決方案塔粒。
- 第四個報錯
[2020-05-29T11:28:37,580][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: jar hell!
class: jdk.packager.services.UserJvmOptionsService
jar1: /Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home/lib/ant-javafx.jar
jar2: /Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home/lib/packager.jar
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:134) ~[main/:?]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:121) ~[main/:?]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:69) ~[main/:?]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) ~[main/:?]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[main/:?]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[main/:?]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) ~[main/:?]
Caused by: java.lang.IllegalStateException: jar hell!
class: jdk.packager.services.UserJvmOptionsService
jar1: /Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home/lib/ant-javafx.jar
jar2: /Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home/lib/packager.jar
at org.elasticsearch.bootstrap.JarHell.checkClass(JarHell.java:282) ~[main/:?]
at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:192) ~[main/:?]
at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:90) ~[main/:?]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:197) ~[main/:?]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:322) ~[main/:?]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:130) ~[main/:?]
... 6 more
應(yīng)該是jdk本身類沖突的問題结借,jdk.packager.services.UserJvmOptionsService這個類有2個。應(yīng)該升級jdk的版本能解卒茬。
解決方法:這里我選擇了一個偷懶的方案船老,直接把JarHell.checkClass這個方法中拋出異常的部分注掉了,后面如果有其他問題再考慮升級jdk圃酵。
- 第五個報錯
[2020-05-29T11:35:28,193][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:134) ~[main/:?]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:121) ~[main/:?]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:69) ~[main/:?]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) ~[main/:?]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[main/:?]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[main/:?]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) ~[main/:?]
Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at java.lang.String.substring(String.java:1967) ~[?:1.8.0_151]
at org.elasticsearch.bootstrap.Security.readPolicy(Security.java:222) ~[main/:?]
at org.elasticsearch.bootstrap.Security.getPluginPermissions(Security.java:179) ~[main/:?]
at org.elasticsearch.bootstrap.Security.configure(Security.java:121) ~[main/:?]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:207) ~[main/:?]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:322) ~[main/:?]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:130) ~[main/:?]
這里是因為柳畔,默認情況下build會采用snapshot的版本,而我們的發(fā)行版都是release的郭赐,所以在security加載module時有文件加載不到薪韩,解決方案有2種。
解決方法1:一個偷懶的方案,直接把org.elasticsearch.bootstrap.Security#readPolicy
這個方法中判斷 snapshot
的地方注掉俘陷,如圖:
解決方法2:因為我們下載的es是正式發(fā)行版罗捎,所以在命名上是不帶
snapshot
的,所以我們將發(fā)行版本中的elasticsearch-6.0.1/modules/reindex/elasticsearch-rest-client-6.0.1.jar
這個文件拉盾,把這個jar包的名稱改成snapshot版本(elasticsearch-rest-client-6.0.1-SNAPSHOT.jar)桨菜,就能正常啟動了。
成功運行
經(jīng)過九九八十一難之后終于啟動運行后捉偏,從日志可以看到啟動之后默認是監(jiān)聽9200端口倒得。后面可以使用127.0.0.1:9200這個端口來測試啦,后面就能進行代碼的調(diào)試拉夭禽。
啟動成功的日志如下:
[2020-05-29T14:07:15,138][INFO ][i.n.u.i.PlatformDependent] Your platform does not provide complete low-level API for accessing direct buffers reliably. Unless explicitly requested, heap buffer will always be preferred to avoid potential system instability.
[2020-05-29T14:07:15,322][INFO ][o.e.t.TransportService ] [80GXGgY] publish_address {127.0.0.1:9300}, bound_addresses {[fe80::1]:9300}, {[::1]:9300}, {127.0.0.1:9300}
[2020-05-29T14:07:15,339][WARN ][o.e.b.BootstrapChecks ] [80GXGgY] initial heap size [268435456] not equal to maximum heap size [4294967296]; this can cause resize pauses and prevents mlockall from locking the entire heap
[2020-05-29T14:07:18,414][INFO ][o.e.c.s.MasterService ] [80GXGgY] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {80GXGgY}{80GXGgYFTRaO0GqmHEG_0Q}{-R2fH2CmSuiKPLR5DOxJfA}{127.0.0.1}{127.0.0.1:9300}
[2020-05-29T14:07:18,421][INFO ][o.e.c.s.ClusterApplierService] [80GXGgY] new_master {80GXGgY}{80GXGgYFTRaO0GqmHEG_0Q}{-R2fH2CmSuiKPLR5DOxJfA}{127.0.0.1}{127.0.0.1:9300}, reason: apply cluster state (from master [master {80GXGgY}{80GXGgYFTRaO0GqmHEG_0Q}{-R2fH2CmSuiKPLR5DOxJfA}{127.0.0.1}{127.0.0.1:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2020-05-29T14:07:18,440][INFO ][o.e.h.n.Netty4HttpServerTransport] [80GXGgY] publish_address {127.0.0.1:9200}, bound_addresses {[fe80::1]:9200}, {[::1]:9200}, {127.0.0.1:9200}
[2020-05-29T14:07:18,440][INFO ][o.e.n.Node ] [80GXGgY] started
[2020-05-29T14:07:18,444][INFO ][o.e.g.GatewayService ] [80GXGgY] recovered [0] indices into cluster_state