【elasticsearch架构系列】elasticsearch 6.8.13源码分析系列(1)

准备工作

软件和源码版本说明

  • elasticsearch 6.8.13源码。
    可以从github checkout。
  • elasticsearch 6.8.13.tar.gz包。
    从elastic官网下载。
  • gradle 5.4.11.zip。
    从网络下载
  • jdk12。
    可以brew或者windows安装包。启动过程会修改jdk的java.security文件(目录在/Library/Java/JavaVirtualMachines/jdk-12.jdk/Contents/Home/conf/security)
  • idea 2020.2.3

编译过程

  • 将下载的 gradle 5.4.11.zip放到源码包的gradle/wrapper/dists目录下(dists目录要新建)
  • 命令行进入源码目录执行./gradlew。
    如果出现Sucessful,就说明初始化成功了。
    如果出现什么bug,直接谷歌找答案。
  • 在idea打开源码项目.
    idea会提示是否导入给gradle project。点是就可以了
  • 解压从elastic官网下载的module、config、plugin包到源码工程下
    注意,在复制之前:把module包下的.iml文件删除
    把plugin包重命名为plugin_bak先。
  • 配置debug参数
    -Des.path.home=/Users/xiaogang/java/elasticsearch-6.8.13
    -Des.path.conf=/Users/xiaogang/java/elasticsearch-6.8.13/config

补充知识

elasticsearch整体架构


ps:图片来源于网络

源码目录

启动过程

elasticsearch服务debug启动参数

/Library/Java/JavaVirtualMachines/jdk-12.jdk/Contents/Home/bin/java 
-agentlib:jdwp=transport=dt_socket,address=127.0.0.1:53251,suspend=y,server=n
-Des.path.home=/Users/xiaogang/java/elasticsearch-6.8.13
-Des.path.conf=/Users/xiaogang/java/elasticsearch-6.8.13/config
-javaagent:/Users/xiaogang/Library/Caches/JetBrains/IntelliJIdea2020.2/groovyHotSwap/gragent.jar
-javaagent:/Users/xiaogang/Library/Caches/JetBrains/IntelliJIdea2020.2/captureAgent/debugger-agent.jar
-Dfile.encoding=UTF-8
-classpath /Users/xiaogang/java/elasticsearch-6.8.13/server/build/classes/java/main:/Users/xiaogang/java/elasticsearch-6.8.13/server/out/production/resources:/Users/xiaogang/java/elasticsearch-6.8.13/libs/x-content/build/classes/java/main:/Users/xiaogang/java/elasticsearch-6.8.13/libs/cli/build/classes/java/main:/Users/xiaogang/java/elasticsearch-6.8.13/libs/core/build/classes/java/main:/Users/xiaogang/java/elasticsearch-6.8.13/libs/secure-sm/build/classes/java/main:/Users/xiaogang/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-core/7.7.3/5faa5ae56f7599019fce6184accc6c968b7519e7/lucene-core-7.7.3.jar:/Users/xiaogang/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-analyzers-common/7.7.3/a76cbf5e21bbbb0c2d6288b042450236248214e/lucene-analyzers-common-7.7.3.jar:/Users/xiaogang/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-backward-codecs/7.7.3/40207d0dd023a0e2868a23dd87d72f1a3cdbb893/lucene-backward-codecs-7.7.3.jar:/Users/xiaogang/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-grouping/7.7.3/f83db9f652630a8b39fbb455a91f59f21ff67779/lucene-grouping-7.7.3.jar:/Users/xiaogang/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-highlighter/7.7.3/e271cfdaae0c1d3987e375ac9315cdc7fccdf203/lucene-highlighter-7.7.3.jar:/Users/xiaogang/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-join/7.7.3/4e8c4b73a82db1dd898735671aa42d6cc839c977/lucene-join-7.7.3.jar:/Users/xiaogang/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-memory/7.7.3/a23d27ab6e928a9e6b8b2a0dceeef8c719d82396/lucene-memory-7.7.3.jar:/Users/xiaogang/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-misc/7.7.3/3aca078edf983059722fe61a81b7b7bd5ecdb222/lucene-misc-7.7.3.jar:/Users/xiaogang/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-queries/7.7.3/f507367487e614b9915cef04761afca72a279047/lucene-queries-7.7.3.jar:/Users/xiaogang/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-queryparser/7.7.3/685fc6166d29eb3e3441ae066873bb442aa02df1/lucene-queryparser-7.7.3.jar:/Users/xiaogang/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-sandbox/7.7.3/8ae1fde038dbdf915a515166c0d0c142bea8bbe5/lucene-sandbox-7.7.3.jar:/Users/xiaogang/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-spatial/7.7.3/6aca987f61da63aad6635ecc8d23de7f2bbb2aea/lucene-spatial-7.7.3.jar:/Users/xiaogang/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-spatial-extras/7.7.3/455c20d05a2955a0cb67089377102cad1750bef2/lucene-spatial-extras-7.7.3.jar:/Users/xiaogang/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-spatial3d/7.7.3/b46e052b83c546c6bfcc331347c06374b8d03ca/lucene-spatial3d-7.7.3.jar:/Users/xiaogang/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-suggest/7.7.3/8d184d913dc6f4181afd61ee8f657909fa38516b/lucene-suggest-7.7.3.jar:/Users/xiaogang/.gradle/caches/modules-2/files-2.1/com.carrotsearch/hppc/0.7.1/8b5057f74ea378c0150a1860874a3ebdcb713767/hppc-0.7.1.jar:/Users/xiaogang/.gradle/caches/modules-2/files-2.1/joda-time/joda-time/2.10.1/9ac3dbf89dbf2ee385185dd0cd3064fe789efee0/joda-time-2.10.1.jar:/Users/xiaogang/.gradle/caches/modules-2/files-2.1/com.tdunning/t-digest/3.2/2ab94758b0276a8a26102adf8d528cf6d0567b9a/t-digest-3.2.jar:/Users/xiaogang/.gradle/caches/modules-2/files-2.1/org.hdrhistogram/HdrHistogram/2.1.9/e4631ce165eb400edecfa32e03d3f1be53dee754/HdrHistogram-2.1.9.jar:/Users/xiaogang/.gradle/caches/modules-2/files-2.1/org.locationtech.spatial4j/spatial4j/0.7/faa8ba85d503da4ab872d17ba8c00da0098ab2f2/spatial4j-0.7.jar:/Users/xiaogang/.gradle/caches/modules-2/files-2.1/org.locationtech.jts/jts-core/1.15.0/705981b7e25d05a76a3654e597dab6ba423eb79e/jts-core-1.15.0.jar:/Users/xiaogang/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/Users/xiaogang/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/Users/xiaogang/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-1.2-api/2.11.1/3aba3398fe064a3eab4331f88161c7480e848418/log4j-1.2-api-2.11.1.jar:/Users/xiaogang/.gradle/caches/modules-2/files-2.1/org.elasticsearch/jna/5.5.0/ade077cbb2618a18bfc6c335413b2b7163d97601/jna-5.5.0.jar:/Users/xiaogang/java/elasticsearch-6.8.13/libs/plugin-classloader/build/classes/java/main:/Users/xiaogang/.gradle/caches/modules-2/files-2.1/org.yaml/snakeyaml/1.17/7a27ea250c5130b2922b86dea63cbb1cc10a660c/snakeyaml-1.17.jar:/Users/xiaogang/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.8.11/876ead1db19f0c9e79c9789273a3ef8c6fd6c29b/jackson-core-2.8.11.jar:/Users/xiaogang/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-smile/2.8.11/d9d1c49c5d9d5e46e2aee55f3cdd119286fe0fc1/jackson-dataformat-smile-2.8.11.jar:/Users/xiaogang/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-yaml/2.8.11/2e77c6ff7342cd61ab1ae7cb14ed16aebfc8a72a/jackson-dataformat-yaml-2.8.11.jar:/Users/xiaogang/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-cbor/2.8.11/8b9826e16c3366764bfb7ad7362554f0471046c3/jackson-dataformat-cbor-2.8.11.jar:/Users/xiaogang/.gradle/caches/modules-2/files-2.1/net.sf.jopt-simple/jopt-simple/5.0.2/98cafc6081d5632b61be2c9e60650b64ddbc637c/jopt-simple-5.0.2.jar:/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar
org.elasticsearch.bootstrap.Elasticsearch

日志输出

Connected to the target VM, address: '127.0.0.1:53251', transport: 'socket'
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
2020-12-12 15:24:37,681 [INFO ][o.e.e.NodeEnvironment ] [node-1] using [1] data paths, mounts [[/ (/dev/disk1s1)]], net usable_space [245.6gb], net total_space [465.6gb], types [apfs]
2020-12-12 15:24:37,688 [INFO ][o.e.e.NodeEnvironment ] [node-1] heap size [4gb], compressed ordinary object pointers [true]
2020-12-12 15:24:37,692 [INFO ][o.e.n.Node ] [node-1] node name [node-1], node ID [ejfYItYER0qc2qaepF3-PA]
2020-12-12 15:24:37,692 [INFO ][o.e.n.Node ] [node-1] version[6.8.13-SNAPSHOT], pid[7977], build[unknown/unknown/Unknown/Unknown], OS[Mac OS X/10.15.7/x86_64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12/12+33]
2020-12-15 22:24:37,693 [INFO ][o.e.n.Node ] [node-1] JVM arguments [-agentlib:jdwp=transport=dt_socket,address=127.0.0.1:53251,suspend=y,server=n, -Des.path.home=/Users/xiaogang/java/elasticsearch-6.8.13, -Des.path.conf=/Users/xiaogang/java/elasticsearch-6.8.13/config, -javaagent:/Users/xiaogang/Library/Caches/JetBrains/IntelliJIdea2020.2/groovyHotSwap/gragent.jar, -javaagent:/Users/xiaogang/Library/Caches/JetBrains/IntelliJIdea2020.2/captureAgent/debugger-agent.jar, -Dfile.encoding=UTF-8]
2020-12-15 22:24:37,693 [WARN ][o.e.n.Node ] [node-1] version [6.8.13-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production
2020-12-15 22:24:44,173 [INFO ][o.e.p.PluginsService ] [node-1] loaded module [aggs-matrix-stats]
2020-12-15 22:24:44,173 [INFO ][o.e.p.PluginsService ] [node-1] loaded module [analysis-common]
2020-12-15 22:24:44,173 [INFO ][o.e.p.PluginsService ] [node-1] loaded module [ingest-common]
2020-12-15 22:24:44,173 [INFO ][o.e.p.PluginsService ] [node-1] loaded module [ingest-geoip]
2020-12-15 22:24:44,173 [INFO ][o.e.p.PluginsService ] [node-1] loaded module [ingest-user-agent]
2020-12-15 22:24:44,173 [INFO ][o.e.p.PluginsService ] [node-1] loaded module [lang-expression]
2020-12-15 22:24:44,173 [INFO ][o.e.p.PluginsService ] [node-1] loaded module [lang-mustache]
2020-12-15 22:24:44,174 [INFO ][o.e.p.PluginsService ] [node-1] loaded module [lang-painless]
2020-12-15 22:24:44,174 [INFO ][o.e.p.PluginsService ] [node-1] loaded module [mapper-extras]
2020-12-15 22:24:44,174 [INFO ][o.e.p.PluginsService ] [node-1] loaded module [parent-join]
2020-12-15 22:24:44,174 [INFO ][o.e.p.PluginsService ] [node-1] loaded module [percolator]
2020-12-15 22:24:44,174 [INFO ][o.e.p.PluginsService ] [node-1] loaded module [rank-eval]
2020-12-15 22:24:44,174 [INFO ][o.e.p.PluginsService ] [node-1] loaded module [reindex]
2020-12-15 22:24:44,174 [INFO ][o.e.p.PluginsService ] [node-1] loaded module [repository-url]
2020-12-15 22:24:44,174 [INFO ][o.e.p.PluginsService ] [node-1] loaded module [transport-netty4]
2020-12-15 22:24:44,174 [INFO ][o.e.p.PluginsService ] [node-1] loaded module [tribe]
2020-12-15 22:24:44,175 [INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-ccr]
2020-12-15 22:24:44,175 [INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-core]
2020-12-15 22:24:44,175 [INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-deprecation]
2020-12-15 22:24:44,175 [INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-graph]
2020-12-15 22:24:44,175 [INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-ilm]
2020-12-15 22:24:44,175 [INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-logstash]
2020-12-15 22:24:44,175 [INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-ml]
2020-12-15 22:24:44,175 [INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-monitoring]
2020-12-15 22:24:44,175 [INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-rollup]
2020-12-15 22:24:44,175 [INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-security]
2020-12-15 22:24:44,176 [INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-sql]
2020-12-15 22:24:44,176 [INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-upgrade]
2020-12-15 22:24:44,176 [INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-watcher]
2020-12-15 22:24:44,176 [INFO ][o.e.p.PluginsService ] [node-1] no plugins loaded
2020-12-15 22:24:44,427 [INFO ][i.n.u.i.PlatformDependent] [node-1] Your platform does not provide complete low-level API for accessing direct buffers reliably. Unless explicitly requested, heap buffer will always be preferred to avoid potential system instability.
2020-12-15 22:24:47,980 [INFO ][o.e.x.s.a.s.FileRolesStore] [node-1] parsed [0] roles from file [/Users/xiaogang/java/elasticsearch-6.8.13/config/roles.yml]
2020-12-15 22:24:48,220 [INFO ][i.n.u.i.PlatformDependent] [node-1] Your platform does not provide complete low-level API for accessing direct buffers reliably. Unless explicitly requested, heap buffer will always be preferred to avoid potential system instability.
2020-12-15 22:24:48,849 [INFO ][o.e.x.m.p.l.CppLogMessageHandler] [node-1] [controller/7980] [Main.cc@114] controller (64 bit): Version 6.8.13 (Build c959f131d0de3b) Copyright (c) 2020 Elasticsearch BV
2020-12-15 22:24:49,779 [DEBUG][o.e.a.ActionModule ] [node-1] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
2020-12-15 22:24:50,078 [INFO ][o.e.d.DiscoveryModule ] [node-1] using discovery type [zen] and host providers [settings]
2020-12-15 22:24:51,247 [INFO ][o.e.n.Node ] [node-1] initialized
2020-12-15 22:24:51,247 [INFO ][o.e.n.Node ] [node-1] starting ...
2020-12-15 22:24:51,505 [INFO ][o.e.t.TransportService ] [node-1] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
2020-12-15 22:24:51,551 [WARN ][o.e.b.BootstrapChecks ] [node-1] initial heap size [268435456] not equal to maximum heap size [4294967296]; this can cause resize pauses and prevents mlockall from locking the entire heap
2020-12-15 22:24:54,612 [INFO ][o.e.c.s.MasterService ] [node-1] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {node-1}{ejfYItYER0qc2qaepF3-PA}{GNbgblpQTFSLDeq5uh5ZpA}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=17179869184, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
2020-12-15 22:24:54,620 [INFO ][o.e.c.s.ClusterApplierService] [node-1] new_master {node-1}{ejfYItYER0qc2qaepF3-PA}{GNbgblpQTFSLDeq5uh5ZpA}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=17179869184, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {node-1}{ejfYItYER0qc2qaepF3-PA}{GNbgblpQTFSLDeq5uh5ZpA}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=17179869184, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
2020-12-15 22:24:54,678 [INFO ][o.e.h.n.Netty4HttpServerTransport] [node-1] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
2020-12-15 22:24:54,679 [INFO ][o.e.n.Node ] [node-1] started
2020-12-15 22:24:54,944 [WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [node-1] Failed to clear cache for realms [[]]
2020-12-15 22:24:54,977 [INFO ][o.e.l.LicenseService ] [node-1] license [1400b5fb-5387-4e18-b7c1-a307cabbbc82] mode [basic] - valid
2020-12-15 22:24:54,990 [INFO ][o.e.g.GatewayService ] [node-1] recovered [0] indices into cluster_state