<meta charset="utf-8">

虛擬機(jī)（centos7）：192.168.198.131

java 1.8

一谍咆、hadoop 安裝

1禾锤、設(shè)置主機(jī)名 master

vim /etc/sysconfig/network

NETWORKING=yes 
HOSTNAME=master

vim /etc/hosts

192.168.198.131 master

重啟生效 reboot

2、關(guān)閉防火墻

systemctl stop firewalld firewall-cmd --state

3摹察、設(shè)置免密碼登錄恩掷，感覺(jué)沒(méi)有必要吧（有必要，后面用到供嚎，后面有設(shè)置）

4黄娘、Hadoop-2.7.4 解壓

[root@master tools]# tar -zxvf hadoop-2.7.4.tar.gz

5峭状、jdk

[root@master hadoop-2.7.4]# java -version

java version "1.8.0_161"

Java(TM) SE Runtime Environment (build 1.8.0_161-b12)

Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)

6、查看hadoop版本

[root@master bin]# ./hadoop version

Error: JAVA_HOME is not set and could not be found.

修改hadoop環(huán)境配置

vim hadoop-env.sh

export JAVA_HOME=/usr/local/tools/jdk1.8.0_161 
export HADOOP_LOG_DIR=/data/hadoop_repo/logs/hadoop

查看版本：

[root@master bin]# ./hadoop version

Hadoop 2.7.4

Subversion https://shv@git-wip-us.apache.org/repos/asf/hadoop.git -r cd915e1e8d9d0131462a0b7301586c175728a282

Compiled by kshvachk on 2017-08-01T00:29Z

Compiled with protoc 2.5.0

From source with checksum 50b0468318b4ce9bd24dc467b7ce1148

This command was run using /usr/local/tools/hadoop-2.7.4/share/hadoop/common/hadoop-common-2.7.4.jar

7逼争、修改配置文件

[root@master hadoop]# pwd

/usr/local/tools/hadoop-2.7.4/etc/hadoop

vim core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://master:9000</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/data/hadoop_repo</value>
    </property>
</configuration>

vim hdfs-site.xml
副本數(shù)量

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

[root@master hadoop]# cp mapred-site.xml.template mapred-site.xml

vim mapred-site.xml

表示在yarn這個(gè)引擎執(zhí)行

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

vim yarn-site.xml

yarn跑哪個(gè)引擎优床，白名單

<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.env-whitelist</name>
        <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
    </property>
</configuration>

8、hdfs使用前需要進(jìn)行格式化（和格式化磁盤(pán)類似）：不要頻繁執(zhí)行誓焦，如果出錯(cuò)胆敞，把hadoop_repo目錄刪除，在執(zhí)行格式化

確保路徑 /data/hadoop_repo 存在

bin/hdfs namenode -format

20/05/05 19:44:45 INFO common.Storage: Storage directory /data/hadoop_repo/dfs/name has been successfully formatted.

9杂伟、環(huán)境變量

vim /etc/profile

HADOOP_HOME=/usr/local/tools/hadoop-2.7.4 
export PATH=${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:${PATH}

10移层、設(shè)置免密碼登錄 ssh-keygen -t rsa

如果不設(shè)置，執(zhí)行 start-all.sh 命令赫粥，會(huì)一直提示：

[root@master sbin]# ./start-all.sh

This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh

Starting namenodes on [master]

The authenticity of host 'master (192.168.198.131)' can't be established.

ECDSA key fingerprint is SHA256:1C+x54x8j+iiBi/mdjnk8mcbfYEH0ilFTHe8DMFohNw.

ECDSA key fingerprint is MD5:db:63:90:91:26:0d:40:d2:61:f2:56:23:b9:75:db:3a.

Are you sure you want to continue connecting (yes/no)? yes

master: Warning: Permanently added 'master,192.168.198.131' (ECDSA) to the list of known hosts.

root@master's password:

master: starting namenode, logging to /data/hadoop_repo/logs/hadoop/hadoop-root-namenode-master.out

The authenticity of host 'localhost (::1)' can't be established.

ECDSA key fingerprint is SHA256:1C+x54x8j+iiBi/mdjnk8mcbfYEH0ilFTHe8DMFohNw.

ECDSA key fingerprint is MD5:db:63:90:91:26:0d:40:d2:61:f2:56:23:b9:75:db:3a.

Are you sure you want to continue connecting (yes/no)? yes

localhost: Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.

root@localhost's password:

root@localhost's password: localhost: Permission denied, please try again.

localhost: starting datanode, logging to /data/hadoop_repo/logs/hadoop/hadoop-root-datanode-master.out

Starting secondary namenodes [0.0.0.0]

The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.

ECDSA key fingerprint is SHA256:1C+x54x8j+iiBi/mdjnk8mcbfYEH0ilFTHe8DMFohNw.

ECDSA key fingerprint is MD5:db:63:90:91:26:0d:40:d2:61:f2:56:23:b9:75:db:3a.

Are you sure you want to continue connecting (yes/no)? yu^H[[3~^[[D[[D^[[C

Please type 'yes' or 'no': yes

0.0.0.0: Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known hosts.

root@0.0.0.0's password:

0.0.0.0: starting secondarynamenode, logging to /data/hadoop_repo/logs/hadoop/hadoop-root-secondarynamenode-master.out

starting yarn daemons

starting resourcemanager, logging to /usr/local/tools/hadoop-2.7.4/logs/yarn-root-resourcemanager-master.out

root@localhost's password:

localhost: starting nodemanager, logging to /usr/local/tools/hadoop-2.7.4/logs/yarn-root-nodemanager-master.out

未設(shè)置ssh免密碼登錄

[root@master sbin]# ssh 192.168.198.131

root@192.168.198.131's password:

Last failed login: Tue May 5 19:51:21 PDT 2020 from localhost on ssh:notty

There was 1 failed login attempt since the last successful login.

Last login: Tue May 5 19:28:21 2020 from 192.168.198.1

設(shè)置ssh免密碼登錄

[root@master sbin]# ssh-keygen -t rsa

三次回車

執(zhí)行

[root@master ~]# cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

然后可以直接ssh了

exit退出

11观话、啟動(dòng)

[root@master sbin]# ./start-all.sh
[root@master sbin]# jps
7202 Jps
6836 NodeManager
6709 ResourceManager
6536 SecondaryNameNode
6201 NameNode
6347 DataNode

12、訪問(wèn)

localhost:8088 localhost:50070

二越平、hive 安裝

1匪燕、解壓

2、配置環(huán)境變量

[root@master apache-hive-2.3.7]# hive --version

Hive 2.3.7

Git git://Alans-MacBook-Air.local/Users/gates/git/hive -r cb213d88304034393d68cc31a95be24f5aac62b6

Compiled by gates on Tue Apr 7 12:42:45 PDT 2020

From source with checksum 9da14e8ac4737126b00a1a47f662657e

3喧笔、

[root@master conf]# cp hive-default.xml.template hive-site.xml

[root@master conf]# vim hive-site.xml

<property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>root</value>
</property>
<property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>root</value>
</property>
<property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://192.168.198.131:3306/hive</value>
</property>
<property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
</property>

4、復(fù)制mysql的驅(qū)動(dòng)程序到hive/lib下面

5龟再、創(chuàng)建mysql下的hive數(shù)據(jù)庫(kù)书闸，然后執(zhí)行

mysql 啟動(dòng)，創(chuàng)建hive庫(kù)利凑，不使用hive自帶的

[root@master mysql]# service mysqld start

/etc/init.d/mysqld: line 239: my_print_defaults: command not found

/etc/init.d/mysqld: line 259: cd: /usr/local/mysql: No such file or directory

Starting MySQL ERROR! Couldn't find MySQL server (/usr/local/mysql/bin/mysqld_safe)

CREATE DATABASE hive;

[root@master bin]# schematool -dbType mysql -initSchema

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/usr/local/tools/apache-hive-2.3.7/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/local/tools/hadoop-2.7.4/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Metastore connection URL: jdbc:mysql://192.168.198.131:3306/hive

Metastore Connection Driver : com.mysql.jdbc.Driver

Metastore connection User: root

Starting metastore schema initialization to 2.3.0

Initialization script hive-schema-2.3.0.mysql.sql

Initialization script completed

schemaTool completed

[root@master bin]#

6浆劲、執(zhí)行hive命令

[root@master apache-hive-2.3.7]# hive

which: no hbase in (/usr/local/tools/apache-hive-2.3.7/bin:/usr/local/tools/hadoop-2.7.4/bin:/usr/local/tools/hadoop-2.7.4/sbin:/usr/local/tools/node/bin:/usr/local/tools/apache-maven-3.6.3/bin:/usr/local/tools/jdk1.8.0_161/bin:/usr/local/tools/node/bin:/usr/local/tools/apache-maven-3.6.3/bin:/usr/local/tools/jdk1.8.0_161/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin)

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/usr/local/tools/apache-hive-2.3.7/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/local/tools/hadoop-2.7.4/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/usr/local/tools/apache-hive-2.3.7/lib/hive-common-2.3.7.jar!/hive-log4j2.properties Async: true

Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.

hive>

7、查看第5步創(chuàng)建的數(shù)據(jù)庫(kù)哀澈，已經(jīng)有了很多表

mysql -uroot -p

目測(cè)主要看兩個(gè)表：TBLS , COLUMNS_V2

8牌借、測(cè)試

[root@master apache-hive-2.3.7]# hive

which: no hbase in (/usr/local/tools/apache-hive-2.3.7/bin:/usr/local/tools/hadoop-2.7.4/bin:/usr/local/tools/hadoop-2.7.4/sbin:/usr/local/tools/node/bin:/usr/local/tools/apache-maven-3.6.3/bin:/usr/local/tools/jdk1.8.0_161/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin)

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/usr/local/tools/apache-hive-2.3.7/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/local/tools/hadoop-2.7.4/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/usr/local/tools/apache-hive-2.3.7/lib/hive-common-2.3.7.jar!/hive-log4j2.properties Async: true

Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.

hive> show tables;

OK

Time taken: 4.426 seconds

hive> create database hive_1;

OK

Time taken: 0.198 seconds

hive> show databases;

OK

default

hive_1

Time taken: 0.03 seconds, Fetched: 2 row(s)

hive>

看看hadoop存儲(chǔ)信息：

[root@master ~]# hadoop fs -lsr /

lsr: DEPRECATED: Please use 'ls -R' instead.

drwx-wx-wx - root supergroup 0 2020-05-05 22:29 /tmp

drwx-wx-wx - root supergroup 0 2020-05-05 22:29 /tmp/hive

drwx------ - root supergroup 0 2020-05-06 00:42 /tmp/hive/root

drwx------ - root supergroup 0 2020-05-06 00:42 /tmp/hive/root/1fed50ca-d9f6-4b5c-b80b-a81a66679812

drwx------ - root supergroup 0 2020-05-06 00:42 /tmp/hive/root/1fed50ca-d9f6-4b5c-b80b-a81a66679812/_tmp_space.db

drwxr-xr-x - root supergroup 0 2020-05-06 00:53 /user

drwxr-xr-x - root supergroup 0 2020-05-06 00:53 /user/hive

drwxr-xr-x - root supergroup 0 2020-05-06 00:53 /user/hive/warehouse

drwxr-xr-x - root supergroup 0 2020-05-06 00:53 /user/hive/warehouse/hive_1.db

三、kafka （偽分布式）安裝割按，前提安裝zookeeper

1膨报、安裝

# tar zxvf kafka_2.11-2.2.1.tgz # mv kafka_2.11-2.2.1 kafka # cd kafka

啟動(dòng)kafka服務(wù)

# nohup bin/kafka-server-start.sh config/server.properties &

創(chuàng)建topic

# bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

查看topic

# bin/kafka-topics.sh --list --zookeeper localhost:2181

2、測(cè)試

使用kafka-console-producer.sh 發(fā)送消息

# bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test

使用kafka-console-consumer.sh消費(fèi)消息

# bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

3适荣、kafka集群

config 下配置多個(gè)server屬性文件现柠，設(shè)置不同的 broker.id

bin/kafka-server-start.sh config/server-1.properties &

需要先啟動(dòng)zookeeper

四、hbase 安裝

1弛矛、解壓够吩，配置環(huán)境變量

[root@master hbase-1.4.13]# vim /etc/profile

HBASE_HOME=/usr/local/tools/hbase-1.4.13
export PATH=${HBASE_HOME}/bin:${PATH}

[root@master hbase-1.4.13]# source /etc/profile

2、修改配置文件

向hbase-env.sh中添加：

export JAVA_HOME=/usr/local/tools/jdk1.8.0_161 export HBASE_MANAGES_ZK=false

修改hbase-site.xml為

<configuration>

<property>
    <name>hbase.rootdir</name>
    <value>hdfs://master:9000/hbase</value>
</property>

<property>
 <name>hbase.zookeeper.property.clientPort</name>
 <value>2181</value>
 <description>Property from ZooKeeper'sconfig zoo.cfg. The port at which the clients will connect.
 </description>
</property>

<property>
    <name>hbase.tmp.dir</name>
    <value>/usr/local/tools/hbase-1.4.13/data</value>
</property>
<!-- zk的位置丈氓，zk偽集群周循，value只有一個(gè)强法，如果是集群，主機(jī)名以逗號(hào)分隔 -->
        <property>
                <name>hbase.zookeeper.quorum</name>
                <value>master</value>
        </property>
<!-- 
false表示單機(jī)模式 true表示集群模式 
此處必須為true湾笛，不然hbase仍用自帶的zk饮怯，若啟動(dòng)了外部的zookeeper，會(huì)導(dǎo)致沖突迄本，hbase啟動(dòng)不起來(lái)
-->
<property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
    <description>The mode the cluster will be in. Possible values are
      false: standalone and pseudo-distributed setups with managed Zookeeper
      true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh)
    </description>
</property>
</configuration>

3硕淑、啟動(dòng)hbase

[root@master bin]# ./start-hbase.sh

訪問(wèn)：查看HBase界面 端口 16010

4、問(wèn)題總結(jié)

1）

running master, logging to /usr/local/tools/hbase-1.4.13/bin/../logs/hbase-root-master-master.out

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0

解決方案：

如報(bào)錯(cuò)所示嘉赎，在hbase-env.sh配置文件中存在某些在jdk8中不存在命令置媳，查看配置文件發(fā)現(xiàn)如下場(chǎng)景：

Configure PermSize. Only needed in JDK7. You can safely remove it for JDK8+

export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS -XX:PermSize=128m -XX:MaxPermSize=128m -XX:ReservedCodeCacheSize=256m"

export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -XX:PermSize=128m -XX:MaxPermSize=128m -XX:ReservedCodeCacheSize=256m"

注釋即可

2）

HMaster和HRegionServer是Hbase的兩個(gè)子進(jìn)程，但是使用jps發(fā)現(xiàn)沒(méi)有啟動(dòng)起來(lái)公条，所以去我們配置的logs查看錯(cuò)誤信息拇囊。提示：

Could not start ZK at requested port of 2181. ZK was started at port: 2182. Aborting as clients (e.g. shell) will not be able to find this ZK quorum.

但是在hbase-env.sh文件中設(shè)置了export HBASE_MANAGES_ZK=false

設(shè)置不使用自帶zookeeper，這一步設(shè)置完按理說(shuō)就可以使用獨(dú)立的zookeeper程序了,但是還是報(bào)錯(cuò)靶橱。很明顯寥袭，這是啟動(dòng)自帶zookeeper與獨(dú)立zookeeper沖突了。因?yàn)榘裩base.cluster.distributed設(shè)置為false关霸，也就是讓hbase以standalone模式運(yùn)行時(shí)传黄，依然會(huì)去啟動(dòng)自帶的zookeeper。

所以要做如下設(shè)置队寇，值為true：

vim conf/hbase-site.xml

<property>

<name>hbase.cluster.distributed</name>

<value>true</value>

</property>

3）

2020-05-07 02:57:17,302 INFO [main-SendThread(192.168.181.131:2181)] zookeeper.ClientCnxn: Opening socket connection to server 192.168.181.131/192.168.181.131:2181. Will not attempt to authenticate using SASL (unknown error)

hbase-site.xml 配置的zookeeper 主機(jī)為hostname（master）膘掰，之前是ip

五、solr集群 solr-7.5.0佳遣，前提识埋，已經(jīng)配置好了zk偽集群

1、解壓

2零渐、啟動(dòng)測(cè)試

solr start

3窒舟、配置zk集群和SOLR_PORT，zokeeper偽集群已經(jīng)配置好诵盼，需要在solr中配置zk和SOLR_PORT

[root@master bin]# vim solr.in.sh

ZK_HOST="192.168.198.131:2181,192.168.198.131:2182,192.168.198.131:2183"

4惠豺、solr 創(chuàng)建 collection

bash $SOLR_HOME/bin/solr create -c vertex_index -d $SOLR_HOME/apache-atlas-conf -shards 2 -replicationFactor 2

bash $SOLR_HOME/bin/solr create -c edge_index -d $SOLR_HOME/apache-atlas-conf -shards 2 -replicationFactor 2

bash $SOLR_HOME/bin/solr create -c fulltext_index -d $SOLR_HOME/apache-atlas-conf -shards 2 -replicationFactor 2

5、solr集群?jiǎn)?dòng)

zk偽集群

solr偽集群

/usr/local/tools/solr-cloud/solr1/bin/solr start -force 
/usr/local/tools/solr-cloud/solr2/bin/solr start -force 
/usr/local/tools/solr-cloud/solr3/bin/solr start -force 
/usr/local/tools/solr-cloud/solr4/bin/solr start -force 

/usr/local/tools/solr-cloud/solr1/bin/solr stop 
/usr/local/tools/solr-cloud/solr2/bin/solr stop 
/usr/local/tools/solr-cloud/solr3/bin/solr stop 
/usr/local/tools/solr-cloud/solr4/bin/solr stop

[root@master bin]#./solr create_collection -c test_collection -shards 2 -replicationFactor 2 -force

-c 指定庫(kù)(collection)名稱

-shards 指定分片數(shù)量,可簡(jiǎn)寫(xiě)為 -s ,索引數(shù)據(jù)會(huì)分布在這些分片上

-replicationFactor 每個(gè)分片的副本數(shù)量

-force 上文已說(shuō)明

加 -force 是因?yàn)閟olr不允許使用 root 進(jìn)行操作的风宁，其他賬戶可不加

solr集群完成

參考：https://blog.csdn.net/qq_37936542/article/details/83113083

六耕腾、apache atlas 獨(dú)立部署開(kāi)始

使用atlas內(nèi)置的hbase和solr

/usr/local/project/apache-atlas-sources-2.0.0/distro/target/apache-atlas-2.0.0-server

不使用atlas內(nèi)置的hbase和solr

/usr/local/project/apache-atlas-sources-2.0.0-alone

[root@master apache-atlas-sources-2.0.0-alone]# mvn clean -DskipTests package -Pdist

編譯完成，使用 distro/target/apache-atlas-2.0.0-server

集成solr到apache atlas

cd /usr/local/project/apache-atlas-sources-2.0.0-alone/distro/target/apache-atlas-2.0.0-server/apache-atlas-2.0.0/conf

[root@master conf]# cp -r solr/ /usr/local/tools/solr-7.5.0/apache-atlas-conf

獨(dú)立部署：主要修改配置文件
atlas-env.sh

export HBASE_CONF_DIR=/usr/local/project/apache-atlas-sources-2.0.0-alone/distro/target/apache-atlas-2.0.0-server/apache-atlas-2.0.0/conf/hbase/conf

atlas-application.properties

#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

#########  Graph Database Configs  #########

# Graph Database

#Configures the graph database to use.  Defaults to JanusGraph
atlas.graphdb.backend=org.apache.atlas.repository.graphdb.janus.AtlasJanusGraphDatabase

# Graph Storage
# Set atlas.graph.storage.backend to the correct value for your desired storage
# backend. Possible values:
#
# hbase
# cassandra
# embeddedcassandra - Should only be set by building Atlas with  -Pdist,embedded-cassandra-solr
# berkeleyje
#
# See the configuration documentation for more information about configuring the various  storage backends.
#
atlas.graph.storage.backend=hbase2
atlas.graph.storage.hbase.table=apache_atlas_janus

#Hbase
#For standalone mode , specify localhost
#for distributed mode, specify zookeeper quorum here
atlas.graph.storage.hostname=master:2181,master:2182,master:2183
atlas.graph.storage.hbase.regions-per-server=1
atlas.graph.storage.lock.wait-time=10000

#In order to use Cassandra as a backend, comment out the hbase specific properties above, and uncomment the
#the following properties
#atlas.graph.storage.clustername=
#atlas.graph.storage.port=

# Gremlin Query Optimizer
#
# Enables rewriting gremlin queries to maximize performance. This flag is provided as
# a possible way to work around any defects that are found in the optimizer until they
# are resolved.
#atlas.query.gremlinOptimizerEnabled=true

# Delete handler
#
# This allows the default behavior of doing "soft" deletes to be changed.
#
# Allowed Values:
# org.apache.atlas.repository.store.graph.v1.SoftDeleteHandlerV1 - all deletes are "soft" deletes
# org.apache.atlas.repository.store.graph.v1.HardDeleteHandlerV1 - all deletes are "hard" deletes
#
#atlas.DeleteHandlerV1.impl=org.apache.atlas.repository.store.graph.v1.SoftDeleteHandlerV1

# Entity audit repository
#
# This allows the default behavior of logging entity changes to hbase to be changed.
#
# Allowed Values:
# org.apache.atlas.repository.audit.HBaseBasedAuditRepository - log entity changes to hbase
# org.apache.atlas.repository.audit.CassandraBasedAuditRepository - log entity changes to cassandra
# org.apache.atlas.repository.audit.NoopEntityAuditRepository - disable the audit repository
#
atlas.EntityAuditRepository.impl=org.apache.atlas.repository.audit.HBaseBasedAuditRepository

# if Cassandra is used as a backend for audit from the above property, uncomment and set the following
# properties appropriately. If using the embedded cassandra profile, these properties can remain
# commented out.
# atlas.EntityAuditRepository.keyspace=atlas_audit
# atlas.EntityAuditRepository.replicationFactor=1


# Graph Search Index
atlas.graph.index.search.backend=solr

#Solr
#Solr cloud mode properties
atlas.graph.index.search.solr.mode=cloud
atlas.graph.index.search.solr.zookeeper-url=master:2181,master:2182,master:2183
atlas.graph.index.search.solr.zookeeper-connect-timeout=60000
atlas.graph.index.search.solr.zookeeper-session-timeout=60000
atlas.graph.index.search.solr.wait-searcher=true

#Solr http mode properties
#atlas.graph.index.search.solr.mode=http
#atlas.graph.index.search.solr.http-urls=http://localhost:8983/solr

# ElasticSearch support (Tech Preview)
# Comment out above solr configuration, and uncomment the following two lines. Additionally, make sure the
# hostname field is set to a comma delimited set of elasticsearch master nodes, or an ELB that fronts the masters.
#
# Elasticsearch does not provide authentication out of the box, but does provide an option with the X-Pack product
# https://www.elastic.co/products/x-pack/security
#
# Alternatively, the JanusGraph documentation provides some tips on how to secure Elasticsearch without additional
# plugins: http://docs.janusgraph.org/latest/elasticsearch.html
#atlas.graph.index.hostname=localhost
#atlas.graph.index.search.elasticsearch.client-only=true

# Solr-specific configuration property
atlas.graph.index.search.max-result-set-size=150

#########  Notification Configs  #########
atlas.notification.embedded=false
atlas.kafka.data=${sys:atlas.home}/data/kafka
atlas.kafka.zookeeper.connect=master:2181/kafka,master:2182/kafka,master:2183/kafka
atlas.kafka.bootstrap.servers=master:9092,master:9093,master:9094
atlas.kafka.zookeeper.session.timeout.ms=400
atlas.kafka.zookeeper.connection.timeout.ms=200
atlas.kafka.zookeeper.sync.time.ms=20
atlas.kafka.auto.commit.interval.ms=1000
atlas.kafka.hook.group.id=atlas

atlas.kafka.enable.auto.commit=false
atlas.kafka.auto.offset.reset=earliest
atlas.kafka.session.timeout.ms=30000
atlas.kafka.offsets.topic.replication.factor=1
atlas.kafka.poll.timeout.ms=1000

atlas.notification.create.topics=true
atlas.notification.replicas=1
atlas.notification.topics=ATLAS_HOOK,ATLAS_ENTITIES
atlas.notification.log.failed.messages=true
atlas.notification.consumer.retry.interval=500
atlas.notification.hook.retry.interval=1000
# Enable for Kerberized Kafka clusters
#atlas.notification.kafka.service.principal=kafka/_HOST@EXAMPLE.COM
#atlas.notification.kafka.keytab.location=/etc/security/keytabs/kafka.service.keytab

## Server port configuration
#atlas.server.http.port=21000
#atlas.server.https.port=21443

#########  Security Properties  #########

# SSL config
atlas.enableTLS=false

#truststore.file=/path/to/truststore.jks
#cert.stores.credential.provider.path=jceks://file/path/to/credentialstore.jceks

#following only required for 2-way SSL
#keystore.file=/path/to/keystore.jks

# Authentication config

atlas.authentication.method.kerberos=false
atlas.authentication.method.file=true

#### ldap.type= LDAP or AD
atlas.authentication.method.ldap.type=none

#### user credentials file
atlas.authentication.method.file.filename=${sys:atlas.home}/conf/users-credentials.properties

### groups from UGI
#atlas.authentication.method.ldap.ugi-groups=true

######## LDAP properties #########
#atlas.authentication.method.ldap.url=ldap://<ldap server url>:389
#atlas.authentication.method.ldap.userDNpattern=uid={0},ou=People,dc=example,dc=com
#atlas.authentication.method.ldap.groupSearchBase=dc=example,dc=com
#atlas.authentication.method.ldap.groupSearchFilter=(member=uid={0},ou=Users,dc=example,dc=com)
#atlas.authentication.method.ldap.groupRoleAttribute=cn
#atlas.authentication.method.ldap.base.dn=dc=example,dc=com
#atlas.authentication.method.ldap.bind.dn=cn=Manager,dc=example,dc=com
#atlas.authentication.method.ldap.bind.password=<password>
#atlas.authentication.method.ldap.referral=ignore
#atlas.authentication.method.ldap.user.searchfilter=(uid={0})
#atlas.authentication.method.ldap.default.role=<default role>


######### Active directory properties #######
#atlas.authentication.method.ldap.ad.domain=example.com
#atlas.authentication.method.ldap.ad.url=ldap://<AD server url>:389
#atlas.authentication.method.ldap.ad.base.dn=(sAMAccountName={0})
#atlas.authentication.method.ldap.ad.bind.dn=CN=team,CN=Users,DC=example,DC=com
#atlas.authentication.method.ldap.ad.bind.password=<password>
#atlas.authentication.method.ldap.ad.referral=ignore
#atlas.authentication.method.ldap.ad.user.searchfilter=(sAMAccountName={0})
#atlas.authentication.method.ldap.ad.default.role=<default role>

#########  JAAS Configuration ########

#atlas.jaas.KafkaClient.loginModuleName = com.sun.security.auth.module.Krb5LoginModule
#atlas.jaas.KafkaClient.loginModuleControlFlag = required
#atlas.jaas.KafkaClient.option.useKeyTab = true
#atlas.jaas.KafkaClient.option.storeKey = true
#atlas.jaas.KafkaClient.option.serviceName = kafka
#atlas.jaas.KafkaClient.option.keyTab = /etc/security/keytabs/atlas.service.keytab
#atlas.jaas.KafkaClient.option.principal = atlas/_HOST@EXAMPLE.COM

#########  Server Properties  #########
atlas.rest.address=http://localhost:21000
# If enabled and set to true, this will run setup steps when the server starts
#atlas.server.run.setup.on.start=false

#########  Entity Audit Configs  #########
atlas.audit.hbase.tablename=apache_atlas_entity_audit
atlas.audit.zookeeper.session.timeout.ms=1000
atlas.audit.hbase.zookeeper.quorum=master:2181,master:2182,master:2183

#########  High Availability Configuration ########
atlas.server.ha.enabled=false
#### Enabled the configs below as per need if HA is enabled #####
#atlas.server.ids=id1
#atlas.server.address.id1=localhost:21000
#atlas.server.ha.zookeeper.connect=localhost:2181
#atlas.server.ha.zookeeper.retry.sleeptime.ms=1000
#atlas.server.ha.zookeeper.num.retries=3
#atlas.server.ha.zookeeper.session.timeout.ms=20000
## if ACLs need to be set on the created nodes, uncomment these lines and set the values ##
#atlas.server.ha.zookeeper.acl=<scheme>:<id>
#atlas.server.ha.zookeeper.auth=<scheme>:<authinfo>



######### Atlas Authorization #########
atlas.authorizer.impl=simple
atlas.authorizer.simple.authz.policy.file=atlas-simple-authz-policy.json

#########  Type Cache Implementation ########
# A type cache class which implements
# org.apache.atlas.typesystem.types.cache.TypeCache.
# The default implementation is org.apache.atlas.typesystem.types.cache.DefaultTypeCache which is a local in-memory type cache.
#atlas.TypeCache.impl=

#########  Performance Configs  #########
#atlas.graph.storage.lock.retries=10
#atlas.graph.storage.cache.db-cache-time=120000

#########  CSRF Configs  #########
atlas.rest-csrf.enabled=true
atlas.rest-csrf.browser-useragents-regex=^Mozilla.*,^Opera.*,^Chrome.*
atlas.rest-csrf.methods-to-ignore=GET,OPTIONS,HEAD,TRACE
atlas.rest-csrf.custom-header=X-XSRF-HEADER

############ KNOX Configs ################
#atlas.sso.knox.browser.useragent=Mozilla,Chrome,Opera
#atlas.sso.knox.enabled=true
#atlas.sso.knox.providerurl=https://<knox gateway ip>:8443/gateway/knoxsso/api/v1/websso
#atlas.sso.knox.publicKey=

############ Atlas Metric/Stats configs ################
# Format: atlas.metric.query.<key>.<name>
atlas.metric.query.cache.ttlInSecs=900
#atlas.metric.query.general.typeCount=
#atlas.metric.query.general.typeUnusedCount=
#atlas.metric.query.general.entityCount=
#atlas.metric.query.general.tagCount=
#atlas.metric.query.general.entityDeleted=
#
#atlas.metric.query.entity.typeEntities=
#atlas.metric.query.entity.entityTagged=
#
#atlas.metric.query.tags.entityTags=

#########  Compiled Query Cache Configuration  #########

# The size of the compiled query cache.  Older queries will be evicted from the cache
# when we reach the capacity.

#atlas.CompiledQueryCache.capacity=1000

# Allows notifications when items are evicted from the compiled query
# cache because it has become full.  A warning will be issued when
# the specified number of evictions have occurred.  If the eviction
# warning threshold <= 0, no eviction warnings will be issued.

#atlas.CompiledQueryCache.evictionWarningThrottle=0


#########  Full Text Search Configuration  #########

#Set to false to disable full text search.
#atlas.search.fulltext.enable=true

#########  Gremlin Search Configuration  #########

#Set to false to disable gremlin search.
atlas.search.gremlin.enable=false


########## Add http headers ###########

#atlas.headers.Access-Control-Allow-Origin=*
#atlas.headers.Access-Control-Allow-Methods=GET,OPTIONS,HEAD,PUT,POST
#atlas.headers.<headerName>=<headerValue>

######### Hive Hook Configs #######
 
atlas.hook.hive.synchronous=false
 
atlas.hook.hive.numRetries=3
 
atlas.hook.hive.queueSize=10000
 
######### Sqoop Hook Configs #######
 
atlas.hook.sqoop.synchronous=false
 
atlas.hook.sqoop.numRetries=3
 
atlas.hook.sqoop.queueSize=10000

storage.cql.protocol-version=3
storage.cql.local-core-connections-per-host=10
storage.cql.local-max-connections-per-host=20
storage.cql.local-max-requests-per-connection=2000
storage.buffer-size=1024

七杀糯、atlas 獨(dú)立部署問(wèn)題總結(jié)

1）

Could not find hbase-site.xml in %s. Please set env var HBASE_CONF_DIR to the hbase client conf dir

軟連接不對(duì) 扫俺？

cd /usr/local/project/apache-atlas-sources-2.0.0-alone/distro/target/apache-atlas-2.0.0-server/apache-atlas-2.0.0

[root@cdh632-worker03 atlas]# ln -s /etc/hbase/conf /opt/module/apache-atlas-sources-2.0.0/distro/target/apache-atlas-2.0.0-server/apache-atlas-2.0.0/conf/hbase/conf

[root@cdh632-worker03 atlas]# pwd

vim atlas-env.sh

export HBASE_CONF_DIR=/usr/local/project/apache-atlas-sources-2.0.0-alone/distro/target/apache-atlas-2.0.0-server/apache-atlas-2.0.0/conf/hbase/conf

2）

啟動(dòng)報(bào)錯(cuò)

2020-05-23 10:30:46,794 WARN - [main:] ~ Unexpected exception during getDeployment() (HBaseStoreManager:399)

java.lang.RuntimeException: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend

修改配置文件

master:2181,master:2182,master:2183

參考：
https://blog.csdn.net/qq_34024275/article/details/105393745

圖數(shù)據(jù)庫(kù)建立流程如下：

配置文件配置圖數(shù)據(jù)庫(kù)的數(shù)據(jù)存儲(chǔ)位置和索引存儲(chǔ)位置：

atlas.graphdb.backend=org.apache.atlas.repository.graphdb.janus.AtlasJanusGraphDatabase

# Graph Storage

atlas.graph.storage.backend=hbase

atlas.graph.storage.port=2181

atlas.graph.storage.hbase.table=atlas-test

atlas.graph.storage.hostname=docker2,docker3,docker4

# Graph Search Index Backend

atlas.graph.index.search.backend=elasticsearch

atlas.graph.index.search.hostname=127.0.0.1

atlas.graph.index.search.index-name=atlas_test

3）

at org.apache.hadoop.hbase.regionserver.HRegion.checkFamily

atlas hbase 是2.0，本地啟動(dòng)的是 1.4.13

解決：

<property>

<name>hbase.rootdir</name>

<value>hdfs://master:9000/hbase</value>

</property>

<property>

<name>hbase.zookeeper.property.clientPort</name>

<value>2181</value>

<description>Property from ZooKeeper'sconfig zoo.cfg. The port at which the clients will connect.

</description>

</property>

<property>

<name>hbase.tmp.dir</name>

<value>/usr/local/tools/hbase-2.2.4/data</value>

</property>

<!-- zk的位置 -->

<property>

<name>hbase.zookeeper.quorum</name>

<value>master</value>

</property>

<property>

<name>hbase.cluster.distributed</name>

<value>true</value>

</property>

4）

2020-05-23 23:52:00,415 WARN - [main:] ~ JanusGraphException: Could not open global configuration (AtlasJanusGraphDatabase:167)

2020-05-23 23:52:00,432 WARN - [main:] ~ Unexpected exception during getDeployment() (HBaseStoreManager:399)

java.lang.RuntimeException: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend

配置文件添加：

storage.cql.protocol-version=3

storage.cql.local-core-connections-per-host=10

storage.cql.local-max-connections-per-host=20

storage.cql.local-max-requests-per-connection=2000

storage.buffer-size=1024

5）

Caused by: org.apache.solr.common.SolrException: Cannot connect to cluster at master:2181,master:2182,master:2183/solr: cluster not found/not ready

at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:385)

at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.connect(ZkClientClusterStateProvider.java:141)

at org.apache.solr.client.solrj.impl.CloudSolrClient.connect(CloudSolrClient.java:383)

at org.janusgraph.diskstorage.solr.Solr6Index.<init>(Solr6Index.java:218)

master:2181,master:2182,master:2183/solr

改成 master:2181,master:2182,master:2183 就可以了

補(bǔ)充：

1固翰、java環(huán)境變量

vim /etc/profile

加上以下代碼：

export JAVA_HOME=/usr/local/tools/jdk1.8.0_161 
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar 
export PATH=$JAVA_HOME/bin:$PATH

然后保存退出狼纬，使配置生效：

source /etc/profile

2刃唐、zookeeper 偽分布式

[root@master zookeeper-01]# cd data/

[root@master data]# touch myid

[root@master data]# echo 1 >> myid

修改配置文件宠叼。把conf目錄下的zoo_sample.cfg文件改名為zoo.cfg（IP號(hào)記得改成你自己的）

server.1=192.168.198.131:2881:3881

server.2=192.168.198.131:2882:3882

server.3=192.168.198.131:2883:3883

zookeeper 集群?jiǎn)?dòng)報(bào)錯(cuò)

org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException

zoo.cfg 需要改成自己創(chuàng)建的data路徑，因?yàn)槔锩嬗衜yid文件

dataDir=/usr/local/tools/zk-cloud/zookeeper01/data

創(chuàng)建啟動(dòng)文件，省的一個(gè)一個(gè)啟動(dòng)

vim zk-start.sh

cd zookeeper01/bin

./zkServer.sh start

cd ../../

cd zookeeper02/bin

./zkServer.sh start

cd ../../

cd zookeeper03/bin

./zkServer.sh start

cd ../../

chmod -R 755 zk-start.sh

zookeeper啟動(dòng)成功封断，查看 zkServer.sh stauts