一垢油、文件下載和解壓
1)下載地址:http://www.alluxio.org/download
2) 解壓命令如下:
$wgethttp://alluxio.org/downloads/files/1.2.0/alluxio-1.2.0-bin.tar.gz
$tar?xvfz?alluxio-1.2.0-bin.tar.gz
$cdalluxio-1.2.0
二、 配置文件更改
目前只是基本配置更改:
1) /data/spark/software/alluxio-1.2.0/conf下的??alluxio-env.sh.template 復制一份為:?alluxio-env.sh
更改如下:
#!/usr/bin/env bash
#
# The Alluxio Open Foundation licenses this work under the Apache License, version 2.0
# (the "License"). You may not use this work except in compliance with the License, which is
# available at www.apache.org/licenses/LICENSE-2.0
#
# This software is distributed on an "AS IS" basis, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
# either express or implied, as more fully set forth in the License.
#
# See the NOTICE file distributed with this work for information regarding copyright ownership.
#
# Copy it as alluxio-env.sh and edit that to configure Alluxio for your
# site. This file is sourced to launch Alluxio servers or use Alluxio shell
# commands.
#
# This file provides one way to configure Alluxio options by setting the
# following listed environment variables. Note that, setting this file will not
# affect jobs (e.g., Spark job or MapReduce job) that are using Alluxio client
# as a library. Alternatively, you can edit alluxio-site.properties file, where
# you can set all the configuration options supported by Alluxio
# (http://alluxio.org/documentation/)which is respected by both external jobs
# and Alluxio servers (or shell).
# The directory where Alluxio deployment is installed. (Default: the parent directory of libexec/).
exportALLUXIO_HOME=/data/spark/software/alluxio-1.2.0
# The directory where log files are stored. (Default: ${ALLUXIO_HOME}/logs).
# ALLUXIO_LOGS_DIR
# Hostname of the master.
# ALLUXIO_MASTER_HOSTNAME
exportALLUXIO_MASTER_HOSTNAME=spark29
# This is now deprecated. Support will be removed in v2.0
# ALLUXIO_MASTER_ADDRESS
#export ALLUXIO_MASTER_ADDRESS=spark29
# The directory where a worker stores in-memory data. (Default: /mnt/ramdisk).
# E.g. On linux,? /mnt/ramdisk for ramdisk, /dev/shm for tmpFS; on MacOS, /Volumes/ramdisk for ramdisk
# ALLUXIO_RAM_FOLDER
exportALLUXIO_RAM_FOLDER=/data/spark/software/alluxio-1.2.0/ramdisk
# Address of the under filesystem address. (Default: ${ALLUXIO_HOME}/underFSStorage)
# E.g. "/my/local/path" to use local fs, "hdfs://localhost:9000/alluxio" to use a local hdfs
# ALLUXIO_UNDERFS_ADDRESS
exportALLUXIO_UNDERFS_ADDRESS=hdfs://spark29:9000
# How much memory to use per worker. (Default: 1GB)
# E.g. "1000MB", "2GB"
# ALLUXIO_WORKER_MEMORY_SIZE
exportALLUXIO_WORKER_MEMORY_SIZE=12GB
# Config properties set for Alluxio master, worker and shell. (Default: "")
# E.g. "-Dalluxio.master.port=39999"
# ALLUXIO_JAVA_OPTS
# Config properties set for Alluxio master daemon. (Default: "")
# E.g. "-Dalluxio.master.port=39999"
# ALLUXIO_MASTER_JAVA_OPTS
# Config properties set for Alluxio worker daemon. (Default: "")
# E.g. "-Dalluxio.worker.port=49999" to set worker port, "-Xms2048M -Xmx2048M" to limit the heap size of worker.
# ALLUXIO_WORKER_JAVA_OPTS
# Config properties set for Alluxio shell. (Default: "")
# E.g. "-Dalluxio.user.file.writetype.default=CACHE_THROUGH"
# ALLUXIO_USER_JAVA_OPTS
2)worker 下面的添加worker節(jié)點的地址
spark24
spark30
spark31
spark32
spark33
三 它呀、主機配置更改
1)在家目錄下更改?.bash_profile 添加一下內(nèi)容:
export TACHYON_HOME=/data/spark/software/alluxio-1.2.0
PATH=$PATH:$HOME/bin:$HADOOP/bin:$JAVA_HOME/bin:$TACHYON_HOME/bin
2) 生效配置
source .bash_profile
四 末早、Spark 添加依賴Jar
1诈闺、在所有的spark主機的spark安裝目錄下的conf目錄下
更改spark-env.sh 后面添加:export SPARK_CLASSPATH="/data/spark/software/spark-1.5.2-bin-hadoop2.6/lib/alluxio-core-client-spark-1.2.0-jar-with-dependencies.jar:$SPARK_CLASSPATH"
五 、分發(fā)到各個Worker節(jié)點上去
1诀拭、alluxio 軟件:scp -r ./alluxio-1.2.0spark30:/data/spark/software/
六、格式化和啟動
1众辨、進入到alluxio的安裝目錄下面的bin目錄端三,執(zhí)行命令:?alluxio format ?進行內(nèi)存格式化。
2鹃彻、啟動集群:./alluxio-start.sh all
七郊闯、可能遇到問題
1、啟動worker報錯蛛株,報錯內(nèi)容:Pseudo-terminal will not be allocated because stdin is not a terminal.
更改:alluxio\bin\alluxio-workers.sh ?的44行內(nèi)容
原始內(nèi)容為:
nohup ssh -o ConnectTimeout=5 -o StrictHostKeyChecking=no -t ${worker} ${LAUNCHER} \
改成如下:
nohup ssh -o ConnectTimeout=5 -o StrictHostKeyChecking=no -tt ?$?{worker} ?${LAUNCHER}?\
2团赁、如果啟動報sudo相關命令錯誤,是因為啟動用戶未在sudoers里面谨履,需要將用戶添加到此文件中欢摄,添加方法搜下root位置,再后面添加即可笋粟。
內(nèi)容如下:
root ALL=(ALL) ALL
spark ALL=(ALL) ALL
同時把此文件中的:#Defaults ? ?requiretty 注釋掉怀挠。
3、如果還報錯害捕,可以在啟動master之后绿淋,一個一個節(jié)點去啟動worker。
八尝盼、 官網(wǎng)安裝說明
官網(wǎng)安裝說明:http://www.alluxio.org/docs/master/cn/Running-Alluxio-on-a-Cluster.html?中文的吞滞,可以看看。
九盾沫、安裝后確認
通過訪問鏈接:http://master:19999/ 去查看安裝是否成功裁赠。