python訪問hdfs(namenode是HA)
背景
本文記錄用python訪問hdfs，hdfs有兩個(gè)namenode HA模式几缭。主要有hdfs和pyhdfs兩種方式雹有，下面分別介紹下。
1.用hdfs
Python的HDFS相關(guān)包有很多颠放，我使用的是hdfs允睹，根據(jù)官方文檔的說法运准，同時(shí)支持hdfs和WebHDFS，默認(rèn)創(chuàng)建client的方式好像是WebHDFS缭受，
需要通過datanode進(jìn)行文件操作胁澳，而HDFS則是通過namenode進(jìn)行文件操作，在這里卡了很久贯涎，也換過snakebite等包听哭，直到把端口換成datanode，才正常連接塘雳。
? learn-python3 git:(master) ? pip install hdfs
/Users/zzy/anaconda2/lib/python2.7/site-packages/cryptography/hazmat/primitives/constant_time.py:26: CryptographyDeprecationWarning: Support for your Python version is deprecated. The next version of cryptography will remove support. Please upgrade to a 2.7.x release that supports hmac.compare_digest as soon as possible.
utils.PersistentlyDeprecated2018,
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting hdfs
Downloading https://files.pythonhosted.org/packages/de/07/9d4af32b643650a368ec29899bcb2a06343810b6ed5c2382d670eb35cc5b/hdfs-2.5.0.tar.gz
Collecting docopt (from hdfs)
Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz
Requirement already satisfied: requests>=2.7.0 in /Users/zzy/anaconda2/lib/python2.7/site-packages (from hdfs) (2.18.4)
Requirement already satisfied: six>=1.9.0 in /Users/zzy/anaconda2/lib/python2.7/site-packages (from hdfs) (1.11.0)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /Users/zzy/anaconda2/lib/python2.7/site-packages (from requests>=2.7.0->hdfs) (3.0.4)
Requirement already satisfied: idna<2.7,>=2.5 in /Users/zzy/anaconda2/lib/python2.7/site-packages (from requests>=2.7.0->hdfs) (2.6)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in /Users/zzy/anaconda2/lib/python2.7/site-packages (from requests>=2.7.0->hdfs) (1.22)
Requirement already satisfied: certifi>=2017.4.17 in /Users/zzy/anaconda2/lib/python2.7/site-packages (from requests>=2.7.0->hdfs) (2019.3.9)
Building wheels for collected packages: hdfs, docopt
Building wheel for hdfs (setup.py) ... done
Stored in directory: /Users/zzy/Library/Caches/pip/wheels/e4/43/cb/5cfa9f13f579e92b6968e1f65624a6fd0e684e03deb163ce20
Building wheel for docopt (setup.py) ... done
Stored in directory: /Users/zzy/Library/Caches/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e
Successfully built hdfs docopt
Installing collected packages: docopt, hdfs
Successfully installed docopt-0.6.2 hdfs-2.5.0
運(yùn)行如下代碼：
client = Client("http://namenode1:50070",root="/",proxy='ikonw',timeout=100,session=False)
dir(client)
報(bào)錯(cuò)：
CryptographyDeprecationWarning: Support for your Python version is deprecated. The next version of cryptography will remove support. Please upgrade to a 2.7.x release that supports hmac.compare_digest as soon as possible.
utils.PersistentlyDeprecated2018,
2.pyhdfs
參考pyhdfs官網(wǎng)陆盘，hosts按照下面的方式傳參，name http host:port 的list或者逗號(hào)分隔的string
? hosts (list or str) – List of NameNode HTTP host:port strings, either as list or a comma separated string. Port defaults to 50070 if left unspecified. Note that in Hadoop 3, the default NameNode HTTP port changed to 9870; the old default of 50070 is left as-is for backwards compatibility.
最終的python腳本如下:
iknow@6-34:~/zzy $ cat HdfsUtil.py
#!/usr/bin/env python
# -*- coding: utf-8 -*-
from pyhdfs import HdfsClient
## ns1
def readFromHdfs():
#client = HdfsClient(hosts="http://namenode1:50070",user_name="iknow")
client = HdfsClient(hosts=[namenode1:50070","namenode2:50070"],user_name="iknow")
#client = HdfsClient(hosts="http://ns1:8020",user_name="iknow")
print("client:",client.list_status('/'))
response = client.open("/flink/test_data/input/iris.csv")
for line in response.read():
print("line:",line)
if __name__ == '__main__':
readFromHdfs()
運(yùn)行結(jié)果：
可以看到在當(dāng)前路徑下生成了iris.csv文件
從hdfs上copy一個(gè)文件到本地
hdfs.client.copy_to_local('/home/data/2018-06-28.out','/local_path/2018-06-28.out')
參考：
