一厕倍、地址和端口問(wèn)題
spark官網(wǎng) 說(shuō)默認(rèn)端口是 4040完沪，使用spark.ui.port 進(jìn)行配置spark配置鳍寂。
我們使用的是CDH，使用 spark.ui.port 查看沒(méi)有找到該配置
image.png
-
搜索
port响蕴，找到如下幾個(gè)端口焕毫，看著也就18088比較像
image.png 使用如下命令訪問(wèn)
curl http://xxx.xxx.xxx.xxx:18088/api/v1/applications?limit=2
得到如下結(jié)果，看著挺像那么回事锭魔，仔細(xì)一看例证，怎么都是歷史的任務(wù)？
[ {
"id" : "application_1618400751843_125338",
"name" : "cat3-title-vec",
"attempts" : [ {
"startTime" : "2021-05-13T13:33:24.965GMT",
"endTime" : "2021-05-13T13:34:41.551GMT",
"lastUpdated" : "2021-05-13T13:34:46.298GMT",
"duration" : 76586,
"sparkUser" : "supdev",
"completed" : true,
"appSparkVersion" : "2.4.0-cdh6.2.0",
"startTimeEpoch" : 1620912804965,
"lastUpdatedEpoch" : 1620912886298,
"endTimeEpoch" : 1620912881551
} ]
}, {
"id" : "application_1618400751843_125318",
"name" : "re-list-demo",
"attempts" : [ {
"startTime" : "2021-05-13T13:26:58.297GMT",
"endTime" : "2021-05-13T13:28:58.500GMT",
"lastUpdated" : "2021-05-13T13:28:58.570GMT",
"duration" : 120203,
"sparkUser" : "supdev",
"completed" : true,
"appSparkVersion" : "2.4.0-cdh6.2.0",
"startTimeEpoch" : 1620912418297,
"lastUpdatedEpoch" : 1620912538570,
"endTimeEpoch" : 1620912538500
} ]
} ]
- 加上狀態(tài)
curl "http://sj-20-207-14:18088/api/v1/applications?limit=2&status=running"
還是歷史
[ {
"id" : "application_1618400751843_99465",
"name" : "Hive on Spark (hiveSessionId = 4f556621-4e68-40d5-a416-8f1fc6a3ea9e)",
"attempts" : [ {
"attemptId" : "1",
"startTime" : "2021-05-08T09:50:00.273GMT",
"endTime" : "1969-12-31T23:59:59.999GMT",
"lastUpdated" : "2021-05-08T10:36:31.654GMT",
"duration" : 0,
"sparkUser" : "jiahang.liu",
"completed" : false,
"appSparkVersion" : "2.4.0-cdh6.2.0",
"startTimeEpoch" : 1620467400273,
"lastUpdatedEpoch" : 1620470191654,
"endTimeEpoch" : -1
} ]
}, {
"id" : "application_1618400751843_93669",
"name" : "Hive on Spark (hiveSessionId = f31a0dec-abbe-4418-bf7f-98428a7cb44a)",
"attempts" : [ {
"attemptId" : "1",
"startTime" : "2021-05-07T13:30:08.725GMT",
"endTime" : "1969-12-31T23:59:59.999GMT",
"lastUpdated" : "2021-05-07T14:58:18.949GMT",
"duration" : 0,
"sparkUser" : "supdev",
"completed" : false,
"appSparkVersion" : "2.4.0-cdh6.2.0",
"startTimeEpoch" : 1620394208725,
"lastUpdatedEpoch" : 1620399498949,
"endTimeEpoch" : -1
} ]
} ]
未完待續(xù)
