GeoSpark-[階段性總結(jié)]

一则披、 Geospark安裝：

我采用的是gradle安裝，安裝環(huán)境是Spark3.0.3+Scala.2.12.3

plugins {
    id 'java'
    id 'scala'
}

group 'org.example'
version '1.0-SNAPSHOT'

configurations {
    jar.archiveName = 'OutputToDeltaV3.jar'
}



repositories {
    maven{ url 'http://maven.aliyun.com/nexus/content/groups/public'}
}


dependencies {
    compile group: 'org.scala-lang', name: 'scala-library', version: '2.12.2'
    compile group: 'org.apache.spark', name: 'spark-core_2.12', version: '3.0.3'
    compile group: 'org.apache.spark', name: 'spark-sql_2.12', version: '3.0.3'
    compile group: 'org.apache.spark', name: 'spark-hive_2.12', version: '3.0.3'
    compile group: 'org.apache.hive',name: 'hive-jdbc',version: '1.2.0'
    compile group: 'io.delta', name: 'delta-core_2.12', version: '0.7.0'
    compile group: 'org.apache.spark', name: 'spark-sql-kafka-0-10_2.12', version: '3.0.3'
    compile group: 'mysql', name: 'mysql-connector-java', version: '5.1.41'
    compile group: 'org.apache.hadoop',name: 'hadoop-common',version: '2.7.7'
    compile group: 'org.apache.hadoop',name: 'hadoop-mapreduce-client-core',version: '2.7.7'
//    compile group: 'org.apache.logging.log4j', name: 'log4j-api', version: '2.14.1'
//    compile group: 'org.apache.logging.log4j', name: 'log4j-core', version: '2.14.1'

// geospark 依賴開(kāi)始的地方
    compile group: 'org.apache.sedona',name: 'sedona-core-3.0_2.12',version: '1.0.1-incubating'
    compile group: 'org.apache.sedona',name: 'sedona-sql-3.0_2.12',version: '1.0.1-incubating'
    compile group: 'org.apache.sedona',name: 'sedona-viz-3.0_2.12',version: '1.0.1-incubating'
    compile group: 'org.locationtech.jts',name: 'jts-core',version: '1.18.0'
    compile group: 'org.datasyslab',name: 'geotools-wrapper',version: 'geotools-24.0'
    compile ('org.wololo:jts2geojson:0.16.1'){
        exclude group: 'org.locationtech.jt',module: 'jts-core'
        exclude group: 'com.fasterxml.jackson.core',module:'*'
    }
// geospark依賴結(jié)束的地方
    compile('ru.yandex.clickhouse:clickhouse-jdbc:0.1.53')
            {
        exclude group: 'com.fasterxml.jackson.core',module: 'jackson-databind'
        exclude group: 'com.fasterxml.jackson.core',module: 'jackson-core'
    }

    compile fileTree(dir:'lib',includes:['*jar'])
    implementation fileTree(dir:'lib',includes:['*jar'])
}

二洗出、 Geospark 能夠讀取文件的結(jié)構(gòu)有：CSV,TSV士复，Shapefile, Geojson;

S1.CSV結(jié)構(gòu)如下：

// 單點(diǎn)坐標(biāo)
-88.331492,32.324142,hotel
-88.175933,32.360763,gas
-88.388954,32.357073,bar
-88.221102,32.35078,restaurant

// 多點(diǎn)坐標(biāo)
-88.331492,32.324142,-88.331492,32.324142,-88.331492,32.324142,-88.331492,32.324142,-88.331492,32.324142,hotel
-88.175933,32.360763,-88.175933,32.360763,-88.175933,32.360763,-88.175933,32.360763,-88.175933,32.360763,gas
-88.388954,32.357073,-88.388954,32.357073,-88.388954,32.357073,-88.388954,32.357073,-88.388954,32.357073,bar
-88.221102,32.35078,-88.221102,32.35078,-88.221102,32.35078,-88.221102,32.35078,-88.221102,32.35078,restaurant

C1.CSV讀取方式：

// 創(chuàng)建單個(gè)點(diǎn)坐標(biāo)的方法
val pointRDDInputLocation = "/Download/checkin.csv"
val pointRDDOffset = 0 // The point long/lat starts from Column 0
val pointRDDSplitter = FileDataSplitter.CSV
val carryOtherAttributes = true // Carry Column 2 (hotel, gas, bar...)
var objectRDD = new PointRDD(sc, pointRDDInputLocation, pointRDDOffset, pointRDDSplitter, carryOtherAttributes)

// 創(chuàng)建多變形RDD
val polygonRDDInputLocation = "/Download/checkinshape.csv"
val polygonRDDStartOffset = 0 // The coordinates start from Column 0
val polygonRDDEndOffset = 9 // The coordinates end at Column 9
val polygonRDDSplitter = FileDataSplitter.CSV
val carryOtherAttributes = true // Carry Column 10 (hotel, gas, bar...)
var objectRDD = new PolygonRDD(sc, polygonRDDInputLocation, polygonRDDStartOffset, polygonRDDEndOffset, polygonRDDSplitter, carryOtherAttributes)

S2. TSV結(jié)構(gòu)如下：

POINT (-88.331492 32.324142)    hotel
POINT (-88.175933 32.360763)    gas
POINT (-88.388954 32.357073)    bar
POINT (-88.221102 32.35078) restaurant

C2. TSV讀取方式：

val inputLocation = "/Download/checkin.tsv"
val wktColumn = 0 // The WKT string starts from Column 0
val allowTopologyInvalidGeometries = true // Optional
val skipSyntaxInvalidGeometries = false // Optional
val spatialRDD = WktReader.readToGeometryRDD(sparkSession.sparkContext, inputLocation, wktColumn, allowTopologyInvalidGeometries, skipSyntaxInvalidGeometries)

S3.Shapefile結(jié)構(gòu)如下:

- shapefile1
- shapefile2
- myshapefile
    - myshapefile.shp
    - myshapefile.shx
    - myshapefile.dbf
    - myshapefile...
    - ...

C3. Shapefile 讀取方式：

val shapefileInputLocation="/Download/myshapefile"
val spatialRDD = ShapefileReader.readToGeometryRDD(sparkSession.sparkContext, shapefileInputLocation)

S4. Geojson結(jié)構(gòu)如下：

{ "type": "Feature", "properties": { "STATEFP": "01", "COUNTYFP": "077", "TRACTCE": "011501", "BLKGRPCE": "5", "AFFGEOID": "1500000US010770115015", "GEOID": "010770115015", "NAME": "5", "LSAD": "BG", "ALAND": 6844991, "AWATER": 32636 }, "geometry": { "type": "Polygon", "coordinates": [ [ [ -87.621765, 34.873444 ], [ -87.617535, 34.873369 ], [ -87.6123, 34.873337 ], [ -87.604049, 34.873303 ], [ -87.604033, 34.872316 ], [ -87.60415, 34.867502 ], [ -87.604218, 34.865687 ], [ -87.604409, 34.858537 ], [ -87.604018, 34.851336 ], [ -87.603716, 34.844829 ], [ -87.603696, 34.844307 ], [ -87.603673, 34.841884 ], [ -87.60372, 34.841003 ], [ -87.603879, 34.838423 ], [ -87.603888, 34.837682 ], [ -87.603889, 34.83763 ], [ -87.613127, 34.833938 ], [ -87.616451, 34.832699 ], [ -87.621041, 34.831431 ], [ -87.621056, 34.831526 ], [ -87.62112, 34.831925 ], [ -87.621603, 34.8352 ], [ -87.62158, 34.836087 ], [ -87.621383, 34.84329 ], [ -87.621359, 34.844438 ], [ -87.62129, 34.846387 ], [ -87.62119, 34.85053 ], [ -87.62144, 34.865379 ], [ -87.621765, 34.873444 ] ] ] } },
{ "type": "Feature", "properties": { "STATEFP": "01", "COUNTYFP": "045", "TRACTCE": "021102", "BLKGRPCE": "4", "AFFGEOID": "1500000US010450211024", "GEOID": "010450211024", "NAME": "4", "LSAD": "BG", "ALAND": 11360854, "AWATER": 0 }, "geometry": { "type": "Polygon", "coordinates": [ [ [ -85.719017, 31.297901 ], [ -85.715626, 31.305203 ], [ -85.714271, 31.307096 ], [ -85.69999, 31.307552 ], [ -85.697419, 31.307951 ], [ -85.675603, 31.31218 ], [ -85.672733, 31.312876 ], [ -85.672275, 31.311977 ], [ -85.67145, 31.310988 ], [ -85.670622, 31.309524 ], [ -85.670729, 31.307622 ], [ -85.669876, 31.30666 ], [ -85.669796, 31.306224 ], [ -85.670356, 31.306178 ], [ -85.671664, 31.305583 ], [ -85.67177, 31.305299 ], [ -85.671878, 31.302764 ], [ -85.671344, 31.302123 ], [ -85.668276, 31.302076 ], [ -85.66566, 31.30093 ], [ -85.665687, 31.30022 ], [ -85.669183, 31.297677 ], [ -85.668703, 31.295638 ], [ -85.671985, 31.29314 ], [ -85.677177, 31.288211 ], [ -85.678452, 31.286376 ], [ -85.679236, 31.28285 ], [ -85.679195, 31.281426 ], [ -85.676865, 31.281049 ], [ -85.674661, 31.28008 ], [ -85.674377, 31.27935 ], [ -85.675714, 31.276882 ], [ -85.677938, 31.275168 ], [ -85.680348, 31.276814 ], [ -85.684032, 31.278848 ], [ -85.684387, 31.279082 ], [ -85.692398, 31.283499 ], [ -85.705032, 31.289718 ], [ -85.706755, 31.290476 ], [ -85.718102, 31.295204 ], [ -85.719132, 31.29689 ], [ -85.719017, 31.297901 ] ] ] } },
{ "type": "Feature", "properties": { "STATEFP": "01", "COUNTYFP": "055", "TRACTCE": "001300", "BLKGRPCE": "3", "AFFGEOID": "1500000US010550013003", "GEOID": "010550013003", "NAME": "3", "LSAD": "BG", "ALAND": 1378742, "AWATER": 247387 }, "geometry": { "type": "Polygon", "coordinates": [ [ [ -86.000685, 34.00537 ], [ -85.998837, 34.009768 ], [ -85.998012, 34.010398 ], [ -85.987865, 34.005426 ], [ -85.986656, 34.004552 ], [ -85.985, 34.002659 ], [ -85.98851, 34.001502 ], [ -85.987567, 33.999488 ], [ -85.988666, 33.99913 ], [ -85.992568, 33.999131 ], [ -85.993144, 33.999714 ], [ -85.994876, 33.995153 ], [ -85.998823, 33.989548 ], [ -85.999925, 33.994237 ], [ -86.000616, 34.000028 ], [ -86.000685, 34.00537 ] ] ] } },
{ "type": "Feature", "properties": { "STATEFP": "01", "COUNTYFP": "089", "TRACTCE": "001700", "BLKGRPCE": "2", "AFFGEOID": "1500000US010890017002", "GEOID": "010890017002", "NAME": "2", "LSAD": "BG", "ALAND": 1040641, "AWATER": 0 }, "geometry": { "type": "Polygon", "coordinates": [ [ [ -86.574172, 34.727375 ], [ -86.562684, 34.727131 ], [ -86.562797, 34.723865 ], [ -86.562957, 34.723168 ], [ -86.562336, 34.719766 ], [ -86.557381, 34.719143 ], [ -86.557352, 34.718322 ], [ -86.559921, 34.717363 ], [ -86.564827, 34.718513 ], [ -86.567582, 34.718565 ], [ -86.570572, 34.718577 ], [ -86.573618, 34.719377 ], [ -86.574172, 34.727375 ] ] ] } },

C4,Geojson 讀取方式：

val inputLocation = "/Download/polygon.json"
val allowTopologyInvalidGeometries = true // Optional
val skipSyntaxInvalidGeometries = false // Optional
val spatialRDD = GeoJsonReader.readToGeometryRDD(sparkSession.sparkContext, inputLocation, allowTopologyInvalidGeometries, skipSyntaxInvalidGeometries)
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者
  • 人面猴
    序言：七十年代末，一起剝皮案震驚了整個(gè)濱河市翩活，隨后出現(xiàn)的幾起案子阱洪，更是在濱河造成了極大的恐慌，老刑警劉巖菠镇，帶你破解...
    沈念sama閱讀 212,454評(píng)論 6贊 493
  • 死咒
    序言：濱河連續(xù)發(fā)生了三起死亡事件冗荸，死亡現(xiàn)場(chǎng)離奇詭異，居然都是意外死亡辟犀，警方通過(guò)查閱死者的電腦和手機(jī)俏竞，發(fā)現(xiàn)死者居然都...
    沈念sama閱讀 90,553評(píng)論 3贊 385
  • 救了他兩次的神仙讓他今天三更去死
    文/潘曉璐 我一進(jìn)店門绸硕，熙熙樓的掌柜王于貴愁眉苦臉地迎上來(lái)，“玉大人魂毁，你說(shuō)我怎么就攤上這事玻佩。” “怎么了席楚？”我有些...
    開(kāi)封第一講書(shū)人閱讀 157,921評(píng)論 0贊 348
  • 道士緝兇錄：失蹤的賣姜人
    文/不壞的土叔 我叫張陵咬崔，是天一觀的道長(zhǎng)。 經(jīng)常有香客問(wèn)我烦秩，道長(zhǎng)垮斯，這世上最難降的妖魔是什么？ 我笑而不...
    開(kāi)封第一講書(shū)人閱讀 56,648評(píng)論 1贊 284
  • ?港島之戀（遺憾婚禮）
    正文 為了忘掉前任只祠，我火速辦了婚禮兜蠕，結(jié)果婚禮上，老公的妹妹穿的比我還像新娘抛寝。我一直安慰自己熊杨，他們只是感情好，可當(dāng)我...
    茶點(diǎn)故事閱讀 65,770評(píng)論 6贊 386
  • 惡毒庶女頂嫁案：這布局不是一般人想出來(lái)的
    文/花漫 我一把揭開(kāi)白布盗舰。 她就那樣靜靜地躺著晶府，像睡著了一般。 火紅的嫁衣襯著肌膚如雪钻趋。 梳的紋絲不亂的頭發(fā)上川陆，一...
    開(kāi)封第一講書(shū)人閱讀 49,950評(píng)論 1贊 291
  • 城市分裂傳說(shuō)
    那天，我揣著相機(jī)與錄音蛮位，去河邊找鬼较沪。 笑死，一個(gè)胖子當(dāng)著我的面吹牛土至，可吹牛的內(nèi)容都是我干的购对。 我是一名探鬼主播，決...
    沈念sama閱讀 39,090評(píng)論 3贊 410
  • 雙鴛鴦連環(huán)套：你想象不到人心有多黑
    文/蒼蘭香墨 我猛地睜開(kāi)眼陶因，長(zhǎng)吁一口氣：“原來(lái)是場(chǎng)噩夢(mèng)啊……” “哼骡苞！你這毒婦竟也來(lái)了？” 一聲冷哼從身側(cè)響起楷扬，我...
    開(kāi)封第一講書(shū)人閱讀 37,817評(píng)論 0贊 268
  • 萬(wàn)榮殺人案實(shí)錄
    序言：老撾萬(wàn)榮一對(duì)情侶失蹤解幽，失蹤者是張志新（化名）和其女友劉穎，沒(méi)想到半個(gè)月后烘苹，有當(dāng)?shù)厝嗽跇?shù)林里發(fā)現(xiàn)了一具尸體躲株，經(jīng)...
    沈念sama閱讀 44,275評(píng)論 1贊 303
  • ?護(hù)林員之死
    正文 獨(dú)居荒郊野嶺守林人離奇死亡，尸身上長(zhǎng)有42處帶血的膿包…… 初始之章·張勛 以下內(nèi)容為張勛視角 年9月15日...
    茶點(diǎn)故事閱讀 36,592評(píng)論 2贊 327
  • ?白月光啟示錄
    正文 我和宋清朗相戀三年镣衡，在試婚紗的時(shí)候發(fā)現(xiàn)自己被綠了霜定。 大學(xué)時(shí)的朋友給我發(fā)了我未婚夫和他白月光在一起吃飯的照片档悠。...
    茶點(diǎn)故事閱讀 38,724評(píng)論 1贊 341
  • 活死人
    序言：一個(gè)原本活蹦亂跳的男人離奇死亡，死狀恐怖望浩，靈堂內(nèi)的尸體忽然破棺而出辖所，到底是詐尸還是另有隱情，我是刑警寧澤磨德，帶...
    沈念sama閱讀 34,409評(píng)論 4贊 333
  • ?日本核電站爆炸內(nèi)幕
    正文 年R本政府宣布缘回，位于F島的核電站，受9級(jí)特大地震影響典挑，放射性物質(zhì)發(fā)生泄漏酥宴。R本人自食惡果不足惜，卻給世界環(huán)境...
    茶點(diǎn)故事閱讀 40,052評(píng)論 3贊 316
  • 男人毒藥：我在死后第九天來(lái)索命
    文/蒙蒙 一您觉、第九天 我趴在偏房一處隱蔽的房頂上張望拙寡。 院中可真熱鬧，春花似錦顾犹、人聲如沸倒庵。這莊子的主人今日做“春日...
    開(kāi)封第一講書(shū)人閱讀 30,815評(píng)論 0贊 21
  • 一樁弒父案炫刷，背后竟有這般陰謀
    文/蒼蘭香墨 我抬頭看了看天上的太陽(yáng)。三九已至郁妈，卻和暖如春浑玛，著一層夾襖步出監(jiān)牢的瞬間，已是汗流浹背噩咪。 一陣腳步聲響...
    開(kāi)封第一講書(shū)人閱讀 32,043評(píng)論 1贊 266
  • 情欲美人皮
    我被黑心中介騙來(lái)泰國(guó)打工顾彰， 沒(méi)想到剛下飛機(jī)就差點(diǎn)兒被人妖公主榨干…… 1. 我叫王不留，地道東北人胃碾。 一個(gè)月前我還...
    沈念sama閱讀 46,503評(píng)論 2贊 361
  • 代替公主和親
    正文 我出身青樓涨享，卻偏偏與公主長(zhǎng)得像，于是被迫代替她去往敵國(guó)和親仆百。 傳聞我的和親對(duì)象是個(gè)殘疾皇子厕隧，可洞房花燭夜當(dāng)晚...
    茶點(diǎn)故事閱讀 43,627評(píng)論 2贊 350

推薦閱讀更多精彩內(nèi)容