GBK格式內(nèi)容轉(zhuǎn)碼
val logRdd = sc.hadoopFile(logPath, classOf[TextInputFormat], classOf[LongWritable], classOf[Text])
.map(p => new String(p._2.getBytes, 0, p._2.getLength, "GBK"))
獲取文件路徑或文件名
hadoop mr 中使用如下方式
InputSplit inputSplit=(InputSplit)context.getInputSplit();
String filename=((FileSplit)inputSplit).getPath().getName();
spark 中
獲取文件名中的ip信息岸售，并將文件轉(zhuǎn)碼
val logRdd = sc.hadoopFile(logPath, classOf[TextInputFormat], classOf[LongWritable], classOf[Text])
.asInstanceOf[HadoopRDD[LongWritable,Text]]
.mapPartitionsWithInputSplit((inputSplit:InputSplit, iterator:Iterator[(LongWritable, Text)]) => {
val file = inputSplit.asInstanceOf[FileSplit]
val fileName = file.getPath.getName()
val reg = "((2(5[0-5]|[0-4]\\d))|[0-1]?\\d{1,2})(\\.((2(5[0-5]|[0-4]\\d))|[0-1]?\\d{1,2})){3}".r
val ip = reg.findFirstIn(fileName)
iterator.map(x => {ip.getOrElse("") + SEPRATOR + new String(x._2.getBytes, 0, x._2.getLength, "GBK")})
})
