Spark SQL读取json文件

DataFrame是什么?

是带有模式(Schema)的RDD, 类似于二维表,  有了Schema可以洞察更多的额信息.

准备json数据, json数据需要是一行代表一个Row

{"id":1, "name":"leo", "age":18} {"id":2, "name":"jack", "age":19} {"id":3, "name":"marry", "age":17}

编写代码, 读取数据

package examples; import org.apache.log4j.Level; import org.apache.log4j.Logger; import org.apache.spark.SparkConf; import org.apache.spark.api.java.JavaSparkContext; import org.apache.spark.sql.DataFrame; import org.apache.spark.sql.SQLContext; public class SQLExample { private static final String filePath = "F:/tmp/spark/users.json"; public static void main(String[] args) { Logger.getLogger("org").setLevel(Level.FATAL); //create spark context SparkConf conf = new SparkConf() .setMaster("local") .setAppName("SQLExample"); JavaSparkContext ctx = new JavaSparkContext(conf); SQLContext sqlCtx = new SQLContext(ctx); DataFrame df = sqlCtx.read().json(filePath); df.show(); } }

结果

+---+---+-----+ |age| id| name| +---+---+-----+ | 18| 1| leo| | 19| 2| jack| | 17| 3|marry| +---+---+-----+

问题

ERROR ShutdownHookManager: Exception while deleting Spark temp dir: C:\Users\415387\AppData\Local\Temp\

方案:

使用winutils, 注意要使用-R选项

winutils.exe chmod -R 777 C:\Users\Administrator\AppData\Local\Temp