一、UDF函数可以直接应用于select语句,对查询结构做格式化处理后,再输出内容。
二、编写UDF函数的时候需要注意一下几点:
a)自定义UDF需要继承org.apache.hadoop.hive.ql.UDF。
b)需要实现evaluate函数,evaluate函数支持重载。
三、编写UDF函数代码
0.要继承org.apache.hadoop.hive.ql.exec.UDF类实现evaluate 方法。
public class NationUDF extends UDF { public static Map<String,String> nationMap = new HashMap<String,String>(); static{ nationMap.put("China", "中国"); nationMap.put("Japan", "小日本"); nationMap.put("USA", "美帝"); } Text t = new Text(); //1000 sum(income) //返回值:中国 getNation(nation) public Text evaluate(Text nation){ String nation_e = nation.toString(); String name = nationMap.get(nation_e); if(name == null){ name = "火星人"; } t.set(name); return t; } }0.1打jar包
右键-->export-->java/jar file-->next-->勾选-->finish;
注:打jar包时,注意jdk版本的问题,centOS的hadoop框架下jdk可以向下兼容,也就说,hadoop框架的jdk版本>=UDFjar的版本。
四、自定义函数调用过程: 1.添加jar包(在hive命令行里面执行) hive> add jar /root/NUDF.jar; 2.创建临时函数(所谓的temporary,就是函数在本会话(此次hive客户端)有效) hive> create temporary function getNation as 'com.heres.hive.NationUDF'; 3.调用 hive> select id, name, getNation(nation) from beauties; 4.利用自定义函数查询结果hive> select id,name,size,getNation(nation) from beauties order by size desc;
Total jobs = 1 Launching Job 1 out of 1 Number of reduce tasks determined at compile time: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapreduce.job.reduces=<number> Starting Job = job_1494132664539_0002, Tracking URL = http://heres04:8088/proxy/application_1494132664539_0002/ Kill Command = /heres/hadoop-2.2.0/bin/hadoop job -kill job_1494132664539_0002 Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1 2017-05-07 13:03:37,424 Stage-1 map = 0%, reduce = 0% 2017-05-07 13:03:51,342 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 2.57 sec 2017-05-07 13:04:00,996 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 4.41 sec MapReduce Total cumulative CPU time: 4 seconds 410 msec Ended Job = job_1494132664539_0002 MapReduce Jobs Launched: Job 0: Map: 1 Reduce: 1 Cumulative CPU: 4.41 sec HDFS Read: 328 HDFS Write: 160 SUCCESS Total MapReduce CPU Time Spent: 4 seconds 410 msec OK 1 bgyjy 56.6565 小日本 4 bing 56.56 火星人 3 liu 45.0 火星人 3 ewrwe 43.9 小日本 1 glm 34.0 火星人 2 lina 30.9 火星人 2 jzmb 23.232 小日本 Time taken: 37.869 seconds, Fetched: 7 row(s)
5、将查询结果保存到HDFS中 create table result row format delimited fields terminated by '\t' as select id, getNation(nation) from beauties;
6、销毁临时函数:
hive>DROP TEMPORARY FUNCTION getNation;