Hadoop WC.txt案例测试
首先查看hdfs集群起来没
[hadoop@hadoop001 ~]$ jps
37605 RunJar
36443 NameNode
36731 SecondaryNameNode
36541 DataNode
41165 Jps
[hadoop@hadoop001 ~]$
创建wordcount文件***## 准备数据
[hadoop@hadoop001 data]$ vim wordcount
ihello world
hello hadoop
hello hdfs
hello yarn
hello test
hello world
hello hadoop
hello hdfs
hello yarn
hello test
hello world
hello hadoop
hello hdfs
hello yarn
hello test
把本地数据放到HDFS上
(1)HDFS上创建input目录，存放数据
[hadoop@hadoop001 data]$ hadoop fs -mkdir -p /user/hadoop/input
(2)把本地数据上传到HDFS
[hadoop@hadoop001 data]$ hadoop fs -put wordcount /user/hadoop
/input/
(3)查看
[hadoop@hadoop001 data]$ hadoop fs -ls /user/hadoop/input
18/10/24 23:29:16 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
-rw-r--r-- 1 hadoop supergroup 178 2018-10-24 23:28 /user/hadoop/input/wordcount
提交作业
(1) 进入到$HADOOP_HOME/bin目录
[hadoop@hadoop001 bin]$ pwd
/home/hadoop/app/hadoop-2.6.0-cdh5.7.0/bin
[hadoop@hadoop001 bin]$
(2)提交作业
[hadoop@hadoop001 bin]$ hadoop jar $HADOOP_HOME/share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar wordcount input output
18/10/25 00:04:59 INFO mapreduce.Job: map 0% reduce 0%
18/10/25 00:05:12 INFO mapreduce.Job: map 100% reduce 0%
18/10/25 00:05:22 INFO mapreduce.Job: map 100% reduce 100%
18/10/25 00:05:24 INFO mapreduce.Job: Job job_1540396662570_0003 completed successfully
18/10/25 00:05:24 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=89
FILE: Number of bytes written=223123
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=297
HDFS: Number of bytes written=56
HDFS: Number of read operations=6
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
(3).查看提交任务执行情况
[hadoop@hadoop001 sbin]$ mapred job -list
18/10/25 00:05:00 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/10/25 00:05:01 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
Total jobs:1
JobId State StartTime UserName Queue Priority UsedContainers RsvdContainers UsedMem RsvdMem NeededMem AM info
job_1540396662570_0003 RUNNING 1540397083947 hadoop root.hadoop NORMAL 2 0 3072M 0M 3072M http://
(4).查看作业执行完成，查看HDFS输出目录
[hadoop@hadoop001 sbin]$ hadoop fs -text output/*
18/10/25 00:10:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
text: `output/output': No such file or directory
hadoop 3
hdfs 3
hello 14
ihello 1
test 3
world 3
yarn 3
结束
