1.本地模式
[root@cdh01 ~]# spark-submit --master local --class org.apache.spark.examples.SparkPi /opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/spark/lib/spark-examples.jar 10 18/10/29 14:39:08 INFO spark.SparkContext: Running Spark version 1.6.0 18/10/29 14:39:09 INFO spark.SecurityManager: Changing view acls to: root 18/10/29 14:39:09 INFO spark.SecurityManager: Changing modify acls to: root 18/10/29 14:39:09 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root) 18/10/29 14:39:09 INFO util.Utils: Successfully started service 'sparkDriver' on port 55692. 18/10/29 14:39:09 INFO slf4j.Slf4jLogger: Slf4jLogger started 18/10/29 14:39:09 INFO Remoting: Starting remoting 18/10/29 14:39:10 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@192.168.50.202:43516] 18/10/29 14:39:10 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkDriverActorSystem@192.168.50.202:43516] 18/10/29 14:39:10 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 43516. 18/10/29 14:39:10 INFO spark.SparkEnv: Registering MapOutputTracker 18/10/29 14:39:10 INFO spark.SparkEnv: Registering BlockManagerMaster 18/10/29 14:39:10 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-2bf97eb7-1a7e-4df7-b221-4e603dc3a55f 18/10/29 14:39:10 INFO storage.MemoryStore: MemoryStore started with capacity 530.0 MB 18/10/29 14:39:10 INFO spark.SparkEnv: Registering OutputCommitCoordinator 18/10/29 14:39:10 INFO util.Utils: Successfully started service 'SparkUI' on port 4040. 18/10/29 14:39:10 INFO ui.SparkUI: Started SparkUI at http://192.168.50.202:4040 18/10/29 14:39:10 INFO spark.SparkContext: Added JAR file:/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/spark/lib/spark-examples.jar at spark://192.168.50.202:55692/jars/spark-examples.jar with timestamp 1540795150401 18/10/29 14:39:10 INFO executor.Executor: Starting executor ID driver on host localhost 18/10/29 14:39:10 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 53969. 18/10/29 14:39:10 INFO netty.NettyBlockTransferService: Server created on 53969 18/10/29 14:39:10 INFO storage.BlockManager: external shuffle service port = 7337 18/10/29 14:39:10 INFO storage.BlockManagerMaster: Trying to register BlockManager 18/10/29 14:39:10 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:53969 with 530.0 MB RAM, BlockManagerId(driver, localhost, 53969) 18/10/29 14:39:10 INFO storage.BlockManagerMaster: Registered BlockManager 18/10/29 14:39:11 INFO scheduler.EventLoggingListener: Logging events to hdfs://cdh01:8020/user/spark/applicationHistory/local-1540795150435 18/10/29 14:39:11 INFO spark.SparkContext: Registered listener com.cloudera.spark.lineage.ClouderaNavigatorListener 18/10/29 14:39:11 INFO spark.SparkContext: Starting job: reduce at SparkPi.scala:36 18/10/29 14:39:11 INFO scheduler.DAGScheduler: Got job 0 (reduce at SparkPi.scala:36) with 10 output partitions 18/10/29 14:39:11 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (reduce at SparkPi.scala:36) 18/10/29 14:39:11 INFO scheduler.DAGScheduler: Parents of final stage: List() 18/10/29 14:39:11 INFO scheduler.DAGScheduler: Missing parents: List() 18/10/29 14:39:11 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:32), which has no missing parents 18/10/29 14:39:12 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 1904.0 B, free 530.0 MB) 18/10/29 14:39:12 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1202.0 B, free 530.0 MB) 18/10/29 14:39:12 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:53969 (size: 1202.0 B, free: 530.0 MB) 18/10/29 14:39:12 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1004 18/10/29 14:39:12 INFO scheduler.DAGScheduler: Submitting 10 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:32) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9)) 18/10/29 14:39:12 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 10 tasks 18/10/29 14:39:12 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, PROCESS_LOCAL, 2036 bytes) 18/10/29 14:39:12 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0) 18/10/29 14:39:12 INFO executor.Executor: Fetching spark://192.168.50.202:55692/jars/spark-examples.jar with timestamp 1540795150401 18/10/29 14:39:12 INFO spark.ExecutorAllocationManager: New executor driver has registered (new total is 1) 18/10/29 14:39:12 INFO util.Utils: Fetching spark://192.168.50.202:55692/jars/spark-examples.jar to /tmp/spark-e7873ccb-d141-4347-abcd-1b263d364be3/userFiles-89bc4061-62e5-41b0-b1c2-cecbc4d3af73/fetchFileTemp4804387182541284155.tmp 18/10/29 14:39:12 INFO executor.Executor: Adding file:/tmp/spark-e7873ccb-d141-4347-abcd-1b263d364be3/userFiles-89bc4061-62e5-41b0-b1c2-cecbc4d3af73/spark-examples.jar to class loader 18/10/29 14:39:12 INFO executor.Executor: Finished task 0.0 in stage 0.0 (TID 0). 877 bytes result sent to driver 18/10/29 14:39:12 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, executor driver, partition 1, PROCESS_LOCAL, 2038 bytes) 18/10/29 14:39:12 INFO executor.Executor: Running task 1.0 in stage 0.0 (TID 1) 18/10/29 14:39:12 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 342 ms on localhost (executor driver) (1/10) 18/10/29 14:39:12 INFO executor.Executor: Finished task 1.0 in stage 0.0 (TID 1). 877 bytes result sent to driver 18/10/29 14:39:12 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, localhost, executor driver, partition 2, PROCESS_LOCAL, 2038 bytes) 18/10/29 14:39:12 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 51 ms on localhost (executor driver) (2/10) 18/10/29 14:39:12 INFO executor.Executor: Running task 2.0 in stage 0.0 (TID 2) 18/10/29 14:39:12 INFO executor.Executor: Finished task 2.0 in stage 0.0 (TID 2). 877 bytes result sent to driver 18/10/29 14:39:12 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, localhost, executor driver, partition 3, PROCESS_LOCAL, 2038 bytes) 18/10/29 14:39:12 INFO executor.Executor: Running task 3.0 in stage 0.0 (TID 3) 18/10/29 14:39:12 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 39 ms on localhost (executor driver) (3/10) 18/10/29 14:39:12 INFO executor.Executor: Finished task 3.0 in stage 0.0 (TID 3). 877 bytes result sent to driver 18/10/29 14:39:12 INFO scheduler.TaskSetManager: Starting task 4.0 in stage 0.0 (TID 4, localhost, executor driver, partition 4, PROCESS_LOCAL, 2038 bytes) 18/10/29 14:39:12 INFO executor.Executor: Running task 4.0 in stage 0.0 (TID 4) 18/10/29 14:39:12 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 42 ms on localhost (executor driver) (4/10) 18/10/29 14:39:12 INFO executor.Executor: Finished task 4.0 in stage 0.0 (TID 4). 877 bytes result sent to driver 18/10/29 14:39:12 INFO scheduler.TaskSetManager: Starting task 5.0 in stage 0.0 (TID 5, localhost, executor driver, partition 5, PROCESS_LOCAL, 2038 bytes) 18/10/29 14:39:12 INFO executor.Executor: Running task 5.0 in stage 0.0 (TID 5) 18/10/29 14:39:12 INFO scheduler.TaskSetManager: Finished task 4.0 in stage 0.0 (TID 4) in 37 ms on localhost (executor driver) (5/10) 18/10/29 14:39:12 INFO executor.Executor: Finished task 5.0 in stage 0.0 (TID 5). 877 bytes result sent to driver 18/10/29 14:39:12 INFO scheduler.TaskSetManager: Starting task 6.0 in stage 0.0 (TID 6, localhost, executor driver, partition 6, PROCESS_LOCAL, 2038 bytes) 18/10/29 14:39:12 INFO scheduler.TaskSetManager: Finished task 5.0 in stage 0.0 (TID 5) in 71 ms on localhost (executor driver) (6/10) 18/10/29 14:39:12 INFO executor.Executor: Running task 6.0 in stage 0.0 (TID 6) 18/10/29 14:39:12 INFO executor.Executor: Finished task 6.0 in stage 0.0 (TID 6). 877 bytes result sent to driver 18/10/29 14:39:12 INFO scheduler.TaskSetManager: Starting task 7.0 in stage 0.0 (TID 7, localhost, executor driver, partition 7, PROCESS_LOCAL, 2038 bytes) 18/10/29 14:39:12 INFO executor.Executor: Running task 7.0 in stage 0.0 (TID 7) 18/10/29 14:39:12 INFO scheduler.TaskSetManager: Finished task 6.0 in stage 0.0 (TID 6) in 32 ms on localhost (executor driver) (7/10) 18/10/29 14:39:12 INFO executor.Executor: Finished task 7.0 in stage 0.0 (TID 7). 877 bytes result sent to driver 18/10/29 14:39:12 INFO scheduler.TaskSetManager: Starting task 8.0 in stage 0.0 (TID 8, localhost, executor driver, partition 8, PROCESS_LOCAL, 2038 bytes) 18/10/29 14:39:12 INFO executor.Executor: Running task 8.0 in stage 0.0 (TID 8) 18/10/29 14:39:12 INFO scheduler.TaskSetManager: Finished task 7.0 in stage 0.0 (TID 7) in 28 ms on localhost (executor driver) (8/10) 18/10/29 14:39:12 INFO executor.Executor: Finished task 8.0 in stage 0.0 (TID 8). 877 bytes result sent to driver 18/10/29 14:39:12 INFO scheduler.TaskSetManager: Starting task 9.0 in stage 0.0 (TID 9, localhost, executor driver, partition 9, PROCESS_LOCAL, 2038 bytes) 18/10/29 14:39:12 INFO executor.Executor: Running task 9.0 in stage 0.0 (TID 9) 18/10/29 14:39:12 INFO scheduler.TaskSetManager: Finished task 8.0 in stage 0.0 (TID 8) in 27 ms on localhost (executor driver) (9/10) 18/10/29 14:39:12 INFO executor.Executor: Finished task 9.0 in stage 0.0 (TID 9). 877 bytes result sent to driver 18/10/29 14:39:12 INFO scheduler.TaskSetManager: Finished task 9.0 in stage 0.0 (TID 9) in 24 ms on localhost (executor driver) (10/10) 18/10/29 14:39:12 INFO scheduler.DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:36) finished in 0.628 s 18/10/29 14:39:12 INFO scheduler.DAGScheduler: Job 0 finished: reduce at SparkPi.scala:36, took 1.046294 s 18/10/29 14:39:12 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool Pi is roughly 3.141903141903142 18/10/29 14:39:13 INFO ui.SparkUI: Stopped Spark web UI at http://192.168.50.202:4040 18/10/29 14:39:13 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 18/10/29 14:39:13 INFO storage.MemoryStore: MemoryStore cleared 18/10/29 14:39:13 INFO storage.BlockManager: BlockManager stopped 18/10/29 14:39:13 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 18/10/29 14:39:13 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 18/10/29 14:39:13 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 18/10/29 14:39:13 INFO spark.SparkContext: Successfully stopped SparkContext 18/10/29 14:39:13 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 18/10/29 14:39:13 INFO util.ShutdownHookManager: Shutdown hook called 18/10/29 14:39:13 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-e7873ccb-d141-4347-abcd-1b263d364be3 18/10/29 14:39:13 INFO Remoting: Remoting shut down 18/10/29 14:39:13 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down.2.yarn模式
[root@cdh01 ~]# spark-submit --master yarn-cluster --class org.apache.spark.examples.SparkPi /opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/spark/lib/spark-examples.jar 10 18/10/29 14:46:07 INFO client.RMProxy: Connecting to ResourceManager at cdh01/192.168.50.202:8032 18/10/29 14:46:07 INFO yarn.Client: Requesting a new application from cluster with 3 NodeManagers 18/10/29 14:46:07 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (1024 MB per container) Exception in thread "main" java.lang.IllegalArgumentException: Required executor memory (1024+384 MB) is above the max threshold (1024 MB) of this cluster! Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or 'yarn.nodemanager.resource.memory-mb'. at org.apache.spark.deploy.yarn.Client.verifyClusterResources(Client.scala:292) at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:139) at org.apache.spark.deploy.yarn.Client.run(Client.scala:1023) at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1083) at org.apache.spark.deploy.yarn.Client.main(Client.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:730) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)报错：Exception in thread "main" java.lang.IllegalArgumentException: Required executor memory (1024+384 MB) is above the max threshold (1024 MB) of this cluster! Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or 'yarn.nodemanager.resource.memory-mb'.
解决办法：
在yarn的配置中将这两项设置为2吉字节，重启yarn。
[root@cdh01 ~]# spark-submit --master yarn-cluster --class org.apache.spark.examples.SparkPi /opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/spark/lib/spark-examples.jar 10 18/10/29 14:58:47 INFO client.RMProxy: Connecting to ResourceManager at cdh01/192.168.50.202:8032 18/10/29 14:58:47 INFO yarn.Client: Requesting a new application from cluster with 3 NodeManagers 18/10/29 14:58:47 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (2048 MB per container) 18/10/29 14:58:47 INFO yarn.Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead 18/10/29 14:58:47 INFO yarn.Client: Setting up container launch context for our AM 18/10/29 14:58:47 INFO yarn.Client: Setting up the launch environment for our AM container 18/10/29 14:58:47 INFO yarn.Client: Preparing resources for our AM container 18/10/29 14:58:48 INFO yarn.Client: Uploading resource file:/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/spark/lib/spark-examples.jar -> hdfs://cdh01:8020/user/root/.sparkStaging/application_1540796200906_0001/spark-examples.jar 18/10/29 14:58:48 INFO yarn.Client: Uploading resource file:/tmp/spark-2dd9e84f-a9ca-49b7-bd8e-1f912deca97c/__spark_conf__7994364669252611040.zip -> hdfs://cdh01:8020/user/root/.sparkStaging/application_1540796200906_0001/__spark_conf__7994364669252611040.zip 18/10/29 14:58:49 INFO spark.SecurityManager: Changing view acls to: root 18/10/29 14:58:49 INFO spark.SecurityManager: Changing modify acls to: root 18/10/29 14:58:49 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root) 18/10/29 14:58:49 INFO yarn.Client: Submitting application 1 to ResourceManager 18/10/29 14:58:49 INFO impl.YarnClientImpl: Submitted application application_1540796200906_0001 18/10/29 14:58:50 INFO yarn.Client: Application report for application_1540796200906_0001 (state: ACCEPTED) 18/10/29 14:58:50 INFO yarn.Client: client token: N/A diagnostics: N/A ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: root.users.root start time: 1540796329247 final status: UNDEFINED tracking URL: http://cdh01:8088/proxy/application_1540796200906_0001/ user: root 18/10/29 14:58:51 INFO yarn.Client: Application report for application_1540796200906_0001 (state: ACCEPTED) 18/10/29 14:58:52 INFO yarn.Client: Application report for application_1540796200906_0001 (state: ACCEPTED) 18/10/29 14:58:53 INFO yarn.Client: Application report for application_1540796200906_0001 (state: ACCEPTED) 18/10/29 14:58:54 INFO yarn.Client: Application report for application_1540796200906_0001 (state: ACCEPTED) 18/10/29 14:58:55 INFO yarn.Client: Application report for application_1540796200906_0001 (state: ACCEPTED) 18/10/29 14:58:56 INFO yarn.Client: Application report for application_1540796200906_0001 (state: ACCEPTED) 18/10/29 14:58:57 INFO yarn.Client: Application report for application_1540796200906_0001 (state: ACCEPTED) 18/10/29 14:58:58 INFO yarn.Client: Application report for application_1540796200906_0001 (state: ACCEPTED) 18/10/29 14:58:59 INFO yarn.Client: Application report for application_1540796200906_0001 (state: ACCEPTED) 18/10/29 14:59:00 INFO yarn.Client: Application report for application_1540796200906_0001 (state: RUNNING) 18/10/29 14:59:00 INFO yarn.Client: client token: N/A diagnostics: N/A ApplicationMaster host: 192.168.50.203 ApplicationMaster RPC port: 0 queue: root.users.root start time: 1540796329247 final status: UNDEFINED tracking URL: http://cdh01:8088/proxy/application_1540796200906_0001/ user: root 18/10/29 14:59:01 INFO yarn.Client: Application report for application_1540796200906_0001 (state: RUNNING) 18/10/29 14:59:02 INFO yarn.Client: Application report for application_1540796200906_0001 (state: RUNNING) 18/10/29 14:59:03 INFO yarn.Client: Application report for application_1540796200906_0001 (state: RUNNING) 18/10/29 14:59:04 INFO yarn.Client: Application report for application_1540796200906_0001 (state: RUNNING) 18/10/29 14:59:05 INFO yarn.Client: Application report for application_1540796200906_0001 (state: RUNNING) 18/10/29 14:59:06 INFO yarn.Client: Application report for application_1540796200906_0001 (state: RUNNING) 18/10/29 14:59:07 INFO yarn.Client: Application report for application_1540796200906_0001 (state: RUNNING) 18/10/29 14:59:08 INFO yarn.Client: Application report for application_1540796200906_0001 (state: FINISHED) 18/10/29 14:59:08 INFO yarn.Client: client token: N/A diagnostics: N/A ApplicationMaster host: 192.168.50.203 ApplicationMaster RPC port: 0 queue: root.users.root start time: 1540796329247 final status: SUCCEEDED tracking URL: http://cdh01:8088/proxy/application_1540796200906_0001/ user: root 18/10/29 14:59:08 INFO util.ShutdownHookManager: Shutdown hook called 18/10/29 14:59:08 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-2dd9e84f-a9ca-49b7-bd8e-1f912deca97c
