Cloudera配置问题

很明显这个问题是因为yarn设置的相关内存不足，有过Apache Hadoop配置经验的人都会去改yarn-site.xml的配置文件，但是对于Cloudera来说它提供的配置文件分在好多目录，让你很迷惑。其实Cloudera是通过网页Cloudera Manager 来配置这些参数的。

Memory errors

Configuring YARN after executor memory errors The following error can occur when the required executor memory is above the maximum threshold: Caused by: com.spss.mapreduce.exceptions.JobException: java.lang.IllegalArgumentException: Required executor memory (1024+384 MB) is above the max threshold (1024 MB) of this cluster! Please increase the value of 'yarn.scheduler.maximum-allocation-mb'. For Cloudera Increase the yarn.nodemanager.resource.memory-mb to 8GB In the Cloudera Manager user interface, go to Yarn service > Configurations > Search Container Memory and increase the value to 8GB. In the Cloudera Manager user interface, go to YARN service > Quick Links and select Dynamic Resource Pools. Under Configuration, click edit for each of the available pools, and under YARN set the Max Running Apps value to 4. Restart YARN, Spark, and the Analytic Server service.