从http://spark.apache.org/下载spark源码包。
然后
(1)解压
$tar -zxvf spark-x.y.z-bin-hadoopx.y.tgz
(2)修改pom.xml文件
该文件在解压目录中的conf文件夹当中，我的部分修改如下：
<java.version>1.8</java.version> <hadoop.version>2.7.3</hadoop.version> <protobuf.version>2.5.0</protobuf.version> <hbase.version>0.98.9-hadoop2</hbase.version> <zookeeper.version>3.4.6</zookeeper.version> <derby.version>10.11.1.1</derby.version>
(3)如果要支持Scala 2.11，请运行以下脚本：-- 该版本太新了，目前还有些组件不支持，所以不建议修改
~spark-2.1.0$sh ./dev/change-version-to-2.11.sh
(4)修改maven环境变量配置
export MAVEN_OPTS="-Xmx4g -XX:MaxPermSize=1024M -XX:ReservedCodeCacheSize=1024m"
(5)maven编译
~spark-2.1.0$./build/mvn -Pyarn -Phadoop-2.7 -Dhadoop.version=2.7.3 -DskipTests clean package 该过程需要很长时间，也可能会出现build failure，多执行几次。
(6)生成安装包
如果不需要hive，则执行
./dev/make-distribution.sh --name "hadoop2-without-hive" --tgz "-Pyarn,hadoop-provided,hadoop-2.7,parquet-provided"
如果需要hive，则执行
./dev/make-distribution.sh --name custom-spark --tgz -Psparkr -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pyarn 我的没有要hive，最后编译成功信息如下： main: [INFO] Executed tasks [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Spark Project Parent POM ........................... SUCCESS [ 19.393 s] [INFO] Spark Project Tags ................................. SUCCESS [ 18.437 s] [INFO] Spark Project Sketch ............................... SUCCESS [ 15.209 s] [INFO] Spark Project Networking ........................... SUCCESS [ 19.325 s] [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 12.823 s] [INFO] Spark Project Unsafe ............................... SUCCESS [ 24.586 s] [INFO] Spark Project Launcher ............................. SUCCESS [ 23.378 s] [INFO] Spark Project Core ................................. SUCCESS [31:01 min] [INFO] Spark Project ML Local Library ..................... SUCCESS [ 39.471 s] [INFO] Spark Project GraphX ............................... SUCCESS [32:14 min] [INFO] Spark Project Streaming ............................ SUCCESS [33:25 min] [INFO] Spark Project Catalyst ............................. SUCCESS [ 01:54 h] [INFO] Spark Project SQL .................................. SUCCESS [ 04:23 h] [INFO] Spark Project ML Library ........................... SUCCESS [ 01:08 h] [INFO] Spark Project Tools ................................ SUCCESS [ 10.568 s] [INFO] Spark Project Hive ................................. SUCCESS [25:59 min] [INFO] Spark Project REPL ................................. SUCCESS [18:52 min] [INFO] Spark Project YARN Shuffle Service ................. SUCCESS [ 42.815 s] [INFO] Spark Project YARN ................................. SUCCESS [ 01:08 h] [INFO] Spark Project Assembly ............................. SUCCESS [ 35.475 s] [INFO] Spark Project External Flume Sink .................. SUCCESS [ 25.452 s] [INFO] Spark Project External Flume ....................... SUCCESS [27:54 min] [INFO] Spark Project External Flume Assembly .............. SUCCESS [ 40.121 s] [INFO] Spark Integration for Kafka 0.8 .................... SUCCESS [10:36 min] [INFO] Spark Project Examples ............................. SUCCESS [06:14 min] [INFO] Spark Project External Kafka Assembly .............. SUCCESS [ 34.261 s] [INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [40:25 min] [INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [ 15.937 s] [INFO] Kafka 0.10 Source for Structured Streaming ......... SUCCESS [18:20 min] [INFO] Spark Project Java 8 Tests ......................... SUCCESS [02:35 min] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 12:49 h [INFO] Finished at: 2017-04-26T21:07:50+08:00 [INFO] Final Memory: 126M/904M [INFO] ------------------------------------------------------------------------ 可以看到花的时间非常久。编译成功后的安装包是在最早的解压目录下，文件名是spark-2.1.0-bin-hadoop2-without-hive.tgz
