Spark、hadoop源码编译

CentOS6.7-64bit编译hadoop2.6.4

1.下载maven（apache-maven-3.3.3-bin.tar.gz）

http://archive.apache.org/dist/maven/maven-3/3.3.3/binaries/apache-maven-3.3.3-bin.tar.gz 2.安装maven tar -zxvf apache-maven-3.3.3-bin.tar.gz -C /usr/local 3.添加环境变量 vim /etc/profile export JAVA_HOME=/usr/local/jdk1.7.0_45 export MAVEN_HOME=/usr/local/apache-maven-3.3.3 export PATH=$PATH:$JAVA_HOME/bin:$MAVEN_HOME/bin 4.加载环境变量 source /etc/proflie mvn -version 5.下载protobuf（https://code.google.com/p/protobuf/downloads/list 或 https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz） 6.安装protobuf编译依赖，为了编译安装protoc，需要使用YUM下载几个依赖的工具 yum install -y gcc gcc-c++ make 7.解压安装protobuf tar -zxvf  protobuf-2.5.0.tar.gz -C /usr/local/src cd /usr/local/src/protobuf-2.5.0   ./configure --prefix=/usr/local/protobuf  make && make install 8将protobuf添加到环境变量 vim /etc/profile export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$MAVEN_HOME/bin:/usr/local/protobuf/bin source /etc/profile protoc --version 9.安装其他编译依赖 yum install -y cmake openssl-devel ncurses-devel zlib-devel yum install -y snappy snappy-devel bzip2 bzip2-devel lzo lzo-devel lzop openssl openssl-devel 10.编译hadoop（进入文件夹里面，里面有一个文件BUILDINT.txt，打开即可看见里面关于编译hadoop的一些环境要求） tar -zxvf hadoop-2.6.4-src.tar.gz -C /usr/local/src/ cd /usr/local/src/hadoop-2.6.4-src mvn package -Pdist,native -DskipTests -Dtar   12.查看编译好的安装包 cd /usr/local/src/hadoop-2.6.4-src/hadoop-dist/target #出现问题 INFO util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 在/etc/profile中添加 export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native" #查看可用的本地库 hadoop checknative -a cat /proc/cpuinfo | grep "physical id" | sort | uniq | wc –l cat /proc/cpuinfo | grep "core id" | sort -u | wc -l cat /proc/cpuinfo | grep MHz | uniq

CentOS6.7-64bit编译spark-1.6.1

tar -zxvf spark-1.6.1.tgz -C /usr/local/src/ cd /usr/local/src/spark-1.6.1/ #设置内存2G export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m" #编译前安装一些压缩解压缩工具 yum install -y snappy snappy-devel bzip2 bzip2-devel lzo lzo-devel lzop openssl openssl-devel #仅仅是为了编译源码, 编译后可以导入idea中 mvn clean package -Phadoop-2.6 -Dhadoop.version=2.6.4 -Phive -Phive-thriftserver -Pyarn -DskipTests #编译后并打包, 打包后可以丢到生产环境了 ./make-distribution.sh --tgz -Phadoop-2.6 -Dhadoop.version=2.6.4 -Phive -Phive-thriftserver -Pyarn -DskipTests