这个系列主要摘自《Tensorflow实战Google深度学习框架》这本书,边学习边整理其中主要内容,以便记忆。
Tensorflow系列一环境搭建
一 Tensorflow的主要依赖包
Protocol BufferBazel 二 Tensorflow安装
使用 Virtualenv安装使用native pip安装使用Docker安装使用Anaconda安装从源码编译安装 三 Tensorflow测试样例
Tensorflow系列一:环境搭建
一 Tensorflow的主要依赖包
1. Protocol Buffer
Protocol Buffer是谷歌开发的处理结构化数据的工具。这里结构化数据指的是拥有多种属性的数据。假设用户信息包括名字,ID和Email地址。
一个用户的信息表示示例
name:张三id: 12345email: zhangsan@abc.com
当要将结构化数据数据持久化或进行网络传输时,需要先将它序列化(将结构化数据变成数据流的格式,简单地说就是变为一个字符串)。如何将结构化数据序列化,并从序列化数据流中还原结构化数据,称为处理结构化数据。
XML和JSON是两种常用的结构化数据处理工具。
<user>
<name>张三
</name>
<id>12345
</id>
<email>zhangsan@abc.com
</email>
</user>
//JSON格式示例
{
"
name":
"张三",
"id":"12345",
"
email":
"zhangsan@abc.com",
}
不同于XML和JSON,Protocol Buffer序列化后得到的数据不是可读的字符串威视二进制流。再者Protocol Buffer需要先定义数据的格式(schema)再还原结构化数据。(Protocol Buffer序列化的数据比XML格式数据小3到10倍,解析时间要快20到100倍)。
message user{
optional
string name =
1;
required
int32 id =
2;
repeated
string email =
3;
}
2. Bazel
Bazel是从谷歌开源的自动化构建工具,用于编译应用。
项目空间(worksapce): Bazel的基本概念,包含了编译一个软件所需要的源代码以及输出编译结果的软连接(symbolic link)地址。项目空间对应的文件夹是项目的根目录,需要有:
WORKSPACE文件:定义了对外部资源的依赖关系,可为空文件;
BULID文件:通过它找到需要编译的目标,该文件中指定可每一个编译目标的输入,输出和编译方式。Bazel的编译方式是事先定义好的,比如Python支持的编译方式只有三种:py_binary(将Python程序编译为可执行文件), py_library(将Python编译为库函数供供py_test或py_binary调用)和py_test(编译Python测试程序)。
下面使用一个简单样例来说明Bazel的工作,如下项目空间有4个文件:
-rw-rw-r-- root root
208 BULID
-rw-rw-r-- root root
48 hello_lib.py
-rw-rw-r-- root root
47 hell0_main.py
-rw-rw-r-- root root
0 WORKSPACE
def print_hello_world():
print(
"Hello World")
import hello_lib
hello_lib.print_hello_world()
'''
name属性:编译目标名字;
srcs属性:编译所需要的源代码,可以是一个列表;
deps属性:编译所需依赖关系
'''
py_library(
name =
"hello_lib",
srcs = [
"hello_lib.py"
]
)
py_binary(
name =
"hello_main",
srcs = [
"hello_main.py",
],
deps = [
":hello_lib",
],
)
在这个项目空间中允许编译操作bazel build :hello_main 将得到编译结果,他们都是通过软连接的形式放在当前的项目空间里,实际编译结果文件保存在~/.cache/bazel目录下,其中bazel-bin目录下寻访了编译产生的二进制文件以及运行该文件所需的里来关系,在当前目录下运行bazel-bin/hello_main实现hello_main.py程序,即在屏幕输出”Hello World“。
二 Tensorflow安装
这部分不再参考书籍内容,而是翻译官网安装文档https://www.tensorflow.org/install/install_linux(installing Tensorflow on Ubuntu)。
1.使用 Virtualenv安装
$ sudo apt-get
install python-pip python-dev python-virtualenv
$ virtualenv --system-site-packages 目标目录(假设为~/tensorflow)
$ source ~/tensorflow/bin/activate
$ source ~/tensorflow/bin/activate.csh
(tensorflow)$ pip
install --upgrade tensorflow
(tensorflow)$ pip3
install --upgrade tensorflow
(tensorflow)$ pip
install --upgrade tensorflow-gpu
(tensorflow)$ pip3
install --upgrade tensorflow-gpu
(tensorflow)$ pip
install --upgrade tenfsorflow安装包路径
(tensorflow)$ pip3
install --upgrade tenfsorflow安装包路径
(tensorflow)$ deactivate
$ rm -r 目标目录
2.使用”native“ pip安装
$ python -
V
$ pip -
V
$ sudo apt-get install python-pip python-dev
$ sudo pip install tensorflow
$ sudo pip3 install tensorflow
$ sudo pip install tensorflow-gpu
$ sudo pip3 install tensorflow-gpu
$ sudo pip install --upgrade tenfsorflow安装包路径
$ sudo pip3 install --upgrade tenfsorflow安装包路径
$ sudo pip uninstall tensorflow
$ sudo pip3 uninstall tensorflow
3.使用Docker安装
#先安装Docker(若要安装支持GPUs的Tensorflow,需要安装nvidia-docker)
#可以创建一个Linux docker group,调用docker可不使用sudo
#运行一个包含Tensorflow镜像的docker容器(cpu)
$ docker run -it -p hostPort:containerPort Tensorflow镜像
#-p hostPort:containerPort是可选的,-p 8888:8888使用Jupyter Notebooks; -p 6006:6006 使用TensorBoard
#Tensorflow镜像(以下按所需选择)
#gcr.io/tensorflow/tensorflow, which is the TensorFlow CPU binary image.
#gcr.io/tensorflow/tensorflow:latest-devel, which is the latest TensorFlow CPU Binary image plus source code.
#gcr.io/tensorflow/tensorflow:version, which is the specified version (for example, 1.0.1) of TensorFlow CPU binary image.
#gcr.io/tensorflow/tensorflow:version-devel
#GPU
$ nvidia-docker run -it -p hostPort:containerPort Tensorflow镜像
#Tensorflow镜像(以下按所需选择)
#gcr.io/tensorflow/tensorflow:lastest-gpu, which is the TensorFlow CPU binary image.
#gcr.io/tensorflow/tensorflow:latest-devel-gpu, which is the latest TensorFlow CPU Binary image plus source code.
#gcr.io/tensorflow/tensorflow:version-gpu, which is the specified version (for example, 1.0.1) of TensorFlow CPU binary image.
#gcr.io/tensorflow/tensorflow:version-devel-gpu
4.使用Anaconda安装
#先安装Anaconda
#创建一个conda环境命名为tensorflow
$ conda create -n tensorflow
#激活
$ source activate tensorflow
#安装Tensorflow
(tensorflow)$ pip install --ignore-installed --upgrade tenfsorflow安装包路径
5.从源码编译安装
#安装Tensorflow依赖的工具包
#1.Bazel(https://bazel.build/versions/master/docs/install-ubuntu.html#install-with-installer-ubuntu)
echo "deb [arch=amd64] http://storage.googleapis.com/bazel-apt stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list
curl https://bazel.build/bazel-release.pub.gpg | sudo apt-key add -
sudo apt-get
update && sudo apt-get install bazel
sudo apt-get upgrade bazel#更新Bazel
#2.Python依赖包(numpy,dev,pip,wheel)
$ sudo apt-get install python-numpy python-dev python-pip python-wheel #for Python 2.7
$ sudo apt-get install python3-numpy python3-dev python3-pip python3-wheel #for Python 3.n
#3.可选的:GPU支持
#Nvidia计算能力大于3.0的GPU
#Nvidia的Cuda Toolkit(>=7.0)和cuDNN(>=v3)
#安装libcupti-dev
$ sudo apt-get install libcupti-dev
#配置Tensoflow编译环境
#下载最新的Tensorflow源代码
$ git clone https://github.com/tensorflow/tensorflow
$ cd tensorflow
$ ./configure
#配置Python的路径
Please specify the location of python. [Default is /usr/bin/python]: /usr/bin/python2.7
Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -march=native]:
Do you wish to use jemalloc as the malloc implementation? [Y/n]
jemalloc enabled
#配置是否支持谷歌云平台
Do you wish to build TensorFlow with Google Cloud Platform support? [y/N]
No Google Cloud Platform support will be enabled for TensorFlow
#配置是否支持Hadoop File System
Do you wish to build TensorFlow with Hadoop File System support? [y/N]
No Hadoop File System support will be enabled for TensorFlow
#配置是否支持XLA just-in-time编译
Do you wish to build TensorFlow with the XLA just-in-time compiler (experimental)? [y/N]
No XLA JIT support will be enabled for TensorFlow
Found possible Python library paths:
/usr/local/lib/python2.7/dist-packages
/usr/lib/python2.7/dist-packages
#配置Python库路径
Please input the desired Python library path to use. Default is [/usr/local/lib/python2.7/dist-packages]
Using python library path: /usr/local/lib/python2.7/dist-packages
#配置是否支持OpenCL
Do you wish to build TensorFlow with OpenCL support? [y/N] N
No OpenCL support will be enabled for TensorFlow
#配置是否支持CUDA
Do you wish to build TensorFlow with CUDA support? [y/N] Y
CUDA support will be enabled for TensorFlow
#配置GPU
Please specify which gcc should be used by nvcc as the host compiler. [Default is /usr/bin/gcc]:
#配置 Cuda SDK版本
Please specify the Cuda SDK version you want to use, e.g. 7.0. [Leave empty to use system default]: 8.0
#配置Cuda toolkit目录
Please specify the location where CUDA 8.0 toolkit is installed. Refer to README.md for more details. [Default is /usr/local/cuda]:
#配置 cuDnn版本
Please specify the cuDNN version you want to use. [Leave empty to use system default]: 5
#配置cuDnn目录
Please specify the location where cuDNN 5 library is installed. Refer to README.md for more details. [Default is /usr/local/cuda]:、
#配置GPU计算能力
Please specify a list of comma-separated Cuda compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus.
Please note that each additional compute capability significantly increases your build time and binary size.
[Default is: "3.5,5.2"]: 3.0
Setting up Cuda include
Setting up Cuda lib
Setting up Cuda bin
Setting up Cuda nvvm
Setting up CUPTI include
Setting up CUPTI lib64
Configuration finished
#使用Bazel编译pip的安装包
$ bazel build --config=opt //tensorflow/tools/pip_package:build_pip_package #cpu
$ bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package #for gpu
$ bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
#安装
$ sudo pip install /tmp/tensorflow_pkg/tensorflow-1.1.0-py2-none-any.whl
三 Tensorflow测试样例
$ python
>>> import tensorflow
as tf
>>> a = tf.constant([
1.0,
2.0],name=
"a")
>>> b = tf.constant([
2.0,
3.0],name=
"b")
>>> c = a + b
>>> sess = tf.Session()
>>> sess.run(c)