200字范文 > 『DL笔记』Ubuntu16.04安装cuda-8.0+cudnn5.1+opencv-3.1+caffe+本地浏览器访问服务器jupyter notebook设置

『DL笔记』Ubuntu16.04安装cuda-8.0+cudnn5.1+opencv-3.1+caffe+本地浏览器访问服务器jupyter notebook设置

时间：2022-12-04 01:34:43

文章目录

一、安转cuda+cudnn1.1、安转cuda1.2、安转cudnn 二、安转opencv-3.12.1、安装依赖库2.2、问题1: 下载ippicv_linux_1201失败2.3、问题2: CUDA 8.0不支持OpenCV的GraphCut算法三、安转caffe(convolutional architecture for feature embedding)3.1、caffe简介3.2、修改Makefile.config文件内容3.3、修改caffe目录下的Makefile文件内容3.4、编译四、caffe简单例子4.1、准备数据4.2、定义网络结构4.3、配置solver文件4.4、评估模型执行的时间4.5、训练模型4.6、分析结果4.7、本地浏览器访问服务器jupyter notebook设置参考文献

一、安转cuda+cudnn

cuda8.0+cudnn5.1百度云下载链接：提取码：0ls9

1.1、安转cuda

安装ssh工具：备注：这一步需要到服务器桌面上的命令窗口输入，这一步完成后，就可以用ssh工具远程连接服务器了，本文使用的是XShell。

sudo apt-get install openssh-server

参考我之前的链接，方法一样的；安装cuda!

1.2、安转cudnn

参考我之前的链接，方法一样的；安装cudnn!

二、安转opencv-3.1

2.1、安装依赖库

可以参考大神：yolov3训练自己的数据集——第一次实操完整记录

首先安装一些依赖包，如下：

sudo apt-get install build-essentialsudo apt-get install cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-devsudo apt-get install python-dev python-numpy libtbb2 libtbb-dev libjpeg-dev libpng-dev libtiff-dev libjasper-dev libdc1394-22-devsudo apt-get install --assume-yes libopencv-dev libdc1394-22 libdc1394-22-dev libjpeg-dev libpng12-dev libtiff5-dev libjasper-dev libavcodec-dev libavformat-dev libswscale-dev libxine2-dev libgstreamer0.10-dev libgstreamer-plugins-base0.10-dev libv4l-dev libtbb-dev libqt4-dev libfaac-dev libmp3lame-dev libopencore-amrnb-dev libopencore-amrwb-dev libtheora-dev libvorbis-dev libxvidcore-dev x264 v4l-utils unzipsudo apt-get install ffmpeg libopencv-dev libgtk-3-dev python-numpy python3-numpy libdc1394-22 libdc1394-22-dev libjpeg-dev libpng12-dev libtiff5-dev libjasper-dev libavcodec-dev libavformat-dev libswscale-dev libxine2-dev libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libv4l-dev libtbb-dev qtbase5-dev libfaac-dev libmp3lame-dev libopencore-amrnb-dev

进入官网链接 , 选择 3.1.0 版本的 source , 下载 opencv-3.1.0.zip下载之后，解压到你要安装的位置，命令行进入已解压的文件夹 opencv-3.1.0 目录下，执行：

mkdir build # 创建编译的文件目录cd buildcmake -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=/usr/local ..

成功的话显示如下：

make -j8 #编译make install

2.2、问题1: 下载ippicv_linux_1201失败

问题1：OpenCV 3.1下载 ippicv_linux_1201失败;安装OpenCv 3.1的过程中要下载ippicv_linux_1201，由于网络的原因，这个文件经常会下载失败。解决的办法是手动下载： ippicv_linux_1201；将下载后的ippicv文件拷到下面这个目录，问题得到解决。

参考这个文章：opencv 解决ippicv下载慢的问题: ippicv__lnx_intel64_general_0723

opencv-3.1.0/3rdparty/ippicv/downloads/linux-808b791a6eac9ed78d32a7666804320e/

最终cmake执行成功显示如下（部分）：

-- Python (for build): /usr/bin/python2.7-- -- Java:--ant:NO--JNI:NO--Java wrappers:NO--Java tests: NO-- -- Matlab: Matlab not found or implicitly disabled-- -- Documentation:--Doxygen: NO--PlantUML:NO-- -- Tests and samples:--Tests: YES--Performance tests: YES--C/C++ Examples: NO-- -- Install path: /usr/local-- -- cvconfig.h is in: /home/zhangkf/opencv-3.1.0/build-- ------------------------------------------------------------------- -- Configuring done-- Generating done-- Build files have been written to: /home/zhangkf/opencv-3.1.0/build

2.3、问题2: CUDA 8.0不支持OpenCV的GraphCut算法

问题2：由于CUDA 8.0不支持OpenCV的 GraphCut 算法，可能出现以下错误：

/home/zhangkf/opencv-3.1.0/modules/cudalegacy/src/graphcuts.cpp:275:146: error: ‘nppiGraphcut8_32f8u’ was not declared in this scopestatic_cast<int>(terminals.step), static_cast<int>(leftTransp.step), sznpp, labels.ptr<Npp8u>(), static_cast<int>(labels.step), state) );^/home/zhangkf/opencv-3.1.0/modules/core/include/opencv2/core/private.cuda.hpp:165:52: note: in definition of macro ‘nppSafeCall’#define nppSafeCall(expr) cv::cuda::checkNppError(expr, __FILE__, __LINE__, CV_Func)^make[2]: *** [modules/cudalegacy/CMakeFiles/opencv_cudalegacy.dir/src/graphcuts.cpp.o] Error 1make[2]: *** Waiting for unfinished jobs....make[1]: *** [modules/cudalegacy/CMakeFiles/opencv_cudalegacy.dir/all] Error 2make[1]: *** Waiting for unfinished jobs....[ 90%] Linking CXX shared library ../../lib/libopencv_photo.so[ 90%] Built target opencv_photomake: *** [all] Error 2

解决方法：修改/opencv-3.1.0/modules/cudalegacy/src/graphcuts.cpp文件内容，如下：

#include "precomp.hpp"#if !defined (HAVE_CUDA) || defined (CUDA_DISABLER)改为#include "precomp.hpp"#if !defined (HAVE_CUDA) || defined (CUDA_DISABLER) || (CUDART_VERSION >= 8000)

最终make -j8 #编译成功显示如下（部分）：

[100%] Building CXX object modules/superres/CMakeFiles/opencv_test_superres.dir/test/test_main.cpp.o[100%] Building CXX object modules/superres/CMakeFiles/opencv_test_superres.dir/test/test_superres.cpp.o[100%] Building CXX object modules/superres/CMakeFiles/opencv_perf_superres.dir/perf/perf_superres.cpp.o[100%] Building CXX object modules/superres/CMakeFiles/opencv_perf_superres.dir/perf/perf_main.cpp.o[100%] Linking CXX executable ../../bin/opencv_test_superres[100%] Built target opencv_test_superres[100%] Linking CXX executable ../../bin/opencv_perf_superres[100%] Built target opencv_perf_superresScanning dependencies of target opencv_videostab[100%] Building CXX object modules/videostab/CMakeFiles/opencv_videostab.dir/src/frame_source.cpp.o[100%] Building CXX object modules/videostab/CMakeFiles/opencv_videostab.dir/src/global_motion.cpp.o[100%] Building CXX object modules/videostab/CMakeFiles/opencv_videostab.dir/src/stabilizer.cpp.o[100%] Building CXX object modules/videostab/CMakeFiles/opencv_videostab.dir/src/inpainting.cpp.o[100%] Building CXX object modules/videostab/CMakeFiles/opencv_videostab.dir/src/fast_marching.cpp.o[100%] Building CXX object modules/videostab/CMakeFiles/opencv_videostab.dir/src/deblurring.cpp.o[100%] Building CXX object modules/videostab/CMakeFiles/opencv_videostab.dir/src/wobble_suppression.cpp.o[100%] Building CXX object modules/videostab/CMakeFiles/opencv_videostab.dir/src/outlier_rejection.cpp.o[100%] Building CXX object modules/videostab/CMakeFiles/opencv_videostab.dir/src/motion_stabilizing.cpp.o[100%] Building CXX object modules/videostab/CMakeFiles/opencv_videostab.dir/src/log.cpp.o[100%] Building CXX object modules/videostab/CMakeFiles/opencv_videostab.dir/src/optical_flow.cpp.o[100%] Linking CXX shared library ../../lib/libopencv_videostab.so[100%] Built target opencv_videostab[100%] Generating pyopencv_generated_include.h, pyopencv_generated_funcs.h, pyopencv_generated_types.h, pyopencv_generated_type_reg.h, pyopencv_generated_ns_reg.h[100%] Generating pyopencv_generated_include.h, pyopencv_generated_funcs.h, pyopencv_generated_types.h, pyopencv_generated_type_reg.h, pyopencv_generated_ns_reg.hScanning dependencies of target opencv_python2Scanning dependencies of target opencv_python3[100%] Building CXX object modules/python2/CMakeFiles/opencv_python2.dir/__/src2/cv2.cpp.o[100%] Building CXX object modules/python3/CMakeFiles/opencv_python3.dir/__/src2/cv2.cpp.o[100%] Linking CXX shared module ../../lib/python3/cv2.cpython-34m.so[100%] Built target opencv_python3[100%] Linking CXX shared module ../../lib/cv2.so[100%] Built target opencv_python2zhangkf@Ubuntu:~/opencv-3.1.0/build$

安装完成后通过查看opencv版本验证是否安装成功：

pkg-config --modversion opencv

opencv安装视频推荐：链接

三、安转caffe(convolutional architecture for feature embedding)

3.1、caffe简介

终于可以安装caffe了，使用Git直接下载Caffe非常简单，或者去/BVLC/caffe下载。由于我习惯去github上找代码，所以就直接去下载的源码。

git clone /BVLC/caffe.git

进入 caffe ，将 Makefile.config.example 文件复制一份并更名为 Makefile.config ，也可以在 caffe 目录下直接调用以下命令完成复制操作：

sudo cp Makefile.config.example Makefile.config

复制一份的原因是编译 caffe 时需要的是 Makefile.config 文件，而Makefile.config.example 只是caffe 给出的配置文件例子，不能用来编译 caffe。然后修改 Makefile.config 文件，在 caffe 目录下打开该文件：

sudo gedit Makefile.config

3.2、修改Makefile.config文件内容

这里把我电脑上用到的具体配置贴出来：

## Refer to /installation.html# Contributions simplifying and improving our build system are welcome!# cuDNN acceleration switch (uncomment to build with cuDNN).USE_CUDNN := 1 ########################### CPU-only switch (uncomment to build without GPU support).# CPU_ONLY := 1# uncomment to disable IO dependencies and corresponding data layersUSE_OPENCV := 1 ########################### USE_LEVELDB := 0# USE_LMDB := 0# This code is taken from /sh1r0/caffe-android-lib# USE_HDF5 := 0# uncomment to allow MDB_NOLOCK when reading LMDB files (only if necessary)#You should not set this flag if you will be reading LMDBs with any#possibility of simultaneous read and write# ALLOW_LMDB_NOLOCK := 1# Uncomment if you're using OpenCV 3OPENCV_VERSION :=3 ########################### To customize your choice of compiler, uncomment and set the following.# N.B. the default for Linux is g++ and the default for OSX is clang++# CUSTOM_CXX := g++# CUDA directory contains bin/ and lib/ directories that we need.CUDA_DIR := /usr/local/cuda-10.0 ########################### On Ubuntu 14.04, if cuda tools are installed via# "sudo apt-get install nvidia-cuda-toolkit" then use this instead:# CUDA_DIR := /usr# CUDA architecture setting: going with all of them.# For CUDA < 6.0, comment the *_50 through *_61 lines for compatibility.# For CUDA < 8.0, comment the *_60 and *_61 lines for compatibility.# For CUDA >= 9.0, comment the *_20 and *_21 lines for compatibility.# CUDA_ARCH := -gencode arch=compute_20,code=sm_20#################################################################CUDA_ARCH :=-gencode arch=compute_30,code=sm_30 \-gencode arch=compute_35,code=sm_35 \-gencode arch=compute_50,code=sm_50 \-gencode arch=compute_52,code=sm_52 \-gencode arch=compute_60,code=sm_60 \-gencode arch=compute_61,code=sm_61 \-gencode arch=compute_61,code=compute_61# BLAS choice:# atlas for ATLAS (default)# mkl for MKL# open for OpenBlasBLAS := atlas ########################### Custom (MKL/ATLAS/OpenBLAS) include and lib directories.# Leave commented to accept the defaults for your choice of BLAS# (which should work)!#BLAS_INCLUDE := /home/asrim/intel/mkl/include#BLAS_LIB := /home/asrim/intel/mkl/lib/intel64# Homebrew puts openblas in a directory that is not on the standard search path# BLAS_INCLUDE := $(shell brew --prefix openblas)/include# BLAS_LIB := $(shell brew --prefix openblas)/lib# This is required only if you will compile the matlab interface.# MATLAB directory should contain the mex binary in /bin.MATLAB_DIR := /usr/local ##########################MATLAB根据自己而定MATLAB_DIR := /usr/local/MATLAB/Rb ##########################MATLAB根据自己而定# NOTE: this is required only if you will compile the python interface.# We need to be able to find Python.h and numpy/arrayobject.h.# PYTHON_INCLUDE := /usr/include/python2.7 \#/usr/lib/python2.7/dist-packages/numpy/core/include# Anaconda Python distribution is quite popular. Include path:# Verify anaconda location, sometimes it's in root.ANACONDA_HOME := $(HOME)/anaconda3/envs/caffe ############################PYTHON_INCLUDE := $(ANACONDA_HOME)/include \$(ANACONDA_HOME)/include/python3.5m \$(ANACONDA_HOME)/lib/python3.5/site-packages/numpy/core/include# Uncomment to use Python 3 (default is Python 2)# PYTHON_LIBRARIES := boost_python-py35 python3.5m# PYTHON_INCLUDE := /usr/include/python3.5m \# /usr/lib/python3.5/dist-packages/numpy/core/include# We need to be able to find libpythonX.X.so or .dylib.# PYTHON_LIB := /usr/libPYTHON_LIB := $(ANACONDA_HOME)/lib #########################LINKFLAGS := -Wl,-rpath,$(ANACONDA_HOME)/lib ##################### Homebrew installs numpy in a non standard path (keg only)# PYTHON_INCLUDE += $(dir $(shell python -c 'import numpy.core; print(numpy.core.__file__)'))/include# PYTHON_LIB += $(shell brew --prefix numpy)/lib# Uncomment to support layers written in Python (will link against Python libs)WITH_PYTHON_LAYER := 1# Whatever else you find you need goes here.INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial ##########LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu/hdf5/serial ########### If Homebrew is installed at a non standard location (for example your home directory) and you use it for general dependencies# INCLUDE_DIRS += $(shell brew --prefix)/include# LIBRARY_DIRS += $(shell brew --prefix)/lib# NCCL acceleration switch (uncomment to build with NCCL)# /NVIDIA/nccl (last tested version: v1.2.3-1+cuda8.0)# USE_NCCL := 1# Uncomment to use `pkg-config` to specify OpenCV library paths.# (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.)# USE_PKG_CONFIG := 1# N.B. both build and distribute dirs are cleared on `make clean`BUILD_DIR := buildDISTRIBUTE_DIR := distribute# Uncomment for debugging. Does not work on OSX due to /BVLC/caffe/issues/171# DEBUG := 1# The ID of the GPU that 'make runtest' will use to run unit tests.TEST_GPUID := 0# enable pretty build (comment to see full commands)Q ?= @

注意一点：我的依赖库都是在anaconda虚拟环境caffe中安装的，这里ANACONDA_HOME := $(HOME)/anaconda3/envs/caffe

1、若使用cudnn，则将

#USE_CUDNN := 1 修改为：USE_CUDNN := 1

2、若使用的opencv版本是3的，则将

#OPENCV_VERSION := 3 修改为：OPENCV_VERSION := 3

3、若要使用python来编写layer，则将

#WITH_PYTHON_LAYER := 1 修改为：WITH_PYTHON_LAYER := 1

4、修改 python 路径(重要的一项)

INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/includeLIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib 修改为： INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serialLIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu/hdf5/serial

3.3、修改caffe目录下的Makefile文件内容

将：NVCCFLAGS +=-ccbin=$(CXX) -Xcompiler-fPIC $(COMMON_FLAGS)替换为：NVCCFLAGS += -D_FORCE_INLINES -ccbin=$(CXX) -Xcompiler -fPIC $(COMMON_FLAGS)

将：LIBRARIES += glog gflags protobuf boost_system boost_filesystem m hdf5_hl hdf5改为：LIBRARIES += glog gflags protobuf boost_system boost_filesystem m hdf5_serial_hl hdf5_serial

然后修改 /usr/local/cuda/include/host_config.h 文件 :

将#error-- unsupported GNU version! gcc versions later than 4.9 are not supported!改为//#error-- unsupported GNU version! gcc versions later than 4.9 are not supported!

3.4、编译

开始编译，在 caffe 目录下执行：

make all -j8

如果显示结果如下（部分），则表示 caffe 已经成功安装。

CXX src/caffe/layers/cudnn_tanh_layer.cppCXX src/caffe/layers/power_layer.cppCXX src/caffe/layers/input_layer.cppCXX src/caffe/layers/log_layer.cppCXX src/caffe/layers/conv_layer.cppCXX src/caffe/layers/lrn_layer.cppCXX src/caffe/layers/loss_layer.cppCXX src/caffe/layers/slice_layer.cppCXX src/caffe/layers/batch_norm_layer.cppCXX src/caffe/layers/cudnn_sigmoid_layer.cppCXX src/caffe/layers/tile_layer.cppCXX src/caffe/layers/mvn_layer.cppCXX src/caffe/layers/spp_layer.cppCXX src/caffe/layers/prelu_layer.cppCXX src/caffe/layers/accuracy_layer.cppCXX src/caffe/layers/clip_layer.cppCXX src/caffe/layers/threshold_layer.cppCXX src/caffe/layers/softmax_loss_layer.cppCXX src/caffe/layers/embed_layer.cppCXX src/caffe/layers/sigmoid_cross_entropy_loss_layer.cppCXX src/caffe/layers/tanh_layer.cppCXX src/caffe/layers/cudnn_lrn_layer.cppCXX src/caffe/layers/elu_layer.cppCXX src/caffe/layers/scale_layer.cppCXX src/caffe/layers/bnll_layer.cppCXX src/caffe/layers/lstm_unit_layer.cppCXX src/caffe/layers/image_data_layer.cppCXX src/caffe/layers/dropout_layer.cppCXX src/caffe/layers/filter_layer.cppCXX src/caffe/layers/swish_layer.cppCXX src/caffe/layers/neuron_layer.cppCXX src/caffe/layers/cudnn_conv_layer.cppCXX src/caffe/layers/hdf5_data_layer.cppCXX src/caffe/layers/reduction_layer.cppCXX src/caffe/layers/data_layer.cppCXX src/caffe/layers/split_layer.cppCXX src/caffe/layers/batch_reindex_layer.cppCXX src/caffe/layers/base_data_layer.cppCXX src/caffe/layers/relu_layer.cppCXX src/caffe/layers/base_conv_layer.cppCXX src/caffe/layers/bias_layer.cppCXX src/caffe/layers/parameter_layer.cppCXX src/caffe/layers/euclidean_loss_layer.cppCXX src/caffe/layers/cudnn_pooling_layer.cppCXX src/caffe/layers/dummy_data_layer.cppCXX src/caffe/layers/recurrent_layer.cppCXX src/caffe/layers/infogain_loss_layer.cppCXX src/caffe/layers/multinomial_logistic_loss_layer.cppCXX src/caffe/layers/sigmoid_layer.cppCXX src/caffe/layers/cudnn_softmax_layer.cppCXX src/caffe/layers/softmax_layer.cppCXX src/caffe/layers/rnn_layer.cppCXX src/caffe/layers/flatten_layer.cppCXX src/caffe/layers/contrastive_loss_layer.cppCXX src/caffe/layers/inner_product_layer.cppCXX src/caffe/layers/hinge_loss_layer.cppCXX src/caffe/layers/exp_layer.cppCXX src/caffe/layers/crop_layer.cppCXX src/caffe/layers/argmax_layer.cppCXX src/caffe/layers/absval_layer.cppCXX src/caffe/layers/cudnn_lcn_layer.cppCXX src/caffe/layers/cudnn_deconv_layer.cppCXX src/caffe/layers/lstm_layer.cppCXX src/caffe/layers/concat_layer.cppCXX src/caffe/layers/deconv_layer.cppCXX src/caffe/layers/hdf5_output_layer.cppCXX src/caffe/layers/im2col_layer.cppCXX src/caffe/layers/eltwise_layer.cppCXX src/caffe/layers/silence_layer.cppCXX src/caffe/layers/memory_data_layer.cppCXX src/caffe/layers/window_data_layer.cppCXX src/caffe/layers/pooling_layer.cppCXX src/caffe/layers/reshape_layer.cppCXX src/caffe/layers/cudnn_relu_layer.cppCXX src/caffe/util/db_lmdb.cppCXX src/caffe/util/math_functions.cppCXX src/caffe/util/benchmark.cppCXX src/caffe/util/db_leveldb.cppCXX src/caffe/util/hdf5.cppCXX src/caffe/util/im2col.cppCXX src/caffe/util/blocking_queue.cppCXX src/caffe/util/io.cppCXX src/caffe/util/insert_splits.cppCXX src/caffe/util/upgrade_proto.cppCXX src/caffe/util/db.cppCXX src/caffe/util/cudnn.cppCXX src/caffe/util/signal_handler.cppCXX src/caffe/parallel.cppCXX src/caffe/net.cppAR -o .build_release/lib/libcaffe.aLD -o .build_release/lib/libcaffe.so.1.0.0CXX/LD -o .build_release/tools/extract_features.binCXX/LD -o .build_release/tools/caffe.binCXX/LD -o .build_release/tools/convert_imageset.binCXX/LD -o .build_release/tools/upgrade_solver_proto_text.binCXX/LD -o .build_release/tools/upgrade_net_proto_binary.binCXX/LD -o .build_release/tools/upgrade_net_proto_text.binCXX/LD -o .build_release/tools/compute_image_mean.binCXX/LD -o .build_release/examples/mnist/convert_mnist_data.binCXX/LD -o .build_release/examples/cpp_classification/classification.binCXX/LD -o .build_release/examples/siamese/convert_mnist_siamese_data.binCXX/LD -o .build_release/examples/cifar10/convert_cifar_data.binzhangkf@wang-ThinkStation-P520:~/caffe$ clear

sudo make test -j8

如果之前的配置或安装出错，那么编译就会出现各种各样的问题，所以前面的步骤一定要细心。编译成功后可运行测试：

sudo make runtest -j8

然后，配置环境变量，将/home/zhangkf/anaconda3/envs/caffe/python加到环境变量:

export PYTHONPATH="/home/zhangkf/anaconda3/envs/caffe/bin/python:$PATH"

然后输入Python

(caffe) zhangkf@wang-ThinkStation-P520:~$ pythonPython 3.4.5 |Continuum Analytics, Inc.| (default, Jul 2 , 17:47:47) [GCC 4.4.7 0313 (Red Hat 4.4.7-1)] on linuxType "help", "copyright", "credits" or "license" for more information.>>> import caffe>>>

上面如果ImportError: No module named caffe，需要把caffe下的Python路径导入环境变量中去；

sudo vim ~/.bashrc# 最后一行加上，这里的路径写上你自己的路径。export PYTHONPATH="home/zhangkf/caffe/python:$PATH"# 记得执行下面的命令，否则的话只能在这个目录下执行Python，导入caffe了；source ~/.bashrc

下面测试caffe是不是安装好想要不再特定目录下正常运行caffe这个命令，需要添加环境变量。这样任何位置输入caffe都可以啦。

alias caffe="/home/zhangkf/caffe/build/tools/caffe"

四、caffe简单例子

4.1、准备数据

mnist数据集下载，执行/home/zhangkf/caffe/data/mnist目录下的get_mnist.sh脚本进行mnist数据集下载

bash get_mnist.sh

解压数据集

(caffe) zhangkf@wang-ThinkStation-P520:~/caffe/data/mnist$ tar -xvzf mnist.tar.gz t10k-images-idx3-ubytetrain-images-idx3-ubytet10k-labels-idx1-ubytetrain-labels-idx1-ubyte

数据集格式转换；下载得到的MNIST数据集是二进制文件,需要转换为LEVELDB或者LMDB才能被Caffe识别. 可以通过Caffe框架的/caffe/example/mnsit/create_mnist.sh将原始数据转换为lmdb/leveldb 格式（注意：需要在caffe路径下运行）

(caffe) zhangkf@wang-ThinkStation-P520:~/caffe$ bash examples/mnist/create_mnist.sh Creating lmdb...I1008 09:52:35.662119 6949 db_lmdb.cpp:35] Opened lmdb examples/mnist/mnist_train_lmdbI1008 09:52:35.662295 6949 convert_mnist_data.cpp:88] A total of 60000 items.I1008 09:52:35.662302 6949 convert_mnist_data.cpp:89] Rows: 28 Cols: 28I1008 09:52:39.572388 6949 convert_mnist_data.cpp:108] Processed 60000 files.I1008 09:52:39.897873 6961 db_lmdb.cpp:35] Opened lmdb examples/mnist/mnist_test_lmdbI1008 09:52:39.898061 6961 convert_mnist_data.cpp:88] A total of 10000 items.I1008 09:52:39.898067 6961 convert_mnist_data.cpp:89] Rows: 28 Cols: 28I1008 09:52:40.528492 6961 convert_mnist_data.cpp:108] Processed 10000 files.Done.

4.2、定义网络结构

定义网络结构：hbk_mnist.prototxt

name: "hbk_mnist"################### train/test lmdb数据层layer {name: "mnist"type: "Data"top: "data"top: "label"include {phase: TRAIN}transform_param {scale: 0.00390625}data_param {source: "/home/zhangkf/caffe/examples/mnist/mnist_train_lmdb" # 训练集的位置batch_size: 64backend: LMDB}}layer {name: "mnist"type: "Data"top: "data"top: "label"include {phase: TEST}transform_param {scale: 0.00390625}data_param {source: "/home/zhangkf/caffe/examples/mnist/mnist_test_lmdb" # 测试集的位置batch_size: 100backend: LMDB}}#################### 全连接层，激活层为ReLU 784->500->10layer {name: "ip1"type: "InnerProduct"bottom: "data"top: "ip1"param {lr_mult: 1}param {lr_mult: 2}inner_product_param {num_output: 500weight_filler {type: "xavier"}bias_filler {type: "constant"}}}################### 激活层layer {name: "relu1"type: "ReLU"bottom: "ip1"top: "re1"}################### 全连接层layer {name: "ip2"type: "InnerProduct"bottom: "re1"top: "ip2"param {lr_mult: 1}param {lr_mult: 2}inner_product_param {num_output: 10weight_filler {type: "xavier"}bias_filler {type: "constant"}}}# 测试验证用，不必须，输出准确率layer {name: "accuracy"type: "Accuracy"bottom: "ip2"bottom: "label"top: "accuracy"include {phase: TEST}}# 代价Cost层layer {name: "loss"type: "SoftmaxWithLoss"bottom: "ip2"bottom: "label"top: "loss"}

网络结构定义完了还是不太直观，我想看看这个网络结构到底是什么样，caffe提供了一个工具这个工具python/draw_net.py,它可以把我们定义的网络结构图形化显示出来。

python ~/caffe/python/draw_net.py hbk_mnist.prototxt aa.png --rankdir=BT# 其中aa.png是要生成的图片名称。# --rankdir=BT BT表示bottom top从底向上。# --rankdir=LR LR表示left right从左向右

4.3、配置solver文件

配置solver文件：hbk_mnist_solver.prototxt

# The train/test net 文件路径net: "hbk_mnist.prototxt" # 网络文件# test_iter specifies how many forward passes the test should carry out.# In the case of MNIST, we have test batch size 100 and 100 test iterations,# covering the full 10,000 testing images.test_iter: 100# 训练迭代多少次执行一次Test验证test_interval: 500# The base learning rate, momentum and the weight decay of the network. 默认随机梯度下降base_lr: 0.01momentum: 0.9weight_decay: 0.0005# The learning rate policylr_policy: "inv"gamma: 0.0001power: 0.75# 多少次迭代输出一次信息display: 100# The maximum number of iterationsmax_iter: 10001# 存储中间结果snapshot: 5000snapshot_prefix: "snapshot"# solver mode: CPU or GPUsolver_mode: GPU

4.4、评估模型执行的时间

评估模型执行的时间大概是多少，运行如下命令（下面这个CPU跑的时间）：

zhangkf@wang-ThinkStation-P520:~/caffe/my_project$ caffe time -model hbk_mnist.prototxt -iterations 10......I1008 12:57:17.865840 8901 caffe.cpp:397] Average time per layer: I1008 12:57:17.865844 8901 caffe.cpp:400]mnistforward: 0.0028 ms.I1008 12:57:17.865847 8901 caffe.cpp:403]mnistbackward: 0.0001 ms.I1008 12:57:17.865867 8901 caffe.cpp:400] ip1forward: 1.8415 ms.I1008 12:57:17.865870 8901 caffe.cpp:403] ip1backward: 1.6795 ms.I1008 12:57:17.865873 8901 caffe.cpp:400]relu1forward: 0.021 ms.I1008 12:57:17.865876 8901 caffe.cpp:403]relu1backward: 0.022 ms.I1008 12:57:17.865880 8901 caffe.cpp:400] ip2forward: 0.1029 ms.I1008 12:57:17.865885 8901 caffe.cpp:403] ip2backward: 0.1368 ms.I1008 12:57:17.865887 8901 caffe.cpp:400] lossforward: 0.0241 ms.I1008 12:57:17.865890 8901 caffe.cpp:403] lossbackward: 0.0008 ms.I1008 12:57:17.865897 8901 caffe.cpp:408] Average Forward pass: 1.995 ms.I1008 12:57:17.865900 8901 caffe.cpp:410] Average Backward pass: 1.8418 ms.I1008 12:57:17.865905 8901 caffe.cpp:412] Average Forward-Backward: 3.8 ms.I1008 12:57:17.865909 8901 caffe.cpp:414] Total Time: 38 ms. # 跑了38毫秒I1008 12:57:17.865914 8901 caffe.cpp:415] *** Benchmark ends ***

GPU跑的时间：

zhangkf@wang-ThinkStation-P520:~/caffe/my_project$ caffe time -model hbk_mnist.prototxt -iterations 10 -gpu 0......I1008 13:00:59.563093 8943 caffe.cpp:397] Average time per layer: I1008 13:00:59.563104 8943 caffe.cpp:400]mnistforward: 0.212448 ms.I1008 13:00:59.563109 8943 caffe.cpp:403]mnistbackward: 0.0014048 ms.I1008 13:00:59.563115 8943 caffe.cpp:400] ip1forward: 0.0407168 ms.I1008 13:00:59.563120 8943 caffe.cpp:403] ip1backward: 0.0318944 ms.I1008 13:00:59.563125 8943 caffe.cpp:400]relu1forward: 0.0118592 ms.I1008 13:00:59.563130 8943 caffe.cpp:403]relu1backward: 0.0077504 ms.I1008 13:00:59.563136 8943 caffe.cpp:400] ip2forward: 0.0259872 ms.I1008 13:00:59.563140 8943 caffe.cpp:403] ip2backward: 0.02536 ms.I1008 13:00:59.563146 8943 caffe.cpp:400] lossforward: 0.0810144 ms.I1008 13:00:59.563151 8943 caffe.cpp:403] lossbackward: 0.0143584 ms.I1008 13:00:59.563159 8943 caffe.cpp:408] Average Forward pass: 0.41905 ms.I1008 13:00:59.563165 8943 caffe.cpp:410] Average Backward pass: 0.113318 ms.I1008 13:00:59.563171 8943 caffe.cpp:412] Average Forward-Backward: 0.56696 ms.I1008 13:00:59.563177 8943 caffe.cpp:414] Total Time: 5.6696 ms. # 跑了5.6696毫秒I1008 13:00:59.563191 8943 caffe.cpp:415] *** Benchmark ends ***

4.5、训练模型

zhangkf@wang-ThinkStation-P520:~/caffe/my_project$ caffe time -model hbk_mnist.prototxt -iterations 10 gpu 0......I1008 13:06:12.997189 8988 solver.cpp:347] Iteration 10000, Testing net (#0)I1008 13:06:13.031898 9002 data_layer.cpp:73] Restarting data prefetching from start.I1008 13:06:13.032205 8988 solver.cpp:414]Test net output #0: accuracy = 0.9752I1008 13:06:13.032243 8988 solver.cpp:414]Test net output #1: loss = 0.0869556 (* 1 = 0.0869556 loss)I1008 13:06:13.032627 8988 solver.cpp:239] Iteration 10000 (1507.71 iter/s, 0.0663259s/100 iters), loss = 0.046366I1008 13:06:13.032657 8988 solver.cpp:258]Train net output #0: loss = 0.0463659 (* 1 = 0.0463659 loss)I1008 13:06:13.032671 8988 sgd_solver.cpp:112] Iteration 10000, lr = 0.00594604I1008 13:06:13.032773 8988 solver.cpp:464] Snapshotting to binary proto file snapshot_iter_10001.caffemodelI1008 13:06:13.037735 8988 sgd_solver.cpp:284] Snapshotting solver state to binary proto file snapshot_iter_10001.solverstateI1008 13:06:13.040637 8988 solver.cpp:332] Optimization Done.I1008 13:06:13.040649 8988 caffe.cpp:250] Optimization Done.

我们发现程序跑的太快，一屏跟不上显示不完。下面把所有的log都输入一个文件里面；其中 2>&1重定向命令解释

zhangkf@wang-ThinkStation-P520:~/caffe/my_project$ caffe train -solver hbk_mnist_solver.prototxt 2>&1 | tee a.logzhangkf@wang-ThinkStation-P520:~/caffe/my_project$ clearzhangkf@wang-ThinkStation-P520:~/caffe/my_project$ ls1hbk_mnist.prototxt snapshot_iter_10000.caffemodel snapshot_iter_10001.caffemodel snapshot_iter_5000.caffemodela.log hbk_mnist_solver.prototxt snapshot_iter_10000.solverstate snapshot_iter_10001.solverstate snapshot_iter_5000.solverstatezhangkf@wang-ThinkStation-P520:~/caffe/my_project$