Install

AWS Cloud Setup

Amazon Linux AMI with NVIDIA GRID and TESLA GPU Driver

NVIDIA GRID and TESLA GPU

The above AMI had been tested with Caffe2 + GPU support on a G2.2xlarge instance that uses a NVIDIA GRID K520 GPU. This AMI comes with CUDA v7.5, and no cuDNN, so we install that manually. The installation is currently a little tricky, but we hope over time this can be smoothed out a bit. This AMI is great though because it supports the latest and greatest features from NVIDIA.

Installation Guide

Note that this guide will help you install Caffe2 on any CentOS distribution. Amazon uses their own flavor of RHEL and they’ve installed CUDA in different spots than normally expected, so keep that in mind if you have to do some troubleshooting. Some of these steps will not be required on vanilla CentOS because things will go in their normal places.

Get your repos set

Many of the required dependencies don’t show up in Amazon’s enabled repositories. Epel is already provided in this image, but the repo is disabled. You need to enable it by editing the repo config to turn it on. Set enabled=1 in the epel.repo file. This enables you to find cmake3 leveldb-devel lmdb-devel.

1

sudo vim /etc/yum.repos.d/epel.repo

Next you should update yum and install Caffe2’s core dependencies. These differ slightly from Ubuntu due to availability of ready-to-go packages.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

sudo yum update
sudo yum install -y \
automake \
cmake3 \
gcc \
gcc-c++ \
git \
kernel-devel \
leveldb-devel \
lmdb-devel \
libtool \
protobuf-devel \
python-devel \
python-pip \
snappy-devel

gflags and glog is not found in yum for this version of Linux, so install from source:

1
2
3
4
5
6
7
8
9
10

git clone https://github.com/gflags/gflags.git && \
cd gflags && \
mkdir build && cd build && \
cmake3 -DBUILD_SHARED_LIBS=ON -DCMAKE_CXX_FLAGS='-fPIC' .. && \
make -j 8 && sudo make install && cd ../.. && \
git clone https://github.com/google/glog && \
cd glog && \
mkdir build && cd build && \
cmake3 -DBUILD_SHARED_LIBS=ON -DCMAKE_CXX_FLAGS='-fPIC' .. && \
make -j 8 && sudo make install && cd ../..

Python Dependencies

Now we need the Python dependencies. Note the troubleshooting info below… the install path with Python can get difficult.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

sudo pip install \
flask \
future \
graphviz \
hypothesis \
jupyter \
matplotlib \
numpy \
protobuf \
pydot \
python-nvd3 \
pyyaml \
requests \
scikit-image \
scipy \
setuptools \
six \
tornado

This may fail with error: pkg_resources.DistributionNotFound: pip==7.1.0

To fix this, upgrade pip, and then update the pip’s config to match the version it upgraded to.

1
2
3
4
5
6

$ sudo easy_install --upgrade pip
Password:
Searching for pip
Reading https://pypi.python.org/simple/pip/
Downloading https://pypi.python.org/packages/11/b6/abcb525026a4be042b486df43905d6893fb04f05aac21c32c638e939e447/pip-9.0.1.tar.gz#md5=35f01da33009719497f01a4ba69d63c9
Best match: pip 9.0.1

Note that in this example, the upgrade was to 9.0.1. Use vim to open the /usr/bin/pip file and change the instances of 7.1.0 to 9.0.1, and this solves the pip error and will allow you to install the dependencies.

1

sudo vim /usr/bin/pip

Once you’ve fixed the config file re-run the sudo pip install flask graphviz... command from above.

Setup CUDA

This image doesn’t come with cuDNN, however Caffe2 requires it. Here we’re downloading the files, extracting them, and copying them into existing folders where CUDA is currently installed.

Note: recent developments indicate that you should try to upgrade to CUDA 8 and cuDNN 6, however these instructions provide a working build with v7.5 and cuDNN 5.1.

1
2
3
4
5

wget http://developer.download.nvidia.com/compute/redist/cudnn/v5.1/cudnn-7.5-linux-x64-v5.1.tgz
tar xfvz cudnn-7.5-linux-x64-v5.1.tgz
sudo rsync -av cuda /opt/nvidia/
rm cudnn-7.5-linux-x64-v5.1.tgz
rm -rf cuda

Now you need to setup some environment variables for the build step.

1
2

export CUDA_HOME=/opt/nvidia/cuda
export LD_LIBRARY_PATH=/opt/nvidia/cuda/lib64:/usr/local/bin

Almost done. Now you need to clone Caffe2 repo and build it (note: update the -j8 with your system’s number of processors; to check this, run nproc from the terminal.):

1
2
3
4

git clone --recursive https://github.com/caffe2/caffe2
cd caffe2 && mkdir build
cd build && cmake3 ..
sudo make -j8 install

Test it out!

To check if Caffe2 is working and it’s using the GPU’s try the commands below. The first will tell you success or failure, and the second should trigger the GPU and output of a bunch of arrays, but more importantly, you should see no error messages! Consult the Troubleshooting section of the docs here and for Ubuntu for some help.

1
2

python -c 'from caffe2.python import core' 2>/dev/null && echo "Success" || echo "Failure"
python -m caffe2.python.operator_test.relu_op_test

Test CUDA

Here are a series of commands and sample outputs that you can try. These will verify that the GPU’s are accessible.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

$ cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module 352.99 Mon Jul 4 23:52:14 PDT 2016
GCC version: gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC)

$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2015 NVIDIA Corporation
Built on Tue_Aug_11_14:27:32_CDT_2015
Cuda compilation tools, release 7.5, V7.5.17

$ nvidia-smi -q | head

==============NVSMI LOG==============

Timestamp : Fri Mar 10 23:15:45 2017
Driver Version : 352.99

Attached GPUs : 1
GPU 0000:00:03.0
Product Name : GRID K520
Product Brand : Grid

That’s it. You’ve successfully built Caffe2!

Setting Up Tutorials & Jupyter Server

If you’re running this all on a cloud computer, you probably won’t have a UI or way to view the IPython notebooks by default. Typically, you would launch them locally with ipython notebook and you would see a localhost:8888 webpage pop up with the directory of notebooks running. The following example will show you how to launch the Jupyter server and connect to remotely via an SSH tunnel.

First configure your cloud server to accept port 8889, or whatever you want, but change the port in the following commands. On AWS you accomplish this by adding a rule to your server’s security group allowing a TCP inbound on port 8889. Otherwise you would adjust iptables for this.

Next you launch the Juypter server.

1

jupyter notebook --no-browser --port=8889

Then create the SSH tunnel. This will pass the cloud server’s Jupyter instance to your localhost 8888 port for you to use locally. The example below is templated after how you would connect AWS, where your-public-cert.pem is your own public certificate and [email protected] is your login to your cloud server. You can easily grab this on AWS by going to Instances > Connect and copy the part after ssh and swap that out in the command below.

1

ssh -N -f -L localhost:8888:localhost:8889 -i "your-public-cert.pem" [email protected]

Troubleshooting

caffe2.python not found | You may have some PATH or PYTHONPATH issues. Add /home/ec2-user/caffe2/build to your path and that can take care of those problems. error while loading shared libraries: libCaffe2_CPU.so: cannot open shared object file: No such file or directory | Try updating your LD_LIBRARY_PATH with export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH undefined reference to `ncclReduceScatter’ | This does not occur on Caffe2 building, but on linking with “libCaffe2_GPU.so” in some external projects. To solve this, you may install NCCL from its source bundled with Caffe2: (under the Caffe2 project directory) cd third_party/nccl && make -j 8 && sudo make install

Dependencies

Try to keep the system and python dependencies at the same version. We’ve encountered issues when the python version is more updated than the system version or vice versa.

Strictly speaking, the core dependencies above are all you need to run the core Caffe2 successfully. However, for real-world deep learning (e.g., image processing, mathematical operations, etc), there are other dependencies that you will want to install in order to experience the full features of Caffe2.

1
2
3
4
5

pip install future ^
            hypothesis ^
            numpy ^
            protobuf ^
            six

1
2
3
4
5
6
7
8
9
10
11
12

pip install flask ^
            glog ^
            graphviz ^
            jupyter ^
            matplotlib ^
            pydot python-nvd3 ^
            pyyaml ^
            requests ^
            scikit-image ^
            scipy ^
            setuptools ^
            tornado

1

git clone --recursive https://github.com/caffe2/caffe2.git

1

conda install -y --channel https://conda.anaconda.org/conda-forge  graphviz hypothesis numpy pydot-ng python-lmdb requests zeromq

1

git clone --recursive https://github.com/caffe2/caffe2.git

1
2

cd caffe2
./scripts/build_ios.sh

1

git clone --recursive https://github.com/caffe2/caffe2.git

1
2

cd caffe2
./scripts/build_android.sh

1
2
3
4
5

docker pull caffe2ai/caffe2
# to test
nvidia-docker run -it caffe2ai/caffe2:latest python -m caffe2.python.operator_test.relu_op_test
# to interact
nvidia-docker run -it caffe2ai/caffe2:latest /bin/bash

1
2
3
4
5

docker pull caffe2ai/caffe2:cpu-minimal-ubuntu14.04 
# to test
docker run -it caffe2ai/caffe2:cpu-minimal-ubuntu14.04 python -m caffe2.python.operator_test.relu_op_test
# to interact
docker run -it caffe2ai/caffe2:cpu-minimal-ubuntu14.04 /bin/bash

1
2

cd ~/caffe2/docker/ubuntu-14.04-cpu-all-options
docker build -t caffe2:cpu-optionals .

1

docker run -it caffe2 /bin/bash

1

docker run -it 5ee1fb669aef /bin/bash

1
2

docker build -t caffe2:cpu-minimal .
docker run -it caffe2:cpu-minimal /bin/bash

1

docker run -it mydocker-repo/mytag ipython

1

sudo docker run -ti --device /dev/nvidia0:/dev/nvidia0 --device /dev/nvidiactl:/dev/nvidiactl --device /dev/nvidia-uvm:/dev/nvidia-uvm mydocker-repo/mytag ipython

1

In [1]: from caffe2.python import workspace

1

docker run -it mydocker-repo/mytag /bin/bash

1

nvidia-docker run -it caffe2 python -m caffe2.python.operator_test.relu_op_test

1

docker run -it -p 8888:8888 caffe2ai/caffe2:cpu-fulloptions-ubuntu14.04 sh -c "jupyter notebook --no-browser --ip 0.0.0.0 /caffe2/caffe2/python/tutorials"

1

git clone --recursive https://github.com/caffe2/caffe2.git

1
2

cd caffe2
./scripts/build_raspbian.sh

1

git clone --recursive https://github.com/caffe2/caffe2.git

1
2

cd caffe2
./scripts/build_tegra_x1.sh