這是我的個人網頁，我來自臺灣，先前就讀於台灣大學物理系、台灣大學應用物理研究所，現在就讀於台灣大學資訊學系博士班，並於中研院CVLAB擔任研究助理
This is my personal website. I come from Taiwan.
NTU Physics bachelor
NTU Applied Physics master
NTU Computer Science and Information Engineering PhD student (PhD candidate now!)
Academia Sinica CVLAB research assistant
ULSEE algorithm engineer (2017/07~2017/12)
Congradutaions to Tsun-Yi Yang, Yi-Husan Huang, Yen-Yu Lin, Pi-Cheng Hsiu, and Yung-Yu Chuang! Our work entitled "SSR-Net: A Compact Soft Stagewise Regression Network for Age Estimation" has been accepted for IJCAI 2018. Project Page:https://github.com/shamangary/SSR-Net
Congradutaions to Tsun-Yi Yang, Jo-Han Hsu, Yen-Yu Lin, and Yung-Yu Chuang! Our work entitled "DeepCD: Learning Deep Complementary Descriptors for Patch Representations" has been accepted for ICCV 2017. Project Page:https://github.com/shamangary/DeepCD
Congradutaions to Tsun-Yi Yang, Yen-Yu Lin, and Yung-Yu Chuang! Our work entitled "Accumulated Stability Voting: A Robust Descriptor from Descriptors of Multiple Scales" has been accepted for CVPR 2016. Project Page:http://shamangary.logdown.com/posts/587520
[Eng. ver.] This is a note for rewriting StarGAN from pytorch to keras. I record my steps and thoughts in this note. If you find anything wrong or something you want to add, please contact me or comment below.
In this note, we first review the model.py first. Since every framework (pytorch, keras, ...) must require their own special coding method, it is pretty clear that this file contains pure pytorch, and we want to edit it first.
Goal
# Pytorch -> Keras
# Sequential model -> functional API
1. Conv2D/Conv2DTranspose
# Pytorch
# ref
torch.nn.Conv2d(in_channels, out_channels, kernel_size, stride=1, padding=0, dilation=1, groups=1, bias=True)
# source code
layers.append(nn.Conv2d(3+c_dim, conv_dim, kernel_size=7, stride=1, padding=3, bias=False))
----------------------------------------------
|
| (rewrite the layer into keras)
|
v
----------------------------------------------
# Keras
# ref
keras.layers.Conv2D(filters, kernel_size, strides=(1, 1), padding='valid', data_format=None, dilation_rate=(1, 1), activation=None, use_bias=True, kernel_initializer='glorot_uniform', bias_initializer='zeros', kernel_regularizer=None, bias_regularizer=None, activity_regularizer=None, kernel_constraint=None, bias_constraint=None)
keras.layers.ZeroPadding2D(padding=(1, 1), data_format=None)
# reimplementation
x = ZeroPadding2D(padding=(3,3))(x)
x = Conv2D(conv_dim ,(7,7), strides=(1,1), use_bias=False)(x)
or
x = Conv2D(conv_dim ,(7,7), strides=(1,1), padding='same', use_bias=False)(x)
There is no need to use input dimension of the layer in keras.
Conv2DTranspose
*** Somehow zero padding with Conv2DTranspose in Keras is not equal to nn.ConvTranspose2d in Pytorch. Use padding='same' directly!!! ***
# Pytorch
# source code
layers.append(nn.ConvTranspose2d(curr_dim, curr_dim//2, kernel_size=4, stride=2, padding=1, bias=False))
----------------------------------------------
|
| (rewrite the layer into keras)
|
v
----------------------------------------------
# Keras
# reimplementation
x = Conv2DTranspose(curr_dim//2 ,(4,4), strides=(2,2), padding='same', use_bias=False)(x)
# Pytorch
# ref
torch.nn.InstanceNorm2d(num_features, eps=1e-05, momentum=0.1, affine=False)
affine: a boolean value that when set to ``True``, gives the layer learnable affine parameters. Default: ``False``
# source code
layers.append(nn.InstanceNorm2d(conv_dim, affine=True))
----------------------------------------------
|
| (rewrite the layer into keras)
|
v
----------------------------------------------
# Keras
# ref
https://github.com/keras-team/keras-contrib/blob/master/keras_contrib/layers/normalization.py
# reimplementation
x = InstanceNormalization()(x)
3. ReLU/LeakyReLU
# Pytorch
# ref
torch.nn.ReLU(inplace=False)
torch.nn.LeakyReLU(negative_slope=0.01, inplace=False)
# source code
layers.append(nn.ReLU(inplace=True))
layers.append(nn.LeakyReLU(0.01, inplace=True))
----------------------------------------------
|
| (rewrite the layer into keras)
|
v
----------------------------------------------
# Keras
# ref
relu(x, alpha=0.0, max_value=None)
keras.layers.LeakyReLU(alpha=0.3)
# reimplementation
x = Activation('relu')(x)
x = LeakyReLU(alpha=0.01)(x)
4. unsqueeze and expand (in Pytorch)
# Pytorch
def forward(self, x, c):
# replicate spatially and concatenate domain information
c = c.unsqueeze(2).unsqueeze(3)
c = c.expand(c.size(0), c.size(1), x.size(2), x.size(3))
x = torch.cat([x, c], dim=1)
return self.main(x)
----------------------------------------------
|
| (rewrite the layer into keras)
|
v
----------------------------------------------
# Keras
inputs = Input(shape=self._input_shape) # remember input image channel is (RGB+mask channel)
fake_c = Input(shape=(self.c_dim,))
def labels_to_maps(x, image_size):
x_temp = K.expand_dims(x,1)
x_temp = K.expand_dims(x_temp,2)
x_temp = K.tile(x_temp, (1, image_size, image_size, 1))
return x_temp
fake_c_2d = Lambda(labels_to_maps, arguments={'image_size': self.image_size})(fake_c)
x = Concatenate(axis=-1)([inputs, fake_c_2d])
Note that unsqueeze() in pytorch is corresponding to expand_dims() in keras.
And expand() in pytorch is corresponding to tile() in keras.
5. squeeze (in Pytorch)
# pytorch
Returns a tensor with all the dimensions of input of size 1 removed.
http://pytorch.org/docs/master/torch.html
# source code
out_real.squeeze(), out_aux.squeeze()
----------------------------------------------
|
| (rewrite the layer into keras)
|
v
----------------------------------------------
# keras
def squeeze_all(x):
x_temp = x
delta_temp = 0
for i in range(1,4):
if x.shape[i]==1:
x_temp = K.squeeze(x_temp,axis=(i-delta_temp))
delta_temp = delta_temp + 1
return x_temp
out_real = Lambda(squeeze_all)(out_real)
out_aux = Lambda(squeeze_all)(out_aux)
Note that the squeeze in tensorflow or keras is not equal to the squeeze in pytorch.
You must define your own custom layer to do that.
<<General CNN training process>>
Step.1 Preprocessing
---------------------------------------------------------------------------
<model or mlp (multi-layer perceptron) loading>
<criterion setting>
<optimState setting>
local w,dE_dw = model:getParameters()
---------------------------------------------------------------------------
Step.2 Training through epoch
---------------------------------------------------------------------------
for epoch = 1, maxEpoch do
for t = 1, nBatches-1 do
xlua.progress(t+1, nBatches) -- disp progress
<inputs per batch setting>
local eval_E = function(w)
<forward and backward propagation>
return loss, dE_dw
end
optim.sgd(eval_E, w, optimState)
end
end
---------------------------------------------------------------------------
<<Propagations>>
local eval_E = function(w)
-- reset gradients
dE_dw:zero()
-- evaluate function for complete mini batch
local outputs = model:forward(inputs)
local loss = criterion:forward(outputs,targets)
-- estimate dE/dW
local dE_dy = criterion:backward(outputs,targets)
model:backward(inputs,dE_dy)
-- return E and dE/dW
return loss,dE_dw
end
In training mode, the network remembers all previous rho (number of time-steps) states. This is necessary for BPTT.
evaluate()
During evaluation, since their is no need to perform BPTT at a later time, only the previous step is remembered. This is very efficient memory-wise, such that evaluation can be performed using potentially infinite-length sequence.
remember([mode])
When mode='neither' (the default behavior of the class), the Sequencer will additionally call forget before each call to forward. When mode='both' (the default when calling this function), the Sequencer will never call forget. In which case, it is up to the user to call forget between independent sequences. This behavior is only applicable to decorated AbstractRecurrent modules. Accepted values for argument mode are as follows :
'eval' only affects evaluation (recommended for RNNs)
'train' only affects training
'neither' affects neither training nor evaluation (default behavior of the class)
'both' affects both training and evaluation (recommended for LSTMs)
forget(offset)
This method brings back all states to the start of the sequence buffers, i.e. it forgets the current sequence. It also resets the step attribute to 1. It is highly recommended to call forget after each parameter update. Otherwise, the previous state will be used to activate the next, which will often lead to instability. This is caused by the previous state being the result of now changed parameters. It is also good practice to call forget at the start of each new sequence.
裡面有一句話：
Note : make sure you call brnn:forget() after each call to updateParameters().
就是每次你update完之後記得要forget()一下。
The post is the setup guildeline of the OpenFace face recoginition sample code. Due to the development of the GPU and the Cuda version, the original installation guideline is not working for my computer. Therefore I record how did I setup the OpenFace step-by-step in order to save the time for the next people who need this.
-- Could NOT find Boost
-- *****************************************************************************************************
-- To compile Boost.Python yourself download boost from boost.org and then go into the boost root folder
-- and run these commands:
-- ./bootstrap.sh --with-libraries=python
-- ./b2
-- sudo ./b2 install
-- *****************************************************************************************************
Checking OpenCV and Dlib
如果上面都處理完了我們來確認一下是否有正確安裝好，OpenFace的官網也提供好了確認的方法
python2
import cv2
import dlib
---------2017/10/02更新-------------
如果遇到cv2 no module named cv2的話，用下面指令安裝即可
pip install opencv-python
如果什麼問題都沒有大概你就是裝好囉(^.<)V，我是說需要的資料庫而已
Torch7
接下來Torch本身也是有許多需要的東西的，官網提供如下
dpnn
nn
optim
csvigo
cutorch and cunn (only with CUDA)
fblualib (only for training a DNN)
tds (only for training a DNN)
torchx (only for training a DNN)
optnet (optional, only for training a DNN)
**Warning: DO NOT USE THE FOLLOWING COMMAND**
for NAME in dpnn nn optim optnet csvigo cutorch cunn fblualib torchx tds; do luarocks install $NAME; done
#lfw:(put the content of this one into ./data/lfw/raw)
All images as gzipped tar file
(173MB, md5sum a17d05bd522c52d84eca14327a23d494)
#lfw-deepfunneled:(put the content of this one into ./data/lfw/deepfunneled)
[new] All images aligned with deep funneling
(111MB, md5sum 68331da3eb755a505a502b5aacb3c201)
##Minimal Working Example to Extract Features
#command (remember to add "python2" and "th")
mkdir -p classify-test/raw/{lennon,clapton}
cp images/examples/lennon-* classify-test/raw/lennon
cp images/examples/clapton-* classify-test/raw/clapton
python2 ./util/align-dlib.py classify-test/raw align outerEyesAndNose classify-test/aligned --size 96
th ./batch-represent/main.lua -outDir classify-test/features -data classify-test/aligned
#warning
OpenBLAS Warning : Detect OpenMP Loop and this application may hang. Please rebuild the library with USE_OPENMP=1 option.
#output
nImgs: 4
Represent: 4/4
#How to fix the warning?
-- Add this line to the function "repBatch()" inside "./batch-represent/batch-represent.lua"
torch.setnumthreads(1)
接下來我們看一下Demo3和Training new neural network models都有的前處理 http://cmusatyalab.github.io/openface/demo-3-classifier/ 1. Create raw image directory
關於dataset，在Minimal Working Example to Extract Features中使用的只是幾張images，這裡我們對之後會使用的dataset進行一個下載的動作。
2. Preprocess the raw images
#Change 8 to however many separate processes you want to run:
for N in {1..8}; do python2 ./util/align-dlib.py <path-to-raw-data> align outerEyesAndNose <path-to-aligned-data> --size 96 & done
#Prune out directories with less than 3 images per class with:
python2 ./util/prune-dataset.py <path-to-aligned-data> --numImagesThreshold 3
Replace <path-to-raw-data> and <path-to-aligned-data> by the path of step.1
看到這裡可以發現"Minimal Working Example to Extract Features"就包含了前三步，可以視為一個完整的範例，但是第四步開始就不一樣了。
4. Create the Classification Model
執行第四步開始出現問題
#command
python2 ./demos/classifier.py train ./classify-test/features
#error1
Traceback (most recent call last):
File "./demos/classifier.py", line 34, in <module>
import pandas as pd
ImportError: No module named pandas
#error2 (after installing pandas)
Traceback (most recent call last):
File "./demos/classifier.py", line 38, in <module>
from sklearn.pipeline import Pipeline
ImportError: No module named sklearn.pipeline
/usr/local/lib/python2.7/dist-packages/sklearn/lda.py:4: DeprecationWarning: lda.LDA has been moved to discriminant_analysis.LinearDiscriminantAnalysis in 0.17 and will be removed in 0.19
"in 0.17 and will be removed in 0.19", DeprecationWarning)
Loading embeddings.
Training for 2 classes.
Saving classifier to './classify-test/features/classifier.pkl'
Classifying New Images
上面好了的話接下來也不會有問題
#command
python2 ./demos/classifier.py infer ./models/openface/celeb-classifier.nn4.small2.v1.pkl images/examples/{carell,adams,lennon}*
#output
/usr/local/lib/python2.7/dist-packages/sklearn/lda.py:4: DeprecationWarning: lda.LDA has been moved to discriminant_analysis.LinearDiscriminantAnalysis in 0.17 and will be removed in 0.19
"in 0.17 and will be removed in 0.19", DeprecationWarning)
=== images/examples/carell.jpg ===
Predict SteveCarell with 0.97 confidence.
=== images/examples/adams.jpg ===
Predict AmyAdams with 0.81 confidence.
=== images/examples/lennon-1.jpg ===
Predict SteveCarell with 0.50 confidence.
=== images/examples/lennon-2.jpg ===
Predict DavidBoreanaz with 0.43 confidence.
function repBatch(paths, inputs, labels, batchSz)
batchNumber = batchNumber + batchSz
-- number of threads
torch.setnumthreads(1)
--[[
if opt.cuda then
inputs = inputs:cuda()
end
--]]
--print(inputs:size())
if opt.cuda then
embeddings = torch.CudaTensor(inputs:size(1),128)
local embeddingsSplit = embeddings:split(10)
for j,v in ipairs(inputs:split(10)) do
v_temp = v:cuda()
embeddingsSplit[j]:copy(model:forward(v_temp))
end
embeddings = embeddings:float()
else
embeddings = model:forward(inputs):float()
end
if opt.cuda then
cutorch.synchronize()
end
if batchSz == 1 then
embeddings = embeddings:reshape(1, embeddings:size(1))
end
for i=1,batchSz do
labelsCSV:write({labels[i], paths[i]})
repsCSV:write(embeddings[i]:totable())
end
print(('Represent: %d/%d'):format(batchNumber, nImgs))
end
#command
cd evaluation
python2 ./lfw.py nn4.small2.v1 lfw.nn4.small2.v1.reps --lfwPairs ../data/lfw/pairs.txt
#output
/usr/local/lib/python2.7/dist-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')
Loading embeddings.
+ Reading pairs.
+ Computing accuracy.
+ 0.9292
Plotting.
1. Go to
http://www.cbsr.ia.ac.cn/english/CASIA-WebFace-Database.html
2. Follow the steps
Download Instructions:
To apply for the database, please follow the steps below:
(1)Download and print the document Agreement for using CASIA WebFace database
(2)Sign the agreement (The agreement must be signed by the director or the delegate of the deparmart of university. Personal applicant is not acceptable.)
(3)Send the agreement to cbsr-request@authenmetric.com
(4)Check your email to find a login account and a password of our website after one day, if your application has been approved.
(5)Download the CASIA WebFace database from our website with the authorized account within 48 hours.
Download facescrub (simple): 直接下載即可，要花一段時間
1. Download files from
https://github.com/faceteam/facescrub
2. Run the command
python download.py
#Facescrub dataset preprocessing
for N in {1..8}; do python ./util/align-dlib.py data/facescrub/download align outerEyesAndNose data/facescrub/dlib-affine-sz:96 --size 96 & done
#CASIA dataset preprocessing
for N in {1..8}; do python ./util/align-dlib.py data/CASIA/CAISA-WebFace align outerEyesAndNose data/CASIA/dlib-affine-sz:96 --size 96 & done
(1): nn.Sequential {
[input -> (1) -> (2) -> (3) -> output]
(1): nn.ConcatTable {
input
|`-> (1): first branch is there to transpose inputs to BHWD, for the bilinear sampler
|
|`-> (2): second branch is the localization network
|
... -> output
}
(2): nn.BilinearSamplerBHWD
(3): nn.Transpose
}
# Concept
Input
|
,-------+-----.
Downsampling 3x3 convolution+dimensionality reduction
| |
v v
Zero-padding 3x3 convolution
| |
`-----( Add )---'
|
Output
# How to add them up? (Only show the important part)
local skip = input
-- Add them together
net = cudnn.SpatialBatchNormalization(nOutChannels)(net)
net = nn.CAddTable(){net, skip}
net = cudnn.ReLU(true)(net)
# CMath Modules perform element-wise operations on a table of Tensors:
CAddTable: addition of input Tensors;
CSubTable: substraction of input Tensors;
CMulTable: multiplication of input Tensors;
CDivTable: division of input Tensors;
# in a terminal, run the commands WITHOUT sudo
git clone https://github.com/torch/distro.git ~/torch --recursive
cd ~/torch; bash install-deps;
./install.sh
# 注意！千萬要執行下一行，不管安裝有沒有成功都一定要，不然你重開機會無法正常進入ubnutu (2017/07/04更新)
# On Linux with bash
source ~/.bashrc
# run luarocks WITHOUT sudo
$ luarocks install image
$ luarocks list
2018/05/16更新
ubuntu的login loop有很多種可能，這次遇到的是裝完nvidia driver後就會發生，而且不管裝哪個版本都會發生，後來才發現原來是螢幕的顯示hdmi插到主機板的插孔而不是GPU的插孔，插對後會沒畫面，重開機就完全正常了，真的非常該死。
DO NOT INSERT HDMI ON THE MOTHERBOARD!!! INSERT IT ON THE HOLE OF GPU!!!!!!
