Skip to content

Repository for our upcoming code, that we used for our "Deep diffeomorphic transformer networks" paper (Accepted to CVPR 2018). Will be update during the spring of 2018.

License

Notifications You must be signed in to change notification settings

SkafteNicki/ddtn

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

32 Commits
 
 
 
 
 
 
 
 

Repository files navigation

ddtn (Deep Diffeomorphic Transformer Networks)

This repo is a Tensorflow implementation of so called continues piecewise affine based (CPAB) transformations by Freifeld et al.. We use these transformations to create a more flexible spatial transformer layer, than the original layer by Jadenberg et al.. The code is written using both Tensorflows c++ and python API. Thus the main contribution of this repo is a so implementation of a ST-layer with diffeomorphic transformations. However, the repo also contains code for other kinds of ST-layers. The following transformation models are included:

  • Affine transformations
  • Diffiomorphic affine transformations
  • Homography transformations
  • CPAB transformations
  • Thin-Plate-Spline (TPS) transformations

The code is based upon the original implementation of the CPAB transformations by Oren Freifeld (Github repo: cpabDiffeo). Additionally, some of the code for doing interpolation is based upon the Tensorflow implementation of the Spatial Transformer Networks (Github repo: spatial-transformer-network).

Author of this software

Nicki Skafte Detlefsen (email: [email protected])

License

This software is released under the MIT License (included with the software). Note, however, that using this code (and/or the results of running it) to support any form of publication (e.g., a book, a journal paper, a conference papar, a patent application ect.) we request you to cite the following papers:

[1] @article{detlefsen2018transformations,
  title = {Deep Diffeomorphic Transformer Networks},
  author = {Nicki Skafte Detlefsen and Oren Freifeld and S{\o}ren Hauberg},
  journal = {Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2018},
  publisher={IEEE}
}

[2] @article{freifeld2017transformations,
  title={Transformations Based on Continuous Piecewise-Affine Velocity Fields},
  author={Freifeld, Oren and Hauberg, Soren and Batmanghelich, Kayhan and Fisher, John W},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year={2017},
  publisher={IEEE}
}

Requirements

  • Generic python packages: numpy, scipy, matplotlib
  • Tensorflow
  • To use the GPU implementation, you need a nvidia GPU and CUDA + cuDNN installed. See Tensorflows GPU installation instructions for more details

The code was testes running python 3.6, tensorflow 1.8.0 and CUDA 9.0. However, the code should be compatible with tensorflow from version 1.4.0.

The code should run on all operating system with or without an GPU. If you are on Linux or MAC and tensorflow is able to detect your GPU then the fast GPU version of the CPAB transformations will be uses. If you are on windows or do have an GPU, an slower (and memory consuming) implementation will be used that is written in pure tensorflow.

Installation

  1. Clone this reposatory to a directory of your choice
git clone https://github.com/SkafteNicki/ddtn
  1. Add this directory to your PYTHONPATH
export PYTHONPATH=$PYTHONPATH:$YOUR_FOLDER_PATH/ddtn
  1. (optional) If you want to use the fast GPU version their is a good chance that you have to recompile the dynamic libraries where the operation is defined. Go to the folder 'ddtn/ddtn/cuda/' and type the following in a command prompt
make clean
make

Using the code

Try opening a python command promt and type in the follow command

import ddtn

This should give you one of the following outputs

----------------------------------------------------------------------
Operating system: linux
Using the fast cuda implementation for CPAB
----------------------------------------------------------------------
or
----------------------------------------------------------------------
Operating system: linux
Using the slow pure tensorflow implementation for CPAB
----------------------------------------------------------------------

The reposatory comes with two scripts you can try to run

python play_with_transformers.py <- show what the different transformers can do
python mnist_classifier.py <- trains a classifier on a distorted mnist dataset

for both script you can type -h after to get the commandline options.

To use the different transformers in your own settings, there are primarily 3 files that are important

  1. ddtn.transformers.transformers

    Defines methods called tf_'name'_transformer eg. tf_Affine_transformer, tf_CPAB_transformer ect. that as input takes a grid of points and a parametrization and outputs the transformed grid.

  2. ddtn.transformers.transformer_layers

    Defines methods called ST_'name'_transformer eg. ST_Affine_transformer, ST_CPAB_transformer ect. that as input takes an image, a parametrization and output the transformed image.

  3. ddtn.transformers.keras_layers

    Defined keras layers called Spatial'name'Layer eg. SpatialAffineLayer, SpatialCPABLayer ect. that can be incorporated into keras models (main used high-level-api for tensorflow).

Known bugs

  1. "Executor failed to create kernel. Not found: Op type not registered 'tf_CPAB_transformer' in binary running on HedonismeBot. Make sure the Op and Kernel are registered in the binary running in this process".

    tf.Defun seems to be working but still give out this error message. Should probably try to find a replacement for tf.Defun. Has something to do with the fact that tensorflow sessions freeze the current graph. Look into tf.RegisterGradient at some point.

About

Repository for our upcoming code, that we used for our "Deep diffeomorphic transformer networks" paper (Accepted to CVPR 2018). Will be update during the spring of 2018.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published