conda build for OS X

[tylerjereddy]

I've produced a preliminary conda build for OS X:

Try this in a clean environment (that doesn't already have any kind of access to MDAnalysis):

conda config --add channels treddy
conda config --add channels MDAnalysis
conda install mdanalysis

and let me know if you have any issues (it seems to work for me). It is particularly annoying to test this if you work with an environment that already has a source version installed for development, etc. Also, I had to remove the testing from the conda build procedure because of the circular dependency situation -- so running the tests on the Mac conda build would also be helpful.

There's a huge discussion in #751 and elsewhere regarding conda builds that I skimmed through. It looks like scripts are being put in the maintainer package path, although mine look a bit different and I don't see a per-architecture breakdown there yet.

[kain88-de]

@tylerjereddy can you share the script you have used to generate the build.

Also, I had to remove the testing from the conda build procedure because of the circular dependency situation

That sounds strange. The installatoin should work. The build.sh script is basically the recommended install way for developers.

I don't see a per-architecture breakdown there yet.

There shouldn't be one. I can't see a reason why the build process should be different between osx and linux right now. Also conda doesn't encourage that.

[tylerjereddy]

The objective was simply to perform a skeleton build from pypi and see if it worked. Out of the box, it didn't, so I edited the code until it did. I don't think it is that strange to have problems--it may just be that one or two other os-x things need to be added to the appropriate channel, but if you start from the skeleton it doesn't just work on my machine. Anyway, I just got the ball rolling--feel free to take over.

meta.yaml:

package:
  name: mdanalysis
  version: "0.14.0"

source:
  fn: MDAnalysis-0.14.0.tar.gz
  url: https://pypi.python.org/packages/source/M/MDAnalysis/MDAnalysis-0.14.0.tar.gz
  md5: 51e62a1df93b4c84957594977acdbdbd
#  patches:
   # List any patch files here
   # - fix.patch

# build:
  # noarch_python: True
  # preserve_egg_dir: True
  # entry_points:
    # Put any entry points (scripts to be generated automatically) here. The
    # syntax is module:function.  For example
    #
    # - mdanalysis = mdanalysis:main
    #
    # Would create an entry point called mdanalysis that calls mdanalysis.main()


  # If this is a new build for the same version, increment the build
  # number. If you do not include this key, it defaults to 0.
  # number: 1

requirements:
  build:
    - python
    - setuptools
    - numpy >=1.5.0
    - biopython >=1.59
    - networkx >=1.0
    - griddataformats >=0.3.2
    - six >=1.4.0

  run:
    - python
    - numpy >=1.5.0
    - biopython >=1.59
    - networkx >=1.0
    - griddataformats >=0.3.2
    - six >=1.4.0

#test:
#  # Python imports
#  imports:
#    - MDAnalysis
#    - MDAnalysis.analysis
#    - MDAnalysis.analysis.hbonds
#    - MDAnalysis.coordinates
#    - MDAnalysis.coordinates.pdb
#    - MDAnalysis.core
#    - MDAnalysis.lib
#    - MDAnalysis.lib.formats
#    - MDAnalysis.migration
#    - MDAnalysis.migration.fixes
#    - MDAnalysis.selections
#    - MDAnalysis.tests
#    - MDAnalysis.topology
#    - MDAnalysis.topology.tpr
#    - MDAnalysis.visualization
#
#  # commands:
#    # You can put test commands to be run here.  Use this to test that the
#    # entry points work.
#
#
#  # You can also put a file called run_test.py in the recipe that will be run
#  # at test time.
#
#  requires:
#    - nose >=1.3.7
#    # Put any additional test requirements here.  For example
#    # - nose

about:
  home: http://www.mdanalysis.org
  license: GNU General Public License (GPL)
  summary: 'An object-oriented toolkit to analyze molecular dynamics trajectories generated by CHARMM, Gromacs, NAMD, LAMMPS, or Amber.'

# See
# http://docs.continuum.io/conda/build.html for
# more information about meta.yaml

and build.sh:

#!/bin/bash

$PYTHON setup.py install

# Add more build steps here, if they are necessary.

# See
# http://docs.continuum.io/conda/build.html
# for a list of environment variables that are set during the build process.

[kain88-de]

I just got the ball rolling--feel free to take over.

Would if I could. But the mac I have access to is at work and I mostly try to do my PhD at work.

Out of the box, it didn't, so I edited the code until it did

yeah we are not very friendly for that. Have a look at my scripts. They are in develop now. You should be able to run it with conda build maintainer/conda/MDAnalysis from the root folder.

[tylerjereddy]

The result (from: conda build maintainer/conda/MDAnalysis) is:

Ran 2241 tests in 89.116s

FAILED (KNOWNFAIL=2, SKIP=2, errors=462)
TESTS FAILED: mdanalysis-0.14.0-py27_0

Because:

ERROR: Failure: ImportError (No module named MDAnalysisTests)

So no, it doesn't 'just work' on Mac OS X.

[tylerjereddy]

Why are the tests being installed with pip in the build script in our development branch? This is causing problems on my machine--it was using version 0.11 of the test suite, and if I updated that manually with pip (which I shouldn't have to do), it still couldn't find testing submodules (so, new errors are like this: ERROR: Failure: ImportError (No module named test_velocities_forces)).

MDAnalysisTests is a requirement of the testing phase of conda build and should be listed as such in meta.yaml. That means we need conda packages for the test suite on different platforms as well. The test suite depends on the main package, and that creates the circular dependency I originally described.

There's something really awkward about this whole workflow really--large libraries with even more complex dependencies seem to be able to avoid this. The test suite should probably just be installed automatically during the conventional setup procedure. I think a philosophical decision to split was made at some stage, and now we are stuck with a conda build process with a testing phase that is pip-dependent (and doesn't even work on my machine).

pip install is also being used for the package itself, which is almost certainly not an ideal thing either--it should simply use the setup.py as in most build.sh scripts.

[kain88-de]

MDAnalysisTests is a requirement of the testing phase of conda build and should be listed as such in meta.yaml.

Nope. The package always builds them both at the same time. I wanted to just call the setup.py but changing the directories didn't work. And conda has it's own packages for pip this shouldn't be a problem.

Why are the tests being installed with pip in the build script in our development branch?

because the build.sh doesn't like when I change directories and it is working from the root of the checkout.

This is causing problems on my machine--it was using version 0.11 of the test suite,

That sounds very very strange. The conda builds are done in a new completely separate conda environment. It shouldn't see anything of other MDAnalysis installations (Assuming you don't have anything installed in $HOME/.local/lib/python2.7/site-packages Which could be exactly the reason why I couldn't build it myself in the first place!).

[kain88-de]

Nope it still fails.

and doesn't even work on my machine

It fails on all macs. Even without using pip and installing them seperately. No idea why!
And we are not stuck with it. osx isn't supported yet officially.

[tylerjereddy]

Note that on linux I also get a failure to build because tempdir is not found in the available channels--is there no way to resolve this without manual addition of the channel first (I suppose not)?

After manually adding the channel, conda build maintainer/conda/MDAnalysis on linux still fails for me:

    error: could not create '/lib/python2.7/site-packages/MDAnalysis': Permission denied

    ----------------------------------------
Command "/sansom/n22/bioc1009/anaconda_python/anaconda/envs/_build/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-_GqEoX-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-UKBOql-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-_GqEoX-build/
Command failed: /bin/bash -x -e /sansom/sc2/bioc1009/MDA_conda/mdanalysis/maintainer/conda/MDAnalysis/build.sh

If conda has its own pip handling and a clean contained environment is used then it should just build (conda build maintainer/conda/MDAnalysis) irrespective of my setup, right?

I agree that changing directories is highly problematic, but $PYTHON setup.py install did work for me on mac if the testing component was removed. Anyway, I'm confused.

[tylerjereddy]

Also,

It fails on all macs.

Did my alternative build also fail, and / or can you install the conda package I built for Mac OS using the skeleton test-stripped route?

[kain88-de]

Did my alternative build also fail, and / or can you install the conda package I built for Mac OS using the skeleton test-stripped route?

I haven't tested that yet.

s not found in the available channels--is there no way to resolve this without manual addition of the channel first (I suppose not)?

Nope it is not in the standard conda repository. I can try to get it accepted there or we just bundle it our self.

on linux still fails for me:

This is a weird error. It shouldn't try to write in that location since that is the global python installation path and not the conda installation. I haven't seen that yet. You could ask on the conda-mailing list for help if you are willing to spend the time on it.

[kain88-de]

Btw a reason why I bundle the tests as well is to have an automated check of the package during the build phase, keeps me from forgetting to do it.

[tylerjereddy]

If I run as root on linux (probably shouldn't have to) it almost works:

Ran 3342 tests in 289.508s

FAILED (KNOWNFAIL=3, SKIP=5, failures=2)
TESTS FAILED: mdanalysis-0.14.0-py27_0

It complains about XTC and TRR offsets, i.e.: FAIL: test_persistent_offsets_readonly (MDAnalysisTests.coordinates.test_xdr.TestXTCReader_offsets)

[tylerjereddy]

Note that conda forge has been getting some buzz lately. Apparently they will soon be added directly to the default conda channel--or rather the default sources will actually migrate to conda forge-- https://t.co/JGv6F5fdzB. Might be worth checking out (I don't fully get it yet, but basically it seems you just make a PR with your recipe and it gets built in VMs for different platforms, and if successful placed in their channel). Not sure what happens for code that doesn't work on Windows (since they do all 3 platforms), but I imagine that can be worked around.

Similarly, I seem to be able to avoid a lot of the above linux build issues by using docker, which (of course) removes the idiosyncrasies of the local python environment from the build process. I had similar issues to what I report above for MDAnalysis with another package, and they all dissipated in a clean docker build.

[kain88-de]

This is on conda forge now.

[orbeckst]

https://anaconda.org/conda-forge/mdanalysis has Linux and OSX packages that are rendered from https://github.com/conda-forge/mdanalysis-feedstock so we can close this as done.