Skip to content

Commit

Permalink
deploy: 2c05b27
Browse files Browse the repository at this point in the history
  • Loading branch information
@elcorto
elcorto committed Jul 19, 2024
1 parent deedb9b commit 2f63bd8
Show file tree
Hide file tree
Showing 5 changed files with 73 additions and 123 deletions.
98 changes: 37 additions & 61 deletions _sources/written/install.rst.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,11 @@ Installation
Quick start
-----------

Debian and derivatives: Fortran compiler, Python headers, lapack
Fortran compiler, Python headers, lapack, meson on Debian/Ubuntu:

.. code-block:: sh
$ sudo apt install python3-dev gfortran liblapack-dev
$ sudo apt install python3-dev gfortran liblapack-dev meson
Then

Expand Down Expand Up @@ -120,15 +120,6 @@ might use ``ase``) won't fail at import time, but later at runtime when e.g.
``crys.Structure.get_ase_atoms()`` is called. What packages are optional might
change depending on usage.

You also need the following to compile extensions. On a Debian-ish system

.. code-block:: sh
# apt
python3-dev # for compiling extensions
gfortran # or ifort, see src/Makefile
liblapack-dev
Also note that you can get most Python packages via your system's package
manager. Debian and derivatives:

Expand Down Expand Up @@ -173,22 +164,13 @@ Running tests

See test/README. Actually, all of these are good examples, too!

Python versions
---------------

Only Python3 is supported, tested: Python 3.6-3.11

The package was developed mostly with Python 2.5-2.7 and ported using 2to3 +
manual changes. Therefore, you might find some crufty Python 2 style code
fragments in lesser used parts of the code base.


.. _extensions:

Fortran extensions and OpenMP notes
-----------------------------------

Use ``src/Makefile``:
Use ``src/_ext_src/Makefile``:

.. code-block:: sh
Expand All @@ -200,25 +182,44 @@ Use ``src/Makefile``:
make ifort-omp # ifort + OpenMP
make ifort-mkl # ifort, Intel MKL lapack, set MKL_LIB
Generates ``*.so`` and ``*.pyf`` (f2py interface) files.
Generates ``*.so`` files

.. code-block:: sh
src/_ext_src/_dcd.so # will be copied to src/pwtools/_dcd.so
src/_ext_src/_flib.so # will be copied to src/pwtools/_flib.so
You need the following to compile extensions. On Debian/Ubuntu

You need:
.. code-block:: sh
# apt
python3-numpy # for f2py
python3-dev # Python headers
gfortran # or ifort, see src/_ext_src/Makefile
liblapack-dev # see below
meson # f2py build backend as of Python 3.12, will also install ninja
* numpy for f2py
* a Fortran compiler
* Python headers (Debian/Ubuntu: python-dev)
* Lapack (Debian: liblapack-dev)
``liblapack-dev`` is only one option, maybe the slowest. On Debian/Ubuntu,
there are other options, i.e. anything that provides a ``liblapack.so`` works.

The module is compiled with f2py (currently part of numpy, tested with numpy
1.1.0 .. 1.21.x).
.. code-block:: sh
$ apt-file search -x 'liblapack.so$'
libatlas-base-dev: /usr/lib/x86_64-linux-gnu/atlas/liblapack.so
liblapack-dev: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so
libmkl-dev: /usr/lib/x86_64-linux-gnu/mkl/liblapack.so
libopenblas-openmp-dev: /usr/lib/x86_64-linux-gnu/openblas-openmp/liblapack.so
libopenblas-pthread-dev: /usr/lib/x86_64-linux-gnu/openblas-pthread/liblapack.so
libopenblas-serial-dev: /usr/lib/x86_64-linux-gnu/openblas-serial/liblapack.so
The default ``make`` target is "gfortran" which tries to build a serial version
using system BLAS and LAPACK (e.g. from ``liblapack-dev``). If you want another
using system LAPACK (e.g. from ``liblapack-dev``). If you want another
target (e.g. ``ifort-mkl``), then

.. code-block:: sh
$ cd src
$ cd src/_ext_src
$ make clean
$ make ifort-mkl
Expand All @@ -229,10 +230,10 @@ or when using ``pip`` (or anything calling ``setup.py``) set
$ PWTOOLS_EXT_MAKE_TARGET=ifort-mkl pip install ...
This will use the Intel ``ifort`` compiler instead fo the default ``gfortran`` and
This will use the Intel ``ifort`` compiler instead of the default ``gfortran`` and
link against the MKL.

In the MKL case, the Makefile uses the env var ``$MKL_LIB`` which sould point
In the MKL case, the Makefile uses the env var ``$MKL_LIB`` which should point
to the location where things like ``libmkl_core.so`` live. You may need to set
this. On a HPC cluster, that could look like this.

Expand All @@ -242,28 +243,8 @@ this. On a HPC cluster, that could look like this.
$ module load intel/20.4
$ MKL_LIB=$MKL_ROOT/lib/intel64 PWTOOLS_EXT_MAKE_TARGET=ifort-mkl pip install ...
See ``src/Makefile`` for more details.

Compiler / f2py
^^^^^^^^^^^^^^^
Instead of letting numpy.distutils pick a compiler + special flags, which is
not trivial and therefore almost never works, it is much easier to simply
define the compiler to use + architecture-specific flags. See F90 and ARCH in
the Makefile.
See ``src/_ext_src/Makefile`` for more details.

Also, numpy.distutils has default -03 for fcompiler. ``--f90flags="-02"`` does NOT
override this. We get ``-O3 -O2`` and a compiler warning. We have to use f2py's
``--opt=`` flag.

On some systems (Debian), you may have:

.. code-block:: sh
/usr/bin/f2py -> f2py2.6
/usr/bin/f2py2.5
/usr/bin/f2py2.6
and such. But usually ``F2PY=f2py`` is fine.

OpenMP
^^^^^^
Expand All @@ -289,13 +270,8 @@ Setting the number of threads:
If this env var is NOT set, then OpenMP uses all available cores (e.g. 4 on a
quad-core box).

IMPORTANT:
Note that we may have found a f2py bug (see test/test_f2py_flib_openmp.py)
re. OMP_NUM_THREADS. We have a workaround for that in pydos.fvacf().

There is also an optional arg 'nthreads' to _flib.vacf(). If this is
supplied, then it will override OMP_NUM_THREADS. Currently, this is the
safest way to set the number of threads.
There is also an optional arg 'nthreads' to ``_flib.vacf()``. If this is
supplied, then it will override ``OMP_NUM_THREADS``.

Tests
^^^^^
Expand Down
1 change: 0 additions & 1 deletion index.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,6 @@ <h1>Welcome to the pwtools documentation<a class="headerlink" href="#welcome-to-
<li class="toctree-l2"><a class="reference internal" href="written/install.html#detailed-instructions">Detailed instructions</a></li>
<li class="toctree-l2"><a class="reference internal" href="written/install.html#dependencies">Dependencies</a></li>
<li class="toctree-l2"><a class="reference internal" href="written/install.html#running-tests">Running tests</a></li>
<li class="toctree-l2"><a class="reference internal" href="written/install.html#python-versions">Python versions</a></li>
<li class="toctree-l2"><a class="reference internal" href="written/install.html#fortran-extensions-and-openmp-notes">Fortran extensions and OpenMP notes</a></li>
</ul>
</li>
Expand Down
2 changes: 1 addition & 1 deletion searchindex.js
View file

Large diffs are not rendered by default.

2 changes: 0 additions & 2 deletions written/index.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -101,9 +101,7 @@ <h3 id="searchlabel">Quick search</h3>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="install.html#running-tests">Running tests</a></li>
<li class="toctree-l2"><a class="reference internal" href="install.html#python-versions">Python versions</a></li>
<li class="toctree-l2"><a class="reference internal" href="install.html#fortran-extensions-and-openmp-notes">Fortran extensions and OpenMP notes</a><ul>
<li class="toctree-l3"><a class="reference internal" href="install.html#compiler-f2py">Compiler / f2py</a></li>
<li class="toctree-l3"><a class="reference internal" href="install.html#openmp">OpenMP</a></li>
<li class="toctree-l3"><a class="reference internal" href="install.html#tests">Tests</a></li>
</ul>
Expand Down
93 changes: 35 additions & 58 deletions written/install.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,6 @@ <h3>Navigation</h3>
<li class="toctree-l2"><a class="reference internal" href="#detailed-instructions">Detailed instructions</a></li>
<li class="toctree-l2"><a class="reference internal" href="#dependencies">Dependencies</a></li>
<li class="toctree-l2"><a class="reference internal" href="#running-tests">Running tests</a></li>
<li class="toctree-l2"><a class="reference internal" href="#python-versions">Python versions</a></li>
<li class="toctree-l2"><a class="reference internal" href="#fortran-extensions-and-openmp-notes">Fortran extensions and OpenMP notes</a></li>
</ul>
</li>
Expand Down Expand Up @@ -102,8 +101,8 @@ <h3 id="searchlabel">Quick search</h3>
<h1>Installation<a class="headerlink" href="#installation" title="Link to this heading"></a></h1>
<section id="quick-start">
<h2>Quick start<a class="headerlink" href="#quick-start" title="Link to this heading"></a></h2>
<p>Debian and derivatives: Fortran compiler, Python headers, lapack</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>$<span class="w"> </span>sudo<span class="w"> </span>apt<span class="w"> </span>install<span class="w"> </span>python3-dev<span class="w"> </span>gfortran<span class="w"> </span>liblapack-dev
<p>Fortran compiler, Python headers, lapack, meson on Debian/Ubuntu:</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>$<span class="w"> </span>sudo<span class="w"> </span>apt<span class="w"> </span>install<span class="w"> </span>python3-dev<span class="w"> </span>gfortran<span class="w"> </span>liblapack-dev<span class="w"> </span>meson
</pre></div>
</div>
<p>Then</p>
Expand Down Expand Up @@ -191,13 +190,6 @@ <h2>Dependencies<a class="headerlink" href="#dependencies" title="Link to this h
might use <code class="docutils literal notranslate"><span class="pre">ase</span></code>) won’t fail at import time, but later at runtime when e.g.
<code class="docutils literal notranslate"><span class="pre">crys.Structure.get_ase_atoms()</span></code> is called. What packages are optional might
change depending on usage.</p>
<p>You also need the following to compile extensions. On a Debian-ish system</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span><span class="c1"># apt</span>
python3-dev<span class="w"> </span><span class="c1"># for compiling extensions</span>
gfortran<span class="w"> </span><span class="c1"># or ifort, see src/Makefile</span>
liblapack-dev
</pre></div>
</div>
<p>Also note that you can get most Python packages via your system’s package
manager. Debian and derivatives:</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>python3-ase
Expand Down Expand Up @@ -240,16 +232,9 @@ <h3>Optional dependencies<a class="headerlink" href="#optional-dependencies" tit
<h2>Running tests<a class="headerlink" href="#running-tests" title="Link to this heading"></a></h2>
<p>See test/README. Actually, all of these are good examples, too!</p>
</section>
<section id="python-versions">
<h2>Python versions<a class="headerlink" href="#python-versions" title="Link to this heading"></a></h2>
<p>Only Python3 is supported, tested: Python 3.6-3.11</p>
<p>The package was developed mostly with Python 2.5-2.7 and ported using 2to3 +
manual changes. Therefore, you might find some crufty Python 2 style code
fragments in lesser used parts of the code base.</p>
</section>
<section id="fortran-extensions-and-openmp-notes">
<span id="extensions"></span><h2>Fortran extensions and OpenMP notes<a class="headerlink" href="#fortran-extensions-and-openmp-notes" title="Link to this heading"></a></h2>
<p>Use <code class="docutils literal notranslate"><span class="pre">src/Makefile</span></code>:</p>
<p>Use <code class="docutils literal notranslate"><span class="pre">src/_ext_src/Makefile</span></code>:</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>$<span class="w"> </span>make<span class="w"> </span><span class="nb">help</span>
make<span class="w"> </span>gfortran<span class="w"> </span><span class="c1"># gfortran, default</span>
make<span class="w"> </span>gfortran-omp<span class="w"> </span><span class="c1"># gfortran + OpenMP</span>
Expand All @@ -259,21 +244,36 @@ <h2>Python versions<a class="headerlink" href="#python-versions" title="Link to
make<span class="w"> </span>ifort-mkl<span class="w"> </span><span class="c1"># ifort, Intel MKL lapack, set MKL_LIB</span>
</pre></div>
</div>
<p>Generates <code class="docutils literal notranslate"><span class="pre">*.so</span></code> and <code class="docutils literal notranslate"><span class="pre">*.pyf</span></code> (f2py interface) files.</p>
<p>You need:</p>
<ul class="simple">
<li><p>numpy for f2py</p></li>
<li><p>a Fortran compiler</p></li>
<li><p>Python headers (Debian/Ubuntu: python-dev)</p></li>
<li><p>Lapack (Debian: liblapack-dev)</p></li>
</ul>
<p>The module is compiled with f2py (currently part of numpy, tested with numpy
1.1.0 .. 1.21.x).</p>
<p>Generates <code class="docutils literal notranslate"><span class="pre">*.so</span></code> files</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>src/_ext_src/_dcd.so<span class="w"> </span><span class="c1"># will be copied to src/pwtools/_dcd.so</span>
src/_ext_src/_flib.so<span class="w"> </span><span class="c1"># will be copied to src/pwtools/_flib.so</span>
</pre></div>
</div>
<p>You need the following to compile extensions. On Debian/Ubuntu</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span><span class="c1"># apt</span>
python3-numpy<span class="w"> </span><span class="c1"># for f2py</span>
python3-dev<span class="w"> </span><span class="c1"># Python headers</span>
gfortran<span class="w"> </span><span class="c1"># or ifort, see src/_ext_src/Makefile</span>
liblapack-dev<span class="w"> </span><span class="c1"># see below</span>
meson<span class="w"> </span><span class="c1"># f2py build backend as of Python 3.12, will also install ninja</span>
</pre></div>
</div>
<p><code class="docutils literal notranslate"><span class="pre">liblapack-dev</span></code> is only one option, maybe the slowest. On Debian/Ubuntu,
there are other options, i.e. anything that provides a <code class="docutils literal notranslate"><span class="pre">liblapack.so</span></code> works.</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>$<span class="w"> </span>apt-file<span class="w"> </span>search<span class="w"> </span>-x<span class="w"> </span><span class="s1">&#39;liblapack.so$&#39;</span>
libatlas-base-dev:<span class="w"> </span>/usr/lib/x86_64-linux-gnu/atlas/liblapack.so
liblapack-dev:<span class="w"> </span>/usr/lib/x86_64-linux-gnu/lapack/liblapack.so
libmkl-dev:<span class="w"> </span>/usr/lib/x86_64-linux-gnu/mkl/liblapack.so
libopenblas-openmp-dev:<span class="w"> </span>/usr/lib/x86_64-linux-gnu/openblas-openmp/liblapack.so
libopenblas-pthread-dev:<span class="w"> </span>/usr/lib/x86_64-linux-gnu/openblas-pthread/liblapack.so
libopenblas-serial-dev:<span class="w"> </span>/usr/lib/x86_64-linux-gnu/openblas-serial/liblapack.so
</pre></div>
</div>
<p>The default <code class="docutils literal notranslate"><span class="pre">make</span></code> target is “gfortran” which tries to build a serial version
using system BLAS and LAPACK (e.g. from <code class="docutils literal notranslate"><span class="pre">liblapack-dev</span></code>). If you want another
using system LAPACK (e.g. from <code class="docutils literal notranslate"><span class="pre">liblapack-dev</span></code>). If you want another
target (e.g. <code class="docutils literal notranslate"><span class="pre">ifort-mkl</span></code>), then</p>
<blockquote>
<div><div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>src
<div><div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>src/_ext_src
$<span class="w"> </span>make<span class="w"> </span>clean
$<span class="w"> </span>make<span class="w"> </span>ifort-mkl
</pre></div>
Expand All @@ -284,34 +284,17 @@ <h2>Python versions<a class="headerlink" href="#python-versions" title="Link to
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>$<span class="w"> </span><span class="nv">PWTOOLS_EXT_MAKE_TARGET</span><span class="o">=</span>ifort-mkl<span class="w"> </span>pip<span class="w"> </span>install<span class="w"> </span>...
</pre></div>
</div>
<p>This will use the Intel <code class="docutils literal notranslate"><span class="pre">ifort</span></code> compiler instead fo the default <code class="docutils literal notranslate"><span class="pre">gfortran</span></code> and
<p>This will use the Intel <code class="docutils literal notranslate"><span class="pre">ifort</span></code> compiler instead of the default <code class="docutils literal notranslate"><span class="pre">gfortran</span></code> and
link against the MKL.</p>
<p>In the MKL case, the Makefile uses the env var <code class="docutils literal notranslate"><span class="pre">$MKL_LIB</span></code> which sould point
<p>In the MKL case, the Makefile uses the env var <code class="docutils literal notranslate"><span class="pre">$MKL_LIB</span></code> which should point
to the location where things like <code class="docutils literal notranslate"><span class="pre">libmkl_core.so</span></code> live. You may need to set
this. On a HPC cluster, that could look like this.</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span><span class="c1"># module only sets MKL_ROOT=/path/to/intel/20.4/mkl</span>
$<span class="w"> </span>module<span class="w"> </span>load<span class="w"> </span>intel/20.4
$<span class="w"> </span><span class="nv">MKL_LIB</span><span class="o">=</span><span class="nv">$MKL_ROOT</span>/lib/intel64<span class="w"> </span><span class="nv">PWTOOLS_EXT_MAKE_TARGET</span><span class="o">=</span>ifort-mkl<span class="w"> </span>pip<span class="w"> </span>install<span class="w"> </span>...
</pre></div>
</div>
<p>See <code class="docutils literal notranslate"><span class="pre">src/Makefile</span></code> for more details.</p>
<section id="compiler-f2py">
<h3>Compiler / f2py<a class="headerlink" href="#compiler-f2py" title="Link to this heading"></a></h3>
<p>Instead of letting numpy.distutils pick a compiler + special flags, which is
not trivial and therefore almost never works, it is much easier to simply
define the compiler to use + architecture-specific flags. See F90 and ARCH in
the Makefile.</p>
<p>Also, numpy.distutils has default -03 for fcompiler. <code class="docutils literal notranslate"><span class="pre">--f90flags=&quot;-02&quot;</span></code> does NOT
override this. We get <code class="docutils literal notranslate"><span class="pre">-O3</span> <span class="pre">-O2</span></code> and a compiler warning. We have to use f2py’s
<code class="docutils literal notranslate"><span class="pre">--opt=</span></code> flag.</p>
<p>On some systems (Debian), you may have:</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>/usr/bin/f2py<span class="w"> </span>-&gt;<span class="w"> </span>f2py2.6
/usr/bin/f2py2.5
/usr/bin/f2py2.6
</pre></div>
</div>
<p>and such. But usually <code class="docutils literal notranslate"><span class="pre">F2PY=f2py</span></code> is fine.</p>
</section>
<p>See <code class="docutils literal notranslate"><span class="pre">src/_ext_src/Makefile</span></code> for more details.</p>
<section id="openmp">
<h3>OpenMP<a class="headerlink" href="#openmp" title="Link to this heading"></a></h3>
<p>We managed to speed up the calculations by sprinkling some OpenMP
Expand All @@ -330,14 +313,8 @@ <h3>OpenMP<a class="headerlink" href="#openmp" title="Link to this heading">¶</
</div>
<p>If this env var is NOT set, then OpenMP uses all available cores (e.g. 4 on a
quad-core box).</p>
<dl class="simple">
<dt>IMPORTANT:</dt><dd><p>Note that we may have found a f2py bug (see test/test_f2py_flib_openmp.py)
re. OMP_NUM_THREADS. We have a workaround for that in pydos.fvacf().</p>
</dd>
</dl>
<p>There is also an optional arg ‘nthreads’ to _flib.vacf(). If this is
supplied, then it will override OMP_NUM_THREADS. Currently, this is the
safest way to set the number of threads.</p>
<p>There is also an optional arg ‘nthreads’ to <code class="docutils literal notranslate"><span class="pre">_flib.vacf()</span></code>. If this is
supplied, then it will override <code class="docutils literal notranslate"><span class="pre">OMP_NUM_THREADS</span></code>.</p>
</section>
<section id="tests">
<h3>Tests<a class="headerlink" href="#tests" title="Link to this heading"></a></h3>
Expand Down

0 comments on commit 2f63bd8

Please sign in to comment.