Isotropic turbulence with Re_lambda=128 computed using NS_shenfun and a mesh of size 300**3 is physical space.
spectralDNS contains a classical high-performance pseudo-spectral Navier-Stokes DNS solver for triply periodic domains. The most notable feature of this solver is that it's written entirely in Python using NumPy, MPI for Python (mpi4py) and pyFFTW. MPI decomposition is performed using either the "slab" or the "pencil" approach and, stripping away unnecessary pre- and post-processing steps, the slab solver is no more than 100 lines long, including the MPI. The code has been found to scale very well in tests on the Shaheen Blue Gene/P supercomputer at KAUST Supercomputing Laboratory. Results of both weak and strong scaling tests are shown below. In addition to incompressible Navier-Stokes there are also solvers for MHD and Navier-Stokes or MHD with variable density through a Boussinesq approximation. The solver is described more thoroughly in this paper:
M. Mortensen and H. P. Langtangen "High performance Python for direct numerical simulations of turbulent flows", Computer Physics Communications 203, p 53-65 (2016) http://arxiv.org/pdf/1602.03638v1.pdf
The efficiency of the pure NumPy/mpi4py solver has been enhanced using Cython for certain routines. The strong scaling results on Shaheen shown below have used the optimized Python/Cython solver, which is found to be faster than a pure C++ implementation of the same solver.
A channel flow solver is implemented using the Shen basis (Jie Shen, SIAM Journal on Scientific Computing, 16, 74-87, 1995) for the scheme described by Kim, Moin and Moser (J. Fluid Mechanics, Vol 177, 133-166, 1987). The solver is described here: https://arxiv.org/pdf/1701.03787v1.pdf
spectralDNS contains a setup.py script and can be installed by cloning or forking the repository and then with regular python distutils
python setup.py install --prefix="path used for installation. Must be on the PYTHONPATH"
or in-place using
python setup.py build_ext --inplace
However, spectralDNS depends on three other modules in the spectralDNS organization: mpiFFT4py, shenfun and mpi4py-fft. And besides that, it requires h5py built with parallel HDF5, for visualizing the results, and cython is used to optimize a few routines. The regular "python setup.py ..." will not install these dependencies for you, and, as such, it is strongly adviced to install spectralDNS using Anaconda, as shown below.
To install using Anaconda, you may compile it yourselves using (from the main directory after cloning)
conda build -c conda-forge -c spectralDNS conf/conda
conda install spectralDNS --use-local
which will also build and install mpiFFT4py, shenfun and mpi4py-fft. You may also use precompiled binaries in the spectralDNS channel on Anaconda cloud. Use for exampel
conda create --name spectralDNS -c conda-forge -c spectralDNS spectralDNS
source activate spectralDNS
which installs both spectralDNS, mpiFFT4py and all required dependencies, most of which are pulled in from the conda-forge channel. There are binaries compiled for both OSX and linux, for either Python version 2.7 or 3.6. To specify the Python version as 3.6 instead of default (used above) you can for exampel do
conda create --name spectralDNS_py3 -c conda-forge -c spectralDNS python=3 spectralDNS
source activate spectralDNS_py3
See the demo folder for extensive usage.
There are different solvers. For example, there are two Navier Stokes solvers for the triply periodic domain. A regular one (solvers/NS.py), and one based on a velocity-vorticity formulation (solvers/VV.py). The solver of your choice is required as an argument when running:
cd demo
mpirun -np 4 python TG.py NS
or
mpirun -np 4 python TG.py VV
There are many different arguments to each solver. They are all described in config.py. Arguments may be specified on the commandline
mpirun -np 4 python TG.py --M 6 6 6 --precision single --dealias '3/2-rule' NS
before the required solver argument. Alternatively, use config.update as shown in demo/TG.py.
The most recent simulations of the pencil version of the NS solver are showing excellent scaling up to 65k cores at KAUST and Shaheen II!
Results also from the old Shaheen BlueGene/P:
Weak scaling of pure numpy/mpi4py solver on Shaheen BlueGene/P. The C++ solver uses slab decomposition and MPI communication is performed by the FFTW library.
Strong scaling of optimized Python/Cython solver on Shaheen BlueGene/P. The C++ solver uses slab decomposition and MPI communication is performed by the FFTW library.
Turbulent channel flow at Re_tau = 2000. Simulations are performed using 512 cores on Shaheen II with the KMM channel flow solver (solvers/KMM.py) using 512x1024x1024 points in real physical space.
Evolution of vorticity. Two-dimensional simulation of Kelvin Helmholtz shear instability using a 2D solver (solvers/NS2D.py)
spectralDNS is developed by
- Mikael Mortensen
- Diako Darian
spectralDNS is licensed under the GNU GPL, version 3 or (at your option) any later version. spectralDNS is Copyright (2014-2018) by the authors.
The latest version of this software can be obtained from
https://github.com/spectralDNS/spectralDNS
Please report bugs and other issues through the issue tracker at: