Skip to content

Commit

Permalink
Merge pull request #705 from nf-core/dev
Browse files Browse the repository at this point in the history
Release:  v1.10 - Copper Camel
  • Loading branch information
ewels authored Jul 30, 2020
2 parents ebf6309 + bd0fbe2 commit 3b62ee1
Show file tree
Hide file tree
Showing 92 changed files with 7,052 additions and 3,354 deletions.
72 changes: 46 additions & 26 deletions .github/CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,70 +2,90 @@

Hi there! Many thanks for taking an interest in improving nf-core/tools.

We try to manage the required tasks for nf-core/tools using GitHub issues, you probably came to this page when creating one. Please use the pre-filled templates to save time.

However, don't be put off by this template - other more general issues and suggestions are welcome! Contributions to the code are even more welcome ;)

> If you need help using or developing nf-core/tools then the best place to ask is the nf-core `tools` channel on [Slack](https://nf-co.re/join/slack/).
If you need help then the best place to ask is the [`#tools` channel](https://nfcore.slack.com/channels/tools) on the nf-core Slack.
You can get an invite on the [nf-core website](https://nf-co.re/join/slack/).

## Contribution workflow

If you'd like to write some code for nf-core/tools, the standard workflow
is as follows:

1. Check that there isn't already an issue about your idea in the
[nf-core/tools issues](https://github.com/nf-core/tools/issues) to avoid
duplicating work.
1. Check that there isn't [already an issue](https://github.com/nf-core/tools/issues) about your idea to avoid duplicating work.
* If there isn't one already, please create one so that others know you're working on this
2. Fork the [nf-core/tools repository](https://github.com/nf-core/tools) to your GitHub account
3. Make the necessary changes / additions within your forked repository
4. Submit a Pull Request against the `dev` branch and wait for the code to be reviewed and merged.

If you're not used to this workflow with git, you can start with some [basic docs from GitHub](https://help.github.com/articles/fork-a-repo/) or even their [excellent interactive tutorial](https://try.github.io/).
If you're not used to this workflow with git, you can start with some [basic docs from GitHub](https://help.github.com/articles/fork-a-repo/).

## Style guide
Google provides an excellent [style guide](https://github.com/google/styleguide/blob/gh-pages/pyguide.md), which
is a best practise extension of [PEP](https://www.python.org/dev/peps/), the Python Enhancement Proposals. Have a look at the
[docstring](https://github.com/google/styleguide/blob/gh-pages/pyguide.md#38-comments-and-docstrings) section, which is in particular
important, as nf-core tool's code documentation is generated out of these automatically.
## Installing dev requirements

In order to test the documentation, you have to install Sphinx on the machine, where the documentation should be generated.
If you want to work with developing the nf-core/tools code, you'll need a couple of extra Python packages.
These are listed in `requirements-dev.txt` and can be installed as follows:

Please follow Sphinx's [installation instruction](http://www.sphinx-doc.org/en/master/usage/installation.html).
```bash
pip install --upgrade -r requirements-dev.txt
```

Once done, you can run `make clean` and then `make html` in the root directory of `nf-core tools`, where the `Makefile` is located.
Then install your local fork of nf-core/tools:

The HTML will then be generated in `docs/api/_build/html`.
```bash
pip install -e .
```

## Code formatting with Black

All Python code in nf-core/tools must be passed through the [Black Python code formatter](https://black.readthedocs.io/en/stable/).
This ensures a harmonised code formatting style throughout the package, from all contributors.

You can run Black on the command line (it's included in `requirements-dev.txt`) - eg. to run recursively on the whole repository:

```bash
black .
```

Alternatively, Black has [integrations for most common editors](https://black.readthedocs.io/en/stable/editor_integration.html)
to automatically format code when you hit save.
You can also set it up to run when you [make a commit](https://black.readthedocs.io/en/stable/version_control_integration.html).

There is an automated CI check that runs when you open a pull-request to nf-core/tools that will fail if
any code does not adhere to Black formatting.

## API Documentation

We aim to write function docstrings according to the [Google Python style-guide](https://github.com/google/styleguide/blob/gh-pages/pyguide.md#38-comments-and-docstrings). These are used to automatically generate package documentation on the nf-core website using Sphinx.
You can find this documentation here: [https://nf-co.re/tools-docs/](https://nf-co.re/tools-docs/)

If you would like to test the documentation, you can install Sphinx locally by following Sphinx's [installation instruction](https://www.sphinx-doc.org/en/master/usage/installation.html).
Once done, you can run `make clean` and then `make html` in the root directory of `nf-core tools`.
The HTML will then be generated in `docs/api/_build/html`.

## Tests

When you create a pull request with changes, [GitHub Actions](https://github.com/features/actions) will run automatic tests.
Typically, pull-requests are only fully reviewed when these tests are passing, though of course we can help out before then.

There are two types of tests that run:

### Unit Tests

The nf-core tools package has a set of unit tests bundled, which can be found in the `tests/` directory.
New features should also come with new tests, to keep the test-coverage high (we use [codecov.io](https://codecov.io/gh/nf-core/tools/) to check this automatically).

You can try running the tests locally before pushing code using the following command:

```bash
python -m pytest .
pytest --color=yes tests/
```

### Lint Tests
nf-core has a [set of guidelines](http://nf-co.re/guidelines) which all pipelines must adhere to.
To enforce these and ensure that all pipelines stay in sync, we have developed a helper tool which runs checks on the pipeline code. This is in the [nf-core/tools repository](https://github.com/nf-core/tools) and once installed can be run locally with the `nf-core lint <pipeline-directory>` command.

The nf-core/tools repo itself contains the master template for creating new nf-core pipelines.
Once you have created a new pipeline from this template GitHub Actions is automatically set up to run lint tests on it.
nf-core/tools contains both the main nf-core template for pipelines and the code used to test that pipelines adhere to the nf-core guidelines.
As these two commonly need to be edited together, we test the creation of a pipeline and then linting using a CI check.
This ensures that any changes we make to either the linting or the template stay in sync.
You can replicate this process locally with the following commands:

```bash
nf-core create -n testpipeline -d "This pipeline is for testing"
nf-core lint nf-core-testpipeline
```

## Getting help
For further information/help, please consult the [nf-core/tools documentation](https://github.com/nf-core/tools#documentation) and don't hesitate to get in touch on the nf-core `tools` channel on [Slack](https://nf-co.re/join/slack/).
15 changes: 10 additions & 5 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,17 @@
<!--
Many thanks for contributing to nf-core/tools!
Please fill in the appropriate checklist below (delete whatever is not relevant). These are the most common things requested on pull requests (PRs).
Please fill in the appropriate checklist below (delete whatever is not relevant).
These are the most common things requested on pull requests (PRs).
Remember that PRs should be made against the dev branch, unless you're preparing a release.
Learn more about contributing: https://github.com/nf-core/tools/tree/master/.github/CONTRIBUTING.md
-->

## PR checklist

- [ ] This comment contains a description of changes (with reason)
- [ ] `CHANGELOG.md` is updated
- [ ] If you've fixed a bug or added code that should be tested, add tests!
- [ ] Documentation in `docs` is updated
- [ ] `CHANGELOG.md` is updated
- [ ] `README.md` is updated

**Learn more about contributing:** https://github.com/nf-core/tools/tree/master/.github/CONTRIBUTING.md
12 changes: 6 additions & 6 deletions .github/RELEASE_CHECKLIST.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
## Before release

1. Check issue milestones to see outstanding issues to resolve if possible or transfer to the milestones for the next release e.g. [`v1.9`](https://github.com/nf-core/tools/issues?q=is%3Aopen+is%3Aissue+milestone%3A1.9)
2. Create a PR to `dev` to bump the version in `CHANGELOG.md` and `setup.py`.
3. Make sure all CI tests are passing!
4. Create a PR from `dev` to `master`
5. Make sure all CI tests are passing again (additional tests are run on PRs to `master`)
6. Request review (2 approvals required)
7. Most importantly, pick an undeniably outstanding [name](http://www.codenamegenerator.com/) for the release where *Prefix* = *Metal* and *Dictionary* = *Animal*.
2. Most importantly, pick an undeniably outstanding [name](http://www.codenamegenerator.com/) for the release where *Prefix* = *Metal* and *Dictionary* = *Animal*.
3. Create a PR to `dev` to bump the version in `CHANGELOG.md` and `setup.py`.
4. Make sure all CI tests are passing!
5. Create a PR from `dev` to `master`
6. Make sure all CI tests are passing again (additional tests are run on PRs to `master`)
7. Request review (2 approvals required)
8. Merge the PR into `master`
9. Once CI tests on commit have passed, create a new release copying the `CHANGELOG` for that release into the description section.

Expand Down
8 changes: 5 additions & 3 deletions .github/markdownlint.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
# Markdownlint configuration file
default: true,
default: true
line-length: false
no-duplicate-header:
siblings_only: true
no-bare-urls: false # tools only - the {{ jinja variables }} break URLs and cause this to error
commands-show-output: false # tools only - suppresses error messages for usage of $ in main README
# tools only - the {{ jinja variables }} break URLs and cause this to error
no-bare-urls: false
# tools only - suppresses error messages for usage of $ in main README
commands-show-output: false
35 changes: 35 additions & 0 deletions .github/workflows/branch.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
name: nf-core branch protection
# This workflow is triggered on PRs to master branch on the repository
# It fails when someone tries to make a PR against the nf-core `master` branch instead of `dev`
on:
pull_request:
branches: [master]

jobs:
test:
runs-on: ubuntu-latest
steps:

# PRs to the nf-core repo master branch are only ok if coming from the nf-core repo `dev` or any `patch` branches
- name: Check PRs
if: github.repository == 'nf-core/tools'
run: |
{ [[ ${{github.event.pull_request.head.repo.full_name}} == nf-core/tools ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]]
# If the above check failed, post a comment on the PR explaining the failure
- name: Post PR comment
if: failure()
uses: mshick/add-pr-comment@v1
with:
message: |
Hi @${{ github.event.pull_request.user.login }},
It looks like this pull-request is has been made against the ${{github.event.pull_request.head.repo.full_name}} `master` branch.
The `master` branch on nf-core repositories should always contain code from the latest release.
Beacuse of this, PRs to `master` are only allowed if they come from the ${{github.event.pull_request.head.repo.full_name}} `dev` branch.
You do not need to close this PR, you can change the target branch to `dev` by clicking the _"Edit"_ button at the top of this page.
Thanks again for your contribution!
repo-token: ${{ secrets.GITHUB_TOKEN }}
allow-repeats: false
8 changes: 7 additions & 1 deletion .github/workflows/create-lint-wf.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,13 @@ jobs:
wget -qO- get.nextflow.io | bash
sudo ln -s /tmp/nextflow/nextflow /usr/local/bin/nextflow
- name: Run nf-core tools
- name: Run nf-core/tools
run: |
nf-core create -n testpipeline -d "This pipeline is for testing" -a "Testing McTestface"
nf-core lint nf-core-testpipeline
nf-core list
nf-core licences nf-core-testpipeline
nf-core sync nf-core-testpipeline/
nf-core schema build nf-core-testpipeline/ --no-prompts
nf-core bump-version nf-core-testpipeline/ 1.1
nf-core modules install nf-core-testpipeline/ fastqc
23 changes: 11 additions & 12 deletions .github/workflows/code-tests.yml → .github/workflows/pytest.yml
Original file line number Diff line number Diff line change
@@ -1,9 +1,16 @@
name: Python tests
# This workflow is triggered on pushes and PRs to the repository.
on: [push, pull_request]
# Only run if we changed a Python file
on:
push:
paths:
- '**.py'
pull_request:
paths:
- '**.py'

jobs:
PythonLint:
pytest:

runs-on: ubuntu-latest
strategy:
Expand All @@ -21,8 +28,8 @@ jobs:

- name: Install python dependencies
run: |
python -m pip install --upgrade pip pytest pytest-datafiles pytest-cov mock jsonschema
pip install .
python -m pip install --upgrade pip -r requirements-dev.txt
pip install -e .
- name: Install Nextflow
run: |
Expand All @@ -31,14 +38,6 @@ jobs:
wget -qO- get.nextflow.io | bash
sudo ln -s /tmp/nextflow/nextflow /usr/local/bin/nextflow
- name: Lint with flake8
run: |
pip install flake8
# stop the build if there are Python syntax errors or undefined names
flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
# exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
- name: Test with pytest
run: python3 -m pytest --color=yes --cov-report=xml --cov-config=.github/.coveragerc --cov=nf_core

Expand Down
17 changes: 17 additions & 0 deletions .github/workflows/python-lint.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
name: Lint Python
on:
push:
paths:
- '**.py'
pull_request:
paths:
- '**.py'

jobs:
PythonLint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2

- name: Check code lints with Black
uses: jpetrucciani/black-check@master
9 changes: 7 additions & 2 deletions .github/workflows/sync.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,10 @@ jobs:
- uses: actions/checkout@v2
name: Check out source-code repository

- name: Set up Python 3.7
- name: Set up Python 3.8
uses: actions/setup-python@v1
with:
python-version: 3.7
python-version: 3.8

- name: Install python dependencies
run: |
Expand All @@ -29,6 +29,11 @@ jobs:
wget -qO- get.nextflow.io | bash
sudo ln -s /tmp/nextflow/nextflow /usr/local/bin/nextflow
- name: Configure git
run: |
git config user.email "[email protected]"
git config user.name "nf-core-bot"
- name: Run synchronisation
if: github.repository == 'nf-core/tools'
env:
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/tools-api-docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@ on:
branches: [master, dev]

jobs:
build-n-publish:
name: Build and publish nf-core to PyPI
api-docs:
name: Build & push Sphinx API docs
runs-on: ubuntu-18.04

steps:
Expand Down
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
testing*
htmlcov/
.tox/
.coverage
Expand Down Expand Up @@ -107,3 +108,4 @@ ENV/

# backup files
*~
*\?
Loading

0 comments on commit 3b62ee1

Please sign in to comment.