Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow storing outputs in database #371

Draft
wants to merge 21 commits into
base: dev-jobqueue-old
Choose a base branch
from

Conversation

janpisl
Copy link

@janpisl janpisl commented Aug 13, 2018

Implement functionality for storing output data from run processes in a PostgreSQL or SQLite database

Overview

This extension allows output data from run processes to be stored in a PostgreSQL or SQLite database. This extension can be enabled in the configuration (section "server", option "store_type"; when set to "db", the extension is active). Credentials for connecting to the database are found in the (new) section "db".

Contribution Agreement

(as per https://github.com/geopython/pywps/blob/master/CONTRIBUTING.rst#contributions-and-licensing)

I'd like to contribute extension Database Output Storage to PyWPS. I confirm that my contributions to PyWPS will be compatible with the PyWPS license guidelines at the time of contribution.

Implement functionality for storing output data from run processes in a PostgreSQL or SQLite database
@cehbrecht
Copy link
Collaborator

@janpisl Please check the travis build. There is an import error:

 File "/home/travis/build/geopython/pywps/pywps/inout/storage/db/sqlite.py", line 11, in <module>
    from . import DbStorage
ImportError: cannot import name 'DbStorage'

... and pep8 is failing.

@coveralls
Copy link

Coverage Status

Coverage decreased (-3.1%) to 69.317% when pulling 43f9042 on janpisl:db into fbd5b3a on geopython:master.

@janpisl
Copy link
Author

janpisl commented Aug 13, 2018 via email

@cehbrecht
Copy link
Collaborator

@janpisl I have tested this PR with my Emu processes and I run into an exception (test with Python 3.6):

Traceback (most recent call last):
  File "/Users/pingu/Documents/GitHub/birdhouse/pywps/pywps/response/execute.py", line 72, in update_status_file                                                       
    self.doc = self._construct_doc()
  File "/Users/pingu/Documents/GitHub/birdhouse/pywps/pywps/response/execute.py", line 189, in _construct_doc                                                          
    doc = template.render(**self.json)
  File "/Users/pingu/Documents/GitHub/birdhouse/pywps/pywps/response/execute.py", line 145, in json                                                                    
    data["process"] = self.process.json
  File "/Users/pingu/Documents/GitHub/birdhouse/pywps/pywps/app/Process.py", line 91, in json                                                                          
    'outputs': [o.json for o in self.outputs],
  File "/Users/pingu/Documents/GitHub/birdhouse/pywps/pywps/app/Process.py", line 91, in <listcomp>                                                                    
    'outputs': [o.json for o in self.outputs],
  File "/Users/pingu/Documents/GitHub/birdhouse/pywps/pywps/inout/outputs.py", line 92, in json                                                                        
    data = self._json_reference(data)
  File "/Users/pingu/Documents/GitHub/birdhouse/pywps/pywps/inout/outputs.py", line 104, in _json_reference                                                            
    data["href"] = self.get_url()
  File "/Users/pingu/Documents/GitHub/birdhouse/pywps/pywps/inout/basic.py", line 711, in get_url                                                                      
    (outtype, storage, url) = self.storage.store(self)
AttributeError: 'NoneType' object has no attribute 'store'

I tried this with the Emu chomsky process. But this exception happens with every process having a complex output.

I have used as config: store_type = file

Currently you have set store_type = db as default. I would keep the file-storage as default.

@janpisl
Copy link
Author

janpisl commented Aug 14, 2018

@cehbrecht thank you. i will look into it.

@cehbrecht
Copy link
Collaborator

@janpisl Please have also a look on the codacy check (see link above next to travis). The complains on the tests can be skipped.

Jan Pišl added 8 commits September 24, 2018 12:56
Use Enum for DATA_TYPE (if not used and DATA_TYPE is only an object, it throws a TypeError: type "object" is not iterable
Copy vector data to a schema specified in cfg instead of public schema.
@cehbrecht cehbrecht marked this pull request as draft November 17, 2020 17:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants