Harvest Job - Solr Issue

[ykhadilkar]

I was able to successfully create the datajson harvester job. However when tried to clear the job, data from solr was not cleared.

Steps to reproduce

1. Start harvester job.
2. While job is running, hit clear button
3. Datasets from postgres are deleted but solr still has those datasets.

I believe datasets from solr should be deleted as well. Please let me know if I am missing something.

Versions
Ckan - 2.2
Postgres - 9.2
Solr - 4.5

[JoshData]

Do you know if this works with other harvesters?

[ykhadilkar]

Yea, it works with other harvesters. We have been using CSW harvester and solr cleanup works with it.

[JoshData]

Hmm. I don't have any particular idea what might be happening (or not happening).

I've spent a lot of time chasing down weird behaviors like this, so I'm probably not going to be able to hunt this one down unless it happens to get in the way of work for HHS.

[syxxzonz]

Yatin,

May I ask if you were the one responsible for the installation? I have an Ubuntu machine with CKAN and an older version of SOLR (the one that comes with sudo apt-get install - 1.4 I think) but when I tried installing ckanext-datajson and access the website, there is an error in the server logs:

[Mon Sep 29 07:46:19 2014] [error] [client 127.0.0.1] mod_wsgi (pid=2683): Target WSGI script '/etc/ckan/default/apache.wsgi' cannot be loaded as Python module.
[Mon Sep 29 07:46:19 2014] [error] [client 127.0.0.1] mod_wsgi (pid=2683): Exception occurred processing WSGI script '/etc/ckan/default/apache.wsgi'.
[Mon Sep 29 07:46:19 2014] [error] [client 127.0.0.1] Traceback (most recent call last):
[Mon Sep 29 07:46:19 2014] [error] [client 127.0.0.1] File "/etc/ckan/default/apache.wsgi", line 5, in
[Mon Sep 29 07:46:19 2014] [error] [client 127.0.0.1] from paste.deploy import loadapp
[Mon Sep 29 07:46:19 2014] [error] [client 127.0.0.1] File "/usr/lib/ckan/default/lib/python2.7/site-packages/paste/deploy/init.py", line 3, in
[Mon Sep 29 07:46:19 2014] [error] [client 127.0.0.1] from paste.deploy.loadwsgi import *
[Mon Sep 29 07:46:19 2014] [error] [client 127.0.0.1] File "/usr/lib/ckan/default/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 8, in
[Mon Sep 29 07:46:19 2014] [error] [client 127.0.0.1] import pkg_resources
[Mon Sep 29 07:46:19 2014] [error] [client 127.0.0.1] File "/usr/lib/ckan/default/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/pkg_resources.py", line 2727, in
[Mon Sep 29 07:46:19 2014] [error] [client 127.0.0.1] add_activation_listener(lambda dist: dist.activate())
[Mon Sep 29 07:46:19 2014] [error] [client 127.0.0.1] File "/usr/lib/ckan/default/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/pkg_resources.py", line 700, in subscribe
[Mon Sep 29 07:46:19 2014] [error] [client 127.0.0.1] callback(dist)
[Mon Sep 29 07:46:19 2014] [error] [client 127.0.0.1] File "/usr/lib/ckan/default/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/pkg_resources.py", line 2727, in
[Mon Sep 29 07:46:19 2014] [error] [client 127.0.0.1] add_activation_listener(lambda dist: dist.activate())
[Mon Sep 29 07:46:19 2014] [error] [client 127.0.0.1] File "/usr/lib/ckan/default/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/pkg_resources.py", line 2230, in activate
[Mon Sep 29 07:46:19 2014] [error] [client 127.0.0.1] map(declare_namespace, self._get_metadata('namespace_packages.txt'))
[Mon Sep 29 07:46:19 2014] [error] [client 127.0.0.1] File "/usr/lib/ckan/default/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/pkg_resources.py", line 1827, in declare_namespace
[Mon Sep 29 07:46:19 2014] [error] [client 127.0.0.1] _handle_ns(packageName, path_item)
[Mon Sep 29 07:46:19 2014] [error] [client 127.0.0.1] File "/usr/lib/ckan/default/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/pkg_resources.py", line 1797, in _handle_ns
[Mon Sep 29 07:46:19 2014] [error] [client 127.0.0.1] loader.load_module(packageName); module.path = path
[Mon Sep 29 07:46:19 2014] [error] [client 127.0.0.1] File "/usr/lib/python2.7/pkgutil.py", line 246, in load_module
[Mon Sep 29 07:46:19 2014] [error] [client 127.0.0.1] mod = imp.load_module(fullname, self.file, self.filename, self.etc)
[Mon Sep 29 07:46:19 2014] [error] [client 127.0.0.1] File "/usr/lib/ckan/default/src/ckanext-datajson/ckanext/datajson/init.py", line 9, in
[Mon Sep 29 07:46:19 2014] [error] [client 127.0.0.1] from plugin import DataJsonPlugin
[Mon Sep 29 07:46:19 2014] [error] [client 127.0.0.1] File "/usr/lib/ckan/default/src/ckanext-datajson/ckanext/datajson/plugin.py", line 1, in
[Mon Sep 29 07:46:19 2014] [error] [client 127.0.0.1] import ckan.plugins as p
[Mon Sep 29 07:46:19 2014] [error] [client 127.0.0.1] File "/usr/lib/ckan/default/src/ckan/ckan/plugins/init.py", line 5, in
[Mon Sep 29 07:46:19 2014] [error] [client 127.0.0.1] import toolkit as _toolkit
[Mon Sep 29 07:46:19 2014] [error] [client 127.0.0.1] File "/usr/lib/ckan/default/src/ckan/ckan/plugins/toolkit.py", line 4, in
[Mon Sep 29 07:46:19 2014] [error] [client 127.0.0.1] import paste.deploy.converters as converters
[Mon Sep 29 07:46:19 2014] [error] [client 127.0.0.1] AttributeError: 'module' object has no attribute 'deploy'

It seems that this has something to do with wsgi and the datajson plugin. I tried adding the import ckanext line in the apache.wsgi file under /etc/ckan/default but I still get this error. Have you encountered it before?

[zr2d2]

for reference, I have responded to #13, which is @syxxzonz's original ticket