Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

problem with option -p and finding rack (branch: multiconv) #41

Open
source-data opened this issue Aug 7, 2019 · 5 comments
Open

problem with option -p and finding rack (branch: multiconv) #41

source-data opened this issue Aug 7, 2019 · 5 comments
Labels
bug Something isn't working

Comments

@source-data
Copy link
Collaborator

It seems that one has now always to specify explicitly where rack is?

root@6da6703ce11a:/workspace/py-smtag# smtag-convert2th -c 190414 -f 5X_L1200_artice_embeddings -X5 -L1200 -E ".//sd-panel"                                     
ERROR:root:                                                                                                                                                     
            #######################################################################################################                                             
            #                                                                                                                                                   
            ERROR: PRODUCTION DIRECTORY DOES NOT EXIST                                                                                                          

            The specified production directory does not exist:

                /opt/conda/lib/python3.6/site-packages/smtag-2.0.2-py3.6.egg/rack


            In order to configure you production directory use one of this options, sorted by lesser precedence:
            1) Set the environment variable `SMTAG_PRODUCTION_DIRECTORY` (check your .env file)
               Examples:
                SMTAG_PRODUCTION_DIRECTORY='/absolute/path/to/resources_folder' python -m smtag.predict.egine --demo
                SMTAG_PRODUCTION_DIRECTORY='./relative/path/to/resources_folder' python -m smtag.predict.egine --demo
            2) Set the `--production_directory` flag or its abbreviated version `-p`
               Examples:
                python -m smtag.predict.egine --demo --production_directory "/absolute/path/to/resources_folder"
                python -m smtag.predict.egine --demo -p "./relative/path/to/resources_folder"
            3) Set the `smtag.config.prod_dir` programatically
               Examples:
                import smtag
                smtag.config.prod_dir = "/absolute/path/to/resources_folder"
                smtag.config.prod_dir = "./relative/path/to/resources_folder"

            If none of the above options is specified, the default directory is assumed to be:

                /opt/conda/lib/python3.6/site-packages/smtag-2.0.2-py3.6.egg/rack


            #
            #######################################################################################################

while this works

root@6da6703ce11a:/workspace/py-smtag# smtag-convert2th -c 190414 -f 5X_L1200_artice_embeddings -X5 -L1200 -E ".//sd-panel" -p rack

with

root@6da6703ce11a:/workspace/py-smtag# ls -thl
total 572M
drwxr-xr-x 63 root root 4.0K Aug  6 20:39 runs
drwxr-xr-x  2 root root 4.0K Aug  4 05:56 smtag.egg-info
drwxrwxr-x  2 1002 1002 4.0K Aug  4 05:55 docs
drwxrwxr-x  5 1002 1002 4.0K Jul 23 06:17 vsearch
-rw-rw-r--  1 1002 1002 1.2K Jul 23 06:08 Dockerfile
drwxrwxr-x  7 1002 1002 4.0K Jul 23 06:08 smtag
drwxr-xr-x  2 root root 4.0K Jul 21 14:34 __pycache__
-rw-rw-r--  1 1002 1002 2.2K Jul 21 14:13 setup.py
-rwxrwxr-x  1 1002 1002  117 Jul 19 04:31 dck
drwxrwxr-x 10 1002 1002 4.0K Jul 18 20:45 testdir
-rw-r--r--  1 1002 1002 572M Jul 18 20:41 testdir.zip
drwxrwxr-x  2 1002 1002 4.0K Jul 18 17:31 rack
drwxr-xr-x  2 root root 4.0K Jul 17 16:59 src
drwxrwxr-x  8 1002 1002 4.0K Jul 17 11:19 resources
drwxr-xr-x  2 root root 4.0K Jul 17 04:26 dist
drwxr-xr-x  4 root root 4.0K Jul  9 10:25 build
-rw-rw-r--  1 1002 1002 1.9K Jul  8 20:22 docker-compose.yml
drwxrwxr-x  2 1002 1002 4.0K Jun 18 07:49 test
-rw-rw-r--  1 1002 1002  537 Jun 18 07:49 setup.cfg
-rw-rw-r--  1 1002 1002   42 Jun 18 07:49 LICENSE.txt
-rw-rw-r--  1 1002 1002  222 Jun 18 07:49 MANIFEST.in
-rw-rw-r--  1 1002 1002  458 Jun 18 07:49 README.md
@source-data source-data added the bug Something isn't working label Aug 7, 2019
@source-data source-data changed the title probalem with option -p and finding rack problem with option -p and finding rack (branch: multiconv) Aug 7, 2019
@ariera
Copy link
Member

ariera commented Aug 8, 2019

hmmm it is working fine for me on multiconv
do you still have the issue?

@ariera
Copy link
Member

ariera commented Aug 8, 2019

i think the problem may have been at the moment of running pip install -e /workspace/py-smtag (see the Dockerfile) the rack folder didn't exist at the root level of the project. In other word the /workspace/py-smtag/rack directory didn't exist.

When we run pip install py-smtag, it copies all the files to its internal destination. In this example to /opt/conda/lib/python3.6/site-packages/smtag-2.0.2-py3.6.egg.

All the files? No: only the files declared in the manifest: https://github.com/source-data/py-smtag/blob/master/MANIFEST.in#L10-L11

If you would, for example, delete the rack folder, and then run pip install then the rack folder will not be copied over to /opt/conda/python3.6...

In summary: as it is right now, rack must always exist, even if it is empty.

Could this have been the issue?

@source-data
Copy link
Collaborator Author

hmm, I stil lhave problem locally...
I tried to remove installation first

$ rm -fr .venv

I have a nicely populated rack:

$ ls
Dockerfile              __pycache__             docs                    runs                    smtag.egg-info          vsearch
LICENSE.txt             build                   rack                    setup.cfg               test
MANIFEST.in             dist                    rack.zip                setup.py                testdir
README.md               docker-compose.yml      resources               smtag                   testdir.zip
mac-embo184:py-smtag lemberge$ ls rack
10X_L1200_disease_articke_embeddings_128-5X_L1200_article_embeddings_128_disease_2019-08-25-21-47.zip
5X_L1200_anonym_not_reporter_article_embeddings_128_intervention_assayed_2019-08-22-16-25.zip
5X_L1200_article_embeddings_128_reporter_2019-08-28-00-08_epoch_23_.zip
5X_L1200_article_embeddings_128_small_molecule_geneprod_subcellular_cell_tissue_organism_assay_2019-08-23-17-46.zip
5X_L1200_emboj_2012_no_viz_panel_stop_2019-08-29-08-31.zip
5X_L1200_molecule_anonym_article_embeddings_128_intervention_assayed_2019-08-28-23-33_epoch_51.zip

And re-installed requirements (for my mac with tensorflow 1.14.0, setuptools 41.0.0 needed by tensorboard, otherwise it does not work):

$ python3 -m venv .venv
mac-embo184:py-smtag lemberge$ source .venv/bin/activate
(.venv) mac-embo184:py-smtag lemberge$ pip install -r smtag/requirements.txt
$ python -m setup install

And then it fails :-( What am I doing wrong?

$ smtag-predict --help
ERROR:root:
            #######################################################################################################
            #
            ERROR: PRODUCTION DIRECTORY DOES NOT EXIST

            The specified production directory does not exist:

                /Users/lemberge/code/py-smtag/.venv/lib/python3.7/site-packages/smtag-2.0.2-py3.7.egg/rack

            
            In order to configure you production directory use one of this options, sorted by lesser precedence:
            1) Set the environment variable `SMTAG_PRODUCTION_DIRECTORY` (check your .env file)
               Examples:
                SMTAG_PRODUCTION_DIRECTORY='/absolute/path/to/resources_folder' python -m smtag.predict.egine --demo
                SMTAG_PRODUCTION_DIRECTORY='./relative/path/to/resources_folder' python -m smtag.predict.egine --demo
            2) Set the `--production_directory` flag or its abbreviated version `-p`
               Examples:
                python -m smtag.predict.egine --demo --production_directory "/absolute/path/to/resources_folder"
                python -m smtag.predict.egine --demo -p "./relative/path/to/resources_folder"
            3) Set the `smtag.config.prod_dir` programatically
               Examples:
                import smtag
                smtag.config.prod_dir = "/absolute/path/to/resources_folder"
                smtag.config.prod_dir = "./relative/path/to/resources_folder"

            If none of the above options is specified, the default directory is assumed to be:

                /Users/lemberge/code/py-smtag/.venv/lib/python3.7/site-packages/smtag-2.0.2-py3.7.egg/rack
            

            #
            #######################################################################################################
        
Traceback (most recent call last):
  File "/Users/lemberge/code/py-smtag/.venv/bin/smtag-predict", line 11, in <module>
    load_entry_point('smtag==2.0.2', 'console_scripts', 'smtag-predict')()
  File "/Users/lemberge/code/py-smtag/.venv/lib/python3.7/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/Users/lemberge/code/py-smtag/.venv/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2843, in load_entry_point
    return ep.load()
  File "/Users/lemberge/code/py-smtag/.venv/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2434, in load
    return self.resolve()
  File "/Users/lemberge/code/py-smtag/.venv/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2440, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/Users/lemberge/code/py-smtag/.venv/lib/python3.7/site-packages/smtag-2.0.2-py3.7.egg/smtag/__init__.py", line 15, in <module>
    config = Config()
  File "/Users/lemberge/code/py-smtag/.venv/lib/python3.7/site-packages/smtag-2.0.2-py3.7.egg/smtag/common/config/configuration.py", line 84, in __init__
    self.prod_dir = fetch_production_directory()
  File "/Users/lemberge/code/py-smtag/.venv/lib/python3.7/site-packages/smtag-2.0.2-py3.7.egg/smtag/common/config/configuration.py", line 94, in prod_dir
    self.__production_directory = validated_production_directory(new_production_directory)
  File "/Users/lemberge/code/py-smtag/.venv/lib/python3.7/site-packages/smtag-2.0.2-py3.7.egg/smtag/common/config/production_directory.py", line 124, in validated_production_directory
    raise ProductionDirectoryDoesNotExistError(path)
smtag.common.config.production_directory.ProductionDirectoryDoesNotExistError

@ariera
Copy link
Member

ariera commented Sep 2, 2019 •
edited
Loading

either set SMTAG_PRODUCTION_DIRECTORY or use -p to indicate where the rack folder you want use lives.

In your case i think:

smtag-predict --help -p rack

or

SMTAG_PRODUCTION_DIRECTORY=rack smtag-predict --help

would work

@source-data
Copy link
Collaborator Author

is it possible to make ./rack/ the default rack so that I don't have to always add -p rack? So sorry to annoy with this... Not a priority anyway.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ariera