adding connectome sideloading #26
Conversation
Nope. Probably better to hack in as direct (robot) merge of relevant OWL files (assuming the are accessible from here) in the goal. If you make these into dependencies, this should be outside of the patsub. |
|
Ref #24 (comment) |
|
I've linked the owl files previously loaded in at collectdata stage. Test still running ATM. |
There was a problem hiding this comment.
I think your edit will cause the job to fail because it will look for non-existent SPARQL queries to run here:
$(RAW_DUMPS_DIR)/%.ttl:
curl -G --data-urlencode "query=`cat $(SPARQL_DIR)/construct_$*.sparql`" $(SPARQL_ENDPOINT) -o $@I assume the connectomics OWL files are no longer loaded into the triplestore. They need to be accessible to this build and the dumps goals need to be edited to merge them.
e.g.
CONNECTOMICS=connectome_fafb.owl connectome_l1em.owl connectome_hemibrain.owl
$(FINAL_DUMPS_DIR)/pdb.owl: $(patsubst %, $(RAW_DUMPS_DIR)/construct_%.owl, $(DUMPS_PDB)) $(RAW_DUMPS_DIR)/inferred_annotation.owl ${CONNECTOMICS}
$(ROBOT) merge $(patsubst %, -i %, $^) -o $@ $(STDOUT_FILTER)
|
If current approach doesn't work, I can implement this function both in this branch and generic_pipeline. I can read EXTERNAL_ONTOLOGIES (aka. CONNECTOMICS) from environment variables and directly feed to the robot. |
|
Well, we've officially made it slower but as we are now doing the merge twice it's not unexpected but it is running very inefficiently as using only 5% of a single CPU and 138G RAM for the last few days but still seems to be going. Could do with --vvv but I'll give it the weekend to finish. |
@Robbie1977 - do you have edits outside this PR that pull in the additional files and merge them? Not seeing anything here that would what we planned. This would #26 (review) (see suggested code in second box). BTW - the merge shouldn't take long. I think it would be best to sprinkle echo statements through the Makefile that print time so that we have a clear idea which steps are slow. |
false means we will not use reasoning and $(INFER_ANNOTATE_RELATION) is the property that already exists in the pdb.owl (coming from inferred_annotation.owl) to get tags.
Most specific, non-redundant list of facets
No description provided.