-
Notifications
You must be signed in to change notification settings - Fork 14
Postprocessing Troubleshooting
Sometimes an upload fails or you didn't originally specify that you wanted to have your results uploaded. As an alternative to running with the --uploadonly
flag, it is possible to manually upload your results to s3 manually.
First, log into an Eagle DAV node (they have more memory and are better than using the traditional login nodes).
ssh eagle-dav.hpc.nrel.gov
Note which node you got routed to. It'll be ed[1-6]
i.e.:
[nmerket@ed6 ~]$
If you are getting permission denied or credentials errors on the upload step, this is why. We cover this in the user documentation, so I'm not going to go over it again here.
Go to the latest release page for s5cmd.
Scroll down to Assets and right click on the one that is like s5cmd_X.X.X_Linux-64bit.tar.gz
and choose "Copy link".
Go back to your Eagle terminal. The following commands will download, extract and place the s5cmd
utility in the correct place. If you have a newer version, you may need to update the commands below to reflect that. (If you could update this wiki page, that would be great too.)
wget https://github.com/peak/s5cmd/releases/download/v1.3.0/s5cmd_1.3.0_Linux-64bit.tar.gz
mkdir -p ~/.local/bin
tar -C ~/.local/bin/ -xzf s5cmd_1.3.0_Linux-64bit.tar.gz s5cmd
rm s5cmd_1.3.0_Linux-64bit.tar.gz
At this point you should be able to run s5cmd -h
and get a help message.
Start a screen
session, which will allow you to leave a terminal running on the DAV node after you've logged out. Read this screen tutorial to learn how to do that.
First you're probably here because your upload failed in the first place. That means you likely have some files on S3, but it didn't complete. Let's clear those out before we try to upload them again.
s5cmd rm s3://{bucket}/{prefix}/*
where bucket
and prefix
are what you put in your project config file for the upload location. Leave out the {}
.
Now you can upload results parquet files:
s5cmd /projects/path/to/your_output_directory/results/parquet/ s3://{bucket}/{prefix}/
It's good to try first with the --dry-run
flag to make sure the correct files will land in the correct place. It will also verify your AWS credentials are available and active.
Once you have started the upload, it will probably take a while to complete. Because you probably don't want to leave a terminal open and connected to Eagle for the next few hours, disconnect from your screen session (Ctrl-a
d
, lift your fingers from the keyboard between the two parts instead of pushing all the buttons at the same time). This will leave your command running in the background. You can then close the connection to the Eagle DAV node and go on about your day.
To check on it later, you'll go directly to the node you noted earlier:
ssh ed6.hpc.nrel.gov # or ed1, ed2, ...
Reconnect to the screen
. See the tutorial linked above for how to do that. You'll either see your upload still running or it will be complete. If it's still running, disconnect. If it's done exit
out of that screen.
Once the results are fully uploaded to AWS, you still need to create the Glue/Athena tables manually.
- Log into the AWS console.
- Make sure you're in the region you want to be (probably Oregon).
- Go to "Services" > "AWS Glue"
- "Add Tables" > "add tables using a crawler"
- Name the crawler something
- Crawler source type: "Data stores"
- Add the data store:
- Data store: s3
- Crawl data in: Specified path in my account
- Include path: The s3 path to the parquet data you just uploaded. You can navigate to it using the little folder icon.
- Don't add another data store
- Choose IAM role:
- Choose existing IAM role: AWSGlueServiceRole-default
- Schedule: Run on demand
- Crawler output:
- Database: Select the database you want
- Prefix: Choose your prefix i.e.
myanalysis_
- Leave other options alone.
- Review and Finish.
- At the top of the list of crawlers there will be a banner asking if you want to run the crawler you just created. Select "yes".
- Wait for the crawler to complete.
- Delete the crawler.
If you are getting an error in postprocessing that is like TypeError: not a path-like object
, do a pull from buildstockbatch/master and recreate your environment. That should clear up the issue. If you’re using the buildstock-0.18
or greater environment on Eagle you should be okay.