Skip to content

Postprocessing Troubleshooting

Jump to bottom
Elaina Present edited this page Jul 16, 2021 · 11 revisions

Postprocessing Troubleshooting

Uploading results to AWS manually from Eagle

Sometimes an upload fails or you didn't originally specify that you wanted to have your results uploaded. As an alternative to running with the --uploadonly flag, it is possible to manually upload your results to s3 manually.

Use a DAV node

First, log into an Eagle DAV node (they have more memory and are better than using the traditional login nodes).

ssh eagle-dav.hpc.nrel.gov

Note which node you got routed to. It'll be ed[1-6] i.e.:

[nmerket@ed6 ~]$

Make sure your AWS credentials are properly set up on Eagle

If you are getting permission denied or credentials errors on the upload step, this is why. We cover this in the user documentation, so I'm not going to go over it again here.

Install s5cmd (only the first time)

Go to the latest release page for s5cmd.

Scroll down to Assets and right click on the one that is like s5cmd_X.X.X_Linux-64bit.tar.gz and choose "Copy link".

Go back to your Eagle terminal. The following commands will download, extract and place the s5cmd utility in the correct place. If you have a newer version, you may need to update the commands below to reflect that. (If you could update this wiki page, that would be great too.)

wget https://github.com/peak/s5cmd/releases/download/v1.3.0/s5cmd_1.3.0_Linux-64bit.tar.gz
mkdir -p ~/.local/bin
tar -C ~/.local/bin/ -xzf s5cmd_1.3.0_Linux-64bit.tar.gz s5cmd
rm s5cmd_1.3.0_Linux-64bit.tar.gz

At this point you should be able to run s5cmd -h and get a help message.

Launch the screen utility

Start a screen session, which will allow you to leave a terminal running on the DAV node after you've logged out. Read this screen tutorial to learn how to do that.

Upload your parquet files using s5cmd

First you're probably here because your upload failed in the first place. That means you likely have some files on S3, but it didn't complete. Let's clear those out before we try to upload them again.

s5cmd rm s3://{bucket}/{prefix}/*

where bucket and prefix are what you put in your project config file for the upload location. Leave out the {}.

Now you can upload results parquet files:

s5cmd cp /projects/path/to/your_output_directory/results/parquet/ s3://{bucket}/{prefix}/

It's good to try first with the --dry-run flag to make sure the correct files will land in the correct place. It will also verify your AWS credentials are available and active.

Once you have started the upload, it will probably take a while to complete. Because you probably don't want to leave a terminal open and connected to Eagle for the next few hours, disconnect from your screen session (Ctrl-a d, lift your fingers from the keyboard between the two parts instead of pushing all the buttons at the same time). This will leave your command running in the background. You can then close the connection to the Eagle DAV node and go on about your day.

Checking in on your upload

To check on it later, you'll go directly to the node you noted earlier:

ssh ed6.hpc.nrel.gov # or ed1, ed2, ...

Reconnect to the screen. See the tutorial linked above for how to do that. You'll either see your upload still running or it will be complete. If it's still running, disconnect. If it's done exit out of that screen.

Create and Run the AWS Glue Crawler

Once the results are fully uploaded to AWS, you still need to create the Glue/Athena tables manually.

  1. Log into the AWS console.
  2. Make sure you're in the region you want to be (probably Oregon).
  3. Go to "Services" > "AWS Glue"
  4. "Add Tables" > "add tables using a crawler"
  5. Name the crawler something
  6. Crawler source type: "Data stores"
  7. Add the data store:
    • Data store: s3
    • Crawl data in: Specified path in my account
    • Include path: The s3 path to the parquet data you just uploaded. You can navigate to it using the little folder icon.
  8. Don't add another data store
  9. Choose IAM role:
    • Choose existing IAM role: AWSGlueServiceRole-default
  10. Schedule: Run on demand
  11. Crawler output:
    • Database: Select the database you want
    • Prefix: Choose your prefix i.e. myanalysis_
    • Leave other options alone.
  12. Review and Finish.
  13. At the top of the list of crawlers there will be a banner asking if you want to run the crawler you just created. Select "yes".
  14. Wait for the crawler to complete.
  15. Delete the crawler.

Common Errors

If you are getting an error in postprocessing that is like TypeError: not a path-like object, do a pull from buildstockbatch/master and recreate your environment. That should clear up the issue. If you’re using the buildstock-0.18 or greater environment on Eagle you should be okay.

Clone this wiki locally