Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chip_size Parameter Issue: Farm Size Dependency in spaceeye_interpolation Workflow #180

Open
click2cloud-sagarB opened this issue Jul 18, 2024 · 5 comments
Assignees
Labels
bug Something isn't working local cluster Issues encountered in local cluster next-release-fix Issues with a fix planned for the next release triage Issues still not triaged by team workflows Issues encountered when running workflows

Comments

@click2cloud-sagarB
Copy link

click2cloud-sagarB commented Jul 18, 2024

In which step did you encounter the bug?

Workflow execution

Are you using a local or a remote (AKS) FarmVibes.AI cluster?

Local cluster

Bug description

Dear Farmvibes Team,

During the execution of the data_ingestion/spaceeye/spaceeye_interpolation workflow, we encountered an issue with the chip_size parameter "ValueError(\nValueError: dim_size=421 cannot be smaller than chip_size=448. Please consider reducing the step/chip size or increasing the input geometry." Specifically, the workflow failed for a 115.5-acre farm while a 0.5-acre farm ran successfully. So many times it is observed that larger farms fail due to the smaller chip_size, whereas smaller farms are unaffected. Could you please let us know the precise cause of this problem?

Can your team please look into this matter and if possible can your team expose step_size parameter just like crop_cycle workflow so that we can decrease step size from 448 to less than 448 to avoid chip_size issue.

I've attached the log files detailing the errors encountered with both the 115.5-acre and 0.5-acre farms for your review.

log_files.zip

Steps to reproduce the problem

No response

@click2cloud-sagarB click2cloud-sagarB added the bug Something isn't working label Jul 18, 2024
@github-actions github-actions bot added triage Issues still not triaged by team local cluster Issues encountered in local cluster workflows Issues encountered when running workflows labels Jul 18, 2024
@rafaspadilha rafaspadilha self-assigned this Jul 30, 2024
@rafaspadilha rafaspadilha removed the triage Issues still not triaged by team label Jul 30, 2024
@rafaspadilha
Copy link
Contributor

Hi, @click2cloud-sagarB. Thanks for opening this issue.

Looks like that issue happens when the input geometry falls very close to the border of a tile and yields a region that is smaller (in image size) than our chip size. When that happens, we try to increase the size of the input geometry/window in order to run SpaceEye, but because we are close to the border of the tile, sometimes that is not possible.

I've opened PR #183 with a fix for that issue. I've tested with the geometries that you shared with me and SpaceEye completed successfully. Once the PR merges, let me know if you are able to test it in your geometries.

@rafaspadilha rafaspadilha added the next-release-fix Issues with a fix planned for the next release label Jul 31, 2024
rafaspadilha added a commit that referenced this issue Aug 1, 2024
… raster's boarder (#183)

In the SpaceEye workflow, if the input geometry yields a window that is
smaller than the chip size, we adjust it by splitting the difference
between the chip's and window's width (height) half way to the
right/left (top/bottom) sides of the original window. However if the
input geometry falls close to the border of the raster, we might not be
able to completely increase the window in one of the sides and end up
with a window smaller than the chip size. Because of that, the workflow
fails with "_dim_size cannot be smaller than chip_size. Please consider
reducing the step/chip size or increasing the input geometry._"

This PR updates the `_adjust_roi_window` method, so the window is
adjusted from both sides until reaching the chip size or reaching the
border of the raster.

This fixes #180.
@rafaspadilha
Copy link
Contributor

PR merged on dev. @click2cloud-sagarB, let me know if you are able to try building the cluster with the dev branch.

I'll close this issue for now. Feel free to reopen if the problem persists.

@click2cloud-sagarB
Copy link
Author

click2cloud-sagarB commented Aug 9, 2024

Hello, @rafaspadilha I noticed that the dev cluster is having some problems. I was unable to determine whether the chip_size issue had been resolved for the provided geometries because the sapeceeye pipeline is failing for dev cluster with the reason
" INVALID_PROTOBUF : Load model from /opt/terravibes/ops/resources/cloud_models/cloud_model2_cpu.onnx failed:Protobuf parsing failed. "
When I am running same farm with building main branch cluster it runs sucessfully.
I am attaching the log file for failure and ss for same.
andrew_dev_cluster_9ed5227e-0b84-47aa-bae9-fd86cbffd015.json

1

@rafaspadilha
Copy link
Contributor

rafaspadilha commented Aug 9, 2024

Hi, @click2cloud-sagarB. Let me see if I can reproduce this error.

Did you deleted your old cluster and recreated a new one using the dev branch?

@click2cloud-sagarB
Copy link
Author

click2cloud-sagarB commented Aug 9, 2024

Yes @rafaspadilha , I deleted old cluster and recreasted a new one using dev branch.
Edit: Task spaceeye.preprocess.cloud.cloud fails in dev cluster

@rafaspadilha rafaspadilha reopened this Sep 3, 2024
@github-actions github-actions bot added the triage Issues still not triaged by team label Sep 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working local cluster Issues encountered in local cluster next-release-fix Issues with a fix planned for the next release triage Issues still not triaged by team workflows Issues encountered when running workflows
Projects
None yet
Development

No branches or pull requests

2 participants