Change perceiver to have history #293
Conversation
| @@ -618,21 +619,24 @@ def __init__(self) -> None: | |||
| self._curr_env: Optional[BaseEnv] = None | |||
| self._waiting_for_observation = True | |||
| self._ordered_objects: List[Object] = [] # list of all known objects | |||
| self._state_history: Deque[State] = deque( | |||
There was a problem hiding this comment.
Use a list for simplicity for both of these histories -- appending to the end is O(1) in both cases of List vs. Deque
predicators/utils.py
Outdated
| # and beyond. | ||
| curr_prompt = prompt[:] | ||
| curr_prompt_imgs = [ | ||
| imgs_timestep[0] for imgs_timestep in imgs_history[-1] |
There was a problem hiding this comment.
This is only giving it the latest state's image, right?
Do we not want imgs_history[-2:] here?
Another thing is -- depending on what sequence of skills we run, we may be in completely different locations/orientations between timestep t-1 and t, so actually giving the images for, say, front-left camera from t-1 and t may not be useful. Say the front-left image at t has objects A and B in it, and the right image has objects A and B in it at t-1. Then we might want to give the right image from t-1 and the front-left image from t...
Also, I think we should add a flag to pass in ALL the images, not just imgs_timestep[0]? Or we can discuss if we'll need this.
No description provided.