Skip to content

Latest commit

 

History

History
202 lines (137 loc) · 6.27 KB

demo_quick_start_guide.md

File metadata and controls

202 lines (137 loc) · 6.27 KB
Raw

2024 ISMR Workshop Demo Quick Start Guide

Hardware Setup

To complete this demo, you would need the following items:

  1. A dVRK PSM Arm
  2. A camera
  3. dVRK Hand-Eye Calibration Kit (a cube with an Aruco marker attached)
  4. A paper with some Aruco markers printed on it
  5. An external light source
  6. Some mechanical support for the system (you can have your own design)

For the camera, the demo is using a depstech borescope.

Preparation

You may need to print some Aruco markers with certain size. You can take advantage of the cv_marker_generator GitHub Repository.

Software Setup

Note: you can skip step 1-3 if you have already had dVRK and its video pipeline properly installed. You can replace ~/catkin_ws_dvrk with your catkin workspace path

  1. create a catkin workspace

Run the following command in your terminal

cd ~
mkdir catkin_ws_dvrk
cd catkin_ws_dvrk
  1. install the dVRK

Please follow the dVRK ROS 1 install instructions

  1. install gscam for dVRK video pipeline

Note: This note is only for Ubuntu 20.04 OS. If you happen to have different OS, please go through the video pipeline README for more information.

Firstly, clone the noetic-devel branch of a customized gscam repository https://github.com/JackHaoyingZhou/gscam into the src folder of your dVRK workspace. And then manually compile it:

cd ~/catkin_ws_dvrk/src
git clone https://github.com/JackHaoyingZhou/gscam.git
cd gscam
git checkout noetic-devel
cd ..
catkin build --summary
  1. Install the dependencies

clone and compile from source to install the aruco_detection and the ISMR demo packages

cd ~/catkin_ws_dvrk/src
git clone https://github.com/JackHaoyingZhou/aruco_detection.git
git clone https://github.com/JackHaoyingZhou/ismr2024_demo_needle_grasping.git
catkin build --summary
  1. Source your catkin workspace
source ~/catkin_ws_dvrk/devel/setup.bash

Calibrations

  1. create a catkin package

You will need to create a new package to store the camera calibration results. At that point, choose a name for your camera. For the demo application, we will use depstech:

cd ~/catkin_ws_dvrk/src
catkin_create_pkg depstech
catkin build --summary
source ~/catkin_ws_dvrk/devel/setup.bash
  1. start the video streaming

This is a video capture based on a video4linux compatible source:

roslaunch ismr2024_demo_needle_grasping gscam_v4l_mjpg.launch camera_name:=depstech image_width:=1920 image_height:=1080
  1. calibrate the camera

The calibration is performed using the ROS provided application, please refer to their documentation for the parameters ROS camera calibration

For the camera calibration, we are using a 12x10 calibration grid and the size of each square is 6mm:

rosrun camera_calibration cameracalibrator.py --size 12x10 --square 0.006 image:=/depstech/image_raw camera:=/depstech

Make sure you hit calibration button then hit commit button.

  1. restart the video using the calibration
roslaunch ismr2024_demo_needle_grasping gscam_v4l_mjpg.launch camera_name:=depstech image_width:=1920 image_height:=1080 mono_proc:=True
  1. hand-eye calibration

The calibration cube has an Aruco marker attached to it, the marker's size is 1.5mm.

calibration script

rosrun dvrk_camera_registration camera_registration.py -p PSM2 -m 0.015 -c /depstech

-p is used to indicate which PSM to use, -m represents the size of the marker, and -c is the namespace for your camera.

After the script starts, you need to manually move the PSM all over your desired configuration space for best hand-eye calibration results.

After collecting the data, the scripts will generate a couple of .json files with the transformation between the PSM and the camera. The file with -open-cv is used for the validation script below.

validation script

rosrun dvrk_camera_registration vis_gripper_pose.py -p PSM2 -c /depstech -H PSM2-registration-open-cv.json

If your validation results are acceptable, you need to do the following two steps:

  1. Efit your dVRK configuration files

The script also generates a PSM<x>_registration-dVRK.json file that contains a transformation which you can copy-paste in your dVRK console-PSM<x>.json to define the PSM base-frame.

  1. Copy the .json files to <ISMR demo>/config folder
cp ./PSM2-registration-dVRK.json ~/catkin_ws_dvrk/src/ismr2024_demo_needle_grasping/config/
cp ./PSM2-registration-open-cv.json ~/catkin_ws_dvrk/src/ismr2024_demo_needle_grasping/config/

How to Run

Firstly, you need to start the video:

roslaunch ismr2024_demo_needle_grasping gscam_v4l_mjpg.launch camera_name:=depstech image_width:=1920 image_height:=1080 mono_proc:=True

Secondly, you need to run the Aruco detection script to enable the tracking of all visible Aruco markers:

rosrun aruco_detection aruco_detector.py

Alternatively, you can set your own parameters based on the developer guide for the above script. After modifications, you have to run through Python:

roscd aruco_detection
cd scripts
python aruco_detector.py <your arguments>

Last but not least, you can run the needle grasping script using default settings:

rosrun ismr2024_demo_needle_grasping grasp_needle.py

If you would like to visualize the frames, please run the following command in a separate terminal:

rosrun ismr2024_demo_needle_grasping show_all_frames.py

Alternatively, you can set your own parameters based on the developer guide for the above two running scripts. After modifications, you have to run through Python:

cd ~/catkin_ws_dvrk/src/ismr2024_demo_needle_grasping/scripts
python grasp_needle.py <your arguments>

and

cd ~/catkin_ws_dvrk/src/ismr2024_demo_needle_grasping/scripts
python show_all_frames.py <your arguments>