Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revisionNext revisionBoth sides next revision | ||
tutorials:demo:fetch_and_place [2020/02/06 15:00] – feedback Alina hawkin | tutorials:demo:fetch_and_place [2021/11/16 13:57] – arthur | ||
---|---|---|---|
Line 3: | Line 3: | ||
This tutorial is from the "Demo Tutorials" | This tutorial is from the "Demo Tutorials" | ||
+ | ===== Important Links ===== | ||
+ | * The {{ : | ||
+ | * The [[https:// | ||
+ | * An explanation of the setup and first tutorial is also available with voice explanations (around 1h) as [[https:// | ||
+ | ===== Motivation ===== | ||
+ | |||
+ | The aim of the tutorial is to give an intuition about the complexity of robot mobile manipulation tasks and about the knowledge that is required to execute actions successfully. | ||
+ | |||
+ | The tutorial: | ||
+ | - Gives an intuition of what knowledge does the robot need (and how much knowledge) to execute even a simple fetch and place, e.g., robot needs to answer questions such as "where should I stand?", | ||
+ | - Shows how many different things can go wrong, and teaches writing simple failure handling strategies. | ||
+ | - Teaches how to call existing actions from the CRAM framework, i.e. how to use designators (but not program new ones). | ||
+ | | ||
===== Setting Up ===== | ===== Setting Up ===== | ||
Line 15: | Line 28: | ||
==== Technical Requirements ==== | ==== Technical Requirements ==== | ||
- | The VM image with VirtualBox has been successfully tested | + | * The VM image with VirtualBox has been successfully tested |
- | + | * Some kind of GPU, dedicated or integrated, is required | |
- | The machine on which you will be running the system needs a graphics card, it can be a built in laptop GPU as well. | + | * At least 8GB of excess |
- | + | ||
- | We recommend | + | |
- | (The VM itself needs 4 GB of RAM, so the host machine should have at least 5 GB but 8 GB or more would be optimal.) | + | |
Line 42: | Line 52: | ||
== Windows == | == Windows == | ||
- | Download [[https:// | + | Download [[https:// |
The CPU needs to support virtualization to run a VM on Windows. The virtualization can be enabled in the BIOS. To get into the BIOS settings restart your machine, hit the settings button while booting (try F2, F8 and F10) and go into the CPU settings. There is something like 'Intel Virtual Technology' | The CPU needs to support virtualization to run a VM on Windows. The virtualization can be enabled in the BIOS. To get into the BIOS settings restart your machine, hit the settings button while booting (try F2, F8 and F10) and go into the CPU settings. There is something like 'Intel Virtual Technology' | ||
Line 48: | Line 58: | ||
== MacOS == | == MacOS == | ||
- | Download [[https:// | + | Download [[https:// |
Hint: If the VirtualBox installer fails with The **installation failed** where it encountered an //error that caused the installation to fail.// Go to System Preferences > Security & Privacy. Click the **‘Allow’** button at the bottom and re-run the installer. | Hint: If the VirtualBox installer fails with The **installation failed** where it encountered an //error that caused the installation to fail.// Go to System Preferences > Security & Privacy. Click the **‘Allow’** button at the bottom and re-run the installer. | ||
Line 54: | Line 64: | ||
==== VirtualBox Setup ==== | ==== VirtualBox Setup ==== | ||
- | First of all, download the Virtual Disk Image [[https:// | + | First of all, download the Virtual Disk Image [[https:// |
Then, launch VirtualBox, which you installed in the previous step. | Then, launch VirtualBox, which you installed in the previous step. | ||
Line 68: | Line 78: | ||
* the OS as ' | * the OS as ' | ||
* the version as ' | * the version as ' | ||
- | Choose the VM's memory size (RAM), depending on your machine' | + | Choose the VM's memory size (RAM), depending on your machine' |
- | We suggest 4096 MB (or you can also give it 8192 MB as shown in the screenshot): | + | We recommend to offer as much as possible, when working with unreal. CRAM on it's own can get by with less. |
{{ : | {{ : | ||
Line 77: | Line 87: | ||
=== Settings (optional) === | === Settings (optional) === | ||
- | For better performance | + | If the VM needs to communicate with the host or other machines in the network, especially when running the UnrealEngine on the host machine, VirtualBox needs to allow Bridged Networking for the VM. Choose the newly created VM and hit ' |
- | {{ : | + | |
- | + | ||
- | Everything is set up. The VM can now be started. The Ubuntu 16.04 system should start and automatically log into the only account, with the username and password | + | |
+ | {{ : | ||
+ | For better performance the VM could use some of the CPU's power. Go to ' | ||
+ | {{ : | ||
+ | Everything is set up. The VM can now be started. The Ubuntu 18.04 system should start and automatically log into the only account, with the username and password ' | ||
+ | === Last adjustments === | ||
+ | There are two versions of CRAM available in the VM. By default, the setup is designed to connect and work with a project from the Unreal Engine, for intricate physics simulation. We don't need any of that for the upcoming tutorials, instead we change to the stable, standalone version of CRAM. In the VM, launch a terminal with Ctrl-Alt-T and open up the file '' | ||
+ | < | ||
+ | nano .bashrc | ||
+ | </ | ||
+ | Put a ''#'' | ||
+ | By doing so, the terminal uses a different ROS workspace. Sourcing a ROS workspace enables the terminal session to access everything from that workspace, like loading a programming environment, | ||
===== Understanding the Basics ===== | ===== Understanding the Basics ===== | ||
Line 127: | Line 144: | ||
To see useful key shortcuts for your Emacs IDE, [[http:// | To see useful key shortcuts for your Emacs IDE, [[http:// | ||
+ | {{ : | ||
==== ROS ==== | ==== ROS ==== | ||
Line 141: | Line 158: | ||
This language has a convenient command line. | This language has a convenient command line. | ||
If you are curious why Lisp, [[http:// | If you are curious why Lisp, [[http:// | ||
+ | |||
+ | |||
+ | |||
+ | |||
+ | |||
+ | ===== Tutorial 1: Simple Fetch and Place ===== | ||
+ | |||
+ | Head to the [[http:// | ||
+ | |||
+ | ===== Tutorial 2 & 3: Failure Handling ===== | ||
+ | |||
+ | When you got the fetch and place plan go to the [[http:// | ||
+ | |||
+ | ===== Tutorial 4: Alternative Grasping Poses (Advanced) ===== | ||
+ | |||
+ | Picking objects from different angles can be helpful in occluded environments. Go to the [[http:// | ||
+ | |||
+ | |||
+ | |||
+ | < | ||
Line 217: | Line 254: | ||
Pressing and dragging the right mouse button translates the camera. | Pressing and dragging the right mouse button translates the camera. | ||
Pressing and dragging the middle click doesn' | Pressing and dragging the middle click doesn' | ||
+ | |||
+ | If the bullet simulation is stuck call the following function in the REPL to reset the window: | ||
+ | <code lisp> | ||
+ | PP-TUT> (btr-utils: | ||
+ | </ | ||
==== Moving Around ==== | ==== Moving Around ==== | ||
Line 439: | Line 481: | ||
(defparameter *base-pose-near-counter* | (defparameter *base-pose-near-counter* | ||
- | (make-pose "base_footprint" ' | + | (make-pose "map" ' |
</ | </ | ||
Line 493: | Line 535: | ||
(defparameter *base-pose-near-counter* | (defparameter *base-pose-near-counter* | ||
- | (make-pose "base_footprint" ' | + | (make-pose "map" ' |
(defparameter *final-object-destination* | (defparameter *final-object-destination* | ||
Line 800: | Line 842: | ||
<code lisp> | <code lisp> | ||
(defun pick-up-object (? | (defun pick-up-object (? | ||
- | (let* ((? | + | (let* ((? |
- | (?grasp (first ? | + | (?remaining-grasps |
- | (setf ?possible-grasps | + | |
- | + | | |
- | (cpl: | + | |
- | ;; Outer handle failure handling arm change | + | |
- | (handle-failure object-unreachable | + | ;; Outer handle-failure handling arm change |
- | ;; Iner handle-failure handling grasp change | + | |
- | ((handle-failure (or manipulation-pose-unreachable gripper-closed-completely) | + | |
+ | (setf ?grasp (first ? | ||
+ | (setf ? | ||
+ | |||
+ | ;; Inner handle-failure handling grasp change | ||
+ | (handle-failure (or manipulation-pose-unreachable gripper-closed-completely) | ||
;; Try to perform the pick up | ;; Try to perform the pick up | ||
| | ||
Line 815: | Line 862: | ||
| | ||
| | ||
- | ;; When pick-up fails this block gets executed | + | |
- | | + | ;; When pick-up fails this block gets executed |
- | e ?grasp ? | + | |
- | | + | ;; |
- | | + | |
- | | + | |
- | | + | |
- | | + | |
- | ?grasp ? | + | |
- | | + | (when (first ?remaining-grasps) |
- | | + | (setf ?grasp (first ?remaining-grasps)) |
- | | + | (setf ?remaining-grasps (rest ?remaining-grasps)) |
- | | + | (format t " |
- | | + | |
- | | + | (cpl: |
- | | + | ;; This will get executed when there are no more elements in the |
- | | + | ;; ? |
- | | + | ;; which will be caught by the outer handle-failure |
- | + | (print | |
- | ;; This is the failure management of the outer handle-failure call | + | (cpl:fail ' |
- | ;; It changes the arm that is used to grasp | + | |
- | (print " | + | |
- | (print e) | + | |
- | | + | ;; This is the failure management of the outer handle-failure call |
- | (cpl: | + | ;; It changes the arm that is used to grasp |
- | ;; if the current grasping arm is right set left, else set right | + | |
- | (setf ? | + | ;; (print e) ; |
- | :left | + | ;; Here we use the retry counter we defined. The value is decremented automatically |
- | :right)) | + | |
- | (cpl: | + | ;; if the current grasping arm is right set left, else set right |
- | ;; When all retries are exhausted print the error message. | + | (setf ? |
- | (print "No more arm change retries left : | + | :left |
- | ? | + | :right)) |
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | ? | ||
</ | </ | ||
With this, the '' | With this, the '' | ||
Line 900: | Line 949: | ||
Save the file, compile and switch back to the REPL. | Save the file, compile and switch back to the REPL. | ||
- | You should see a result that looks like the one below. | + | You should see a result that looks like the one below. |
+ | [Some messages | ||
<code lisp> | <code lisp> | ||
PP-TUT> (move-bottle ' | PP-TUT> (move-bottle ' | ||
- | [(PICK-PLACE PICK-UP) INFO] 1550504321.279: Opening gripper | + | … |
- | [(PICK-PLACE PICK-UP) INFO] 1550504321.279: Reaching | + | [(PICK-PLACE PICK-UP) INFO] 1621250741.518: Opening gripper |
- | [(GRASP-FAILURE) WARN] Failed to grasp from LEFT-SIDE | + | [(PICK-PLACE PICK-UP) INFO] 1621250741.522: Reaching |
- | [(TRYING-NEW-GRASP) INFO] 1550504800.749: | + | … |
- | [(PICK-PLACE PICK-UP) INFO] 1550504800.789: Opening gripper | + | Grasping failed with RIGHT arm and LEFT-SIDE |
- | [(PICK-PLACE PICK-UP) INFO] 1550504800.789: Reaching | + | Retrying with RIGHT arm and RIGHT-SIDE |
- | [(GRASP-FAILURE) WARN] Failed to grasp from RIGHT-SIDE | + | [(PICK-PLACE PICK-UP) INFO] 1621250742.709: Opening gripper |
- | [(TRYING-NEW-GRASP) INFO] 1550504801.577: Trying to grasp from BACK using RIGHT arm | + | [(PICK-PLACE PICK-UP) INFO] 1621250742.709: Reaching |
- | [(PICK-PLACE PICK-UP) INFO] 1550504801.601: Opening gripper | + | … |
- | [(PICK-PLACE PICK-UP) INFO] 1550504801.602: Reaching | + | Grasping failed with RIGHT arm and RIGHT-SIDE |
- | [(PICK-PLACE PICK-UP) INFO] 1550504801.939: Gripping | + | Retrying with RIGHT arm and FRONT grasp |
- | [(PICK-PLACE PICK-UP) INFO] 1550504801.973: Assert grasp into knowledge base | + | [(PICK-PLACE PICK-UP) INFO] 1621250744.157: Opening gripper |
- | [(PICK-PLACE PICK-UP) INFO] 1550504801.974: Lifting | + | [(PICK-PLACE PICK-UP) INFO] 1621250744.159: |
- | [(PICK-PLACE PLACE) INFO] 1550504802.356: Reaching | + | … |
- | [(PICK-PLACE PLACE) INFO] 1550504802.508: Putting | + | Grasping failed with RIGHT arm and FRONT grasp |
- | [(PICK-PLACE PLACE) INFO] 1550504802.619: Opening gripper | + | Retrying with RIGHT arm and BACK grasp |
- | [(PICK-PLACE PLACE) INFO] 1550504802.655: Retract grasp in knowledge base | + | [(PICK-PLACE PICK-UP) INFO] 1621250745.425: Opening gripper |
- | [(PICK-PLACE PLACE) INFO] 1550504802.660: Retracting | + | [(PICK-PLACE PICK-UP) INFO] 1621250745.430: Reaching |
+ | [(PICK-PLACE PICK-UP) INFO] 1621250746.218: Gripping | ||
+ | [(PICK-PLACE PICK-UP) INFO] 1621250746.273: Assert grasp into knowledge base | ||
+ | [(PICK-PLACE PICK-UP) INFO] 1621250746.277: Lifting | ||
+ | [(PICK-PLACE PLACE) INFO] 1621250746.751: Reaching | ||
+ | [(PICK-PLACE PLACE) INFO] 1621250746.961: Putting | ||
+ | [(PICK-PLACE PLACE) INFO] 1621250747.064: Opening gripper | ||
+ | [(PICK-PLACE PLACE) INFO] 1621250747.142: Retract grasp in knowledge base | ||
+ | [(PICK-PLACE PLACE) INFO] 1621250747.185: Retracting | ||
</ | </ | ||
{{: | {{: | ||
+ | |||
+ | The robot has once again succeeded in grasping the object. | ||
+ | |||
+ | Now let's move the bottle even further away from the robot so that it is out of reach of the right arm. This time, after trying all of the grasp poses with the right arm, it should switch to the left arm. | ||
+ | <code lisp> | ||
+ | PP-TUT> (spawn-object ' | ||
+ | </ | ||
+ | You should see a result that looks like the one below, succeeding in grasping the bottle with the first grasp (LEFT-SIDE) using the LEFT arm. | ||
+ | [Again, some messages have been suppressed here for readability.] | ||
+ | <code lisp> | ||
+ | PP-TUT> (move-bottle ' | ||
+ | … | ||
+ | [(PICK-PLACE PICK-UP) INFO] 1621251445.989: | ||
+ | [(PICK-PLACE PICK-UP) INFO] 1621251445.993: | ||
+ | … | ||
+ | Grasping failed with RIGHT arm and LEFT-SIDE grasp | ||
+ | Retrying with RIGHT arm and RIGHT-SIDE grasp | ||
+ | [(PICK-PLACE PICK-UP) INFO] 1621251447.290: | ||
+ | [(PICK-PLACE PICK-UP) INFO] 1621251447.291: | ||
+ | … | ||
+ | Grasping failed with RIGHT arm and RIGHT-SIDE grasp | ||
+ | Retrying with RIGHT arm and FRONT grasp | ||
+ | [(PICK-PLACE PICK-UP) INFO] 1621251448.443: | ||
+ | [(PICK-PLACE PICK-UP) INFO] 1621251448.454: | ||
+ | … | ||
+ | Grasping failed with RIGHT arm and FRONT grasp | ||
+ | Retrying with RIGHT arm and BACK grasp | ||
+ | [(PICK-PLACE PICK-UP) INFO] 1621251449.628: | ||
+ | [(PICK-PLACE PICK-UP) INFO] 1621251449.650: | ||
+ | … | ||
+ | [(PICK-PLACE PICK-UP) INFO] 1621251450.450: | ||
+ | [(PICK-AND-PLACE GRIP) WARN] 1621251450.480: | ||
+ | Retrying | ||
+ | [(PICK-AND-PLACE GRIP) WARN] 1621251450.491: | ||
+ | #< | ||
+ | Grasping failed with RIGHT arm and BACK grasp | ||
+ | "No more grasp retries left : | ||
+ | #< | ||
+ | [(PICK-PLACE PICK-UP) INFO] 1621251450.606: | ||
+ | [(PICK-PLACE PICK-UP) INFO] 1621251450.608: | ||
+ | [(PICK-PLACE PICK-UP) INFO] 1621251451.151: | ||
+ | [(PICK-PLACE PICK-UP) INFO] 1621251451.234: | ||
+ | [(PICK-PLACE PICK-UP) INFO] 1621251451.235: | ||
+ | [(PICK-PLACE PLACE) INFO] 1621251451.709: | ||
+ | [(PICK-PLACE PLACE) INFO] 1621251451.892: | ||
+ | [(PICK-PLACE PLACE) INFO] 1621251451.988: | ||
+ | [(PICK-PLACE PLACE) INFO] 1621251452.027: | ||
+ | [(PICK-PLACE PLACE) INFO] 1621251452.074: | ||
+ | </ | ||
+ | {{: | ||
The robot has once again succeeded in grasping the object. | The robot has once again succeeded in grasping the object. | ||
Line 1087: | Line 1195: | ||
===== Defining a New Grasp ===== | ===== Defining a New Grasp ===== | ||
- | We discussed predefined possible grasps for a bottle in the previous section, which were 4 in number. Even though there are a lot of different possible grasps in real life, using only 4 limited | + | We discussed predefined possible grasps for a bottle in the previous section, which were 4 in number. Even though there are a lot of different possible grasps in real life, using only 4 limits |
- | There are two things to be carried out during any grasp pose: First is the translation | + | The grasps |
- | You car refer the image below again to see what axes correspond to what face of the object. | + | |
- | {{ : | + | |
- | For a front-left-diagonal | + | The gripper frame in '' |
+ | |||
+ | There are two things that need to be calculated for any grasp pose: the translation | ||
+ | |||
+ | To calculate the translation part, we'll define some offsets as parameters. The values are taken from the predefined grasp offset values in | ||
<code lisp> | <code lisp> | ||
Line 1104: | Line 1214: | ||
</ | </ | ||
- | As for the orientation, first, we need to know what the axes of the gripper | + | For the orientation, |
+ | A front-left-diagonal grasp means that the gripper | ||
+ | You car refer to the image below again to see what axes correspond to what face of the object. | ||
+ | {{ : | ||
- | To visualize this easily, open your left palm with the thumb perpendicular | + | The coordinate frame of a standard two-finger parallel gripper is defined as following: Z-axis points towards the object, X is defined along the translation of the fingers (the axis that connects the left and the right finger), and Y is defined according |
+ | |||
+ | {{: | ||
+ | |||
+ | Thus, to make the front-left diagonal grasp, the Z-axis of the gripper should point 45 degrees in between the X and Y-axes of the object. And since we are grasping across the Z-axis of the bottle | ||
<code lisp> | <code lisp> | ||
Line 1117: | Line 1234: | ||
(0 1 0))) | (0 1 0))) | ||
</ | </ | ||
- | The rotation matrix given above is the rotation matrix required to bring the gripper from the identity pose to the pose that we need to make a left-diagonal grasp. This is achieved by rotating the identity pose along the x-axis by +90 degrees first, and then rotating the new pose-along | + | The rotation matrix given above is the rotation matrix required to bring the gripper from the identity pose to the pose that we need to make a left-diagonal grasp. This is achieved by rotating the identity pose around |
Note:-Since sin(pi/4) = cos(pi/4), we have only defined one variable and used it interchangeably in the defined rotation matrix | Note:-Since sin(pi/4) = cos(pi/4), we have only defined one variable and used it interchangeably in the defined rotation matrix | ||
- | Now let's define the method that tie up all this and let ROS know that a new grasp is available | + | Now let's define the method that tie up all this and let CRAM know that a new grasp is available |
<code lisp> | <code lisp> | ||
(cram-object-interfaces: | (cram-object-interfaces: | ||
: | : | ||
: | : | ||
- | : | + | : |
- | : | + | :2nd-pregrasp-offsets `(, |
+ | : | ||
+ | :2nd-lift-offsets *lift-offset*) | ||
</ | </ | ||
- | The summary of this code is pretty simple, we have defined a grasp called '' | + | The summary of this code is pretty simple, we have defined a grasp called '' |
Now that we have defined our new grasp let's see it in action. First, spawn the bottle again and position the robot ready to pick up. | Now that we have defined our new grasp let's see it in action. First, spawn the bottle again and position the robot ready to pick up. | ||
Line 1171: | Line 1290: | ||
| | ||
</ | </ | ||
- | You will see that PR2 will successfully grasp the bottle from the grasp pose that we defined. | + | You will see that PR2 will successfully grasp the bottle from the grasp pose that we defined. Another useful documentation on this point is [[http:// |
+ | |||
+ | {{ : | ||
+ | |||
+ | To visualize the gripper coordinates, | ||
+ | |||
+ | <code lisp> | ||
+ | (visualize-coordinates (btr: | ||
+ | </ | ||
+ | where ''" | ||
+ | |||
+ | There is one caveat: PR2's gripper coordinate frame is not defined according to the common industry standard that we just described. But in order to keep our grasps applicable to any gripper, we define the grasps for the standard gripper and each robot that transforms the standard grasp into its own gripper frame. Therefore, even if we're defining the grasps for the PR2, whose gripper frames are defined as following: | ||
+ | |||
+ | {{ : | ||
+ | |||
+ | we still define them for a standard gripper coordinate frame. PR2 transforms our standard grasp into its own grasp under the hood through the CRAM hardware abstraction layer. | ||
- | {{: | ||
Line 1198: | Line 1331: | ||
===== Credits ===== | ===== Credits ===== | ||
- | This tutorial was created with the combined efforts of Amar Fayaz, Arthur Niedzwiecki and Gayane Kazhoyan. | + | This tutorial was created with the combined efforts of Amar Fayaz, Arthur Niedzwiecki, and Gayane Kazhoyan. |
+ | --></ | ||