manipulation/deep_perception.html at master · siilats/manipulation · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
<!DOCTYPE html>

<html>

  <head>
    <title>Ch. 10 - Deep Perception for
Manipulation</title>
    <meta name="Ch. 10 - Deep Perception for
Manipulation" content="text/html; charset=utf-8;" />
    <link rel="canonical" href="http://manipulation.csail.mit.edu/deep_perception.html" />

    <script src="https://hypothes.is/embed.js" async></script>
    <script type="text/javascript" src="chapters.js"></script>
    <script type="text/javascript" src="htmlbook/book.js"></script>

    <script src="htmlbook/mathjax-config.js" defer></script>
    <script type="text/javascript" id="MathJax-script" defer
      src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-chtml.js">
    </script>
    <script>window.MathJax || document.write('<script type="text/javascript" src="htmlbook/MathJax/es5/tex-chtml.js" defer><\/script>')</script>

    <link rel="stylesheet" href="htmlbook/highlight/styles/default.css">
    <script src="htmlbook/highlight/highlight.pack.js"></script> <!-- http://highlightjs.readthedocs.io/en/latest/css-classes-reference.html#language-names-and-aliases -->
    <script>hljs.initHighlightingOnLoad();</script>

    <link rel="stylesheet" type="text/css" href="htmlbook/book.css" />
  </head>

<body onload="loadChapter('manipulation');">

<div data-type="titlepage" pdf="no">
  <header>
    <h1><a href="index.html" style="text-decoration:none;">Robotic Manipulation</a></h1>
    <p data-type="subtitle">Perception, Planning, and Control</p>
    <p style="font-size: 18px;"><a href="http://people.csail.mit.edu/russt/">Russ Tedrake</a></p>
    <p style="font-size: 14px; text-align: right;">
      &copy; Russ Tedrake, 2020-2023<br/>
      Last modified <span id="last_modified"></span>.</br>
      <script>
      var d = new Date(document.lastModified);
      document.getElementById("last_modified").innerHTML = d.getFullYear() + "-" + (d.getMonth()+1) + "-" + d.getDate();</script>
      <a href="misc.html">How to cite these notes, use annotations, and give feedback.</a><br/>
    </p>
  </header>
</div>

<p pdf="no"><b>Note:</b> These are working notes used for <a
href="http://manipulation.csail.mit.edu/Fall2023/">a course being taught
at MIT</a>. They will be updated throughout the Fall 2023 semester.  <!-- <a
href="https://www.youtube.com/channel/UChfUOAhz7ynELF-s_1LPpWg">Lecture  videos are available on YouTube</a>.--></p>

<table style="width:100%;" pdf="no"><tr style="width:100%">
  <td style="width:33%;text-align:left;"><a class="previous_chapter" href=segmentation.html>Previous Chapter</a></td>
  <td style="width:33%;text-align:center;"><a href=index.html>Table of contents</a></td>
  <td style="width:33%;text-align:right;"><a class="next_chapter" href=rl.html>Next Chapter</a></td>
</tr></table>

<script type="text/javascript">document.write(notebook_header('deep_perception'))
</script>
<!-- EVERYTHING ABOVE THIS LINE IS OVERWRITTEN BY THE INSTALL SCRIPT -->
<chapter style="counter-reset: chapter 9"><h1>Deep Perception for
Manipulation</h1>

  <p>In the previous chapter, we discussed deep-learning approaches to object
  detection and (instance-level) segmentation; these are general-purpose tasks
  for processing RGB images that are used broadly in computer vision. Detection
  and segmentation alone can be combined with geometric perception to, for
  instance, estimate the pose of a known object in just the segmented point
  cloud instead of the entire scene, or to run our point-cloud grasp selection
  algorithm only on the segmented point cloud in order to pick up an object of
  interest.</p>

  <p>One of the most amazing features of deep learning for perception is that
  we can pre-train on a different dataset (like ImageNet or COCO) or even a
  different task and then fine-tune on our domain-specific dataset or task. But
  what are the right perception tasks for manipulation?  Object detection and
  segmentation are a great start, but often we want to know more about the
  scene/objects to manipulate them. That is the topic of this chapter.</p>

  <p>There is a potential answer to this question that we will defer to a later
  chapter: learning end-to-end "visuomotor" policies, sometimes affectionately
  referred to as "pixels to torques". Here I want us to think first about how
  we can combine a deep-learning-based perception system with the powerful
  existing (model-based) tools that we have been building up for planning and
  control.</p>

  <todo> maybe a system diagram which includes perception, planning and
  control? So far we’ve had two version - grasp candidates and/or object pose…
  </todo>

  <p>I'll start with deep-learning versions of two perception tasks we've
  already considered: object pose estimation and grasp selection.</p>

  <!-- representations of uncertainty should be a theme i'll carry throughout.
  make analogy to object detection outputs (e.g. in mask-rcnn) -->

  <section><h1>Pose estimation</h1>

    <!-- some notes in 2021 lecture 11 ithoughts. -->

    <!-- probably reconstruction loss (for instance, as used by the scoring
    function in BOP) is better than e.g. quaternion loss.  -->

    <!-- NOCS in here? -->

  </section>

  <section><h1>Grasp selection</h1>

  </section>

  <section><h1>(Semantic) Keypoints</h1>

    <!-- affordances -->
    <!-- category level -->

    <!-- wei's keypoint labeling tool -->

    <todo>Example: corner keypoints for boxes.  (also pose+shape estimation
    from keypoints?)</todo>
  </section>

  <section><h1>Dense Descriptors</h1>

    <!-- dense correspondence -->
    <!-- anthony's neural descriptor fields -->

  </section>

  <section><h1>Task-level state</h1>

    <!-- tcc.  chris paxton "red block is on the blue block"-->

  </section>

  <!-- Can I do a putting it all together section here?? Or in each section? -->

  <section><h1>Other perceptual tasks / representations</h1>

    <p>My coverage above is necessarily incomplete and the field is moving
    fast. Here is a quick "shout out" to a few other very relevant ideas.</p>

    <p>More coming soon...</p>
    <!-- shape estimation / completion -->
    <!-- transporter nets -->
    <!-- estimating dynamic parameters (mass / friction / etc) -->

    <!-- danny's compositional nerf? -->
    <!-- probably defer learning dynamics models to another chapter -->
    <!-- M0m from lpk -->
    <!-- Particles? or deformables more generally… -->

  </section>

  <!-- maybe add a “looking forward” section?  Cracking an egg, etc.  what
  should the perception system output?  How do we write the planning/control
  system.  This is where full-stack deep learning approaches are winning.  But
  i don’t know that being data-driven (or deep) is fundamental here.

  Maybe future chapter on state representations?  In the model-learning
  section? -->

  <section id="exercises"><h1>Exercises</h1>
    <exercise id="don_contrastive"><h1>Deep Object Net and Contrastive Loss</h1>
      <p>In this problem you will further explore Dense Object Nets, which were introduced in lecture. Dense Object Nets are able to quickly learn consistent pixel-level representations for visual understanding and manipulation. Dense Object Nets are powerful because the representations they predict are applicable to both rigid and non-rigid objects. They can also generalize to new objects in the same class and can be trained with self-supervised learning. For this problem you will work in <script>document.write(notebook_link('deep_perception', d=deepnote, link_text='this notebook', notebook='contrastive'))</script> to first implement the loss function used to train Dense Object Nets, and then predict correspondences between images using a trained Dense Object Net.</p>
    </exercise>
  </section>

</chapter>
<!-- EVERYTHING BELOW THIS LINE IS OVERWRITTEN BY THE INSTALL SCRIPT -->

<div id="references"></div>

<table style="width:100%;" pdf="no"><tr style="width:100%">
  <td style="width:33%;text-align:left;"><a class="previous_chapter" href=segmentation.html>Previous Chapter</a></td>
  <td style="width:33%;text-align:center;"><a href=index.html>Table of contents</a></td>
  <td style="width:33%;text-align:right;"><a class="next_chapter" href=rl.html>Next Chapter</a></td>
</tr></table>

<div id="footer" pdf="no">
  <hr>
  <table style="width:100%;">
    <tr><td><a href="https://accessibility.mit.edu/">Accessibility</a></td><td style="text-align:right">&copy; Russ
      Tedrake, 2023</td></tr>
  </table>
</div>


</body>
</html>