Tile Detection

As mentioned, tile recognition without AR tags was dropped from the final system after it became apparent the tile pose would be difficult to estimate. For example, consider the two images shown below. The one on the left is a raw camera image, and the one on the right is the same image after processing:

  1. Convert the image to grayscale to reduce unnecessary computation.
  2. Convolve the image with a Gaussian kernel to smooth out high-frequency artifacts.
  3. Apply Sobel edge detection.
  4. Highlight points with high intensities in both the edge and grayscale images.

While the processing did noticeably reduce glare from reflective surfaces like the table, other objects like the whiteboard and floor could still be confused for tiles.

Raw image example

Filtered image example

Pick-and-place

The first demonstration on the left shows the pick-and-place module moving a square piece, which can be done fairly reliably, since the square has a large contiguous flat area. Note the initial sweep the left hand holding the camera performs to pick up the AR tags. So long as the board and tile are in correct approximate locations, this allows the robot to have no a priori knowledge of their exact locations. Also, the board and tile need not coincide with the base frame, which is used as a "world" reference frame all transforms are relative to.

To ensure the tile is placed in the correct orientation, Baxter will rotate its wrist while picking up the tile so its wrist frame coincides with the tile's AR tag frame. Baxter then rotates its wrist (and the tile AR tag) to coincide with the board frame (far right AR tag), with the addition of some multiple of 90 degrees.

The second video illustrates a common failure mode where the board is out of reach of the gripper.

Our project was able to recognize the tiles and the board, solve the puzzle, grasp the object, and rotate into the correct orientation consistently. However, due to the sensitivity of other tiles' AR tags to lighting and position, we were unable to reliably pick up the other types of tiles and complete the puzzle.