Robot Media

 

Co-Lab: Robot Media Studio
School of Media Studies
The New School
Spring 2014

Instructor: Peter Asaro asarop AT newschool.edu
TAs: Eva Jacobus ejacobus AT gmail.com
Thomas Charging Hawk chart058 AT newschool.edu
Sepand Ansari ansas091 AT newschool.edu

Time: Thursday, 7:00 - 9:45 pm
Location: 306 University Center, 63 Fifth Ave.

Course webpage is here: http://peterasaro.org/courses/2014Studio.html

Course blog is here: http://robotstudio2014.wordpress.com

Course Description

This course explores the potential of robotic media platforms and computer vision for cinematic expression. As a Co-Lab, students will work in collaborative groups that will utilize the latest robotic and computer vision technologies to make short films. The first half of the semester will consist of an introduction to these technologies and in-class group exercises that will familiarize you with advanced digital camera techniques, and robotic camera control. These camera techniques and platforms will include advanced computer vision techniques such as Time-lapse, High Dynamic Range Imagery, Motion Magnification, Facial Recognition, Object Tracking, Optic Flow, and others, as well as 3D active-vision systems such as the Xbox Kinect. Robotic camera control will be explored through the use of remote-operated and computer-controlled servo-driven cameras, including RC vehicles, mobile robot dollys, robotic arms, and quadrotors (drones). We will explore a variety of control methods from remote control to pre-programmed and 3D model-driven control, as well as how these can be combined with vision techniques for the interactive control of cameras. We will also explore how these cinematographic techniques relate to visual storytelling and expression. In the second half of the course, students will pursue projects of their own design in groups, with the goal of producing a short experimental or narrative video utilizing these techniques. Previous programming experience is not required, but students will be expected to learn and apply basic programming skills in this course, and will be introduced to programming languages such as Python and Java, and programming platforms and libraries such as Arduino, ROS and OpenCV.

OFFICE HOURS: By Appointment

Please email me to setup an appointment.

 

COURSE REQUIREMENTS & GRADING:

Class Attendance and Participation: 20%
In-Class Studio Exercises: 40%
Final Project: 40%

 

Class Attendance and Participation: 20%

You are expected to have thoroughly and thoughtfully read the assigned texts and to have prepared yourself to contribute meaningfully to the class discussions. Your participation will be evaluated in terms of both quantity and quality.

As this is a studio course, and a co-lab, regular attendance is essential. Because you will be doing group projects, if you do not come to class your fellow group members will be at a disadvantage. You will be permitted two excused absences (you must notify me of your inability to attend before class, via email or phone). Any subsequent absences and any un-excused absences will adversely affect your grade.

In-Class Studio Exercises: 40%

For the first half of the class, studio time will be devoted to in-class group exercises that are designed to teach you fundamental concepts, techniques and how to apply these towards and aesthetic goal. The results of your exercise should be posted to the course blog, along with a description. Generally, this will be a short video, program/software, or both. You should also note the names of the group members, and their roles, in your description.

You will be required to create an account on WordPress, and send me an email with the EMAIL ADDRESS used to creat the account, so that you can be added as authors for the collective course blog. Everyone will be posting to a common blog page, and this will be readable by your classmates, but access will be limited to only other class members, and not the whole internet. When writing and making comments, you are expected to treat other students with the same respect and courtesy as you should in the classroom. You are also expected to respect rules of academic integrity, research ethics, and copyright when posting to the blog.

At the start of each class, we will review questions and concerns from the previous week, as well as review and critique the films that you have produced.

 

Final Project: 40%
(Proposal=10%, Paper=30%)

Proposals Due: April 17
Paper Due: May 15

There will be no final exam. Instead, your Final Project will be due, May 15th at 7:00PM. If that time will not work for you, you need to make other arrangements by May 8th at the latest.

The Final Project could take different forms, but should contain two key elements: robots and video/media. Final Projects will be evaluated in terms of technological and aesthetic innovation and quality. Ideally, aesthetics goals should drive the technology.

Some examples of good Final Projects:

Use of robotic platforms and/or computer vision for a short film

Use of robots in a short film

Development of advanced tools/technology for robotic/computational filmmaking

Development of an interactive media project that employs robotics and/or computer vision

A research paper that explores some advanced aspect of robotic media

In addition to submitting the final video clip, software and/or hardware that you created, you will also be asked to write-up an extended blog entry or webpage that describes how you created and used technologies to create your final project. These will be posted to the course blog. There will also be an event in the Media Center towards the end of the semester, where we will show all of the films that students have created, and you will be asked to present them to the audience. You are also encouraged to submit your films to the Robot Film Festival, which will be held in New York in July.

GROUPS & QUESTIONNAIRE

Because we have limited number of various robotics technologies and computers, and in an effort to build teamwork early, all of the studio projects will be done in groups of 3 or 4 people (depending on the number of students). On the first day of class, you will be asked to fill out a questionnaire to identify your skills and interests. You will then be put in a group which complements those skills. The goal is to have at least one person with film/video experience in each group, and at least one person with some programming experience in each group.

You will be in the same group from all of the in-class exerciese over the first 10 weeks. If one or more people miss a class, we may adjust the groups as needed.

You are free to form you own groups for the Final Project, or to continue with your exercise group.

EQUIPMENT

The IFP Made in NY Media Center does not have computers for us to use, but does have WiFi. Because we will need Linux (Ubuntu Studio 12.04) for the in-class exercises, there will be laptops provided each week. Depending on the number of students, each group will get at least 1 laptop.

For various exercises, there will be more limited amounts of equipment. In these cases there will be a sign-up sheet in advance. So if your group is eager to work with a particular item, be sure to sign up early for it.

Equipment will be signed out at the beginning of class, and signed back in at the end, during the period of exercises. After that, or by special arrangement, equipment can be checkout out for longer periods to complete the Final Project outside of class.

SAFETY

We will be programming real robots and drones which move around in the world. They are mostly small and safe, but you should always use care and caution when working with them to protect yourself, classmates, and the public.

Don't run code if you are unsure of what it will do, or if you think it might be hazardous!

Anyone who behaves recklessly or endangers others will not be allowed to work with the robots any more.

We will review more detailed safety considerations when we start flying the quadrotors.

READINGS

All readings will be available electronically, via the web, in PDF, MS Word, HTML, or similar format.

FILMS & DVDS

Most of the films and TV programs that will be assigned are available from a variety of sources. Many are available through the New School Library on DVD. In addition, they can be purchased from most book or video stores, rented from most video shop, or found through Netflix. For the videos which cannot be obtained easily in these ways, other means will be provided for you to view these films prior to class.

 

 

Introduction

Special Event: January 29
Robot Dialogues Panel

This event will feature lively dialogues between human interlocutors and artificial beings -- bots, robots, and telepresences. The panelists will discuss the nature of subjectivity, identity and sociality -- and the role of dialogue and communication in creating and shaping these. The "Robot Dialogues" panel will showcase technological innovations in artificial intelligence and expression, but it will also provide an opportunity to historicize development in the field and reflect on its implications. Participants will include Lynn Hershman Leeson (Dorothy H. Hirshon Director-in-Residence), Heather Knight (Carnegie Mellon, Marilyn Monrobot Labs), and Wendell Wallach (The Center for Bioethics, Yale University), with Peter Asaro (School of Media Studies faculty) as moderator.

http://events.newschool.edu/event/the_rights_of_robots

Week 1: January 30
Course Introduction

Course and Syllabus Overview
Questionnaire

Studio:

How to use Ubuntu and the Linux terminal.

http://www.ubuntu.com/desktop/take-the-tour
http://ubuntustudio.org/tour/
https://help.ubuntu.com/community/UsingTheTerminal

Programming with Processing

http://processing.org/overview/
http://www.openprocessing.org/

Reference: Learning Processing, Daniel Schiffman

Assignment:

Do the Fun Programming tutorials, starting with "2. Download Processing. Use point() and line()", and work through as many of these as you have time for: 2-4, 9-12, 18-23, 29-32, 35-41, 47, 54, 74, 94, 104, 121
http://funprogramming.org/

If you prefer the textbook format, feel free to work through the Schiffman book above instead.
If you are already familiar with the Processing language, come talk to the TAs or professor.
Once you get comfortable with the language, or finish the exercises, trying writing your own original program, and post it to the blog!

Required Reading:

Bill Gates (2007) "A Robot in Every Home: The leader of the PC revolution predicts that the next hot field will be robotics," Scientific American, January 2007.
Hans Moravec (2009) "Rise of the Robots--The Future of Artificial Intelligence," Scientific American, March 23, 2009
Watch: Rodney Brooks says robots will invade our lives, Ted Talk 2003, 19 min.

Recommended:

Watch: BBC Horizon, Where's My Robot?, BBC, 2008, 50 min.
Watch: Dennis Hong: My seven species of robot, Ted Talk 2009, 16 min.
http://processing.org/
http://processing.org/tutorials/
http://processingjs.org/
http://processingjs.org/learning/
http://www.codecademy.com/tracks/python

Part I: Sensing

Week 2: February 6
The Sensor: Inside the Camera

Studio:

Camera Hacking
OpenCV Processing
Image Segmentation, Edge detection, Object/face/gesture recognition/tracking
https://github.com/atduskgreg/opencv-processing
http://urbanhonking.com/ideasfordozens/2013/07/10/announcing-opencv-for-processing/

Assignment:

You should already have Processing installed, if not: Install Processing

Install OpenCV for Processing:

1. In the Processing IDE development environment, go to “Import Library” under the “Sketch” menu

2. Search for ‘OpenCV’

3. Install “OpenCV for Processing” by Greg Borenstein.

Follow the Edge Detection and Face Detection Exercises

Required:

Listen: http://www.radiolab.org/story/211119-colors/
http://en.wikipedia.org/wiki/High_dynamic_range_imaging
http://en.wikipedia.org/wiki/Time-lapse_photography

Recommended:

Sebastian Thrun (2010) "Toward robotic cars," Communications of the ACM, Volume 53 Issue 4, April 2010, pp. 99-106.
Erico Guizzo, (2011) "How Google's Self-Driving Car Works," IEEE Spectrum Automation Blog, October 18, 2011.
Watch: The Evolution of Self-Driving Cars, YouTube, 20 min.
Explore:
Navlab: The Carnegie Mellon University Navigation Laboratory
Ernst D. Dickmanns (1997) "Vehicles Capable of Dynamic Vision," Proceedings IJCAI, 1997.
Watch: Ernst Dickmanns (2011) Keynote Lecture, YouTube, 77 min.

Week 3: February 13
Cancelled Due to Snow

Week 4: February 20
Seeing Like Robot

Studio:

Follow these instructions: Studio3

HDR with Canon Powershot: http://www.ragingsloth.com/photography/S95/s95hdrhowto.html
HDR with Canon EOS: http://www.youtube.com/watch?v=pLOME_WVm-A
http://www.learn.usa.canon.com/resources/articles/2012/5d3_hdr_capabilities.shtml

Required:

OpenCV http://opencv.org/

Recommended:

Intro to HDR (1 hour 30 min) http://www.youtube.com/watch?v=v3CPavb2NWs
HDR with RAW file in Photoshop http://www.youtube.com/watch?v=YY7cYyXjQ-I
http://en.wikipedia.org/wiki/CCD_camera

http://en.wikipedia.org/wiki/Active_pixel_sensor

http://en.wikipedia.org/wiki/Digital_photography

Watch:
https://vimeo.com/27582408
http://en.wikipedia.org/wiki/Camera_lens

http://photo.net/learn/optics/lensTutorial

Part II: Actuation

Week 5: February 27
Drones I: Remote Piloting

Studio:

Prior to class:

If you have an iPhone, iPad or Android device, please download and install this app from iTunes, GooglePlay or the Android Market: AR.FreeFlight

https://itunes.apple.com/us/app/freeflight/id373065271?mt=8
http://ardrone2.parrot.com/apps/

It is also strongly recommended that you download this simulator, and practice with it before class: ARDrone Sim
Note: This app costs $1.99 (iOS) or $2.39 (Android)

https://play.google.com/store/apps/details?id=com.DDI.ARDroneSim&hl=en
https://itunes.apple.com/us/app/ardrone-sim/id522378610?mt=8

For the Simulator options choose: AR.Drone 2.0, Indoor Hull, Standard Battery

Press the Green Takeoff button to launch and hover, the Green Landing button for an automatic controlled landing.

The Left joystick tilts the drone, and causes it to move laterally.

The Right joystick causes the drone to go up and down, and rotate clockwise or counter clockwise.

The Camera icon will shift your viewing perspective.

If you crash, it will simulate damage, so you may need to go to Settings on the main screen and Restore Defaults to fly straight again.

Required:

Crane & Aerial shots
Paul Virilio, "Cinema isn't I See, it's I Fly," in War and Cinema: The Logistics of Perception, London: Verso, 1989: 11-30.
Beth Herst, "Review: The Disembodied Eye," PAJ: A Journal of Performance and Art, Vol. 24, No. 1, Intelligent Stages: Digital Art
and Performance (Jan., 2002), pp. 122-126.

Recommended:

ROS & Parrot AR Drone
http://robohub.org/up-and-flying-with-the-ar-drone-and-ros-getting-started/
http://www.ros.org/news/2010/10/ros-interface-for-the-parrot-ardrone.html
http://vision.in.tum.de/data/software/tum_ardrone
http://www.diydrones.com/profiles/blogs/new-ros-package-for-ar-drone
Watch: "Spidercam US Open 2010 Opening," YouTube, 1 min.
Watch: "Spidercam European swimming Championships Budapest," YouTube, 3 min.
Watch: "Spidercam Real Madrid Barcelona El Classico," YouTube, 5 min.
Watch: "How Does Skycam Work?" YouTube, 2 min.

Week 6: March 6
Drones II: Programmed Control

Studio:

ROS & Parrot, Processing & Parrot
Studio 4

Required:

Watch: Omer Fast (2011)5,000 Feet is the Best, 30 min. [select from "Online Preview" menu]
Watch: Faultlines (2011)"Robot Wars," Al Jazeera English, 30 min.

Recommended:

Watch: DocZone, (2011) Remote Control War, CBC, 45 min. (only viewable in Canada)

Week 7: March 13
Sensing the 3rd Dimension

Studio:

Kinect & JNI

Studio 5

Required:

Open NI http://www.openni.org/about/
J. J. Gibson
http://www.pointclouds.org/documentation/tutorials/using_kinfu_large_scale.php

Recommended:

Time-of-Flight Camera http://en.wikipedia.org/wiki/Time-of-flight_camera
Bullet Time http://en.wikipedia.org/wiki/Bullet_time
Free Viewpoint Television http://en.wikipedia.org/wiki/Free_viewpoint_television

Week 8: March 20
Point-Clouds

Studio:

Kinect and OpenNI
Capturing 3D data and replaying it from different angles.

Studio 6

Required:

Microsoft Robotics Studio
Willow Garage ROS (Robot Operating System)
ROS & Blender
http://www.openrobots.org/morse/doc/stable/morse.html
http://opengrasp.sourceforge.net

Recommended:

ROShttp://www.ros.org/wiki/
ROS & Arduino http://wiki.ros.org/rosserial
http://www.ros.org/news/2013/03/new-package-ros-arduino-bridge.html
http://wiki.ros.org/arbotix
Watch: Craig Gillespie, Lars and the Real Girl, 2007, 106 min.
Watch: David Hanson: Robots that "show emotion", 2010 TED talk, 5 min.
Watch: Caleb Chung plays with Pleo, 2010 TED talk

 

Spring Break March 22-30
Week of March 27
NO CLASS

Part III: Group Projects!

Week 9: April 3
Studio Day
Project Proposals Due

Week 10: April 10
Robotic Art

Required:

Steve Dixon, "Metal Performance: Humanizing Robots, Returning to Nature, and Camping About," and "A Brief History of Robots and Automata," TDR: The Drama Review, Volume 48, Number 4 (T 184), Winter 2004, pp. 15-46.
Eduardo Kac, "Foundation and Development of Robotic Art," Art Journal, Vol. 56, No. 3, Digital Reflections: The Dialogue of Art and Technology, (Autumn, 1997), pp. 60-67.
Edward A. Shanken, "Tele-Agency: Telematics, Telerobotics, and the Art of Meaning,"Art Journal, Vol. 59, No. 2 (Summer, 2000), pp. 65-77.
Watch: Alex Rivera, Sleep Dealer, Likely Story, 2008, 90 min.

Recommended:

Watch: Jonathan Mostow, Surrogates, Touchstone Pictures, 2009, 89 min.

Week 11: April 17
Robots, Drones, Surveillance & Privacy
Guest Lecture: Nina & Georgi Tushev, Tushev's Aerials

Required:

Ryan Calo (2012) "Robots and Privacy," Robot Ethics: The Ethical and Social Implications of Robotics, MIT Press, pp. 187-201.
Ryan Calo (2011) "The Drone as Privacy Catalyst," Stanford Law Review Online, 29, December 12, 2011, pp. 64.
American Civil Liberties Union (2011) Protecting Privacy From Aerial Surveillance: Recommendations for Government Use of Drone Aircraft, ACLU, December 2011.

Recommended:

"Unblinking eyes in the sky," The Economist, March 3, 2012.
Watch: "Protester Films Polish Riots Using Drone 2011" YouTube, 3 min.
Meghan Keneally (2012) "Drone plane spots a river of blood flowing from the back of a Dallas meat packing plant," Daily Mail, January 24, 2012.
Watch: "Citizens Shoot Down Animal Rights Group's Surveillance Drone" The Blaze, 3 min.

Week 12: April 24
Studio Day

Required:

Belton, John. "The Bionic Eye: Zoom Esthetics." Cineaste (1980): 20-27.

Brown, Garrett, "It's a Bird... It's a Plane... It's a... Camera!", American Film (Archive: 1975-1992) 8.10 (Sept. 1, 1983): 59-61.

 

Week 13: May 1
Robots in Film
Guest Lecture: Alex Rivera, http://alexrivera.com/

Required:

Watch: Alex Rivera, Sleep Dealer, Likely Story, 2008, 90 min.

http://alexrivera.com/films/

Recommended:

Watch: http://www.ted.com/talks/guy_hoffman_robots_with_soul.html
http://en.wikipedia.org/wiki/Uncanny_valleyBrian Fung (2012) "The Uncanny Valley: What Robot Theory Tells Us About Mitt Romney," The Atlantic, January 31, 2012.
Watch: Errol Morris, Fast, Cheap and Out of Control, Sony Pictures Classics, 1997, 80 min.

Week 14: May 8
Final Project Presentations

Week 15: May 15
NO CLASS MEETING
Final Projects Due

Robot Film Festival: July

http://robotfilmfestival.com/submit/