Co-Lab: Robot Media Studio
School of Media Studies
The New School
Spring 2020

Instructor: Peter Asaro asarop AT newschool.edu
Teaching Assistant: Jill Shah shahj225 AT newschool.edu
UAV FAA Consultant: Will Chatham wtchatham AT gmail.com
Course Numbers: NMDS 5283 CRN 7332
Time: Monday, 7:00 - 9:40 pm
Location: 6 East 16th St., Room 603

Course webpage is here: http://peterasaro.org/courses/2020Studio.html

Course blog is here: http://robotstudio2020.wordpress.com

Course Description

This course explores the potential of robotic media platforms and computer vision for cinematic expression. As a Co-Lab, students will work in collaborative groups that will utilize the latest robotic and computer vision technologies to make short films. The first half of the semester will consist of an introduction to these technologies and in-class group exercises that will familiarize you with advanced digital camera techniques, and robotic camera control. These camera techniques and platforms will include advanced computer vision techniques such as Time-lapse, High Dynamic Range Imagery, Motion Magnification, Facial Recognition, Object Tracking, Optic Flow, and others, as well as 3D active-vision systems such as the Xbox Kinect. Robotic camera control will be explored through the use of remote-operated and computer-controlled servo-driven cameras, including RC vehicles, mobile robot dollys, robotic arms, and quadrotors (drones). We will explore a variety of control methods from remote control to pre-programmed and 3D model-driven control, as well as how these can be combined with vision techniques for the interactive control of cameras with gestures. We will also explore how these cinematographic techniques relate to visual storytelling and expression. In the second half of the course, students will pursue projects of their own design in groups, with the goal of producing a short experimental or narrative video utilizing these techniques. Previous programming experience is not required, but students will be expected to learn and apply basic programming skills in this course, and will be introduced to programming languages such as Processing, Python and Java, and programming platforms and libraries such as Arduino, ROS and OpenCV.

OFFICE HOURS: By Appointment

Please email me to setup an appointment.

COURSE REQUIREMENTS & GRADING:

Class Attendance and Participation: 20%
In-Class Studio Exercises: 40%
Final Project: 40%

Class Attendance and Participation: 20%

You are expected to have thoroughly and thoughtfully read the assigned texts and to have prepared yourself to contribute meaningfully to the class discussions. Your participation will be evaluated in terms of both quantity and quality.

As this is a studio course, and a co-lab, regular attendance is essential. Because you will be doing group projects, if you do not come to class your fellow group members will be at a disadvantage. You will be permitted two excused absences (you must notify me and your group partners of your inability to attend before class, via email or phone). Any subsequent absences and any un-excused absences will adversely affect your grade.

In-Class Studio Exercises: 40%

For the first half of the class, studio time will be devoted to in-class group exercises that are designed to teach you fundamental concepts, techniques and how to apply these towards an aesthetic goal. Your results for each exercise should be posted to the course blog, along with a description. Generally, this will be a short video, program/software, or both. You should also note the names of the group members, and their roles, in your description. This should be done by the end of each class, but you may edit or update it later. You should also create or own an account on YouTube or Vimeo to uplaod any videos that you produce.

You will be required to create an account on WordPress, and send me an email with the EMAIL ADDRESS used to create the account, so that you can be added as authors for the collective course blog. Everyone will be posting to a common blog page, and this will be publicly accessible. When writing and making comments, you are expected to treat other students with the same respect and courtesy as you should in the classroom. You are also expected to respect rules of academic integrity, research ethics, and copyright when posting to the blog.

At the start of each class, we will review questions and concerns from the previous week, as well as review and critique the films that you have produced.

Final Project: 40%
(Proposal=10%, Paper=30%)

Proposals Due: April 17
Paper Due: May 15

There will be no final exam. Instead, your Final Project will be due, May 13th at 7:00PM. If that time will not work for you, you need to make other arrangements by May 4th at the latest.

The Final Project could take different forms, but should contain two key elements: robots and video/media. Final Projects will be evaluated in terms of technological and aesthetic innovation and quality. Ideally, aesthetics goals should drive the technology.

Some examples of good Final Projects:

Use of robotic platforms and/or computer vision for a short film

Use of robots in a short film

Development of advanced tools/technology for robotic/computational filmmaking

Development of an interactive media project that employs robotics and/or computer vision

A research paper that explores some advanced aspect of robotic media

Check out previous class projects here:

http://robotstudio2018.wordpress.com

http://robotstudio2017.wordpress.com

http://robotstudio2014.wordpress.com

In addition to submitting the final video clip, software and/or hardware that you created, you will also be asked to write-up an extended blog entry or webpage that describes how you created and used technologies to create your final project. These will be posted to the course blog. There may also be opportunities to submit films to festivals and events..

GROUPS & QUESTIONNAIRE

Because we have limited number of various robotics technologies and computers, and in an effort to build teamwork early, all of the studio projects will be done in groups of 3 or 4 people (depending on the number of students). On the first day of class, you will be asked to fill out a questionnaire to identify your skills and interests. You will then be put in a group which complements those skills. The goal is to have at least one person with film/video experience in each group, and at least one person with some programming experience in each group.

You will be in the same group from all of the in-class exerciese over the first 10 weeks. If one or more people miss a class, we may adjust the groups as needed.

You are free to form you own groups for the Final Project, or to continue with your exercise group.

EQUIPMENT

We will need fast computers for the in-class exercises. You may use your own, or there will be Mac laptops available for checkout form the equipment center each week. Depending on the number of students, each group will get at least 1 laptop. Some exercises can only be done on a PC, and we will bring PCs for that, but most will work on the University Macs.

Equipment will be signed out at the beginning of class, and signed back in at the end, during the period of exercises. After that, or by special arrangement, equipment can be checkout out for longer periods to complete the Final Project outside of class.

SAFETY

We will be programming real robots and drones which move around in the world. They are mostly small and safe, but you should always use care and caution when working with them to protect yourself, classmates, and the public.

Don't run code if you are unsure of what it will do, or if you think it might be hazardous!

Anyone who behaves recklessly or endangers others will not be allowed to work with the robots any more.

We will review more detailed safety considerations when we start flying the quadrotors.

READINGS

All readings will be available electronically, via the web, in PDF, MS Word, HTML, or similar format.

FILMS & DVDS

Most of the films and TV programs that will be assigned are available from a variety of sources. Many are available through the New School Library on DVD. In addition, they can be purchased from most book or video stores, rented from most video shop, or found through Netflix. For the videos which cannot be obtained easily in these ways, other means will be provided for you to view these films prior to class.

Grading Standards

A Work of exceptional quality
A- Work of high quality
B+ Very good work
B Good work; satisfies course requirements
Satisfactory completion of a course is considered to be a grade of B or higher.
B- Below-average work
C+ Less than adequate work
C Well below average work
C- Poor work; lowest possible passing grade
F Failure
GM Grade missing for an individual

Grades of D are not used in graduate level courses.

Grade of W
The grade of W may be issued by the Office of the Registrar to a student who officially withdraws from a course within the applicable deadline. There is no academic penalty, but the grade will appear on the student transcript. A grade of W may also be issued by an instructor to a graduate student (except at Parsons and Mannes) who has not completed course requirements nor arranged for an Incomplete.

Grade of Z
The grade of Z is issued by an instructor to a student who has not attended or not completed all required work in a course but did not officially withdraw before the withdrawal deadline. It differs from an "F," which would indicate that the student technically completed requirements but that the level of work did not qualify for a passing grade.

Grades of Incomplete
The grade of I, or temporary incomplete, may be granted to a student under unusual and extenuating circumstances, such as when the student's academic life is interrupted by a medical or personal emergency. This mark is not given automatically but only upon the student's request and at the discretion of the instructor. A Request for Incomplete form must be completed and signed by student and instructor. The time allowed for completion of the work and removal of the "I" mark will be set by the instructor with the following limitations: [You should include one the following standards, depending on the level of your course].

Undergraduate students: Work must be completed no later than the seventh week of the following fall semester for spring or summer term incompletes and no later than the seventh week of the following spring semester for fall term incompletes. Grades of "I" not revised in the prescribed time will be recorded as a final grade of "F" by the Registrar's Office.

Graduate students: Work must be completed no later than one year following the end of the class. Grades of "I" not revised in the prescribed time will be recorded as a final grade of "N" by the Registrar's Office.

Electronic Devices
The use of electronic devices (phones, tablets, laptops, cameras, etc.) is permitted when the device is being used in relation to the course's work. All other uses are prohibited in the classroom and devices should be turned off before class starts.

Academic Honesty and Integrity
Compromising your academic integrity may lead to serious consequences, including (but not limited to) one or more of the following: failure of the assignment, failure of the course, academic warning, disciplinary probation, suspension from the university, or dismissal from the university.

Students are responsible for understanding the University's policy on academic honesty and integrity and must make use of proper citations of sources for writing papers, creating, presenting, and performing their work, taking examinations, and doing research. It is the responsibility of students to learn the procedures specific to their discipline for correctly and appropriately differentiating their own work from that of others. The full text of the policy, including adjudication procedures, is found at http://www.newschool.edu/policies/# Resources regarding what plagiarism is and how to avoid it can be found on the Learning Center's website: http://www.newschool.edu/university-learning-center/student-resources/

The New School views "academic honesty and integrity" as the duty of every member of an academic community to claim authorship for his or her own work and only for that work, and to recognize the contributions of others accurately and completely. This obligation is fundamental to the integrity of intellectual debate, and creative and academic pursuits. Academic honesty and integrity includes accurate use of quotations, as well as appropriate and explicit citation of sources in instances of paraphrasing and describing ideas, or reporting on research findings or any aspect of the work of others (including that of faculty members and other students). Academic dishonesty results from infractions of this "accurate use". The standards of academic honesty and integrity, and citation of sources, apply to all forms of academic work, including submissions of drafts of final papers or projects. All members of the University community are expected to conduct themselves in accord with the standards of academic honesty and integrity. Please see the complete policy here.

More on your Intellectual Property Rights

Introduction

Week of January 27: NO CLASS!

Week 1: February 3
Course Introduction

Course and Syllabus Overview
Questionnaire

Watch: Robot Media Videos:

https://www.youtube.com/watch?v=zINbyGTOFls

Marmalade

Bot & Dolly "Box"

Behind the Scenes: Bot & Dolly "Box"

Sun Yuan & Peng Yu's "Can't Help Myself"

Behind the Scenes: RC Car chase

OK Go "I Won't Let You Down" music video

#JetLagPonyClub

Neill Blomkamp "Tetra Vaal"

Spike Jonez "I'm Here"

Self-guided Studio Assignment:
Studio 1: Fundamentals of Programming

Part I: Sensing

Week 2: Febuary 10
The Sensor: Inside the Camera

How Digital Cameras Work How Digital Cameras Work on Astropix

Watch (in class):

The History and Science of Lenses

The Science of Camera Sensors

Capturing Digital Images

A Bit of History on Data

How Blurs & Filters Work

Finding the Edges

Canny Edge Detectors

Geometric Face Detection

More about Camera Hacking
OpenCV Processing
Image Processing and Feature Detection in OpenCV
Canny Edge Dector in OpenCV
https://github.com/atduskgreg/opencv-processing (should be installed)
http://urbanhonking.com/ideasfordozens/2013/07/10/announcing-opencv-for-processing/

Studio Assignment:
Studio 2: OpenCV for Computer Vision

Week of February 17: NO CLASS! President's Day

Week 3: February 24
Seeing the 3rd Dimension

Watch (in class):

How Kinect 3D works

The New Xbox Kinect

Kinect and Processing Tutorials

The Depth Image in Kinect and Processing

Studio Assignment:
Studio 3 Part I: Pointclouds with Kinect
Studio 3 Part II: 3D Scanning with Kinect & Skanect

Week 4: March 2
Recognizing Objects & Structures: Skeletons & Particle Clouds

Watch (in class):

The Science of Rendering Photorealistic CGI

Point Clouds in Kinect and Processing

Hand tracking in Kinect and Processing

Videos on SLAM

Studio Assignment: Studio 4: Skeleton Detection with Kinect

Part II: Actuation

Week 5: March 9
Drones I: Ground School

Prior to class:

If you have an iPhone, iPad or Android device, please download and install this app from iTunes, GooglePlay or the Android Market: AR.FreeFlight

https://itunes.apple.com/us/app/freeflight/id373065271?mt=8
http://ardrone2.parrot.com/apps/

It is also strongly recommended that you download this simulator, and practice with it before class: ARDrone Sim
Note: This app costs $1.99 (iOS) or $2.39 (Android)

https://play.google.com/store/apps/details?id=com.DDI.ARDroneSim&hl=en

For the Simulator options choose: AR.Drone 2.0, Indoor Hull, Standard Battery

Press the Green Takeoff button to launch and hover, the Green Landing button for an automatic controlled landing.

The Left joystick tilts the drone, and causes it to move laterally.

The Right joystick causes the drone to go up and down, and rotate clockwise or counter clockwise.

The Camera icon will shift your viewing perspective.

If you crash, it will simulate damage, so you may need to go to Settings on the main screen and Restore Defaults to fly straight again.

Studio:

FAA Drone Regulations

Where to Fly Drones in NYC

Drone Safety

AR 2.0 Tutorial #1 SETUP

AR 2.0 Tutorial #2 PILOT

AR 2.0 Tutorial #3 RECORD