back to Computer Vision (CSE 40535/60535)

Overview

Each project has three stages with deadlines:

1. Data acquisition, processing (if needed) and description of the proposed solution (10 points, due by September 29)
2. Feature extraction, selection and classification (30 points; due by November 20)
3. Performance evaluation (10 points; due by December 7)

You are free to choose the programming language and libraries that you like for the semester project. In particular, we recommend either Matlab 2017a or OpenCV 3.3 (used directly in C++ or - way better - through the Python wrapper).

More details on requirements for deliverables can be found on SAKAI.

Projects proposed by the instructor

Project 1: Scavenger Hunt 

Create a game of scavenger hunt with pictures of objects inside ND campus (or inside a building). The game should be played in pairs. The person "hiding" the objects will take pictures of specific objects (statues, doors, paintings) on campus. The hunter should look for these places or objects and take pictures of them. Then the pictures should be matched to the original images, to verify if the object is the same that was "hidden". Ideally, the game should be implemented to run on mobile devices, but alternatively, it can run on a computer with pictures from a mobile phone.

Project 2: Driving coach

Create an application that analyzes dashcam videos and identifies and tracks objects relevant to the driver, like cars, lanes, signs, etc. Look at an example here. The software should be able to detect/track multiple objects at a time, on a stream of video.

Suggestions for possible video sources/datasets: KITTI/KITTI Road, YouTube 8M, BDD-V. Consider collecting ~30 different videos for training, ~10 different videos for validation and ~10 different videos for presentation of final performance.

Students assigned to this project: DG+BJ+AL (team), MK, MMR, KW+TS (team)

Project 3: Where am I? Identification of ND campus buildings

Create an application that can recognize and identify (some) ND Campus buildings. The application should be able to analyze pictures taken from ND buildings, and identify the name of the building. Ideally, the application should be implemented to run on mobile devices, but alternatively, it can run on a computer with pictures from a mobile phone. Optionally, the application can show the location of the building on a map.

Suggestion for dataset: Select three to five buildings, and collect 20-30 at different angles and distances.

Students assigned to this project: KHHT, JRA, HL

Project 4: Hand gestures recognition

In this project you will write a program that recognizes a hand gesture (one of a few pre-defined). You can assume a uniform background and a vertical placement of a hand. You will need to implement a segmentation algorithm, propose and select features and do the classification. The final solution should work in real-time for a video stream captured by the camera in your laptop ("real-time" means 1 frame per second, or faster).

Suggestions for possible image sources/datasets: Idiap, University of Padova, Silesian University of Technology. Consider recognizing at least 3 different gestures. Collect (or download) at least 20 different samples per one gesture for training, at least 10 different samples per gesture for validation and at least 10 different samples per gesture for presentation of final performance.

Students assigned to this project: EP, DS (focused on sign language recognition)

Project 5: Face recognition

In this project you will write a program that uses a face detection method (e.g., Viola-Jones) and one of the image feature descriptors (e.g., LBP) to recognize your face. Your solution should correctly recognize you when tested on a small set of people, for instance your classmates.

Suggestions for possible image sources/datasets: FRGC, BioID, University of Essex. Or you can collect images of your face + a face of your teammate. If you collect your own data, prepare at least 20 different samples per one face for training, at least 10 different samples per face for validation and at least 10 different samples per face for presentation of final performance.

Students assigned to this project: MT+SO (team), JD, JL, JK

Project 6: Object tracking in outdoor environment

In this project you will write a program that detects and tracks selected object observed in outdoor environment. One example are the planes that approach SBN and can be easily observed on the sky from ND campus. Assume that you are filming an object with a camera that shakes and then you need to detect the object's position in each frame. If you don't like planes, pick an object you like, for instance a car. You can use some pre-trained deep learning models to extract useful features.

Suggestion for dataset: Collect your own object movies, or select them from YouTube (or YouTube 8M). Consider collecting ~30 different videos for training, ~10 different videos for validation and ~10 different videos for presentation of final performance.
 
Students assigned to this project: JS, JZ, SR+BDG+RF (team), VS

Project 7: Recognition of handwritten digits

In this project you will write a program that first learns to recognize handwritten digits (0,...,9) on a database, and then can recognize handwritten digits written by you. You can't use your writing samples in the training.

Suggestions for possible image sources/datasets: MNIST, Bogazici University, Semeion. Consider collecting at least 10 different samples per digit (0,1,...,9) for for training, at least 5 different samples per digit for validation and at least 5 different samples per digit for presentation of final performance. Additionally, you should evaluate your final solution using your handwritten digits, for instance 5 samples per digit. Samples composing each set can be taken from different datasets to increase the generalization capabilities of your solution

Students assigned to this project: SZ, MM, CB+DW (team), KF, NS+KH (team), BL

Individual projects agreed with the instructor

Individual project 1 (DK): Deep learning in semantic segmentation of iris images

The aim of this project is to use CNN-based models (including a CNN that performs dilated convolutions) to solve a semantic segmentation task. The use case is iris recognition and its first step: image segmentation.

Individual project 2 (JK): Automatic learning-free neural image sparse segmentation and reconstruction

Microscopic image data (microCT X-ray, sSEM, SCoRe, etc.) is difficult to properly segment due to a lack of ground truth annotations for training learning models. This project aims to improve upon the quality of reconstructions generated by classical (learning-free) image processing techniques by developing an optimization technique that adapts to local image conditions and iteratively selects the highest-quality segmentation.

Individual project 3 (ML): CAPTCHA solver

The aim of this project is to write an algorithm that recognizes CAPTCHA letters/digits from a few different CAPTCHA sources.

Individual project 4 (KS + MC): recognition of face expressions

The aim of this project is to recognize a facial expression (out of a set of pre-defined expressions).

Individual project 5 (BB): recognition of cities based on skyline pictures

The aim of this project is to write a program that accepts a picture with a skyline and outputs which city the picture was taken in. A small set of cities, say 10, will be selected to reduce the difficulty of the problem.

Individual project 6 (SM): deep-learning-based semantic segmentation for skin lesion detection

This project will follow task 1 of the challenge on skin lesion segmentation.

Individual project 7 (FT): recognition of numbers in lottery tickets

This project will focus on automatic detection and recognition of numbers on various lottery tickets. Tickets from at least three different types of lotteries will be used. A nice complication for this semester project is that the program will not be looking at a particular region for numbers.

Individual project 8 (KL): seam carving

Seam carving, or content-aware image resizing, is a technique that allows image retargeting (displaying them on media of various sizes) by removing "unimportant" image areas and using only the "important" ones. This is done by introducing or removing seams. In this project students will apply object segmentation and recognition to generate a new kind of content-aware "energy" function that is used in content-aware image scaling.