Return to Teaching

AIR – Analysis, Interpretation and Recognition of 2D (touch) and 3D Gestures for New Man-Machine Interactions

Description

With the development of touch screen and motion capture technology, new human-computer interaction gains in popularity in the recent years:  human-machine interactions are evolving. Several methods of artificial intelligence have been designed to take advantage of the new interaction potential offered by 2D and 3D action gestures. These gestural controls allow the user to execute many actions simply by doing 2D or 3D Gestures. Recognition of human actions (2D and 3D action gestures) has recently become an active research topic in Artificial Intelligence, Computer Vision, Pattern Recognition and Man-Machine Interaction.In this course, we address this emerging scientific topic: Analysis, Interpretation and Recognition of 2D (touch) and 3D Gestures for new Man-Machine Interactions. Technically, an action is a sequence generated by a human subject during the performance of a task. Action recognition deals with the process of labelling such motion sequence with respect to the depicted motions. The course will expose the specificity of the motion capture and modelisation as well as the recognition process of these two kind of actions (2D and 3D action gestures) but also the potential convergence of the scientific approaches used for each of them. We want also to address in this course some notion of user-centered design, user needs, acceptability and user testing to illustrate the importance of considering the user when we develop such new human-computer interaction.

Key-words

Geste 2D, Geste 3D, classification, Reconnaissance, Analyse, Interaction Homme-Machine, Computer Vision, Pattern Recognition, Man-Machine Interaction

Prerequisite

None

Content

  • Signal acquisition, Pre-processing and Normalization

    • Motion capture (MoCap) systems to extract 3D joint positions by using markers and high precision camera array.
    • Microsoft Kinect or Leap Motion sensor: Shotton algorithm largely eases the task of extracting 3D joint positions.
    • Pen-based and Multi-Touch Capture on touch screen: smartphone, tablet PC and tangible surface which support simultaneous participation of multiple users
    • Morphology normalisation pre-processing
    • Joint trajectory modelling

  • Feature Extraction

    • 2D and 3D feature extraction
    • Sub-stroke representation
    • Temporal, shape and motion relation between Sub-stroke

  • Artificial Intelligence for 2D and 3D Action recognition

    • Eager and lazy Recognition
    • Skeleton-based human action recognition
    • Several Recognition and Machine Learning Approaches:
      1. Graph modelling, matching and embedding algorithm
      2. Dynamic Time Warping (DTW)
      3. Hidden Markov Model (HMM)
      4. Support Vector Machine (SVM)
      5. Neural Network (NN)
      6. Reject Option…

  • 2D and 3D Segmentation and action detection

    • Direct manipulation and indirect commands
    • Early detection of an action, in an unsegmented stream
    • Temporal segmentation methods
    • Sliding Window approach

  • Human-centered design (ISO 9241-210) and test protocol

    • The goal of the user-centered design process is to obtain a product that is functional, operational and satisfies the user applying humans factors, ergonomics, and knowledge and technics of usability.
    • Test protocols
    • Data analysis

  • Example and demo

Acquired skills

Comprehensive vision of a processing chain from signal acquisition, pre-processing, classification, interpretation and user feedback.
Link between pattern recognition issues and human-machine interaction.
Link between 2D and 3D gesture recognition approaches.

Teachers

Eric Anquetil (responsable), Richard Kulpa, Nathalie Girard

Organization 2022-2023

Room:

    • Guernesey

Dates (dd/mm/yyyy):

  1. Signal acquisition, Pre-processing and Normalization (PDF):
    • 15/11/2022 (16h15 – 18h15 ; B. 02B, E209)
    • 22/11/2022 (10h15 – 12h15)
    • 24/11/2022 (16h15 – 18h15)
  2. 2D and 3D Segmentation and action detection (PDF):
    • 22/11/2022 (16h15 – 18h15)
    • 29/11/2022 (16h15 – 18h15)
    • 6/12/2022 (16h15 – 18h15)
  3. Human-centered design (ISO 9241-210) and test protocol (PDF): 
    • 08/12/2022 (16h15 – 18h15)
    • 13/12/2022 (10h15 – 12h15) 9/01/2023 (14h – 16h)
    • 15/12/2022 (16h15 – 18h15) 12/01/2023 (16h15 – 18h15)

Evaluations:

This module will be evaluated by two tests:

  • Written Exam:
    • Duration : 1h30
    • Date: 19/01/2023 (16h15 – 18h15 ; B. 12D i50)
    • Exam 2017-2018
  • Homework :
    • The work consists of reading a paper of the literature that is given by the teachers and finding (2)3 other papers on the same subject. Then to make a synthesis that will be presented during the defense .
    • The homework will be carried out and defended by pairs or trinomials.
    • Duration: 15’/20′ of presentation and 5′-10′ questions.
    • Date: 17/01/2023 (10h15 – 12h15 ; B. 12D Guernesey)