KIHT: Kaligo-based Intelligent Handwriting Teacher – French/German bilateral ANR research project

Project Presentation

The aim of the KIHT project is to develop an intelligent learning device (« DigiPen » electronic pen) for digitized handwriting. The electronic « DigiPen » hardware is designed by STABILO with the help of the German KIT institute, which is responsible for integrating artificial intelligence algorithms.

IRISA’s IntuiDoc team is responsible for the design and development of the deep learning AI that reconstructs online handwriting trajectories from the DigiPen’s kinematic sensors (accelerometers, gyroscope, magnetometer, force sensors).

At the end, the goal is to integrat this new acqusition device into the « Kaligo » application from the French Learn&Go company, which until now has been dependent on expensive Pen-Based tablet.

The fusion of the electronic « DigiPen » from the STABILO company and the Kaligo app is intended to make a cost-effective learning aid for the development of automated handwriting from which as many children as possible can benefit.

In addition, the Digipen not only enables the use of commercially available tablets, but also writing on normal paper, which is a great advantage for the children’s writing learning.

Role of the Intuidoc team

At IRISA lab, we work on AI algorithms based on deep learning to synthetize on-line handwriting trajectories from kinematic sensors (accelerometers, gyroscope, magnetometer, force sensors). In other words, we use the kinematic sensors’ signals of the Digipen stylus of Stabilo as input to our learnable algorithm to produce the online handwriting trajectory signal that will be processed by the Kaligo application from the Learn&Go company during the learning cycle of the handwriting.

We set up a complete pipeline to achieve the trajectory reconstruction. We propose preprocessing, especially an alignment method between ground truth and sensor data, a neural network architecture and a training strategy. We also define an evaluation protocol to enable optimal assessment of the results.

Current work is focused on developing a system that is robust whatever the medium on which it is written (inclined, paper, with different noises/friction). To do that we use domain adaptation methods.

Publications and communications

Article in an international peer-reviewed journal (+ presentation at ICDAR 2023 – Journal Track)

[1] Online handwriting trajectory reconstruction from kinematic sensors using temporal convolutional network. Wassim Swaileh, Florent Imbert, Yann Soullard, Romain Tavenard, Eric Anquetil, IJDAR (2023).

Communication at the SIFED symposium in France without proceeding

[2] Toward Deep neural network for pen trajectory reconstruction from kinematic sensors, Florent Imbert, Eric Anquetil, Romain Tavenard, Yann Soullard, Wassim Swaileh, Symposium International Francophone sur l’Ecrit et le Document (SIFED’2022), Oct 2022, Rennes, France

Conference poster at the SIFED symposium in France without proceeding

[3] Adaptation de domaine pour la reconstruction de la trajectoire du stylo à partir de capteurs cinématiques, Florent Imbert, Eric Anquetil, Romain Tavenard, Yann Soullard, Symposium International Francophone sur l’Ecrit et le Document (SIFED’2023), Juin 2023, Paris, France

IntuiDoc team researchers involved in the KIHT project