Cognito System

The COGNITO system is complex and involves novel customized hardware with multiple sensors which will deliver a large amount of data at high speed. The software system will deal with streams of heterogeneous sensor data in real-time and moreover offer rendering capabilities in an augmented reality display. The goal is to develop a mobile and wearable system. This requires hardware miniaturization, good ergonomics and optimization of the processing load.

COGNITO building blocks

Five main building blocks of the COGNITO system have been identified:Process scheme

  • On-Body Sensor Network (BSN) and Head-Mounted Display (HMD): inertial measurement units (IMUs), cameras, and RGB-D cameras are combined in a sensor network. A monocular head-mounted display with integrated eye-tracking cameras provides the system feed-back and user assistance information.
  • Low-Level Sensor Processing: processes the measurements from the BSN and provides estimates of the operator's motion and the poses of her hands and relevant objects in the environment. The responsibility is shared between two modules.
    • Sensor Fusion: responsible for the integration of the inertial sensors and high level vision information.
    • Computer Vision: responsible for processing the raw image and depth data from the cameras.
  • Workflow Recovery and Monitoring: receives a sequence of instantaneous configurations for the operator, her hands, objects and parts of objects contained within a work space from the Computer Vision. It processes it and provides the marker (start and stop timestamp) for ongoing/future atomic events in the current workflow sequence. It also estimates the position of objects in the same workspace and provides feedback to the Computer Vision.
  • Graphics: is used to produce the proper graphics for editing workflows as well as aiding the user during task execution using augmented reality. This allows context sensitive output.