Kinect and IMU data to Android App

In this example we send a fire extinguisher position and IMU data to an android app using Arduino.

We use MatLab and Arduino to acquire and send real-world information to mobile applications or virtual reality games. This allows us to have the potential to complement and enrich the experiences in our developments. We can send information to mobile devices: Data from Inertial measurement units, on-off buttons, potentiometers, Image segmentation data, as well as Kinect for Windows sensors data.
Regarding the use of MatLab we use some of its basic commands, in addition to some of their toolbox for our developments:

• MatLab basic commands are used to process information from the sensors. For example, it is necessary to process the data received from multiple IMUs to estimate rotation and position. It is also used to graph the results that are obtained in each iteration, which allows us to have control over information all the time.

• The Image Acquisition toolbox is used for getting Kinect sensor data, including color image, depth image, as well as metadata containing the information of the detected users and skeletons.

• We use the Image Processing Toolbox and the Computer Vision toolbox to find potential objects of interest in the Kinect RGBD data. It was necessary to use segmentation techniques and point-cloud processing.

• Additionally, we use the support for serial communication, which allows us to send and receive data with Arduino. For this, it was necessary to create our own communication protocol to encode the data of the detected skeletons, the objects identified in the scene and the processed sensor data.

We use two types of Arduino boards: one Arduino Mega and multiple Arduino Nano, each one with a different purpose. We wanted the sensor system to be small-sized, portable and wireless in order for the virtual experience to be comfortable for users.
• The purpose of the Arduino Nano is to read the sensors signals. Those signals can come from multiple IMUs, multiple buttons or/and multiple potentiometers. Additionally, the Arduino Nano transmits this information via Bluetooth to the Arduino Mega.
• The Arduino Mega has a dual purpose in our work: It is responsible for wirelessly receiving information obtained by Arduino Nano; in addition, it transmits this information via USB to MatLab. Finally, the Arduino Mega receives the information processed by MatLab and sends it via Bluetooth to multiple mobile devices.

https://www.youtube.com/watch?v=jwIwJjR-sHQ