Human motion estimation using wearable RGB-D camera and inertial sensors

Abstract

Recently, human motion estimation is an important task and attracting significant attention in sports and medical applications, as it presents both theoretical and practical interest from bio-mechanical, computer vision and robotics perspectives. We focus on developing algorithms to: 1) estimate gait parameters, and 2) analysis human motion during walking activity, with reduced number of sensor count. In this dissertation, gait parameters are estimated as well as human lower-limb motion is reconstructed using a single waist-mounted Intel Realsense Depth Camera D455, which has an integrated 6-DOF Inertial Measurement Unit (IMU).

An inertial navigation algorithm is primarily proposed with only IMU data to obtain spatio- temporal gait parameters, such as: attitude, position, velocity of a human body; and stance phase duration, stride length, and walking distance. However, low-cost inertial sensors come with noise and bias that lead to unavoidable accumulative errors. Therefore, straight-line walking with a constant speed constraints is imposed in the filtering algorithm to improve attitude estimation. Visual odometry (VO) algorithm provides relative pose from image sequence, which plays as a measurement updating role for the filter. Detected stance foot from color and depth image data are used as landmarks to update foot position. Finally, a smoothing algorithm is proposed as a linear optimization problem to minimize estimation errors. Stance foot position is derived and other gait parameters can be calculated from step information.

A deep learning-based segmentation model with custom dataset is proposed to improve foot detection not only in stance phases but also in swing phases. 3-D dual foot trajectories are then calculated from proposed filter results and relative position of dual foot with respect to the camera. However, foot position in between Toe-Off and Mid-Stance phases are missing due to obscurity. To tackle this problem, a Graph Convolutional Network (GCN) based model is proposed to predict pose from previous poses. Complete foot trajectories are finally reconstructed with only a single waist-mounted RGB-D camera.

Through experiments, the proposed system shows promising results and could be applied for human motion estimation in real application with a considerable accuracy.