Retargeted body-model generation to simulate human pose detection in real time for Amazon Astro
· New York, NY · 2017 – 2021
To teach a home robot to see people, Amazon’s Lab126 (the project was codenamed Vesta) needed synthetic humans moving through synthetic homes in Unreal Engine 4. The pipeline retargeted the CMU motion-capture library onto Body Labs skeletons, combined them with a range of body shapes into FBX skeleton and animation files, and imported them into Unreal as characters spawned into the scene, across walking, standing, sitting and other actions.
Models trained on the synthetic humans generalized poorly to real footage, and the search for that domain gap led to the keypoints. The keypoint “sockets” on the body came from three placements: at a joint (wrists, elbows, knees, ankles), whose position is a linear regression over the body’s vertices; at a surface vertex (nose, eyes, ears), parented to the neck; or interpolated between a joint and a vertex (hips, shoulders), parented to the joint. That held for the 2D image cases it was built for, but in 3D the hip and shoulder positions drifted, and extreme angles exaggerated the error. The limitation was parenting: a socket placed right relative to its joint moves wrong when that joint rotates.
My fix was to give the sockets their own linear regression over the mesh vertices, the way the joints already had: place the sockets on the shape templates in Unreal, export them as ordered vertices with a parent list, and regress from there. Lab126 adopted the plan as its next steps.
C++ · Unreal · Blender