Therefore, we designed MediaPipe Holistic as a multi-stage pipeline, which treats the different regions using a region appropriate image resolution.įirst, MediaPipe Holistic estimates the human pose with BlazePose’s pose detector and subsequent keypoint model. But if one were to crop the hand and face regions from that image to pass to their respective models, the image resolution would be too low for accurate articulation. The pose estimation model, for example, takes a lower, fixed resolution video frame (256x256) as input. However, because of their different specializations, the input to one component is not well-suited for the others. The MediaPipe Holistic pipeline integrates separate models for pose, face and hand components, each of which are optimized for their particular domain. Note, that our solution consistently identifies a hand as either right (blue color) or left (orange color). Top: MediaPipe Holistic results on sport and dance use-cases. We are also introducing MediaPipe’s new ready-to-use APIs for research ( Python) and web ( JavaScript) to ease access to the technology. MediaPipe Holistic is being released as part of MediaPipe and is available on-device for mobile (Android, iOS) and desktop.

When including all three components, MediaPipe Holistic provides a unified topology for a groundbreaking 540+ keypoints (33 pose, 21 per-hand and 468 facial landmarks) and achieves near real-time performance on mobile devices. MediaPipe Holistic consists of a new pipeline with optimized pose, face and hand components that each run in real-time, with minimum memory transfer between their inference backends, and added support for interchangeability of the three components, depending on the quality/speed tradeoffs.

Today, we are excited to announce MediaPipe Holistic, a solution to this challenge that provides a novel state-of-the-art human pose topology that unlocks novel use cases. Combining them all in real-time into a semantically consistent end-to-end solution is a uniquely difficult problem requiring simultaneous inference of multiple, dependent neural networks.

MediaPipe, an open-source framework designed specifically for complex perception pipelines leveraging accelerated inference (e.g., GPU or CPU), already offers fast and accurate, yet separate, solutions for these tasks. Real-time, simultaneous perception of human pose, face landmarks and hand tracking on mobile devices can enable a variety of impactful applications, such as fitness and sport analysis, gesture control and sign language recognition, augmented reality effects and more. Posted by Ivan Grishchenko and Valentin Bazarevsky, Research Engineers, Google Research

Mesh enabler for inventor 2012