1. Mediapipe Holistic Unity
2. Mediapipe Unity Software
3. Mediapipe Hands
4. Mediapipe Unity
5. Mediapipe Unity Android
6. Mediapipe Unity Pro

layout	title	nav_order
Home

MediaPipe Unity Plugin This is a Unity (2019.4.23f1) Plugin to use MediaPipe (0.8.3.1). Some MediaPipe C# codes are based on terryky/tfliteglesapp; Model Licenses. 'Tf Lite Unity Sample' and other potentially trademarked words, copyrighted images and copyrighted readme contents likely belong to the legal entity who owns the 'Asus4' organization. Overview MediaPipe Objectron is a mobile real-time 3D object detection solution for everyday objects. It detects objects in 2D images, and estimates their poses through a machine learning (ML) model, trained on the Objectron dataset.

Cross-platform ML solutions made simple

MediaPipe is the simplest way for researchers and developers to build world-class ML solutions and applications for mobile, desktop/cloud, web and IoT devices.

End-to-End acceleration: built-in fast ML inference and processing accelerated even on common hardware	Build one, deploy anywhere: Unified solution works across Android, iOS, desktop/cloud, web and IoT
Ready-to-use solutions: Cutting-edge ML solutions demonstrating full power of the framework	Free and open source: Framework and solutions both under Apache 2.0, fully extensible and customizable

ML solutions in MediaPipe

Mediapipe-unity-hand-tracking Project overview Project overview Details Activity Releases Repository Repository Files Commits Branches Tags Contributors Graph.

Face Detection	Face Mesh	Hands	Hair Segmentation

Object Detection	Box Tracking	Objectron	KNIFT

Android	iOS	Desktop	Web	Coral
Face Detection	✅	✅	✅	✅	✅
Face Mesh	✅	✅	✅
Hands	✅	✅	✅	✅
Hair Segmentation	✅	✅	✅
Object Detection	✅	✅	✅	✅
Box Tracking	✅	✅	✅
Objectron	✅
KNIFT	✅
AutoFlip	✅
MediaSequence	✅
YouTube 8M	✅

MediaPipe on the Web

MediaPipe on the Web is an effort to run the same ML solutions built for mobile and desktop also in web browsers. The official API is under construction, but the core technology has been proven effective. Please see MediaPipe on the Web in Google Developers Blog for details.

Mediapipe Holistic Unity

You can use the following links to load a demo in the MediaPipe Visualizer, and over there click the 'Runner' icon in the top bar like shown below. The demos use your webcam video as input, which is processed all locally in real-time and never leaves your device.

Getting started

Learn how to install MediaPipe and build example applications, and start exploring our ready-to-use solutions that you can further extend and customize.

The source code is hosted in the MediaPipe Github repository, and you can run code search using Google Open Source Code Search.

Publications

• MediaPipe KNIFT: Template-based feature matching in Google Developers Blog
• Alfred Camera: Smart camera features using MediaPipe in Google Developers Blog
• Real-Time 3D Object Detection on Mobile Devices with MediaPipe in Google AI Blog
• AutoFlip: An Open Source Framework for Intelligent Video Reframing in Google AI Blog
• MediaPipe on the Web in Google Developers Blog
• Object Detection and Tracking using MediaPipe in Google Developers Blog
• On-Device, Real-Time Hand Tracking with MediaPipe in Google AI Blog

Videos

Events

• Google Industry Workshop at ICIP 2019, 24 Sept 2019, Taipei, Taiwan (presentation)

Community

• Awesome MediaPipe - A curated list of awesome MediaPipe related frameworks, libraries and software
• Slack community for MediaPipe users
• Discuss - General community discussion around MediaPipe

Alpha disclaimer

MediaPipe is currently in alpha at v0.7. We may be still making breaking API changes and expect to get to stable APIs by v1.0.

Contributing

We welcome contributions. Please follow these guidelines.

We use GitHub issues for tracking requests and bugs. Please post questions to the MediaPipe Stack Overflow with a mediapipe tag.

This is a Unity (2019.4.23f1) Plugin to use MediaPipe (0.8.3.1).

Prerequisites

To use this plugin, you need to build native libraries for the target platforms (Desktop/UnityEditor, Android, iOS).If you’d like to build them on your machine, below commands/tools/libraries are required (not required if you use Docker).

• Python >= 3.9.0
• Bazel >= 3.7.2, (< 4.0.0 for iOS)
• GCC/G++ >= 8.0.0 (Linux, macOS)

Platforms

• Linux Desktop (tested on ArchLinux)
• Android
• iOS
• macOS (CPU only)
• Windows 10 (CPU only, experimental)

Example Graphs

Android	iOS	Linux (GPU)	Linux (CPU)	macOS	Windows
Face Detection	✅	✅	✅	✅	✅	✅
Face Mesh	✅	✅	✅	✅	✅	✅
Iris	✅	✅	✅	✅	✅	✅
Hands	✅	✅	✅	✅	✅	✅
Pose	✅	✅	✅	✅	✅	✅
Holistic (with iris)	✅	✅	✅	✅	✅	✅
Hair Segmentation	✅	✅
Object Detection	✅	✅	✅	✅	✅	✅
Box Tracking	✅	✅	✅	✅	✅	🔺*1
Instant Motion Tracking	✅	🔺	✅
Objectron	✅
KNIFT

*1: crashes sometimes when the graph exits.

Installation Guide

Run commands at the project root if not specified otherwise.
Also note that you need to build native libraries for Desktop CPU or GPU to run this plugin on UnityEditor.

Docker for Linux (experimental)

1. Build a Docker image

2. Run a Docker container

3. Run build command inside the container

If the command finishes successfully, required files will be installed to your host machine.

Linux

1. Install OpenCV

   By default, it is assumed that OpenCV 3 is installed under /usr (e.g. /usr/lib/libopencv_core.so).
   If your version or path is different, please edit third_party/opencv_linux.BUILD and WORKSPACE.

   For example, if you use ArchLinux and opencv3-opt, OpenCV 3 is installed under /opt/opencv3.
   In this case, your WORKSPACE will look like this.

   On the other hand, if you use Ubuntu, probably OpenCV’s shared libraries is installed under /usr/lib/x86_64-linux-gnu/.
   In that case, your opencv_linux.BUILD would be like this.

2. Install Bazelisk and NuGet, and ensure you can run them

3. Install numpy

4. (Optional) Install Android SDK and Android NDK, and set environment variables

5. Run build command

Docker for Windows (experimental)

Desktop/UnityEditor

1. Switch to windows containers

   Note that Hyper-V backend is required (that is, Windows 10 Home is not supported).

2. Build a Docker image

   This process will hang when MSYS2 is being installed.
   If this issue occurs, remove C:ProgramDataDockertmphcs*Files$Recycle.Bin manually (hcs* is random name).
   cf. https://github.com/docker/for-win/issues/8910

3. Run a Docker container

4. Run build command inside the container

If the command finishes successfully, required files will be installed to your host machine.

Android

1. Switch to Linux containers

   Note that you cannot build native libraries for Desktop with Linux containers.

2. Build a Docker image

3. Run a Docker container

4. Run build command inside the container

If the command finishes successfully, required files will be installed to your host machine.

Windows

Desktop/UnityEditor

1. Follow mediapipe’s installation guide andinstall MSYS2, Python, Visual C++ Build Tools 2019, WinSDK and Bazel (step1 ~ step6).

2. Install Opencv

   By default, it is assumed that OpenCV 3.4.10 is installed under C:opencv.
   If your version or path is different, please edit third_party/opencv_windows.BUILD and WORKSPACE.

3. Install NuGet, and ensure you can run them

4. Install numpy

5. Set PYTHON_BIN_PATH

   When the path includes space characters (e.g. C:Program FilesPython39python.exe), it’s reported that build command will fail.
   In that case, install python to another directory as a workaround (it’s unnecessary to set the path to %PATH%, but don’t forget to install numpy for the new Python).

6. Run build command

Android

You cannot build native libraries for Android on Windows 10, so use WSL 2 instead.
Installation steps are the same as Linux.

macOS

1. Install Homebrew

2. Install OpenCV 3

3. Install Python

4. Install Bazelisk and NuGet

5. (Optional) Install Xcode

6. (Optional) Install Android SDK and Android NDK, and set environment variables

7. Run build command

Build command

Run python build.py --help and python build.py build --help for more details.

Run example scenes

UnityEditor

Select Mediapipe/Samples/Scenes/DesktopDemo and play.

Desktop

If you’d like to run graphs on CPU, uncheck Use GPU from the inspector window.

Troubleshooting

DllNotFoundException: mediapipe_c

OpenCV’s path may not be configured properly.

If you’re sure the path is correct, please check on Load on startup in the plugin inspector, click Apply button, and restart Unity Editor.
Some helpful logs will be output in the console.

Mediapipe Unity Software

InternalException: INTERNAL: ; eglMakeCurrent() returned error 0x3000

Mediapipe Hands

If you encounter an error like below and you use OpenGL Core as the Unity’s graphics APIs, please try Vulkan.

Debug MediaPipe

If you set an environment variable GLOG_v before loading native libraries (e.g. libmediapipe_c.so),MediaPipe will output verbose logs to log files (e.g. Editor.log, Player.log).

You can also setup Glog so that it writes logs to files.

Mediapipe Unity

TODO

• Prepare API Documents
• Implement cross-platform APIs to send images to MediaPipe
• use CVPixelBuffer on iOS
• Box Tracking (on Windows)
• Objectron
• KNIFT

LICENSE

MIT

Mediapipe Unity Android

Note that some files are distributed under other licenses.

Mediapipe Unity Pro

• MediaPipe (Apache Licence 2.0)