MAD Gaze GLOW & GLOW Plus: Hand Gesture SDK for Unity

v2021.2 (Release Date: 06/18/2021)

  • Updates [Scene Tracking SDK]
  1. Standardize application flow for Scene Tracking
  2. Developers can now customise the Animation and Virtual Planes on their apps.
  3. Different callbacks among all state changes are now available.
  4. Two control modes are now available, Arbitary-Placement mode and Center-Placement mode.


Build History

Date Package File (.unitypackage) Demo Project
06/18/2021
02/10/2021
08/19/2020
07/10/2020
06/29/2020

Capabilities

Hand Gesture SDK for Unity provides the following functions:


Prerequisite

*This SDK is applicable to both MAD Gaze GLOW and MAD Gaze GLOW Plus model.

The application shall be

  • an Android application
  • with Target SDK at 27 (Android 8.1 ‘Oreo’)
  • with Minimum SDK at 27 (Android 8.1 ‘Oreo’)
  • with Landscape Left/Right orientation

GLOW Service app shall be installed on your Android phone. (Check

)

Tested Unity Editor Versions

  • Unity 2019.4.1
  • Unity 2019.3.12
  • Unity 2019.2.3

Getting Started

  1. Download the Unity Package

    .

  2. Double Tap on the downloaded package and import the files.

  3. Download the Android Service application and Install the application on your Android phone.

    GLOW Service (Android APK)
  4. You are ready to develop Glow-enabled apps.


Hand Gesture Recognition

MAD Gaze SDK provides different types of Hand Gesture recognition for your application. AR/MR Smartglasses is the next big trend and Hand Gesture Control will allow users to interact with virtual objects in a more direct way.

MAD Gaze Hand Gesture System captures the video with a single camera, and process the recognition with AI technology.

We provide different types of Hand Gesture recognition to enrich the user experience for your application, you can also selectively enable the particular features for better performance.

Hand Signal Recognition

Hand Signal is suitable for the application with simple-and-direct controls.

Hand Signal Types

SIGNAL_ONE SIGNAL_TWO
SIGNAL_THREE SIGNAL_FOUR
SIGNAL_FIVE SIGNAL_FIST
SIGNAL_OK -
-

Hand Signal Events

HandSignal: OnTracked

OnTracked(HandSignal.Type type, HandSignal.Direction direction, Vector3 pos1, Vector3 pos2)

Parameters

  • type: indicates the type of tracked Hand Signal
  • direction: determines whether the tracked hand is a Left Hand or a Right Hand
  • pos1: indicates the palm of tracked hand in Screen Space Position
  • pos2: indicates the feature point of tracked hand signal in Screen Space Position (This is only available with SIGNAL_ONE and SIGNAL_OK)
Hand Signal Feature Point Description
SIGNAL_ONE The finger tip of Forefinger
SIGNAL_OK The intersection point of Forefinger and Thumb

HandSignal: OnTrackedLost

OnTrackedLost()

Parameters

None


Enabling Hand Signals with Inspector View

  1. Drag prefab MGHandGesture from MADGaze/Demo/Prefabs into your scene.

  2. Toggle the Hand Signal Control button on Inspector view and set it to ON.

  3. Check Enable on Startup under Hand Signal Control section.

  4. Create a Script file and insert with following code segments.

    4.1 Add MADGazeSDK namespaces.

    using MADGazeSDK;
    

    4.2 Insert following code segments for the callbacks for Hand Signal Recognition.

    void HandSignal_OnTracked(HandSignal.Direction direction, Vector3 pos1, Vector3 pos2) {
    	
    }
    
    void HandSignal_OnTrackedLost() {
    	
    }
    
  5. Create an object and add the created script into the object.

  6. Bind the created methods to corresponding callbacks on Inspector view.


Enabling Hand Signals with Coding

  1. Drag prefab MGHandGesture from MADGaze/Demo/Prefabs into your scene.

  2. Toggle the Hand Signal Control button on Inspector view and set it to ON.

  3. Check Enable on Startup under Hand Signal Control section.

  4. Create a Script file and insert with following code segments.

    4.1 Add MADGazeSDK namespaces.

    using MADGazeSDK;
    

    4.2 Insert following code segments for the callbacks for Hand Signal Recognition.

    HandGestureManager.Instance.Controller<HandSignalController>().registerCallback(
    	(HandSignal.Type gestureType, HandSignal.Direction direction, Vector3 pos1, Vector3 pos2)=>{
    		//OnTracked
    	},
    	()=>{
    		//Hand Lost
    	}
    );
    
  5. Create an object and add the created script into the object.


Hand Signals Reference Scene

You can check HandGestureDemo in MADGaze/Demo/Scenes.


Hand Cursor 2 Recognition

Hand Cursor 2 allows you to interact with actual environment. User will be able to control a cursor with familiar gesture such as drags and clicks.

Hand Cursor 2 Events

OnTracked OnMoved
OnClicked OnDragStarted
OnDragging OnDragEnded
OnTrackedLost -
-

HandCursor2: OnTracked

OnTracked(HandCursor2.Direction direction, Vector3 pos1)

Parameters

  • direction: determines whether the tracked hand is a Left Hand or a Right Hand
  • pos1: indicates the palm of tracked hand in Screen Space Position

HandCursor2: OnClicked

OnClicked(HandCursor2.Direction direction, Vector3 pos1)

Parameters

  • direction: determines whether the tracked hand is a Left Hand or a Right Hand
  • pos1: indicates the palm of tracked hand in Screen Space Position

HandCursor2: OnMoved

OnMoved(HandCursor2.Direction direction, Vector3 pos1, Vector3 pos2)

Parameters

  • direction: determines whether the tracked hand is a Left Hand or a Right Hand
  • pos1: indicates the palm of tracked hand in Screen Space Position
  • pos2: indicates the previous palm of tracked hand in Screen Space Position

HandCursor2: OnDragStarted

OnDragStarted(HandCursor2.Direction direction, Vector3 pos1)

Parameters

  • direction: determines whether the tracked hand is a Left Hand or a Right Hand
  • pos1: indicates the palm of tracked hand in Screen Space Position

HandCursor2: OnDragging

OnDragging(HandCursor2.Direction direction, Vector3 pos1, Vector3 pos2)

Parameters

  • direction: determines whether the tracked hand is a Left Hand or a Right Hand
  • pos1: indicates the palm of tracked hand in Screen Space Position
  • pos2: indicates the previous palm of tracked hand in Screen Space Position

HandCursor2: OnDragEnded

OnDragEnded(HandCursor2.Direction direction, Vector3 pos1)

Parameters

  • direction: determines whether the tracked hand is a Left Hand or a Right Hand
  • pos1: indicates the palm of tracked hand in Screen Space Position

HandCursor2: OnTrackedLost

OnTrackedLost()

Parameters

None


Enabling Hand Cursor 2 with Inspector View

  1. Drag prefab MGHandGesture from MADGaze/Demo/Prefabs into your scene.

  2. Toggle the Hand Cursor 2 Control button on Inspector view and set it to ON.

  3. Check Enable on Startup under Hand Cursor 2 Control section.

  4. Create a Script file and insert with following code segments.

    4.1 Add MADGazeSDK namespaces.

    using MADGazeSDK;
    

    4.2 Insert following code segments for the callbacks for Hand Cursor 2 Recognition.

    void HandCursor2_OnTracked(HandCursor2.Direction direction, Vector3 pos1) {
    	
    }
    
    void HandCursor2_OnClicked(HandCursor2.Direction direction, Vector3 pos1) {
    	
    }
    
    void HandCursor2_OnMoved(HandCursor2.Direction direction, Vector3 pos1, Vector3 pos2) {
    	
    }
    
    void HandCursor2_OnDragStarted(HandCursor2.Direction direction, Vector3 pos1) {
    
    }
    
    void HandCursor2_OnDragging(HandCursor2.Direction direction, Vector3 pos1, Vector3 pos2) {
    	
    }
    
    void HandCursor2_OnDragEnded(HandCursor2.Direction direction, Vector3 pos1) {
    		
    }
    
    void HandCursor2_OnTrackedLost() {
    
    }
    
  5. Create an object and add the created script into the object.

  6. Bind the created methods to corresponding callbacks on Inspector view.


Enabling Hand Cursor 2 with Coding

  1. Drag prefab MGHandGesture from MADGaze/Demo/Prefabs into your scene.

  2. Toggle the Hand Cursor 2 Control button on Inspector view and set it to ON.

  3. Check Enable on Startup under Hand Cursor 2 Control section.

  4. Create a Script file and insert with following code segments.

    4.1 Add MADGazeSDK namespaces.

    using MADGazeSDK;
    

    4.2 Insert following code segments for the callbacks for Hand Cursor 2 Recognition.

    HandGestureManager.Instance.Controller<HandCursor2Controller>().registerCallback(
    	(HandCursor2.Direction direction, Vector3 pos1)=>{
    		//OnTracked
    	}, 
    	(HandCursor2.Direction direction, Vector3 pos1, Vector3 pos2)=>{
    		//OnMoved
    	},
    	(HandCursor2.Direction direction, Vector3 pos1, Vector3 pos2)=>{
    		//Clicked
    	},
    	()=>{
    		//Hand Lost
    	}
    )
    
  5. Create an object and add the created script into the object.


Hand Cursor 2 Reference Scene

You can check HandGestureDemo in MADGaze/Demo/Scenes.


Hand Grab Recognition

Hand Grab allows you to interact with actual environment. User will be able to perform picking up an object and placing it in to specific location.

Hand Grab Events

OnStarted OnMoved
OnEnded OnCancelled

HandGrab: OnStarted

OnStarted(HandGrab.Direction direction, Vector3 pos1)

Parameters

  • direction: determines whether the tracked hand is a Left Hand or a Right Hand
  • pos1: indicates the palm of tracked hand in Screen Space Position

HandGrab: OnMoved

OnMoved(HandGrab.Direction direction, Vector3 pos1, Vector3 pos2)

Parameters

  • direction: determines whether the tracked hand is a Left Hand or a Right Hand
  • pos1: indicates the palm of tracked hand in Screen Space Position
  • pos2: indicates the previous palm of tracked hand in Screen Space Position

HandGrab: OnEnded

OnEnded(HandGrab.Direction direction, Vector3 pos1)

Parameters

  • direction: determines whether the tracked hand is a Left Hand or a Right Hand
  • pos1: indicates the palm of tracked hand in Screen Space Position

HandGrab: OnCancelled

OnCancelled()

Parameters

None


Enabling Hand Grab with Inspector View

  1. Drag prefab MGHandGesture from MADGaze/Demo/Prefabs into your scene.

  2. Toggle the Hand Grab Control button on Inspector view and set it to ON.

  3. Check Enable on Startup under Hand Grab Control section.

  4. Create a Script file and insert with following code segments.

    4.1 Add MADGazeSDK namespaces.

    using MADGazeSDK;
    

    4.2 Insert following code segments for the callbacks for Hand Grab Recognition.

    void HandGrab_OnStarted(HandGrab.Direction direction, Vector3 pos1) {
    	
    }
    
    void HandGrab_OnMoved(HandGrab.Direction direction, Vector3 pos1, Vector3 pos2) {
    	
    }
    
    void HandGrab_OnEnded(HandGrab.Direction direction, Vector3 pos1) {
    	
    }
    
    void HandGrab_OnCancelled() {
    
    }
    
  5. Create an object and add the created script into the object.

  6. Bind the created methods to corresponding callbacks on Inspector view.


Enabling Hand Grab with Coding

  1. Drag prefab MGHandGesture from MADGaze/Demo/Prefabs into your scene.

  2. Toggle the Hand Grab Control button on Inspector view and set it to ON.

  3. Check Enable on Startup under Hand Grab Control section.

  4. Create a Script file and insert with following code segments.

    4.1 Add MADGazeSDK namespaces.

    using MADGazeSDK;
    

    4.2 Insert following code segments for the callbacks for Hand Grab Recognition.

    HandGestureManager.Instance.Controller<HandGrabController>().registerCallback(
    	(HandGrab.Direction direction, Vector3 pos1)=>{
    		//onStarted
    	}, 
    	(HandGrab.Direction direction, Vector3 pos1, Vector3 pos2)=>{
    		//onMoved
    	}, 
    	(HandGrab.Direction direction, Vector3 pos1)=>{
    		//onEnded
    	}, 
    	()=>{
    		//onCancelled
    	}
    )
    
  5. Create an object and add the created script into the object.


Hand Grab Reference Scene

You can check HandGestureDemo in MADGaze/Demo/Scenes.


Hand Tracking Recognition

Hand Tracking will track your hands and provide the position of all feature points.

Hand Tracking Events

HandTracking: OnTracking

OnTracking(TrackedHand h1, TrackedHand h2)

Parameters

  • h1: The Tracked Hand information (usually the Left Hand)
  • h2: The Tracked Hand information (usually the Right Hand)
TrackedHand
degree determine the angle of the 2 intersecting lines between Point A and Point B
direction determines whether the tracked hand is a Left Hand or a Right Hand
getPosition(FeaturePointType) obtains the position in Screen Space Position of particular feature point
isBackHand determine whether the tracked hand is at its palm-side or dorsal-side

The degree is defined as the angle of the two intersecting lines between Point A (Bottom of the wrist) and Point B (Bottom of the middle finger joint). A fixed horizon line will be sitting at Point A regardless of its movement, degree is mainly contributed by the amount of turn of the vertical line that sits at point B. This rule applies to both palm-side and dorsal-side of hand.

FeaturePointType
THUMB_FINGER_TIP indicates the position of Thumb Tip in Screen Space Position
FORE_FINGER_TIP indicates the position of Forefinger in Screen Space Position
MIDDLE_FINGER_TIP indicates the position of Middlefinger in Screen Space Position
RING_FINGER_TIP indicates the position of Ring Finger in Screen Space Position
LITTLE_FINGER_TIP indicates the position of Little Finger in Screen Space Position
PALM_CENTER indicates the position of Plam in Screen Space Position

Enabling Hand Tracking with Inspector View

  1. Drag prefab MGHandGesture from MADGaze/Demo/Prefabs into your scene.

  2. Toggle the Hand Tracking Control button on Inspector view and set it to ON.

  3. Check Enable on Startup under Hand Tracking Control section.

  4. Create a Script file and insert with following code segments.

    4.1 Add MADGazeSDK namespaces.

    using MADGazeSDK;
    

    4.2 Insert following code segments for the callbacks for Hand Tracking Recognition.

    void HandTracking_OnTracking(TrackedHand h1, TrackedHand h2) {
    	
    }
    
  5. Create an object and add the created script into the object.

  6. Bind the created methods to corresponding callbacks on Inspector view.


Enabling Hand Tracking with Coding

  1. Drag prefab MGHandGesture from MADGaze/Demo/Prefabs into your scene.

  2. Toggle the Hand Tracking Control button on Inspector view and set it to ON.

  3. Check Enable on Startup under Hand Tracking Control section.

  4. Create a Script file and insert with following code segments.

    4.1 Add MADGazeSDK namespaces.

    using MADGazeSDK;
    

    4.2 Insert following code segments for the callbacks for Hand Tracking Recognition.

    HandGestureManager.Instance.Controller<HandTrackingController>().registerCallback((TrackedHand h1, TrackedHand h2)=>{
    	
    });
    
  5. Create an object and add the created script into the object.


Hand Tracking Reference Scene

You can check HandGestureDemo in MADGaze/Demo/Scenes.


Troubleshooting

  • Q: Why “BindService error: Device not support Hand Gesture” is shown?

    You need to install MAD Gaze GLOW Service

    first. It the problem persists, open MAD Gaze Service app and try again.

  • Q: Why the permission dialog never prompt on my Android 10 device?

    It is because the native bug doesn’t allow you to prompt the permission dialog on Android Q devices with Target SDK newer than 28. You have to set Target SDK at 27 or below in order to make it work.


Changelog

v2021.2 (Release Date: 06/18/2021)

  • Updates [Scene Tracking SDK]
  1. Standardize application flow for Scene Tracking
  2. Developers can now customise the Animation and Virtual Planes on their apps.
  3. Different callbacks among all state changes are now available.
  4. Two control modes are now available, Arbitary-Placement mode and Center-Placement mode.

v2021.1 (Release Date: 02/10/2021)

v1.3.0 (Release Date: 08/19/2020)

  • First Release
    1. : This SDK allows user to recognize table-sized plain surface patterns and place the target to the tracked surface.
  • Updates
    1. : Support USB Connection with MAD Gaze Adapters.

v1.1.0 (Release Date: 07/10/2020)

  • First Release
    1. : This SDK allows users to control the application via operating with head rotation.

v1.0.0 (Release Date: 06/29/2020)

  • First Release
    1. : This SDK allows users to access the camera with codes.

Disclaimer

This library is licensed under the

.