MAD Gaze GLOW & GLOW Plus: Hand Gesture SDK for Unity

v2021.2 (Release Date: 06/18/2021)

• Updates [Scene Tracking SDK]

1. Standardize application flow for Scene Tracking
2. Developers can now customise the Animation and Virtual Planes on their apps.
3. Different callbacks among all state changes are now available.
4. Two control modes are now available, Arbitary-Placement mode and Center-Placement mode.

View historical changes

Build History

Capabilities

Hand Gesture SDK for Unity provides the following functions:

• Hand Signal Recognition
• Hand Cursor 2 Recognition
• Hand Grab Recognition
• Hand Tracking Recognition

Prerequisite

*This SDK is applicable to both MAD Gaze GLOW and MAD Gaze GLOW Plus model.

The application shall be

• an Android application
• with Target SDK at 27 (Android 8.1 'Oreo')
• with Minimum SDK at 27 (Android 8.1 'Oreo')
• with Landscape Left/Right orientation

GLOW Service app shall be installed on your Android phone. (Check Getting Started)

Tested Unity Editor Versions

• Unity 2019.4.1
• Unity 2019.3.12
• Unity 2019.2.3

Getting Started

1. Download the Unity Package here.

2. Double Tap on the downloaded package and import the files.

3. Download the Android Service application and Install the application on your Android phone.

   GLOW Service (Android APK)
   Download here

4. You are ready to develop Glow-enabled apps.

Hand Gesture Recognition

MAD Gaze SDK provides different types of Hand Gesture recognition for your application. AR/MR Smartglasses is the next big trend and Hand Gesture Control will allow users to interact with virtual objects in a more direct way.

MAD Gaze Hand Gesture System captures the video with a single camera, and process the recognition with AI technology.

We provide different types of Hand Gesture recognition to enrich the user experience for your application, you can also selectively enable the particular features for better performance.

Hand Signal Recognition

Hand Signal is suitable for the application with simple-and-direct controls.

Hand Signal Types

• 
  SIGNAL_ONE
• 
  SIGNAL_TWO
• 
  SIGNAL_THREE
• 
  SIGNAL_FOUR
• 
  SIGNAL_FIVE
• 
  SIGNAL_FIST
• 
  SIGNAL_OK

Hand Signal Events

HandSignal: OnTracked

OnTracked(HandSignal.Type type, HandSignal.Direction direction, Vector3 pos1, Vector3 pos2)

Parameters

• type: indicates the type of tracked Hand Signal
• direction: determines whether the tracked hand is a Left Hand or a Right Hand
• pos1: indicates the palm of tracked hand in Screen Space Position
• pos2: indicates the feature point of tracked hand signal in Screen Space Position (This is only available with SIGNAL_ONE and SIGNAL_OK)

HandSignal: OnTrackedLost

OnTrackedLost()

Parameters

None

Enabling Hand Signals with Inspector View

1. Drag prefab MGHandGesture from MADGaze/Demo/Prefabs into your scene.

2. Toggle the Hand Signal Control button on Inspector view and set it to ON.

3. Check Enable on Startup under Hand Signal Control section.

4. Create a Script file and insert with following code segments.

   4.1 Add MADGazeSDK namespaces.

   using MADGazeSDK;
   

   4.2 Insert following code segments for the callbacks for Hand Signal Recognition.

   void HandSignal_OnTracked(HandSignal.Direction direction, Vector3 pos1, Vector3 pos2) {
   	
   }
   
   void HandSignal_OnTrackedLost() {
   	
   }
   

5. Create an object and add the created script into the object.

6. Bind the created methods to corresponding callbacks on Inspector view.

   

Enabling Hand Signals with Coding

1. Drag prefab MGHandGesture from MADGaze/Demo/Prefabs into your scene.

2. Toggle the Hand Signal Control button on Inspector view and set it to ON.

3. Check Enable on Startup under Hand Signal Control section.

   

4. Create a Script file and insert with following code segments.

   4.1 Add MADGazeSDK namespaces.

   using MADGazeSDK;
   

   4.2 Insert following code segments for the callbacks for Hand Signal Recognition.

   HandGestureManager.Instance.Controller<HandSignalController>().registerCallback(
   	(HandSignal.Type gestureType, HandSignal.Direction direction, Vector3 pos1, Vector3 pos2)=>{
   		//OnTracked
   	},
   	()=>{
   		//Hand Lost
   	}
   );
   

5. Create an object and add the created script into the object.

Hand Signals Reference Scene

You can check HandGestureDemo in MADGaze/Demo/Scenes.

Hand Cursor 2 Recognition

Hand Cursor 2 allows you to interact with actual environment. User will be able to control a cursor with familiar gesture such as drags and clicks.

Hand Cursor 2

• 
  OnTracked
• 
  OnMoved
• 
  OnClicked
• 
  OnDragStarted
• 
  OnDragging
• 
  OnDragEnded
• 
  OnTrackedLost

HandCursor2: OnTracked

OnTracked(HandCursor2.Direction direction, Vector3 pos1)

Parameters

• direction: determines whether the tracked hand is a Left Hand or a Right Hand
• pos1: indicates the palm of tracked hand in Screen Space Position

HandCursor2: OnClicked

OnClicked(HandCursor2.Direction direction, Vector3 pos1)

Parameters

• direction: determines whether the tracked hand is a Left Hand or a Right Hand
• pos1: indicates the palm of tracked hand in Screen Space Position

HandCursor2: OnMoved

OnMoved(HandCursor2.Direction direction, Vector3 pos1, Vector3 pos2)

Parameters

• direction: determines whether the tracked hand is a Left Hand or a Right Hand
• pos1: indicates the palm of tracked hand in Screen Space Position
• pos2: indicates the previous palm of tracked hand in Screen Space Position

HandCursor2: OnDragStarted

OnDragStarted(HandCursor2.Direction direction, Vector3 pos1)

Parameters

• direction: determines whether the tracked hand is a Left Hand or a Right Hand
• pos1: indicates the palm of tracked hand in Screen Space Position

HandCursor2: OnDragging

OnDragging(HandCursor2.Direction direction, Vector3 pos1, Vector3 pos2)

Parameters

• direction: determines whether the tracked hand is a Left Hand or a Right Hand
• pos1: indicates the palm of tracked hand in Screen Space Position
• pos2: indicates the previous palm of tracked hand in Screen Space Position

HandCursor2: OnDragEnded

OnDragEnded(HandCursor2.Direction direction, Vector3 pos1)

Parameters

• direction: determines whether the tracked hand is a Left Hand or a Right Hand
• pos1: indicates the palm of tracked hand in Screen Space Position

HandCursor2: OnTrackedLost

OnTrackedLost()

Parameters

None

Enabling Hand Cursor 2 with Inspector View

1. Drag prefab MGHandGesture from MADGaze/Demo/Prefabs into your scene.

2. Toggle the Hand Cursor 2 Control button on Inspector view and set it to ON.

3. Check Enable on Startup under Hand Cursor 2 Control section.

4. Create a Script file and insert with following code segments.

   4.1 Add MADGazeSDK namespaces.

   using MADGazeSDK;
   

   4.2 Insert following code segments for the callbacks for Hand Cursor 2 Recognition.

   void HandCursor2_OnTracked(HandCursor2.Direction direction, Vector3 pos1) {
   	
   }
   
   void HandCursor2_OnClicked(HandCursor2.Direction direction, Vector3 pos1) {
   	
   }
   
   void HandCursor2_OnMoved(HandCursor2.Direction direction, Vector3 pos1, Vector3 pos2) {
   	
   }
   
   void HandCursor2_OnDragStarted(HandCursor2.Direction direction, Vector3 pos1) {
   
   }
   
   void HandCursor2_OnDragging(HandCursor2.Direction direction, Vector3 pos1, Vector3 pos2) {
   	
   }
   
   void HandCursor2_OnDragEnded(HandCursor2.Direction direction, Vector3 pos1) {
   		
   }
   
   void HandCursor2_OnTrackedLost() {
   
   }
   

5. Create an object and add the created script into the object.

6. Bind the created methods to corresponding callbacks on Inspector view.

   

Enabling Hand Cursor 2 with Coding

1. Drag prefab MGHandGesture from MADGaze/Demo/Prefabs into your scene.

2. Toggle the Hand Cursor 2 Control button on Inspector view and set it to ON.

3. Check Enable on Startup under Hand Cursor 2 Control section.

   

4. Create a Script file and insert with following code segments.

   4.1 Add MADGazeSDK namespaces.

   using MADGazeSDK;
   

   4.2 Insert following code segments for the callbacks for Hand Cursor 2 Recognition.

   HandGestureManager.Instance.Controller<HandCursor2Controller>().registerCallback(
   	(HandCursor2.Direction direction, Vector3 pos1)=>{
   		//OnTracked
   	}, 
   	(HandCursor2.Direction direction, Vector3 pos1, Vector3 pos2)=>{
   		//OnMoved
   	},
   	(HandCursor2.Direction direction, Vector3 pos1, Vector3 pos2)=>{
   		//Clicked
   	},
   	()=>{
   		//Hand Lost
   	}
   )
   

5. Create an object and add the created script into the object.

Hand Cursor 2 Reference Scene

You can check HandGestureDemo in MADGaze/Demo/Scenes.

Hand Grab Recognition

Hand Grab allows you to interact with actual environment. User will be able to perform picking up an object and placing it in to specific location.

Hand Grab Events

• 
  OnStarted
• 
  OnMoved
• 
  OnEnded
• 
  OnCancelled

HandGrab: OnStarted

OnStarted(HandGrab.Direction direction, Vector3 pos1)

Parameters

• direction: determines whether the tracked hand is a Left Hand or a Right Hand
• pos1: indicates the palm of tracked hand in Screen Space Position

HandGrab: OnMoved

OnMoved(HandGrab.Direction direction, Vector3 pos1, Vector3 pos2)

Parameters

• direction: determines whether the tracked hand is a Left Hand or a Right Hand
• pos1: indicates the palm of tracked hand in Screen Space Position
• pos2: indicates the previous palm of tracked hand in Screen Space Position

HandGrab: OnEnded

OnEnded(HandGrab.Direction direction, Vector3 pos1)

Parameters

• direction: determines whether the tracked hand is a Left Hand or a Right Hand
• pos1: indicates the palm of tracked hand in Screen Space Position

HandGrab: OnCancelled

OnCancelled()

Parameters

None

Enabling Hand Grab with Inspector View

1. Drag prefab MGHandGesture from MADGaze/Demo/Prefabs into your scene.

2. Toggle the Hand Grab Control button on Inspector view and set it to ON.

3. Check Enable on Startup under Hand Grab Control section.

4. Create a Script file and insert with following code segments.

   4.1 Add MADGazeSDK namespaces.

   using MADGazeSDK;
   

   4.2 Insert following code segments for the callbacks for Hand Grab Recognition.

   void HandGrab_OnStarted(HandGrab.Direction direction, Vector3 pos1) {
   	
   }
   
   void HandGrab_OnMoved(HandGrab.Direction direction, Vector3 pos1, Vector3 pos2) {
   	
   }
   
   void HandGrab_OnEnded(HandGrab.Direction direction, Vector3 pos1) {
   	
   }
   
   void HandGrab_OnCancelled() {
   
   }
   

5. Create an object and add the created script into the object.

6. Bind the created methods to corresponding callbacks on Inspector view.

   

Enabling Hand Grab with Coding

1. Drag prefab MGHandGesture from MADGaze/Demo/Prefabs into your scene.

2. Toggle the Hand Grab Control button on Inspector view and set it to ON.

3. Check Enable on Startup under Hand Grab Control section.

   

4. Create a Script file and insert with following code segments.

   4.1 Add MADGazeSDK namespaces.

   using MADGazeSDK;
   

   4.2 Insert following code segments for the callbacks for Hand Grab Recognition.

   HandGestureManager.Instance.Controller<HandGrabController>().registerCallback(
   	(HandGrab.Direction direction, Vector3 pos1)=>{
   		//onStarted
   	}, 
   	(HandGrab.Direction direction, Vector3 pos1, Vector3 pos2)=>{
   		//onMoved
   	}, 
   	(HandGrab.Direction direction, Vector3 pos1)=>{
   		//onEnded
   	}, 
   	()=>{
   		//onCancelled
   	}
   )
   

5. Create an object and add the created script into the object.

Hand Grab Reference Scene

You can check HandGestureDemo in MADGaze/Demo/Scenes.

Hand Tracking Recognition

Hand Tracking will track your hands and provide the position of all feature points.

Hand Tracking Events

HandTracking: OnTracking

OnTracking(TrackedHand h1, TrackedHand h2)

Parameters

• h1: The Tracked Hand information (usually the Left Hand)
• h2: The Tracked Hand information (usually the Right Hand)

The degree is defined as the angle of the two intersecting lines between Point A (Bottom of the wrist) and Point B (Bottom of the middle finger joint). A fixed horizon line will be sitting at Point A regardless of its movement, degree is mainly contributed by the amount of turn of the vertical line that sits at point B. This rule applies to both palm-side and dorsal-side of hand.

Enabling Hand Tracking with Inspector View

1. Drag prefab MGHandGesture from MADGaze/Demo/Prefabs into your scene.

2. Toggle the Hand Tracking Control button on Inspector view and set it to ON.

3. Check Enable on Startup under Hand Tracking Control section.

4. Create a Script file and insert with following code segments.

   4.1 Add MADGazeSDK namespaces.

   using MADGazeSDK;
   

   4.2 Insert following code segments for the callbacks for Hand Tracking Recognition.

   void HandTracking_OnTracking(TrackedHand h1, TrackedHand h2) {
   	
   }
   

5. Create an object and add the created script into the object.

6. Bind the created methods to corresponding callbacks on Inspector view.

   

Enabling Hand Tracking with Coding

1. Drag prefab MGHandGesture from MADGaze/Demo/Prefabs into your scene.

2. Toggle the Hand Tracking Control button on Inspector view and set it to ON.

3. Check Enable on Startup under Hand Tracking Control section.

   

4. Create a Script file and insert with following code segments.

   4.1 Add MADGazeSDK namespaces.

   using MADGazeSDK;
   

   4.2 Insert following code segments for the callbacks for Hand Tracking Recognition.

   HandGestureManager.Instance.Controller<HandTrackingController>().registerCallback((TrackedHand h1, TrackedHand h2)=>{
   	
   });
   

5. Create an object and add the created script into the object.

Hand Tracking Reference Scene

You can check HandGestureDemo in MADGaze/Demo/Scenes.

Troubleshooting

• Q: Why "BindService error: Device not support Hand Gesture" is shown?

  You need to install MAD Gaze GLOW Service here first. It the problem persists, open MAD Gaze Service app and try again.

• Q: Why the permission dialog never prompt on my Android 10 device?

  It is because the native bug doesn't allow you to prompt the permission dialog on Android Q devices with Target SDK newer than 28. You have to set Target SDK at 27 or below in order to make it work.

Changelog

v2021.2 (Release Date: 06/18/2021)

• Updates [Scene Tracking SDK]

1. Standardize application flow for Scene Tracking
2. Developers can now customise the Animation and Virtual Planes on their apps.
3. Different callbacks among all state changes are now available.
4. Two control modes are now available, Arbitary-Placement mode and Center-Placement mode.

v2021.1 (Release Date: 02/10/2021)

• First Release
  1. Device Control SDK: This SDK allows user to perform the basic control to your device.
  2. MADConnect SDK: This SDK enables interconnection between you and your target users who download your apps.
• Updates
  1. Hand Gesture SDK: Support HandCursor2 Gesture, Palm Detection, Hand Angle Calculation.
  2. Head Tracking SDK: Added Sensor Status API.

v1.3.0 (Release Date: 08/19/2020)

• First Release
  1. Scene Tracking SDK: This SDK allows user to recognize table-sized plain surface patterns and place the target to the tracked surface.
• Updates
  1. All SDK: Support USB Connection with MAD Gaze Adapters.

v1.1.0 (Release Date: 07/10/2020)

• First Release
  1. Head Tracking SDK: This SDK allows users to control the application via operating with head rotation.

v1.0.0 (Release Date: 06/29/2020)

• First Release
  1. Camera SDK: This SDK allows users to access the camera with codes.

Disclaimer

This library is licensed under the MAD Gaze - Terms of Use.