Bringing back the Human into the interface.
When designing a new User Interface, the goal is to produce an easy, efficient, and enjoyable method to operate a digital device to achieve a desired result. A successful interface design is one where the user needs to provide minimal input to achieve the desired output, and the interaction is natural, intuitive, and seamless.
Recently we’ve announced that we will be offering the Mudra Band for Apple Watch with a new functionality – Mudra Air-Touch.
Air-Touch provides touchless control of the iPhone. Air-Touch is a combination of air-mouse and finger gestures, where an air-mouse pointer is used for navigation and finger gestures function as computer mouse buttons. A user can swipe screens, launch applications, scroll website content, adjust controls, and interact with all phone applications using a simple and intuitive set of subtle finger movements.
The basic input types are through text, navigation, and digital element interaction. A keyboard is commonly used to input text, while a computer mouse or a hand-held controller are used for screen navigation and element interaction. Newer touchless interfaces are now becoming ubiquitous, such as voice assistants to input text, and gesture recognition cameras for navigation and element interaction.
Defining a new user input method requires research on the use-case and the user scenarios:
A use case is a description of how a user uses a process or system to accomplish a goal. It focuses on the User Interface.
A scenario is a specific situation in which the process or system could potentially be used. It focuses on the User Interaction.
A simple use-case can be touchlessly controlling a playlist on the Spotify iPhone app. Multiple user scenarios may include controlling the playlist when the hands are wet, the phone is out of reach, the user is chilling on the sofa, or running in the park. While the interface - the iPhone touchscreen - may be the same, the interaction method and the user’s attention level may be different for each scenario - touching the screen, using voice commands, or using touchless gestures.
A straight-forward method to categorize the user’s attention level can be by describing the user interaction as:
Hands free – interaction which needs only limited use of hands, or for which the controls are positioned so that the hands can occupy themselves with another task.
Touchless – interaction without physically touching a screen, keyboard, or hand-held device, via body motion, gestures, or voice.
Eyes Free – interaction that operates effectively without vision for command or for feedback, on an interface with little or no graphical component.
The Mudra technology supports simple hand and finger movements, to bind each gesture with a control function or input. Mudra supports discrete gestures, continuous gestures, and Air-Mouse gestures, to create the Air-Touch input method:
Discrete gestures. Moving a single finger or a soft finger tap, e.g. moving the index finger, or tapping the index finger on the thumb. This can be used for select, go back.
Continuous gestures. Applying fingertip pressure, e.g. apply pressure between the index and the thumb. This can be used for swipe, scroll, drag and drop.
Air-mouse. Moving and controlling a pointer (cursor location) by moving the wrist in mid-air to a desired location.
Air-Touch. Combining a sequence of a discrete gesture, a continuous gesture, and air-mouse, e.g. a soft tap of the index on the thumb, then applying fingertip pressure, and moving the hand to the right. This can be used as a “slide-to-lock” gesture.
The Mudra Band connects as a pointer device to the iPhone using the iPhone AssistiveTouch functionality. The Air-Touch functionality of the Mudra Band lets you touchlessly operate your iPhone using simple and intuitive gestures. You can scroll, swipe and perform multiple operation system, application, and menu commands.
The following gestures are used to touchlessly operate the iPhone:
Pointer – Use this gesture to navigate or highlight an area of interest using a cursor.
With the Mudra Band, move the forearm, when the elbow is fixed with a straight wrist.
On the iPhone Touchscreen, hover your finger over the touchscreen above a desired location.
Tap – Use this gesture to select an object or activate a button.
With the Mudra band, tap your Index finger with the Thumb finger.
On the iPhone screen, a light tap of the fingertip on the screen.
Touch and Hold – Use this gesture on an app icon to preview contents and perform quick actions.
With the Mudra Band, move the cursor above an app icon, then tap and keep applying fingertip pressure between the fingers.
On the iPhone screen, touch and hold an app icon to open a quick actions menu.
Swipe – Use this gesture to swipe left or right on iPhone screens.
With the Mudra Band, tap your fingers and keep applying fingertip pressure, then move your arm to the left or the right. When finished, release the pressure.
On the iPhone screen, tap on the screen, then move your finger to the left or to the right while pressing the screen. When finished, release the pressure.
Scroll - Use this gesture to scroll up or down on a browser or a menu.
With the Mudra Band, tap your fingers and keep applying fingertip pressure, then move your arm up or down. When finished, release the pressure.
On the iPhone screen, tap on the screen, then move your finger up or down while pressing the screen. When finished, release the pressure.
Drag and Drop – Use this gesture to move items, apps and widgets.
With the Mudra Band, tap on the item and continue applying pressure between the fingers, move the item to its new location, and release fingertip pressure.
On the iPhone screen, tap with your finger on the object, drag the object to a new location, then release.
Twist - Use this gesture to go back to the iPhone Home screen.
With the Mudra Band, twist your wrist back and forth to go back to your iPhone home screen.
On the iPhone screen, swipe your finger quickly from the bottom upwards.. Alternatively, in some phone models you can press the Home button.
Orienting the Hand for a Comfortable Interaction
The user can input commands with the Mudra Band while standing, sitting, laying back, or while multitasking. When performing hand and finger movements, these possible scenarios need to be accounted for to ensure a comfortable hand movement.
By tapping the Mudra Band complication on the Apple Watch screen, the device will run a short calibration. This calibration centers the pointer at the iPhone screen and adjusts the correct hand rotation direction. While calibrating, the user points the hand towards the center of the iPhone screen and keeps it still while the calibration runs.
Turning Air-Touch On and Off
Unlike traditional input devices where the user presses a button or touches a screen, the Mudra Band is a wearable interface that provides input through a set of always-on sensors, based on the user’s movement, aim and perspective – translating the user’s intentions and inputs into actions.
For a smooth operation of Air-Touch and to avoid unintended input commands, the user can quickly turn Air-Touch on or off by:
Turn Air-Touch On - tapping on the Mudra Band complication
Turn Air-Touch Off - tapping the Apple Watch digital crown, or exiting the complication
The Air-Touch functionality holds a bigger promise than merely a touchless control of the iPhone. In the near future ambient computing - where all the devices are connected to each other - Air-Touch and the Mudra neural input technology offer a natural, intuitive and simple method to interact with digital devices and digital elements. Controlling a TV, a computer, a smart home system, and additional devices with a simple interface - a link from our body to the digital world - using a mixture of air-touch interactions and finger movement gestures is, as we believe, the perfect realization of our goal – setting the input standard for the Metaverse.
Read more: