青青草App

In this era of hyper-connectivity, we’ve all experienced some version of this problem: We’re presented with an advanced new device, but it just doesn't work the way it’s supposed to. Whether it has too many (or too few) buttons, a confusing touchscreen or improperly calibrated voice controls, the quality and simplicity of a device’s human machine interface (HMI) shapes a user’s opinion of the entire product — no matter how impressive the technology inside. 

The HMI technology market is on track to become a space by 2027, and much of this success can be attributed to smart home technology. Devices in the home are increasingly using holographic projections, proximity sensing, natural language processing and other technologies designed to think, make decisions and learn, making HMI design and manufacturing front and center in product discussions. 

For example, home security systems are using artificial intelligence (AI)-driven facial recognition technology to distinguish between known guests, intruders or the residents of a house. Voice assistants allow us to control everything from thermostats to smart locks and lights without ever touching a button. HMIs in the smart home are transforming how we interact with devices, threading these technologies seamlessly into our everyday lives. 

What is the Next Generation of HMI? 

Just as the brain sends signals to our muscles to move, our mouths to speak and our ears to hear, an HMI does the same for devices. The human machine interface is the brain of a device, telling it to act when the end user needs it to. HMI technologies range from the traditional single-touch display mounted on a machine to technologies as advanced as gesture recognition sensors. 

Evolving HMIs and related functionalities are growing consumer demands for smart home devices. In 青青草App and SIS International Research’s survey of 200 smart home solution and device decision-makers, 39% said intuitive human machine interfaces would be the most important factor to the success of their smart home solution. 

Download the 2023 Smart Home Technology Trends Survey Report

HMIs are bidirectional communication channels that generally fall into one of four sensory groups: sight, sound, touch or a combination of the three. Consumers have come to expect the level of interactivity and attention to detail they have become accustomed to with their smartphones in every HMI they encounter, from ATMs to medical devices like insulin pumps. 

Across most device categories, survey respondents agreed that HMIs should be included on the device itself. Home appliances led these responses, with 81% ranking on-device HMI as a 4 or 5 out of 5 in significance. Roughly three-quarters said significant HMIs should be included on entertainment and media (78%), health management (75%), security and access control (74%), and hubs and controller (73%) devices.

There are about six different types of HMI that we see used most often in smart home devices or that OEMs are exploring for their next generation of solutions. Here is a little about the form, fit and function of each HMI design:

1. Multitouch and Display Screens 

Touch screen functionality is made possible with capacitive sensing technology that can detect and measure anything that is conductive or has a dielectric different from air. Multitouch technologies allow you to control a device with a simple swipe or touch. 

For a screen or any device to be considered capacitive, it must contain an insulating layer that can also be transparent, such as glass or plastic.  

Suppose the insulating layer has a thin trace of the transparent conductive material used to form electrical patterns inside the insulating layer. When you swipe a capacitive touchscreen with your finger, you create a change in a specific region of the magnetic field on the screen. This is measured by the circuit and triggers functions such as on/off, volume control, temperature monitoring and system maintenance checks.  

In the 青青草App survey, 84% of respondents said they currently use multitouch and display screens in their smart home solution, the second-most common response, while 57% said screens hold the most compelling business value of all HMI technologies. Additionally, 43% predict these types of screens will be used by their next-generation customers for product interactions. 

According to one respondent who uses a multitouch screen HMI, “An intuitive touchscreen display, simple controls that make it easy to use and thoughtful design ensures [the device] is easy to clean and maintain.” 

2. Voice Recognition 

Voice (or speech) recognition is software that can decode human voices. While voice assistants like Google Home and Amazon Alexa are simple to use, they elegantly conceal HMI technologies. These devices are designed with voice recognition technology to analyze, filter and digitize the information they receive. The device must be able to distinguish voices, understand verbal inflection and decipher requests in complex environments where background noise is present. 

Voice HMIs are particularly useful for hands-free applications. Imagine you’re clearing the table after dinner and have your hands full; with a simple command, you could open the dishwasher. Or, if you’re grilling outside, verbally adjust the temperature or turn off the gas. 

The more a person interacts with voice-activated devices, the more trends and patterns the system identifies based on the information it receives. The data gathered can determine user preferences and tastes, a long-term selling point for making a home smarter. More than a quarter (76%) of survey respondents said they use HMI interactions, like voice commands, to capture data through their smart home solutions. 

Nearly two-thirds (64%) of survey respondents said they currently use voice recognition in their smart home solutions, while 29% believe this form of HMI holds the most compelling value for their business. Many also expect it could play a significant role in the evolution of smart home solutions, as 49% said their next-generation customers are interacting with their devices using voice recognition. 

Some respondents identified that they are integrating this capability through platforms like Google Home or Amazon’s Alexa, while one noted that they are developing custom solutions that leverage “voice control and minimal user interfaces.” 

3. Holographic Interfaces 

The COVID-19 pandemic created massive consumer demand for contactless HMIs that allowed for the same level of interactivity with appliances as a touchscreen without the potential for contamination. Manufacturers of especially high-touch devices like elevator panels, ATMs and interactive kiosks were forced to rethink how consumers could eliminate or minimize their contact with these shared surfaces. 

One of the most effective ways to accomplish that is with holographic projections. These displays use light to project a mid-air screen image through a holographic plate that tracks the user's hand motions using sophisticated time-of-flight sensors. The system can even provide tactile effects that mimic the feel of touching physical buttons. 

While manufacturers of shared touchpoints are already embracing holographic HMIs for pathogen avoidance and mitigation, smart home OEMs are exploring uses for devices like coffee makers and smart fitness machines. Still, this is a growing area for smart home HMIs as only 4% of survey respondents are currently using holographs in their solutions. 

4. Gesture Recognition 

Instead of twisting knobs or tapping on a touch screen, gesture recognition involves a motion sensor that perceives and interprets movements as the primary data input. For example, let's say your hands are dirty from cooking, but you need to scroll on your phone or tablet to see the rest of a recipe. Gesture recognition technology would allow you to swipe and maneuver an HMI screen without touching the device. 

The technology is complex; it uses motion sensors that interpret movements as the primary data input source. By using 3D sensing technologies like , which captures an image the same way human eyes do, gesture recognition within devices can accurately depict motions from a human the same way people can.  

As with holographic HMIs, the impact of COVID-19 has made gesture recognition an area of growing interest. Just over a third (34%) of survey respondents are currently using this form of HMI; however, they were slightly less certain about how it could be leveraged in future solutions. Only 3% of respondents said their next-generation customers are using gesture recognition to interact with their products.

5. Remote Devices 

“Remotes” are no longer just for your TV. Remote devices — including tablets and smartphones — dominate the way we interact with smart home solutions. Nearly nine in 10 (89%) of survey respondents said they currently use remote device HMIs for their solutions, while 62% indicated they are the HMI with the most compelling business value — the top answer among all choices.  

Additionally, more than half (52%) said their next-generation customers are using remote devices to interact with their products. And it’s not necessarily just smartphones and tablets: “We are seeing expansion in the number of devices,” noted one respondent. 

One of these growing areas is wearable devices. While most people use wearables to track their health patterns and physical activity, they can also serve as a smart home essential. A true smart home consists of devices that can interpret data to understand your preferences and anticipate your needs; all smart technology should be able to respond to your dynamically changing actions. 

Wearable devices like smartwatches, body monitors and fitness trackers can send a signal to your smart home network when you've entered the perimeter of your home. The device can act according to that signal by powering up the living room lights, TV or air conditioner. 

In the coming years, we could also see a radical shift toward wearable displays — think augmented or virtual reality glasses or other eyewear that allows you to interact with the devices around you. Companies like and are already making major plays in this augmented reality (AR) eyewear space, while newer entrants will soon take steps into wearables for the first time. 

6. Facial Recognition and Eye Tracking 

Similar to the way voice recognition HMIs leverage microphones, facial recognition HMIs use cameras to recognize a user’s face and track their eye movements. When the user looks at a specific area of the HMI or device, a certain response is delivered. 

Eye tracking can be used to reinforce other touchless interfaces like voice control. For example, as a driver was driving past a particular restaurant, they could ask their car’s voice assistant to make a reservation at that restaurant. By tracking the driver’s eye movement, the HMI would know exactly which restaurant they were referring to. This type of HMI could also make devices and appliances more accessible for people with movement or speech disabilities; someone could simply look at a control panel to change the temperature of a room or the volume of the TV instead of using a remote or speaking a command. Using artificial intelligence and machine learning, the HMI would understand and make the desired change. 

Respondents in 青青草App’s Smart Home Technology Trends Survey indicated they are exploring and even deploying uses of artificial intelligence and machine learning that can provide more interaction back to users. One in 10 said their next-generation HMIs will be connected with the help of AI. As one respondent put it, these solutions will work “with simplicity, through voice interaction, and the products should learn from experience (machine learning and AI).” 

While OEMs are already developing the next generation of HMIs, the most basic consideration has not changed much: How does the interface help users interact with the device? 

HMI Design for the Smart Home 

While design is one of the most important considerations of HMI development, our survey discovered that it often presents a challenge for smart home OEMs. Nearly two-thirds (65%) of survey respondents said HMI design challenges impacted the development of their smart home solution.  

Modern HMI design for the smart home requires a clear understanding of its users. Engineers must first decide on the visual language of the IoT device. For instance, should the device use icons or colors? It's vital to make early on, so users can operate the device intuitively. 

Engineers need to consider communication functionality in HMI system design as well. Usually, the less text and the more visual cues used for the HMI, the easier it is for the consumer to understand it. The brain processes visuals faster than text, so using icons, lights or numbers instead of words whenever possible can ramp up visual communication and eliminate language barriers. 

Form follows function. Many smart home and appliance brands want to start the design process by creating device aesthetics. However, it's wise for engineers to keep a clear focus on the actual device interface first. The graphical user interface (GUI) and user experience of a device are most important when designing smart home products. With advancements in AR, it’s even possible for brands to superimpose a display on a device that doesn’t have a functional or aesthetically pleasing place to put a physical display panel. 

User experience consulting firm Semantic Studios suggests using these as a checklist to ensure a device has a compelling user experience: 

When designing a human machine interface, also keep these tips in mind: 

• Provide real-time feedback to users to help reverse or rectify a situation. 
• Include incremental safety-critical steps instead of having them all take place on the same screen or at once. 
• Differentiate between the error messages involved with overriding safety-critical and non-safety-critical actions. 
• Ensure tasks require some level of active involvement from the end-user while minimizing repetitive or passive actions. 

The current address some of these tips, like using minimal color to make abnormal situations stand out. But we could soon see the development of additional standards for HMI controls, like haptic feedback and voice and gesture. At this point, it’s widely understood that users can swipe across the screen to scroll or pinch and squeeze to zoom on their smartphone. However, more sophisticated control gestures that differ across HMIs and brands may need to be developed and standardized to ease the adoption process. 

The choice of materials and finishes can significantly impact the strength, durability and suitability of HMI technology. But to make HMIs smart, the right software and connectivity infrastructure needs to be realized. 

Leveraging Off-the-Shelf Solutions to Create Out-of-the-Box HMIs 

Supply chain constraints are always a concern for OEMs, but the layered challenges we’ve all faced in recent years have emphasized the need for flexible HMI solutions that aren’t dependent on specific components. Additionally, 67% of survey respondents named software design as their biggest challenge in smart home HMI development, the top-ranked choice. 

One solution is buying modules or technologies to complement their company’s internal expertise or meet time-to-market expectations for new HMI designs. 青青草App’s System-on-Module, or JSOM, is one such option that can be programmed to meet the software needs of a smart home device and modified to fit the hardware available. A purchased software module like JSOM could also help OEMs overcome one of the smart home industry’s toughest obstacles — connectivity. 

Connectivity brings the most challenges to engineers and OEMs. In 青青草App's smart home technology trends survey, 78% said challenges with creating interoperability across the smart home ecosystem are impacting the development of their solutions, while 68% said device-to-internet connectivity is a hurdle.

For certain household devices, there may not be a clear value proposition for why the appliance should be connected. It is critical to discuss whether or not connectivity is a requirement for the HMI design to function as intended. In an industry where cost is king, connectivity discussions (and the intended value) need to be top of mind. 

Advancements in HMIs, and the IoT technologies powering them, are impelling the growth of smart homes at an unprecedented rate. While 77% of survey respondents believe a device needs an HMI to be considered “smart,” 87% believe it needs artificial intelligence, and 91% say it should have connectivity — providing a sneak peek into the next evolution of smart home solutions.

Engineers should always have the user top of mind when designing and manufacturing an HMI terminal for a smart home. A user might be someone who can't walk and can only complete daily tasks by speaking, someone who cannot hear and only communicates through gestures or someone too busy balancing life to remember to restock the fridge. 

While HMIs are life-improving to most, for others, they are life-altering. It's up to smart home and appliance companies to see that each HMI system is effective, purposeful and continuously innovative.