Rising expectations for an “aerial entry display” amidst the COVID-19 pandemic

In response to growing COVID-19 infections, efforts are accelerating to provide “touchless services” in various aspects of our lives.This has led to an increasing number of people switching to cashless payment which do not need to touch money. Some unique items that allow us to press elevator buttons and turn door knobs without any direct contact are also getting popular.

Under these circumstances, Hitachi-Omron Terminal Solutions, Corp.* has developed an aerial entry display that projects the buttons into the air operated without any physical contact. The company has officially released this new device in October 2020, and it is showing great promise as a technology that may prevent the spread of COVID-19 infection. Here’s a story about a Hitachi developer of this new technology, with various challenges encountered during its development.

The development team in charge of the aerial entry display was first set up in November 2017. At that time, Hitachi-Omron Terminal Solutions was trying to devote its one fifth of work to the development of new technologies and services.This was when Mr. Makoto Iida, a member of the Solutions Development Division, came up with the idea of developing a solution that could make use of the company's expertise in sensing technology, and to project text, images, and other visual information into the air but also enabling it to be "touched" by hand.

Although contactless technologies that functions by waving one's hand over a sensor to open automatic doors or flush toilets are already a familiar sight elsewhere, most are specialized for a single type of use. Technologies that bring together “aerial displays” that not only displays information in the air but also provide flexible “information entry,” in this case by “touching the aerial image”, were not commonplace at the time.

According to Mr. Iida, "Actually, aerial display technology itself was already established 30 years ago. Shortly after that, we started providing ATMs with touch panels. Ever since then, researchers in Japan have been investigating aerial displays. A key feature of our current development project is how it combines aerial display—a legacy technology—with information entry, which is a strength of our company."

This ultimately led to professionals of device design and sensing technology, who were interested in Mr. Iida's idea, gathering as a team and starting up an interdisciplinary project team. Market surveys and technology development were initiated. However, this journey, has not been a smooth one, says Mr. Iida.

Development team members and the demonstration aerial entry display (center)

The first roadblock that the project team encountered was the difficulty of adjusting the position of the aerial image that is projected into the air. To enable touching the aerial image required displaying the image not behind the projector machine, but rather in front of it. Such positional adjustment was tremendously difficult.

Referring to various international research papers and using an aerial image formation and display technology called Aerial Display with Aerial Imaging by Retro-Reflection (AIRR), the project succeeded in projecting an image situated in front of the machine. Key to this success was the adoption of a retro reflection material made by Nippon Carbide Industries—a company possessing brilliant expertise in advanced aerial display technology.

As Mr. Iida expounds, "In order to try many different materials, we made inquiries to a number of manufacturers with whom we had no previous connection, and having communicated our passion toward development, we were able to get them to supply some samples. This is how we eventually arrived at this material."

However, when the aerial display was demonstrated internally, it was met with harsh criticism, that although aerial display was a remarkable technology, there was no conceivable way to introduce it to the market. The reason for this reaction was that it would take a device measuring as much as 50 cm on all sides to display an aerial image of about 10 inches (24 cm x 17 cm), which was deemed impractical. Looking back, Mr. Iida also recalls that not only the aerial display but also enabling the image to be “touched” engendered a number of different issues.

Mr. Iida states that "With touch panel displays, the coordinate axis is specified by physically touching the display with your finger. However, in the case of aerial image projection, individual sight differences, due to variance in users’ height and visual acuity, make it incredibly difficult to enable all people to accurately ascertain the position they are seeking to touch. Also, to make it possible for people to obtain a precise stereoscopic view of a floating image, we had to repeat the process of calculation and experiment over and over again."

Aerial entry display that enables the aerial operation of buttons

In this way, the team spent more than two years in a continuousdaily struggle to miniaturize the display and improve its operability. These efforts finally led to success in reducing the size of the aerial image to 10 cm on all sides and the size of the projector device to 20 cm all around.

The team also made numerous improvements that allow intuitive operation. This was accomplished by developing sensing technology that uses infrared sensors responding to specific gestures such as "press" and "turn a page", in addition to hand-wave selection. This process also benefited from the utilization of knowledge gained and refined through the manufacturing of ATMs and automatic fare adjustment machines.

Just as the route to commercialization came into view, the COVID-19 pandemic hit the market. The project team responded to this situation by beginning considerations on adding aerial entry display to automatic payment machines for medical institutions.

They believed that making automatic payment machines—which an unspecified number of people might touch —contactless would contribute to the prevention of the infection spread. Development was drastically sped up by using 3D printers, delivering a prototype in just two months.

Ootaya Clinic, where demonstration testing took place, permits contactless automatic payment.

Next, with a view toward practical application at medical institutions where countermeasures to prevent infection are most prioritized, proof of concept (PoC) was conducted at a medical clinic in Yamanashi Prefecture, Japan, in July 2020. Aerial entry displays were connected to automatic reception and payment machines which has previously been installed and has operated for three days.

Although many patients at the clinic were senior citizens, when given instructions on how to use the device by project members, they were able to accomplish reception and easily make payments with the aerial entry display. Project members were assured from the patients’ response that the device was providing a sense of comfort and safety.

Speaking about his expectations for the aerial entry display, Dr. Takanori Sato of Ootaya Clinic, where the PoC took place, says that "I realized that this was a mechanism that leads to patients’ safety. It may well prove to be a step toward infection prevention in medical institutions."

Since August 2020, the aerial entry display has also been installed on a trial basis at an orthopedic clinic in Tokyo. Receiving many user feedbacks, the project team has been working on various improvements on the visibility of aerial image and the ease of touching it. After making adjustments to the display, sensor sensitivity, and other various factors, the aerial entry display was finally commercialized from October 2020.

The commercialization of the aerial entry display has begun in October.

As expectations for the aerial entry display continue to rise with the spread of COVID-19 infections, its use is being considered in various scenarios, including ATMs of financial institutions, ticket machines at train stations, and even digital signage at public facilities. Mr. Iida talks about the future prospects of this technology, as follows:

"In the past three years, we have achieved the development of a new type of display that projects information in the air that can be 'touched'. Looking forward to the next 5 to 10 years, however, the concept of a physical display itself may disappear, and we may be able to display and manipulate information anywhere around us. "

Aerial entry displays are garnering the attention of various industries as a technology for driving contactless solution during and after the COVID-19 pandemic. Researchers’ challenges will continue in providing safety and security and improving people’s quality of life through the development of new technology.

*Company name changed to “Hitachi Channel Solutions, Corp.” as of July 1st, 2021