The Metaverse: Supporting Social Infrastructure by Reproducing Railway Tracks and Trains

If you shake your game controller in a live music venue reproduced in a virtual space, your avatar also shakes a penlight. When most people hear the "metaverse," they probably imagine someone wearing a head-mounted display (or goggles) and attending an event in a virtual space. You can see a performance that would not be possible in the real world, and you can gather together with people from afar and have fun. While people are increasingly having fun in the metaverse, supporting by the metaverse is also steadily increasing. But what does it mean to provide "support" by the metaverse? This article asks this question to Katsuyuki Nakamura, Department Manager of the Intelligent Vision Research Department, Advanced Artificial Intelligence Innovation Center, Hitachi, Ltd.

──What kind of research are you doing?

Katsuyuki Nakamura: I am researching the "railway metaverse," which is a metaverse (virtual space) applied to railway manufacturing and maintenance. There is one space in my lab that is enclosed by screens. The back is an only part not covered by a screen. The enclosed space has 2.5 meters high, 2.5 meters long, and 4 meters wide. And the railway tracks and trains generated by the metaverse are projected onto this space.

For example, there is the projection of the railway tracks: you can see the railway tracks extending from our feet to the front, with houses and growing brush to the side. No shadows of us appears on the front, left, or right sides. While images are projected onto the ceiling and floor from a projector installed in the center of the space, images are projected onto the front and sides from the back of the screen, it enhances the sense of immersion.

──So what’s the point of projecting railway tracks in the metaverse?

Nakamura: Railway tracks that need maintenance can be quickly identified. The tracks reproduced in the metaverse are colored in green or red in certain places. Green indicates that the track is normal, while red indicates that the track is abnormal; that is, it needs maintenance. Abnormalities are automatically determined by artificial intelligence (AI) from the real world.

First, a train equipped with a front camera runs along the tracks in the real world and captures the tracks in their normal condition. The equipment of tracks is worn out as it is used. For example, the sleepers of the members that are laid under the rails may corrode. When a train with a camera attached runs on the same tracks again, images containing the corrosion will be taken. By comparing the two images of normal and abnormal conditions, the AI automatically detects the abnormality and color-codes the tracks reproduced in the metaverse.

──Does the color-coding mean that workers can easily identify abnormalities?

Nakamura: Yes, workers can easily identify which tracks are abnormal. Rare abnormal conditions, such as rails that are emitting smoke or that have collapsed, also appear on the metaverse tracks. Even new workers who are not accustomed to maintenance can experience and learn how to respond to abnormal conditions that are normally not easy to experience.

Even if it is impossible to color-code long tracks to signal abnormalities for maintenance workers in the real world, it makes possible in the virtual space of the metaverse.

──Is there anything else that is only possible in digital space?

Nakamura: It is also possible to reproduce trains. We can change the colors of the seats in an instant, which can drastically change the atmosphere of the vehicle and can use as a reference when deciding on the interior design. Additionally, if you want to check the structure of the vehicle itself, the seats can be removed exposing just the frame.

Furthermore, green and red spheres will float on the frame-only vehicle displaying the expertise that Hitachi has gained in design and manufacturing.

──What sort of expertise are you referring to?

Nakamura: Terms such as "Design Change" and "Operational Information” will be displayed in the spheres. When you approach the sphere labeled "Design Change," you will get a paper design document, which indicates that the specifications of the component have been changed.

A tool appears not only design document, the tool and the grooves on the vehicle door begin to flash in red. This is a warning sign to indicate that the tool could not fit into the grooves in the original design, and it displays easy-to-understand reasons for the design change described in the design document.

──What made you interested in the metaverse?

Nakamura: It all started with a metaverse game. I enjoyed the metaverse just as a game. In the game, players could help each other and make progress by building social infrastructure such as roads and bridges. If you build social infrastructure, you will be praised by the other players.

Conversely, those who maintain social infrastructure are not so well appreciated in our real life. To change this, I wished to show the amazing technologies that support our daily lives in an easy way. I felt that the metaverse would be able to reproduce this.

──Why the metaverse?

Nakamura: Although paper documents are gradually being converted to data, people may not even know where other departments' data is stored, or even that it exists in the first place. By systematizing expertise, everyone can learn from past insights, failures, and successes.

The key is the space. Through the space of metaverse, we can create a system that can be easily understood as experiences by multiple people. If expertise is not systematized, it is difficult to pass it on to the next generation.

──Why did you wish to project the metaverse into real life?

Nakamura: We received many comments from on-site workers who did not like the feeling of wearing goggles. By projecting the metaverse into reality, workers can gather together and discuss with what they are seeing. They can of course also join in from remote locations wearing goggles. The use of a mixture of the physical (real life) and the remote takes the bests from both worlds.

──How will the metaverse be used in the railway business moving forward?

Nakamura: I believe that by taking advantage of the railway metaverse, workers will be able to experience the improvement of their skills more through experiential learning and to create horizontal connections to enhance each other's skill. This will also lead to improved wellbeing for those who support social infrastructure.

Furthermore, we are thinking about creating a metaverse of tracks, trains, stations, and command centers to use them for planning, operations, and maintenance. To cope with the shortage of human resources, it is not a dream to operate and maintain from overseas using the time difference by metaverse.

I believe that the metaverse will become the social infrastructure of the future. It has the potential to support not only railways but also the entire social infrastructure. I would like to create a world in which there is no accident or problem with social infrastructure by leveraging the metaverse.