NP Inc opens new horizons with 9bo's Lobster Swimming music video

Summary

Led by NP xR stage's Technical Director Park Yeh-been, the team produced a fantastic and dreamy environment for the video, delivering an immersive seascape for the artist to perform in.

The creative brief called for a visual reflection of the song’s lyrics:

"I opened my eyes, but I still feel like I'm in a dream. A turtle, an anemone, and a sea cucumber, passing by me".

A real-time interactive environment was needed to bring the underwater world to life: a stingray flying over the city centre of the sea and the artist jumping from the stingray onto the hand of a giant, all in real-time and visible on the LED screens for the artist.

NP Inc collaborated with KOVAC (Korea VR AR Complex) to achieve volumetric capture for the 3D environment, making use of the disguise RenderStream protocol to drive vivid and splendid Unreal Engine graphics realistically in a 4K resolution onto the xR stage.

Giant volumetric capture

In order to create the fantastical dream world, sea creatures and giants were selected as key images to animate the world and interact with the artist throughout the video.

“The overwhelming size difference between the giant 3D characters and the real-life artist was important for directing, and we decided that it could be shown well with xR, so we selected a shooting composition that revived the three dimensional effect of the huge volume and filmed it,” says Park Yeh-been.

The team also played around with the captured volume data, making multiple duplicates of the artist and applying stone-like textures, in order to achieve a realistic portrayal of underwater statues.

A volumetric 3D model was delivered in voxel format, which was quite new for the studio. However, in collaboration with Real Peach - the UE content production team, NP Inc. managed to establish a workflow of downsizing the data and designing creative scenes with the captured data.

Realistic lighting

Another challenge was achieving homogeneous lighting as a way to highlight the song’s lyrics about the impossibility to distinguish between the dream and reality. As a result, the team wanted to eliminate the sense of difference between the virtual environment and the real LED environment to further reflect the lyrics.

The studio tested different lighting workflows using outer frustum and DMX integration for realistic lighting. Going further, the studio decided to look through different lighting fixtures and select the working combination for each Unreal Engine scene.

Though time consuming, allocating more of the production workflow into lighting allowed NP Inc to merge the real and the virtual world together. A shimmering translucent light reflected onto the talent, for example, was possible only with the right lighting fixture. This made the experience priceless, allowing room for more testing in the design phase, which is of great importance in an LED xR or in camera VFX workflow.

Artist immersion

Shooting the video on an xR stage allowed the artist to act off the real-time scenes displayed on the LED walls surrounding her as if they were real. Scenes such as the one where the giant extends its hand and snatches the artist, where she reaches for the whale, or flies over the underwater city on a stingray then jumps onto the giant's hand were made seamless thanks to xR’s ability to immerse the artist into the virtual scene with no need for reference monitors or media training.

“With xR and LED, it was easier to match the position and line of sight of the artist compared to green screen workflows, so much more realistic and natural acting was possible as a result,” says Park Yeh-been.

Success

NP Inc found that the ability to deliver the entire project end-to-end in-house had a massive impact on the outcome of the project, bringing not only time efficiencies but also a high level of customer satisfaction. It was an opportunity to promote internal cooperation between team members and discover the various creative and technical opportunities of xR in music video production.

The entire content, including the outer frustum, was rendered in 4K resolution to achieve high-quality graphics. With the use of Unreal Engine sequencer and disguise Object Preset Animation, xR shots of the artist passing by vast virtual background was realized without making the artist move a bit in reality.

The team also found that the R&D work they had done on the convergence of volumetric data and xR established an effective workflow for the creation and implementation of digital humans in the virtual world, which is only expected to increase in demand.