As a technological change in filmmaking today, virtual filmmaking continues to disrupt the industry with its powerful interactivity and flexibility.
I. Digital development of film production Due to the physicality of the material, the film production process in the film period tended to be linear and stable. With the development of digital technology, film production gradually moved from the physical space into the virtual space, roughly going through three stages: Non-Linear Editing, Digital Intermediate and Virtual Production.
Digital non-linear editing, which emerged in Hollywood in the 1980s, is a virtualisation of the traditional film editing process. It involves converting all material in the editing process into a digital signal stored on a computer hard drive, where the material can be accessed and edited at random. This phase of virtualisation focused on film editing.
Digital intermediate film matured and became popular in the 2000s, when film and digital coexisted. It is a high-resolution digitisation of the entire film, on the basis of which a series of post-production tasks such as editing, colour grading and visual effects are completed, resulting in a digital release master, which is of course often output to film media for non-digital projection. Compared to the first stage of digital non-linear editing, digital intermediate film broadens the virtualisation to the entire film post-production and provides great convenience for the distribution and screening of the film.
Virtual filmmaking has been popularised since James Cameron’s Avatar in 2010 and has now become the production method for many mainstream blockbusters. Integrating new technologies and processes from the games and interactive entertainment industries, Virtual Production further applies digital technology to all aspects of filmmaking, and is a relatively broad term for a range of computer-assisted production and visual filmmaking methods.
2. Classification of virtual film production There are four types of virtual production: visualisation, performance capture, hybrid virtual production and live LED wall in-camera, which we will briefly look at.
Visualisation has been around for a long time. Its predecessor, pre-visualization, is widely known as the visualisation of some or all of a film’s footage before it actually goes into large-scale production, in order to explore story ideas and test subsequent technical workability. Today, visualisation is no longer limited to pre-production, but can take the form of pitch previews, visual effects previews, virtual sets, technical previews, stunt previews and post-production previews. The focus here is on the virtual set, which uses VR to provide a fully virtualised scene for the producer to interact with. The virtual set includes adjustable props and a virtual camera, allowing for pre-planned shots and set construction. Virtual set building and scheduling was used for the glass office scene in 《John Wick：Chapter3》, where the complex fight scenes would have been unimaginable from a flat concept drawing alone. In high quality VR scenes, producers can get a first-person view of the set environment, making it easier to make camera decisions and switch to a virtual camera to preview visual effects in real time.
Performance capture is essentially a 2D transcoder technique applied to animated films. Depending on the creative needs of the film, it can take the form of facial capture and full body animation. Facial capture, as the name suggests, captures an actor’s facial performance and applies it to another character, such as the performance of Brad Pitt’s aged appearance in 《The Curious Case of Benjamin Button》. Full-body animation is the transfer of an actor’s entire movement to another character, as in 《Rise of the Planet of the Apes》,《Alita: Battle Angel》etc. This type of virtual production also involves digital performances, kinetic simulations and other issues that need to be constantly explored to make digital character performances more realistic.
Hybrid virtual production combines green screen camera elements with CG elements through real-time camera tracking, providing a real-time preview for the cinematographer to refine in post-production, or directly as the final image. Hybrid VFX can be used for both live production, similar to ‘virtual studio technology’, and post-production, to provide a real-time visual reference for post-production visual effects and compositing.
Live LED wall in-camera is the most advanced technology nowadays, projecting real-time images on the LED wall behind the actors, and the image output of the real-time engine combined with camera tracking to produce a final image located entirely in-camera. Compared to green screens, LED backdrops no longer have colour overflow problems, and they present a high-precision image that is equivalent to a real scene, allowing actors and cinematographers to perform and take scenes more naturally, avoiding the uncertainty that comes with imagination. The concept is very reminiscent of the live image projections used in the past when filming car driving scenes, but it has the fatal disadvantage of parallax, where the camera has to be in a fixed perspective. This fundamental disadvantage is solved by the real-time engine used in the Live LED Wall, where the image can change perspective to synchronise with the camera’s parallax. In addition, the real-time engine’s real-time ray-tracing is also a core technology, so that on the one hand the realistic lighting effects in the digital scene can be matched with the physical lighting effects, and on the other hand the lighting from the screen has a real-time dynamic effect, enhancing immersion and realism.
Virtual filmmaking demonstrates the endless possibilities of filmmaking. While digital technology liberates the imagination of creators, it also reminds them to keep pace with the advance of technology and create new images that match or even surpass the times.
Application of LED curtain wall in virtual film and television production
The year 2020 is destined to be an extraordinary year, with the COVID-19 epidemic coming out of nowhere and spreading across the globe. This not only caused a major impact on the global economy, but also brought a devastating blow to the LED rental industry.
During this period, all types of live events could no longer take place in the same way as before. The restrictions on going out/gathering due to the epidemic directly drove demand for LED displays in the Virtual Production, XR stage market. Many customers began to look for alternative solutions to break the stalemate caused by the epidemic.
The virtual production of films and studios is one of the most active teams.
In 2007, Alex McDowell, one of the UK’s leading art directors, introduced the concept of virtual production in a holistic virtual space and ‘Hybrid Filmmaking’. The concept of ‘Hybrid Filmmaking’.
The most iconic of these is of course the American series The Mandalorian, which started last year. The Mandalorian, the Star Wars series on Disney’s streaming platform, won several awards upon its broadcast. Its stunning visual effects were made possible by new filming and production techniques, including the use of 3D LED screens instead of traditional green screens, virtual production and StageCraft.
The advantages of using virtual production for film production or acting events were explored even further during the epidemic.
Following the success of MANDALORIAN, numerous producers have seen the absolute advantage of replacing green screen productions with LEDs. In the recently filmed third season of WESTWORLD, the famous director Nolan, who directed INTRACTIONS and STAR TREK, and other producers have also chosen to use LED screens to create several of the magnificent scenes in the film.
In addition to the big hits such as The Mandalorian, Katy Perry’s stunning ‘Daisies’ performance on American Idol was also noteworthy.
Katy Perry’s ‘Daisies’ performance on ABC’s American Idol was created in full comic book style and was the most comprehensive use of extended reality (XR) technology in television history. Interact with Katy Perry in this virtual 3D interactive environment throughout
The use of LEDs for virtual productions or extended reality may seem like an alternative solution to adopt during the epidemic, but it is in fact a necessity for the industry to evolve.
Particularly in the film and broadcast sector, LED has unique features such as real-time physical effects and immersive experiences. With virtual production technology, otherwise obscure green screens can be shown in real time with very complete camera art within the footage. Performers can also return to the story scene and see their own play right there in the scene, without having to rely on their imagination.
This also resolves the contradiction between post-production visual effects and pre-filming concepts, saving on revision and production costs. Of course, the advantages of LED virtual production go far beyond this, it is a field full of endless possibilities.
References: