In early December 2024, a promotional video appeared online, where the person vigorously promoting a food product was actually Dr. Zhang Wenhong, the director of the National Center for Infectious Disease Medicine and head of the Infectious Diseases Department at Huashan Hospital affiliated with Fudan University.
Many netizens, trusting Dr. Zhang, not only purchased the product themselves but also actively shared the video in family groups.However,Dr. Zhang quickly clarified that the video was not recorded by him and was likely fabricated using AI.
Network Screenshot
In recent years, with the rapid development of artificial intelligence technology, such incidents of AI-forged videos have frequently come to light. Criminals utilize AI technology to impersonate celebrities for fraud, release false advertisements, and even create forged videos to deceive people out of substantial assets, seriously threatening the social trust system and information security.
Today, we will analyze such incidents from a technical perspective.
Technology Behind AI-Forged Videos:
From Generative Adversarial Networks to Deepfake
The technology used for these AI-forged videos is not new; research on facial replacement and image generation techniques began in academia in the 1990s. In 2014, Ian Goodfellow proposed Generative Adversarial Networks (GANs), allowing computers to generate more realistic and higher-quality images.
Generative Adversarial Networks consist of two parts: the generator and the discriminator. The generator creates content while the discriminator detects whether the created content is real and provides feedback to the generator. During the training process, the two continually challenge each other, enhancing their capabilities until the generator can produce very realistic images.
In 2017, someone created an account named “deepfakes” on the online community Reddit and published celebrity face-swapping videos made using GAN technology. Since then, the term “Deepfake” has been widely used, typically referring to the technology used for these face-swapping videos.
In January 2018, a desktop application called FakeApp was released. Subsequently, open-source tools like Faceswap and DeepFaceLab with similar functionalities emerged. These tools significantly lowered the barrier to creating Deepfakes, enabling ordinary users without specialized knowledge to easily generate face-swapping videos.
With the continuous advancement of related technologies, today’s Deepfakes not only generate videos with higher resolution and more natural facial expression synchronization but also require less data and shorter training times.Early Deepfake training often required hundreds to thousands of images of the target person or several minutes to hours of video to capture facial features from different angles and expressions. However, with the development of GAN technology itself, techniques like Transfer Learning and Few-Shot Learning have emerged, allowing Deepfake videos to be generated with just a few dozen or even a single photo.
Moreover, early Deepfakes could only generate visuals, but now, combined with voice cloning and voice style transfer technologies, they can also generate lifelike voice replicas of the target person, ensuring that the lip movements in the video match the audio.
In simple terms, now it only takes one photo and a few seconds of audio to generate a Deepfake video. Of course, if more photos and longer audio are available, the generated video will be even more realistic.
Positive Applications and Negative Impacts of Deepfake Technology
Although Deepfake is “forgery,” it can have many positive applications when the consent of the person being “forged” is obtained. For example, in 2019, British football star David Beckham released a video calling for the eradication of malaria. In the video, he used nine languages, including Swahili and Yoruba, with all but English generated using Deepfake technology.
Additionally, the various technologies used in Deepfake have broad applications in fields like digital humans, video streaming, film production, education and training, and psychological rehabilitation.
However, any technology can be misused. Before the aforementioned fake Dr. Zhang incident, there had already been cases of fake Jin Dong, fake Jack Ma, fake Lei Jun, and fake Yu Donglai. Impersonating celebrities for selling products is not the worst; Deepfake technology has also been used in many more malicious scenarios, such as fraud.
On December 20, 2024, the BBC reported a story titled “Love Scammer Uses Deepfake to Steal £17,000 from Me.” The victim was 77-year-old Nikki MacLeod. The scammer told her he worked on an oil drilling platform and asked Nikki to purchase Steam gift cards and transfer money via Paypal to obtain internet access on the drilling platform and cover travel costs to meet her in Scotland. Nikki was initially skeptical but believed the scammer after seeing the video from the drilling platform.
In January 2024, an employee of a Hong Kong company transferred $25 million from the company’s account to a scammer. The scammer conducted a video call with him posing as the CFO, requesting the transfer. During the video call, the employee saw not only the “CFO” but also other “colleagues.”
According to a report published in May 2024 by the renowned accounting firm Deloitte, Deepfake fraud in the U.S. increased by 700% in 2023, resulting in losses of $12.3 billion, and this figure could reach $40 billion by 2027.
Besides fraud, Deepfake technology can also be used to deceive facial recognition systems. Many mobile applications use facial recognition to verify user identity. To prevent photos from impersonating faces, current facial recognition technology typically performs liveness detection. Technologies like 3D structured light require special hardware for liveness recognition, which only some phones support.
Currently, many environments still use 2D facial recognition based on the front camera of mobile phones. For 2D facial recognition, liveness detection mainly requires users to perform actions like blinking or nodding, and the screen flashes specific colors of light. Thus, some have used Deepfake technology combined with other methods to deceive facial recognition and steal online accounts.
How to Identify and Detect Deepfake Videos?
Poorly made Deepfake videos can often be identified with the naked eye. For instance, unnatural facial expressions or eye movements, infrequent blinking, blurred facial edges, or unnatural transitions with the background, where the lighting effects on the face do not match the surrounding environment, can all indicate a forgery. However, as Deepfake technology progresses, these abnormal features are becoming less common.
Currently known Deepfake technology can imitate facial expressions but cannot replicate the deformations that occur when a face is subjected to external pressure. Therefore, during video calls, if fraud is suspected, one can ask the other party to press one side of their nostril or one side of their cheek.
In addition to visual identification, using AI to recognize AI-generated content is also a popular research direction.For example, synthetic videos may exhibit discontinuities between frames, and temporal consistency analysis may reveal abnormalities. Additionally, a person’s heartbeat causes subtle color changes in the skin that are rhythmically consistent with their pulse; Deepfake videos may lack this characteristic.
However, we must also recognize that the GAN technology used in Deepfakes consists of both a generator and a discriminator, meaning any detection technology could also be incorporated into the Deepfake discriminator, thus creating content that is difficult to detect.
In recent years, China has implemented regulations such as the “Management Regulations on Deep Synthesis of Internet Information Services” and the “Interim Measures for the Management of Generative Artificial Intelligence Services,” but these laws mainly constrain the platforms providing related services. With the development of software and hardware technology, Deepfake can now be completed using models running on personal computers.
Therefore, to address the various issues posed by Deepfakes,we need a comprehensive governance mechanism for the entire process, requiring multi-faceted collaboration between technology, platforms, and laws in the future.Impersonating celebrities using Deepfake to deceive others into purchasing products may not only violate Article 1019 and Article 1024 of the Civil Code, infringing upon others’ portrait rights and reputation rights but may also involve fraud under Article 266 of the Criminal Law and false advertising under Article 222 of the Criminal Law. Actions involving illegal or criminal use of Deepfake must be addressed according to the law.
Planning and Production
Author: Yu Yang, Head of Tencent Xuanwu Laboratory
Reviewed by: Yu Naigong, Head of the Robotics Engineering Program at Beijing University of Technology, Director of the Robotics Research Center at the Beijing Academy of Artificial Intelligence, PhD Supervisor
Zhao Jingwu, Associate Professor at the School of Law, Beihang University, Deputy Director of the Key Laboratory of Law and Strategy for Industrial and Information Technology of the Ministry of Industry and Information Technology, Deputy Director of the International Governance Research Base for Cyberspace
Editor: Lin Lin
Proofread by: Xu Lai
Science Popularization China
