Research Progress and Prospects of Generative Adversarial Networks (GAN)

GAN is an idea proposed by Goodfellow in 2014, which is a relatively straightforward approach. For example, imagine we have two images, one real and one fake. How can humans determine whether the painting is forged or real? And why can we think this way? For instance, in Figure 1, the issue with the forged image is that the “person in the painting” is not depicted correctly; it is a rabbit instead of a person, so we can consider it fake. As for the forger, they will think: the part where the person should be is drawn incorrectly; if they improve this part next time, they can create a more realistic painting. The next time they draw it, people may find another issue somewhere else. This iterative process of continuous improvement enhances our generator, which is the person creating the fake painting in this story, and simultaneously improves the detective’s skills.

Figure 1

In fact, GAN did not develop well for a long time after its proposal. The image shows the publication results obtained by searching for the keyword GAN on Arxiv. Before the 2016 NIPS conference, not many articles on GAN were published, but after 2016, it developed significantly, currently showing an exponential growth trend.

Among the authors of GAN papers, the first is Bengio, followed by Goodfellow and Professor Yu Yong from China. Their main research content is still focused on computer vision, and it can be said that the research hotspots in the past two years have mainly concentrated on aspects of image generation. In addition, we can see that GAN also involves many other areas, such as encryption and security, robotics, and even astrophysics. Therefore, it can be seen that the development direction of GAN is continuously expanding, moving from traditional computer vision to other fields.

In an interview, LeCun mentioned that for deep learning, the most noteworthy idea in the next ten years is GAN, and we will discuss why later. First, let’s look at some familiar applications of GAN. In Figure 2, GAN is used to generate the next frame in a video; compared to the MSE method, which may result in the next frame lacking ears on the character, the GAN method clearly shows that the character has ears.

Figure 2

Figure 3

In Figure 3, GAN is used to generate images for autonomous driving, i.e., using GAN to generate images and then using these images for training in autonomous driving. Additionally, in the following series of images, we can see that GAN is used to generate super-resolution images, which is also a common task in the field of computer vision. Adobe’s work goes beyond the traditional work done by GAN, automatically generating scenes like mountains and grasslands from hand-drawn lines. There are also studies that have done similar work, where to add hair to a person, one only needs to doodle a bit, and GAN can automatically generate it. The image shows the generation of a 3D model, which is a relatively rare task compared to previous methods, where a new 3D model can be generated from a single image, noise, or by adding some hidden variables. The work shown in the image is image transformation, where we input an image and obtain an abstract image, sketch, or photo; this task is also not addressed by previous GAN work.

Artificial intelligence initially started from perception and gradually developed towards cognition. At first, we hoped that machines could discern what things were and where they were, but ultimately we hoped they could perform some creative tasks, that is, they could understand what something is and what they can accomplish. It can be seen that after the development of machine learning, traditional recognition problems have already made significant breakthroughs. Therefore, the next step for artificial intelligence is to develop creative and application-oriented work.

Feynman said: “What I cannot create, I do not understand.” If we cannot create something, we cannot truly understand it. We also know that when Goodfellow published his research on adversarial samples in December 2014, he mentioned that neural network image classifiers, although they seem to know what they are classifying, being able to classify cats, dogs, and various things with higher accuracy than humans, do not truly understand what these things are. Why? Because by adding a small amount of noise, we can deceive it. This indicates that neural networks do not truly understand what they classify; they merely establish a pattern based on data, which is significantly different from the real world. If we add a creative function to it, it will help enhance its understanding ability. When talking about GAN, its development also has a long historical evolution.

Long before GAN, there were many works on generative models. A generative model generally refers to a model that randomly generates observed data, which is different from what we typically refer to as discriminative models. A generative model is a full probabilistic model of all variables, while a discriminative model seeks the probability of the target variable given the observed variable values.

For example, given an image, what is the probability that it is a cat? This is a discriminative model, a conditional probability model. Such a model is not a generative model; it provides little information or random information and seeks to obtain a real image. For instance, in the common example of a cat model, using GAN to generate an image of a cat. In traditional work, we might only determine whether it is a cat or a dog, rather than generating how it looks. GAN models have many applications, such as representing and processing high-dimensional distributions, embedding in reinforcement learning to generate data for agents to learn from, or using it for semi-supervised learning, and many other problems can also utilize it.

Generally speaking, the starting point of artificial intelligence research has two parts: one is to study the generated things from the perspective of human understanding of data. For human understanding of data, we know that human experience is very limited, and we must learn from some mathematically or intuitively perceivable examples from reality. In the past, when we did generative models, I would first set a distribution, such as a Gaussian distribution, assuming that our images conform to this distribution, but we still do not know what the parameter distribution is; we can fit this distribution through data. However, there is also another possibility, where we can use machines or models to directly understand this data. That is, we do not make any assumptions; we simply let a model generate data and then judge whether this data is correct or wrong, whether it is similar to real data or too far from real data, and we will repeatedly correct this model based on this judgment. This is one perspective on understanding machine data. It can be seen that previous research on generative models mainly focused on the perspective of human understanding of data; we always hoped to use an explicit distribution assumption to design our models. For example, we might design a Gaussian distribution or, as done in VAB, add some noise of a certain distribution to the data to let our fitting function estimate its distribution. However, GAN can be said to be the first widely known model that fits data from a data perspective. We no longer stick to any assumptions and do not give it any data distribution; we learn directly.

Of course, such work obviously cannot be separated from the development of existing technologies, namely the well-known development of deep learning.

We know that the development of artificial intelligence has gone through three waves, and has experienced two valleys. We still do not know when the third valley will occur. In the beginning, AI mainly focused on theorem proving and some checkers programs. It can be seen that this mainly used a rule-based approach, where we manually designed its rules. However, there are several issues with this approach. First, the rules do not seem to apply well; our human experience is limited, and the rules we design are constrained by our personal experiences, which cannot encompass everything, leading to the system’s inability to adapt to new data. Additionally, since rule usage requires manual design, it also results in slow updates that cannot adapt to a large amount of data. The second wave also aimed to solve this problem. The idea at that time was to use expert systems to invite existing domain experts to handle these issues. At this time, neural networks had already begun to develop, but everyone knows that Minsky proved that traditional perceptrons could not solve the XOR problem, which had a significant impact, leading to the second valley period. After this second valley period, the third wave began. Firstly, in 1991, Deep Blue defeated the chess champion, which was an IBM project, and in fact, this work was still based on rule-based models but used powerful computing power. We can see that the beginning of the third wave of AI development is not only related to algorithm design but also closely related to hardware. Today, the explosion of AI is mainly based on three aspects: cloud computing, big data, and deep learning. It can be seen that both cloud computing and big data are based on the development of the internet and our existing database designs, which have significantly enhanced our ability to collect data.

In the first two waves, our data needed to be collected manually, which consumed a lot of time. However, during the third wave of AI explosion, we already had the ability to automate data collection. On this basis, deep learning was proposed to handle this large-scale data, which also benefited from the development of GPUs, allowing neural networks that previously took a long time to train to be trained quickly. The most famous examples are AlphaGo defeating Lee Sedol last year and this year defeating Ke Jie. At this time, artificial intelligence, or deep learning, became widely known. The image shows a brief history of the development of deep learning, starting from 1958, where theologians and psychologists conducted research on this topic, studying neural networks. However, soon Minsky began to oppose this research, which led to a stagnation. Until after 2007, around 2011-2012, ICNet achieved nearly 10% higher accuracy than traditional methods on the ImageNet dataset, after which major companies began to promote this new technology.

Finally, there is adversarial thinking. Traditionally, we might think that the objective function used during neural network training needs to be designed manually. We might design an MSE or other methods and need to supervise it during learning. However, more and more people have discovered that we do not need to specify a very clear problem; we can use a large feedback mechanism to control it. Just like in AlphaGo, we use two networks to play against each other; we train a similar network, and then the network that loses adjusts itself. On the other hand, as I mentioned earlier, there is the adversarial problem. Although there is no clear relationship with GAN, they do the same thing. Personally, I believe that we can clearly see in the case of adversarial samples that neural networks have an inherent flaw, which is that they cannot truly understand what they see. This problem will drive the next stage of our work’s development.

As can be seen, the above content represents adversarial thinking, combining neural networks, our current deep models, and the development of GPUs. Thus, our research work naturally transitions to the next stage, which should use deep learning models to conduct research on generative models.

In light of the limitations mentioned above, Goodfellow proposed the idea of GAN. His basic idea is to design a game involving two players, one being the Generator, whose job is to generate images that appear to come from the training samples, while the other player is the Discriminator, whose goal is to judge whether the input image is indeed from the training samples rather than generated. The overall framework is roughly as follows: a random variable, a latent space random variable—this may be sampled from a Gaussian distribution, or it may carry some information significance as a latent variable, whose dimension may be lower than that of our real samples in the real space. This latent variable is then input into our generator, which is a differentiable function G(z). We will input both G(z) and the real sample G into our Discriminator, which tries to determine whether our generated image is a fake image. Meanwhile, G will try to make our Discriminator mistakenly believe that the image is from real images. However, whether D’s judgment is correct, it will backpropagate its gradient to both G and D, which is why we emphasize that both G and D are differentiable functions. If they are not differentiable, then our errors cannot be backpropagated. Through this method, we have designed the original GAN structure.

Of course, there are many improved models; the original GAN had both its Discriminator and Generator as fully connected networks for generating images. Now, convolutional neural networks (CNNs) are mainly used to design our input and output images.

For tasks related to natural language understanding, for instance, if we have a piece of text. We know that text is not continuous data, and we cannot train it like images using RGB values. There are many solutions; SeqGAN is a relatively common model that actually improves the original language model generated using reinforcement learning to make it suitable for integration with our GAN.

The advantages of GAN, compared to traditional models, are that the complexity of the generated data is linearly related to our dimensions. This means that if you want to generate a larger image, you do not face an exponential increase in computational load as with traditional models; it is merely a linear increase in the neural network process. Secondly, its prior assumptions are very few, which is the greatest advantage compared to traditional models. That is, we do not make any assumptions about the data; we do not assume it follows any distribution. Thirdly, it can generate higher quality samples. However, the reason for this is not yet well explained. Of course, compared to traditional models, it has advantages, but it also has disadvantages. Traditional discriminative models are generally optimization functions, and we know that for convex optimization, there is an optimal solution that we can achieve. However, GAN is essentially searching for a Nash equilibrium point in a two-player game. A Nash equilibrium point, if for a definite strategy, such as a neural network, is a definite strategy; we input a quantity and surely get a definite output. At this time, you cannot be sure that you can find a Nash equilibrium point. The research on searching and optimizing the Nash equilibrium point for GAN is currently still insufficient, and this work is actually quite challenging.

Now, specifically discussing how we determine whether D is correct and how we backpropagate gradients, in a paper, an original performance function for the Discriminator was proposed. It is actually a standard binary classification cross-entropy loss function. The difference is that its training data is divided into two parts, one part from the real dataset and the other from the generator. The first half aims to ensure that the real data is classified correctly, while the second part hopes to effectively determine whether the data is fake.

Having talked about the loss function of the Discriminator, let’s discuss the loss function of the Generator. GAN is designed as a two-player zero-sum game. Thus, the loss of the Generator is the gain of the Discriminator. Naturally, we can set the loss function of the Generator as the negative of the loss function of the Discriminator. Under this setting, we further proved the J-S measure between the real distribution and the generated distribution of GAN. However, there is a problem here: if we reject G with very high confidence, the gradient of our formula will approach zero. We observe that the main source of this data will come from the first term, while the second term, regarding each related aspect, actually has a very small gradient. In this case, your training may be very slow. Therefore, a compromise solution was proposed: we no longer consider the first term for the loss function of G, that is, we no longer consider whether the Discriminator correctly classifies real samples, but only consider its classification of generated samples. That is, using this exponential function, the second term of the formula, the maximization objective of our Generator is the error rate of our Discriminator.

Furthermore, we consider this problem from a probabilistic perspective, that is, minimizing the K-L divergence between the generative model and the real data. Minimizing the divergence in the formula is equivalent to minimizing the K-L divergence between the model and the data. We will not train the generator to optimality before using it; in practical use, we generally use f-divergence, which is to replace the summation symbol with an f function.

Now, combining all these elements, we can specifically discuss the GAN algorithm. There are several issues here. The first issue is during sampling; we generally use the mini-batch method to sample generated data and optimize. This means we will process it in batches to avoid large fluctuations if we train one sample at a time. The second issue is that the algorithm in the original paper has been improved in various ways, including gradient clipping and other techniques.

Let’s take a look at the modern development of GAN and the issues it faces. The main example is the non-convergence issue of GAN; if you have experimented, you might know that GAN convergence is very challenging. It often requires setting many parameters, such as learning rates and network structures, to properly adjust the capabilities of G and D to eventually achieve convergence. The main reasons stem from two parts. First, there is the gradient vanishing problem, which means that when optimizing, the loss functions of our generator and discriminator will experience gradient vanishing issues. Thus, we need to design better loss functions to alleviate the gradient vanishing problem. The second issue is the mode collapse problem, which means our generator may produce the same data rather than diverse data. For example, when generating cats, it may only produce black cats or generate all zeros in MNIST data. This issue mainly arises because our gradient descent approach does not distinguish between min-max and max-min, leading our generator to prefer generating some repeated but safe samples. For instance, it may generate some black cats or zeros, or it might even reach a point where it generates something that is exactly the same as real samples, but it only fits this specific case. This is not what we want; we want the generator to produce diverse samples that are as varied as the real samples.

So what can we do? We will address this problem later, but here we will briefly mention it.

First, for the first problem, we can now use some improved objective functions. This includes a paper from 2016 that uses reference and virtual batch normalization. The basic idea is to perform gradient descent on not just one sample but on a batch of samples. Additionally, there are LSGAN, which is Least Squares GAN, WGAN, and Loss-Sensitive GAN, all of which utilize different loss functions to process the original loss function, making it smoother and less prone to gradient vanishing. Another more direct approach is to provide additional information to GAN, including class-condition GAN, which inputs samples to GAN and hopes that the generated content can also be classified by the discriminator as the same sample. There is also info GAN.

In addition to the theoretical issues of GAN, we also have application issues with GAN. Here, I will mainly discuss the application of GAN in reinforcement learning. In 2016, a paper discussed the similarities between GAN and Actor-Critic. As shown in this image, we can see that the network structure of GAN is similar to the left side, while Actor-Critic, as a method of reinforcement learning, has the structure on the right.

For GAN, the z input goes to G, and the output from G is fed into the Discriminator for judgment of whether it is correct or not. For Actor-Critic, we input the state from the environment into π, which outputs to Q, that is, the Credit evaluator assesses our strategy. We see the difference between reinforcement learning and GAN lies in z, because we know that the state of the environment has some randomness, while in GAN, these two are separated. We see that these two structures have consistency, but do their training methods also have consistency? In fact, they do; most methods that can be used in Actor-Critic have been studied to see how to transfer stable performance methods from reinforcement learning to GAN. On the other hand, GAN can provide some data for reinforcement learning. For reinforcement learning, data is scarce; we may take a long time to obtain such data, or our data sampling may be very expensive, requiring high costs to collect it. Therefore, we can design a framework called enhanced GAN, where we use the data generated by GAN for reinforcement learning.

For imitation learning, the 2016 paper requires us to learn a reward function from expert samples and then use this reward function to train our reinforcement learning agent. In this paper, the method is straightforward; we do not directly learn the reward function but directly learn the mapping from state to action (s_t, a_t), where s_t is equivalent to G in GAN, and a_t is equivalent to G(z) in GAN. We define the expert’s behavior as the mapping directly without going through intermediate steps.

This is the current progress of traditional GAN, and later we will introduce the progress of GAN in our work.

Everyone knows AlphaGo; I believe that AlphaGo’s success does not come from deep learning but from adversarial thinking, similar to GAN.

Computers have developed to this day based on Turing’s hypothesis, which is unprovable. However, von Neumann proposed this hypothesis that all computable numbers can be computed using Turing’s theory; this is unprovable, and Turing can only compute a finite number, while in reality, the computable numbers should be infinite. However, based on this hypothesis, from von Neumann to today’s computer industry and information industry, the effect produced by AlphaGo is similar to the effect of the Church-Turing Thesis.

AlphaGo has provided a path, a path from Newton to Merton, from small data to big data to small knowledge, marking the arrival of the era of intelligent technology and intelligent industry. After AlphaGo, IT is no longer just information technology; it is intelligent technology. Two hundred years ago, IT meant industrial technology, which is closely related to Popper’s theory of three worlds. I only know the physical world and the psychological world, but Popper told us there is a third world, the artificial world, so today is the era of artificial intelligence.

Surrounding the development of physical technology, we have transitioned from agricultural technology to industrial foundations. Surrounding the psychological world, we have developed information technology. Today, we are beginning to develop the third world, the artificial world, where big data becomes a resource, and populations and intelligence become resources. This leads to IT becoming new IT. From the industrial age to the information age to the intelligent age, this is a parallel world.

The foundational structure of this society has come about this way; the entire foundational structure of society initially was a transportation network. In 2000, the world’s first high-speed road network was established. It was only later when the industrial age arrived that we had power grids, energy networks, and telephone networks, developing into today’s internet, and now starting to create the internet of things. However, I believe that the internet of things is a transition; we should soon enter the era of intelligent networks. Today, it is already unnecessary for me to explain what intelligent industry is; this intelligent industry requires intelligent infrastructure, which is CPSS. When various 5.0 systems integrate social space, physical space, information space, and virtual space, only then can human-machine integration be achieved, allowing for philosophical unity of knowledge and action, where the knowledge you possess aligns with the actions you take, ultimately reaching a realm of unity between the virtual and the real.

31 years ago, Minsky’s “Society of Mind” proposed profound changes to artificial intelligence. What connection does it have with Karl Popper’s three worlds? These five networks bring the three worlds together. We build transportation networks around the physical world, information networks around the psychological world, now called the virtual world or artificial world, and intelligent networks. In between, there have been transitions, such as network 2.0, energy networks, and network 4.0, the internet of things.

Now, we are naturally moving towards Industry 5.0. Industry 1.0 was the steam engine, so there were mechanical engineering departments in universities. Later, with the advent of electric motors, universities began to have electrical engineering departments. Now, how many universities still research electric motors? Then came computers, and universities started having computer science departments, but now those studying algorithms and computational structures have also become a minority. The internet of things and the rise of routers have led many schools to establish IoT departments, but I believe this is just a transition; we have already entered the era of balance, that is, the intelligent age.

This is not meant to counter Germany’s Industry 4.0; Industry 4.0 is a very good concept, slogan, and marketing strategy. Its essence is ICT + CPS. Our understanding of these two terms is different. What is ICT? What is CPS? 4.0 is an understanding from the past; it refers to information and communication technology, while we view intelligent connected technology as information physical systems, leading to different understandings and different eras.

This brings about a new era. Karl Jaspers and Marx share the same name, and Marx was born in the year the People’s Republic of China was founded. In that year, he wrote “The Origin and Goal of History,” mentioning the concept of the “Axial Age,” which lasted from 800 BC to 600 BC, during which humanity experienced a great rise, with figures like Laozi, Mencius, Confucius, Zhuangzi, and Sunzi emerging in China, and Socrates, Plato, and Aristotle in ancient Greece, along with Judaism, Hinduism, and Middle Eastern religions, marking the first Axial Age and a philosophical breakthrough. At the end of the Axial Age, the Western Han dynasty began the Silk Road, marking the first globalization, but unfortunately, the technology at that time did not support it. Later came the second Axial Age, which many believe has just begun, marked by religious transformation. I believe the Renaissance brought forth a rise of rationality from Copernicus to Galileo, Newton, and Einstein, marking the second Axial Age. At the beginning of this era, which lasted 600 years, from 1400 to 1900, China also began its second globalization with Zheng He’s voyages to the West, but ultimately brought nothing back, while Columbus’s discovery of America achieved true first globalization, which I term composite globalization due to colonization, where guns were pointed at you.

At the end of this era, after World War II, the United States began the Marshall Plan, and the world started free trade, which was zero-sum; they only sold you things without providing technology, and you had to obediently stay in your original position, and they would not recognize your peaceful rise. However, now this era has begun, stemming from Gödel’s incompleteness theorem. It is because of Gödel’s incompleteness theorem that everyone began to develop artificial intelligence and computers, leading to a great rise of intelligence, which brings about today’s intelligent technology.

The third Axial Age has begun; China has started the “Belt and Road” initiative. This Axial Age is characterized by harmony and inclusiveness. The foremost figure in artificial intelligence in China is Wang Hao, who previously studied philosophy with Feng Youlan and Jin Yuelin at Southwest United University, and later studied philosophy and logic at Harvard. In the 1960s, the simplest computer proved hundreds of definitions from Russell and Whitehead’s “Principia Mathematica” in just a few minutes, becoming the first to prove theorems. Wang Hao was awarded the first Herbert Award, and later Wu Wenjun also received this award.

In his later years, he studied Gödel’s definitions, establishing the world’s first Gödel Society, and served as its first president. He wrote two books, stating that Gödel’s thoughts ultimately aim to prove that his theorem is not only applicable in mathematics but also suitable for human society, which he generalized. The generalized Gödel theorem states that the intelligence expressed by algorithms is always less than the intelligence that humans can express through language, and the intelligence that can be expressed in language is always less than the intelligence existing in the brain. This is why I have never believed in the singularity theory and have always doubted that artificial intelligence will surpass humans.

The first two sentences of Laozi’s “Tao Te Ching” are “The way that can be told is not the eternal way,” which is actually a generalized Gödel theorem. Later, Gödel studied how to generalize his theorem to the social domain.

Why am I telling you this? Because we are entering a new era, which I call the third Axial Age. We are currently in an explosion of intelligence; rationality has reached its limits, and we must rely on human intelligence. We have entered the third world, transitioning from rational recognition to data recognition, which is the Merton theorem.

Therefore, I believe that algorithms like GAN have only opened the door. I certainly hope that everyone can master such methods and create their own methods, transforming small data into big data and big data into small knowledge. Achieving a leap from the Newtonian system to the Merton system. The foundational infrastructure for this is the interaction between the virtual and the real.

Thus, what we are doing here is artificial organization, which is a model. There are not only physical models but also behavioral models. How do we convert big data? It relies on computational experiments, many experiments that cannot be conducted physically due to constraints of cost, legality, morality, and scientific feasibility. By providing large feedback between the model’s behavior and the actual behavior on CPS, parallel execution can be achieved.

Thus, in the future, all systems should be actual plus artificial. It can be one-to-one, many-to-one, one-to-many, or many-to-many. The difference from traditional parallelism is that parallelism divides and conquers, while parallelism expands and governs.

The difference between the Newtonian system and the Merton system is that in the Newtonian system, your analysis does not change its behavior, while in the Merton system, your analysis will necessarily change its behavior. A huge gap will arise between the model and actual behavior. This gap is filled by data and supplemented by parallelism.

In fact, GAN is the simplest parallel system; it is incomplete on both sides. It uses a discriminator to implement the physical system and employs a generator to produce the artificial system. To me, this is the future of parallel machines. It bridges the three worlds: the physical world, the artificial world, and cyberspace.

Thus, in the future, everything should be parallel: driving should be parallel, vision should be parallel, parallel data, parallel materials, etc.

Therefore, I hope that after the class, everyone can think along the lines of parallelism, turning this approach into a special case from Newton to Merton. I am also very willing to support everyone in this direction. Thank you!