An Introduction to Synthetic Image Generation from Text Data

Synthetic Image generation is the creation of artificially generated images that look as realistic as real images. These images can be created by Generation Adversarial Networks(GAN) which use a generator-discriminator architecture to train, generate and rate synthetic images that create a creation-feedback loop that runs multiple times until the generated synthetic image can fool the discriminator enough to be considered a real image. Another way to create synthetic images would be with Variational Autoencoders, and more recently, Vector Quantized Variational Autoencoders (VQ-VAE), which create a discrete latent representation and create more variety of images and is easier to train compared to GANs.

What if images can be generated by simply typing a short description of the image? That’s now possible with the advancement in GANs and Variational Autoencoders. In this article, we will look at several such architectures and models that can make this possible.

Impact of Synthetic Image Generation

Synthetic Image Generation can have varying impacts depending on how it’s used. It can be utilized in several machine learning business problems as well as help solve the scarcity of real data.

Pros of Synthetic Image Generation

• Synthetic Image generation can solve the business problem of scarcity or unavailability of image data by creating synthetic image data.
• Text to image generation can be used in conversational chatbots to generate contextual images based on user input.
• Synthetic images can be utilized to train ML models where the existing real image data does not have much variety. Synthetic images can be generated to add more variation to the existing image dataset before training the model.
• Text to image generation can be sued for search engines to create copyright-free images on the go as well as create synthetic images when there are limited search results.

Cons of Synthetic Image Generation

• Synthetic Image Generation can be used to create fake doctored images that can deceive the public.
• Synthetic Images, if not trained and generated with good accuracy and realism, can reduce the quality of the existing image dataset instead of improving it.

Text to Image Generation

Several types of architectures and models can be trained to generate synthetic images based on a string of text data. We will look at some of these in detail below.

DALL-E

DALL-E is a neural network created by OpenAI which is trained on image-text pairs with 12 billion parameters and can generate synthetic images from any text description. It is capable of various types of image generation aspects like animals and objects with their anthropomorphic versions, transforming and adding variety to images, and can also combine several aspects and details of images that are not related in a plausible manner.

Some examples of images generated by DALL-E:

An armchair in the shape of an avocado

DALL-E architecture is based on Vector Quantized Variational Autoencoder (VQ-VAE) that uses vector quantization to obtain a discrete latent representation. It consists of an encoder-decoder architecture and compared to standard autoencoders, VQ-VAE adds a discrete codebook component to the network. The codebook is a list of vectors associated with a corresponding index used to quantize the bottleneck of the autoencoder. The encoder network output is then obtained and it is then matched with the codebook vectors present, and the nearest vector in Euclidean distance in the codebook is then sent to the decoder.

DALL-E Architecture

Since OpenAI has not released the full details of DALL-E, DALL-E mini can be tested to understand how the model works and also try out a visual interface where you can test input keywords and generate synthetic images.

Text to Image with CLIP

Contrastive Language-Image Pre-Training (CLIP) is a neural network trained on a variety of (image, text) pairs. The neural network can be trained in natural language to make predictions based on an image about the most relevant textual description. The model learns the relationship between a whole sentence and the image it describes; in a sense that when the model is trained, given an input sentence it will be able to generate the most accurate images related to the sentence.

CLIP Architecture is based on Zero-shot learning, where the model attempts to predict a class it saw zero times in the training data. So a model trained on exclusively cats and dogs can be then used to detect rabbits. A model like CLIP, because of how it uses the text information in the (image, text) pairs, tends to do well with zero-shot learning — even if the image it’s looking at is different from the training images, the CLIP model will likely be able to give a good guess for the caption for that image.

CLIP Architecture

Below we will see how to generate synthetic images with CLIP:

Install and Import Necessary Libraries

Install the necessary libraries in the Colab notebook and clone the CLIP repository. Import all the necessary modules and set the torch version suffix based on the CUDA version.

!pip install torch==1.7.1 torchvision==0.8.2 ftfy regex

!git clone https://github.com/openai/CLIP.git
%cd /content/CLIP/

Initialization

Set the text string input and the resolution of the image to be generated, and tokenize the user input.

# text data based on which image will be generated
prompt = 'dog wearing a suit'

#image resolution
resolution = "256" #[128, 256, 512]
res = int(resolution)
im_shape = [res, res, 3]
sideX, sideY, channels = im_shape

#tokenize the user input text
tx = clip.tokenize(prompt)

Define some helper functions for image preprocessing, padding and writing the image to disk.

# Define some helper functions
def displ(img, pre_scaled=True):
 img = np.array(img)[:,:,:]
 img = np.transpose(img, (1, 2, 0))
 if not pre_scaled:
 #scale the image
 img = scale(img, 48*4, 32*4)
 imageio.imwrite('result.png', np.array(img))
 file_path = '/content/CLIP/frames/{}.png'.format(str(len(os.listdir('/content/CLIP/frames'))).zfill(5))
 imageio.imwrite(file_path, np.array(img))
 return display.Image('result.png')

def card_padded(im, to_pad=3):
 return np.pad(np.pad(np.pad(im, [[1,1], [1,1], [0,0]],constant_values=0), [[2,2], [2,2], [0,0]],constant_values=1),
 [[to_pad,to_pad], [to_pad,to_pad], [0,0]],constant_values=0)

Neural Network Model

Define the Sine Layer class along with the methods for initializing the weights and getting the tensor with the sine of the elements from the input.

class SineLayer(nn.Module):
 def __init__(self, in_features, out_features, bias=True,
 is_first=False, omega_0=30):
 super().__init__()
 self.omega_0 = omega_0
 self.is_first = is_first
 self.in_features = in_features
 self.linear = nn.Linear(in_features, out_features, bias=bias)
 self.init_weights()

#initialize weights
 def init_weights(self):
 with torch.no_grad():
 if self.is_first:
 self.linear.weight.uniform_(-1 / self.in_features, 
 1 / self.in_features) 
 else:
 self.linear.weight.uniform_(-np.sqrt(6 / self.in_features) / self.omega_0, 
 np.sqrt(6 / self.in_features) / self.omega_0)

#tensor with the sine of the elements of input
 def forward(self, input):
 return torch.sin(self.omega_0 * self.linear(input))
 def forward_with_intermediate(self, input): 
 # For visualization of activation distributions
 intermediate = self.omega_0 * self.linear(input)
 return torch.sin(intermediate), intermediate

class Siren(nn.Module):
 def __init__(self, in_features, hidden_features, hidden_layers, out_features, outermost_linear=True, 
 first_omega_0=30, hidden_omega_0=30.):
 super().__init__()
 self.net = []
 self.net.append(SineLayer(in_features, hidden_features, 
 is_first=True, omega_0=first_omega_0))
 for i in range(hidden_layers):
 self.net.append(SineLayer(hidden_features, hidden_features, 
 is_first=False, omega_0=hidden_omega_0))
 if outermost_linear:
 final_linear = nn.Linear(hidden_features, out_features)
 with torch.no_grad():
 final_linear.weight.uniform_(-np.sqrt(6 / hidden_features) / hidden_omega_0, 
 np.sqrt(6 / hidden_features) / hidden_omega_0)
 self.net.append(final_linear)
 else:
 self.net.append(SineLayer(hidden_features, out_features, 
 is_first=False, omega_0=hidden_omega_0))
 self.net = nn.Sequential(*self.net)
 def forward(self, coords):
 coords = coords.clone().detach().requires_grad_(True)
 output = self.net(coords.cuda())
 return output.view(1, sideX, sideY, 3).permute(0, 3, 1, 2)#.sigmoid_()
 def forward_with_activations(self, coords, retain_grad=False):
 '''Returns not only model output, but also intermediate activations.
 Only used for visualizing activations later!'''
 activations = OrderedDict()
 activation_count = 0
 x = coords.clone().detach().requires_grad_(True)
 activations['input'] = x
 for i, layer in enumerate(self.net):
 if isinstance(layer, SineLayer):
 x, intermed = layer.forward_with_intermediate(x)
 if retain_grad:
 x.retain_grad()
 intermed.retain_grad()
 activations['_'.join((str(layer.__class__), "%d" % activation_count))] = intermed
 activation_count += 1
 else: 
 x = layer(x)
 if retain_grad:
 x.retain_grad()
 activations['_'.join((str(layer.__class__), "%d" % activation_count))] = x
 activation_count += 1
 return activations

def get_mgrid(sidelen, dim=2):
 '''Generates a flattened grid of (x,y,...) coordinates in a range of -1 to 1.
 sidelen: int
 dim: int'''
 tensors = tuple(dim * [torch.linspace(-1, 1, steps=sidelen)])
 mgrid = torch.stack(torch.meshgrid(*tensors), dim=-1)
 mgrid = mgrid.reshape(-1, dim)
 return mgrid

model = Siren(in_features=2, out_features=3, hidden_features=256, 
 hidden_layers=16, outermost_linear=False)
model.cuda()
LLL = []
eps = 0
optimizer = torch.optim.Adam(model.parameters(), .00001)

def checkin(loss):
 #print(loss)
 with torch.no_grad():
 al = nom(model(get_mgrid(sideX)).cpu()).numpy()
 for allls in al:
 displ(allls)
 clear_output()
 pic_num = str(len(os.listdir('/content/CLIP/frames')))
 print(f'Picture number {pic_num}n')
 if int(pic_num) == 1:
 print("The first image is less accurate.")

 display.display(display.Image('result.png'))
 print('Please wait a few seconds until the next picture')

def ascend_txt():
 out = model(get_mgrid(sideX))
 cutn = 64
 p_s = []
 for ch in range(cutn):
 size = torch.randint(int(.5*sideX), int(.98*sideX), ())
 offsetx = torch.randint(0, sideX - size, ())
 offsety = torch.randint(0, sideX - size, ())
 apper = out[:, :, offsetx:offsetx + size, offsety:offsety + size]
 apper = torch.nn.functional.interpolate(apper, (224,224), mode='bilinear')
 p_s.append(nom(apper))
 into = torch.cat(p_s, 0)
 iii = perceptor.encode_image(into)
 t = perceptor.encode_text(tx.cuda())
 return -100*torch.cosine_similarity(t, iii, dim=-1).mean()

def train(epoch, i):
 loss = ascend_txt()
 optimizer.zero_grad()
 loss.backward()
 optimizer.step()
 if itt % fr == 0:
 checkin(loss)
nom = torchvision.transforms.Normalize((0.48145466, 0.4578275, 0.40821073), (0.26862954, 0.26130258, 0.27577711))
itt = 0
for epochs in range(10000):
 for i in range(1000):
 train(eps, i)
 itt+=1
 eps+=1