Background and concept

Neural Style Transfer, formulated by Leon Gatys, Alexander Ecker, and Matthias Bethge in 2015, shows that deep convolutional networks trained on object recognition encode content geometry and artistic style in distinct internal representations.

StylizeIt is a modular PyTorch implementation of this algorithm. The tool allows users to supply a base photograph and a reference artwork, optimizing an output image to match the spatial arrangement of the photograph while adopting the textures, brushwork, and colors of the artwork.

Mathematical formulation and loss functions

Unlike standard neural network training where weights are updated to fit a dataset, Neural Style Transfer freezes the network weights and updates the pixel values of the target image directly.

The total loss minimized during optimization is a linear combination of three objectives:

1. Content loss: Measures the mean squared error between feature representations of the content image and the generated image. These features are extracted from intermediate convolutional layer conv4_2, preserving high-level semantic structures without enforcing exact low-level pixel matches.
2. Style loss: Measures differences between Gram matrices across multiple layers (conv1_1, conv2_1, conv3_1, conv4_1, and conv5_1). A Gram matrix computes the inner product between vectorized feature maps, capturing spatial correlation patterns and texture statistics independent of where objects appear in the frame.
3. Total variation loss: Acts as a spatial regularizer by penalizing differences between neighboring horizontal and vertical pixels, suppressing high-frequency noise and visual grain in the final render.

Network architecture and optimization

The system uses a pretrained VGG19 model from torchvision.models with parameters set to evaluation mode:

• Feature extraction: The convolutional features of VGG19 serve as perceptual filters. The implementation provides an option to substitute standard max-pooling operations with average pooling, which yields smoother gradient updates and reduces checkerboard artifacts.
• Initialization: Users can initialize the optimization canvas either with the original content image (which converges faster) or with random uniform noise (which gives greater stylistic freedom).
• Optimization routine: The image tensor is iteratively updated using gradient descent, with configurable learning rates and iteration limits managed in neuralStyleTransfer.py.

Command-line configuration

All training parameters are exposed via command-line arguments in exec.py:

# Basic usage
python exec.py "images/content_imgs/bridge.jpg" "images/style_imgs/colors.jpg" "images/stylized_imgs/output.jpg"

# Custom hyperparameters
python exec.py "images/content_imgs/bridge.jpg" "images/style_imgs/colors.jpg" "images/stylized_imgs/output.jpg" \
  --num_iterations 1000 \
  --learning_rate 0.05 \
  --content_weight 1.0 \
  --style_weight 200.0 \
  --total_variation_weight 15.0

Available flags also allow adjusting layer-specific weights (--content_layer_weights, --style_layer_weights), capping maximum image dimensions for memory management (--max_image_size), and exporting periodic checkpoints (--checkpoint_interval).

Example/"demo"

If we take this as the content image:

And this as the style image:

We get the following result: