Haven’t you seen LoRA?

Nevidal is a concept for an open-world narrative RPG built on the principles of tabletop role-playing games (D&D). The game unfolds in a universe inspired by the ancient Chinese treatise, «Catalog of Mountains and Seas, ” as reinterpreted by Henri Volokhonovsky in „Mountains and Rivers.“ The open world, numerous detailed characters, and unique locations create an ideal environment for training an AI model to maintain a consistent aesthetic style.

This research aims to train the model in my artistic style and simplify game development, given the necessity of creating a large variety of characters and locations.

To prepare for model training, I executed the nvidia-smi command to check which GPU was available and to ensure the environment was ready for heavy computation. Next, I installed all necessary libraries: bitsandbytes for efficient memory utilization, transformers and accelerate for model handling, and peft (Parameter-Efficient Fine-Tuning)—a library that implements the LoRA method. Separately, I installed diffusers from the GitHub repository and downloaded a specific script, train_dreambooth_lora_sdxl.py, which will manage the entire training process.

The dataset with my images was named nevidali_dataset. I created this directory in the Colab environment to store the data. Then, I uploaded the images and wrote a simple function, image_grid, which allowed me to visually assess the dataset. I outputted a grid of 10 images to ensure all files were uploaded correctly and that the quality of the source material met my expectations.

Simply feeding pictures to a neural network is not enough for training. You must explain what is depicted in them. Manually labeling 50–100 images is tedious work. Therefore, I automated this process by using the BLIP (Bootstrapping Language-Image Pre-training) model. I loaded this model onto the GPU and wrote a function, caption_images, which takes an image path and returns its textual description. This allowed me to describe the entire dataset quickly and consistently.

I devised a custom prefix to describe my style. Next, I wrote a script that iterated through all the files in the nevidali_dataset folder, generated a description for each one using BLIP, and concatenated it with my prefix. As a result, each image received a prompt in the format: prefix [BLIP description]. All these pairs (filename and prompt) were saved into metadata.jsonl, which served as the instruction set for the model during training.

I ran the script train_dreambooth_lora_sdxl.py with a carefully selected set of parameters. I chose the powerful stabilityai/stable-diffusion-xl-base-1.0 model as the base. I specified the path to my dataset and instructed the script to look for prompts in the «prompt» column of the metadata.jsonl file. I selected a resolution of 512×512, a batch size of 4, and a learning rate of 1e-4. The key parameter here was max_train_steps=800—just enough optimization steps for the model to absorb the style without overfitting. After 50 minutes of waiting (depending on the GPU), the process was complete.

Final test. To check how well the model had absorbed the style, I devised five of the most unusual and abstract prompts that had no direct counterparts in the training dataset. These were descriptions of strange animals. I generated the images at a high resolution of 768×768. The result exceeded my expectations—the neural network didn’t just depict the described creatures; it did so in my unique style: bright, high-contrast colors, abstract forms, and a characteristic texture. All generated images were saved and displayed for a final review.

5 test prompts for checking:

a snail smoking a cigar a dog with a feathered head a rubber whale a snail smoking a cigar a dog with a feathered tail

Even on a small dataset, the model successfully learned the style, capturing the key visual features: uneven, coarse contours of a digital brush with a «dripping» effect at the ends of the lines, color saturation, and textured strokes mimicking fluidity and melting.

For the final prompts, I used translations of descriptions from Henri Volokhonsky’s «Mountains and Rivers.»

The descriptions used for these prompts were:

1. Bumblebees there with pointed ends. Their tongues are turned out.
2. Or Owls with three eyes and ears, whose whistle resembles a rasp.

Clearly, the model simplifies these elaborate and strange descriptions to generate more simple and understandable characters.

But if the prompt is clear and not absurd (well, maybe just a little off), the character comes out clear and even matches the description:

There are Hedgehogs the color of fire

Prompts

A Hare with a bird’s beak has owl eyes and a snake’s tail

Three green beasts whose torsos are fused. This is why they are called «Sroshi.»

Prompts:

Special Fish: It is called «Salmon» because it sucks elk/A special fish: called «Salmon» because it sucks elk

A beast with a horse’s body, bird wings, human face, and snake tail/A beast with a horse’s body, bird wings, human face, and snake tail

A strange creature: thrown-back skull, erect ears, pig’s snout under a human brow, a unicorn’s body, and large fused legs with cloven hooves like a boar’s/A strange creature: thrown-back skull, erect ears, pig’s snout under a human brow, unicorn’s body, and large fused legs with cloven hooves like a boar’s

A six-legged snake with four wings/A six-legged snake with four wings

Greenish-black Bulls/Greenish-black Bulls

Beyond the Heavenly Pillar Mountain lives a bay Horse without a head/Beyond the Heavenly Pillar Mountain lives a bay Horse without a head

A creature like a hare with a mouse’s face. It is called «Flying Mouse"/A creature like a hare with a mouse’s face. It’s called «Flying Mouse.»

The model successfully learned the style, accurately reproducing the main characteristics: uneven brushstrokes, colors, dripping effects, and a spooky-cute atmosphere. However, prompts should be formulated not from the original treatise—its descriptions are too ornate and archaic—but based on simplified, contemporary interpretations of these fantastical creatures for clarity and generation accuracy.