Diffusers welcomes FLUX. 3 Image Klein to its open-source library

Diffusers welcomes FLUX. 3 Image Klein to its open-source library
3 min read

Black Forest Labs has introduced FLUX.3 Image Klein, a model that handles both text-to-image generation and image editing from one or more reference images. It adds a new feature, layout control, which allows specifying where each element of the image should be located. Two open checkpoints are available, and the model works in the Diffusers library from launch.

To define this layout, the user writes a list of regions in JSON format in their prompt, after the image description. Each region has an identifier, a bounding box (bbox), and a description. Coordinates are written as [y_min, x_min, y_max, x_max], with y first, on a scale from 0 to 1000. This normalized scale allows using the same box regardless of the image size.

The distilled checkpoint generates a 1024 × 1024 pixel image in 16 steps by default. According to published measurements, text-to-image generation takes 1.93 seconds on a single NVIDIA H100 80GB GPU, with a memory peak of 23.8 GiB. A same-size reference image nearly doubles the computation time, to 3.87 seconds. For machines with less memory, the single-file format variants, fp8r and nvfp4, reduce requirements. The post-inference memory peak reaches 15.3 GiB for nvfp4 on an RTX6000, compared to 16.27 GiB for fp8r and 20.72 GiB for bf16 on H100.

A two-stage architecture

The model handles three types of tokens: text, the noisy image being generated, and optionally, reference images. The text encoder is Qwen3-VL, a vision-language model. FLUX.3 does not only take the last layer of this model: it reads the hidden states from eight of its 36 layers and stacks them, giving each text token a vector of 20,480 dimensions, later projected to a width of 3072.

The transformer processes these tokens in two stages. The first stage consists of 5 blocks, with weights specific to each stream, which do not see each other. The second stage combines the streams into a single sequence, “text | image | reference”, and passes it through 28 blocks with shared weights. In this second phase, the image can leverage the text and references. The text and reference tokens are then discarded, and only the image tokens go through the output layers.

Another particularity concerns the noise schedule. In FLUX.1 and FLUX.2, a single timestep embedding modulates all tokens. In FLUX.3, the modulation depends on the token type: image tokens receive the current timestep, while text and reference tokens always receive timestep zero, meaning “clean, nothing to denoise.”

Training LoRAs

The base model runs in 50 steps with classifier-free guidance, and it is this one that the team recommends for fine-tuning. Diffusers provides LoRA training scripts, these small adapters that attach to a model, for both text-to-image generation and image editing. The source also cites Ostris’ AI Toolkit as another possible tool. An example presented in the article trains a tarot card style from a public domain dataset, with 1500 training steps and a rank of 16. The produced adapters are available on the Hub, under the name diffusers-modular/flux3-tarot-lora.

To write good prompts, Black Forest Labs publishes an official prompting guide. The article notes that FLUX.3 responds better to a complete description of the scene, including the subject, setting, lighting, composition, and style.

S’abonner
Notification pour
guest

0 Commentaires
Le plus ancien
Le plus récent Le plus populaire
0
Nous aimerions avoir votre avis, veuillez laisser un commentaire.x