Hugging Face releases scripts and containers to fine-tune TRELLIS.2, a popular 3D generation model, using one’s own 3D files. These tools can run locally or on Hugging Face Jobs. In an experiment on shoes, the authors report that the soles generated by the adapted model are significantly closer to the reference scans.
A generalist model, imprecise details
3D generation is used notably in robotics, simulation, media, and entertainment. In these fields, the produced objects must adhere to strict rules, which remains difficult to achieve with a general-purpose model. The TRELLIS model, and later TRELLIS.2 from the same team, uses a common latent representation that can be converted into different formats, such as GLB or PLY. However, in its current state, it does not allow for easy adaptation to a specific task.
The authors therefore tested fine-tuning on a subset of 188 footwear objects from Google Scanned Objects (GSO), a dataset containing 1,030 objects in total. In initial tests, the base model generally produced shoes with correct appearance, but with inaccurate soles. The goal was to verify if a LoRA adapter, i.e., a small module trained while keeping the original model’s weights fixed, could improve sole accuracy.
Which parts of the model to train
TRELLIS.2 operates in five steps: background removal from the image, encoding the image into ‘tokens’ (data units processed by the model), predicting the coarse structure, generating detailed geometry, then generating materials. The authors added LoRA adapters only to the diffusion transformer that predicts the coarse structure and to the one that generates the shape, as these two components determine the geometry.
The texturing step was left untouched. The dataset contains color maps that already incorporate lighting, not the separate material maps required for this training. The authors also fear that training on this data would cause the model to learn lighting as a surface property, or to forget what it already knew, a phenomenon called ‘catastrophic forgetting’. The chosen configuration totals about 9.3 million trainable parameters for the two components.
The preparatory work, done with Blender on an a100-large job, took about an hour for the 188 objects. Nine objects were reserved for evaluation, and 179 were used for training. Each training run operated on an 80GB A100. The structure branch took 7.2 hours, and the shape branch 3.6 hours.
Higher scores on the reserved objects
On the nine test objects, the average F-score, which measures both surface precision and coverage, increased from 0.276 for the base model to 0.803 for the model with adapters. The Chamfer distance, which measures the gap between two surfaces (the lower, the better), decreased from 0.174 to 0.033. Normal consistency, which describes the agreement of surface directions, climbed from 0.530 to 0.804. The F-score is better on all nine objects.
The authors note that training loss curves do not directly measure the quality of the generated geometry. The adapters, their configuration, and the trained adapters are published, and the associated GitHub repository allows modifying the scripts. The authors finally cite other avenues, such as Pixal3D, UniLat3D, HunyuanWorld 2.0 and NVIDIA Lyra 2.0.
