Skip to content
BranchesTags

Repository files navigation

Minecraft Skin Generator

Generate 64×64 Minecraft player skins from a text prompt and/or a reference image, using a fine-tuned SDXL + PixelArt-XL + IP-Adapter pipeline.

status python license hardware

TL;DR

text  ─┐
       ├─►  SDXL + PixelArt-XL fuse + pa_v2 LoRA  ──►  64×64 RGBA  ──►  Minecraft
ref  ──┘     (+ optional IP-Adapter-plus)              skin PNG         player skin
  • Web UI with Generate / Edit tabs, 3D in-browser viewer, multi-variation gallery
  • Pixel editor built in — touch up the AI output before exporting
  • 38 curated example prompts included in examples/showcase_skins/
  • Honest about limits: outputs are interesting but not yet commercial-quality. See § Known Limitations.

This is a research prototype, not a polished product. I'm publishing it to (a) document what worked and what didn't across ~3 weeks of training experiments, and (b) get feedback / ideas from the community on where to take it next. See § Help Wanted.

Table of contents

Demo

Browse examples/showcase_skins/contact_sheet.png for 38 sample outputs across categories (knight, anime, pirate, animal, sci-fi, etc.) generated with the pa_v2 LoRA at default settings.

The web UI (scripts/web_ui.py) provides:

  • Text → skin generation (1–9 variations per prompt)
  • Optional reference image input (IP-Adapter)
  • 3D in-browser viewer (via skinview3d)
  • Pixel editor for manual touch-up
  • One-click Minecraft skin PNG download

Quick start

Requirements

  • Python 3.11+
  • NVIDIA GPU with ≥ 16 GB VRAM (RTX 3090/4080/4090, A5000, RTX 5070 Ti, etc.)
  • CUDA 12.x
  • ~10 GB disk for model weights + dependencies

Install

# 1. Clone
git clone https://github.com/wjddusrb03/minecraft-skin-generator
cd minecraft-skin-generator

Option A — uv (recommended):

uv sync                       # installs everything including CUDA-12.8 PyTorch

Option B — pip:

PyTorch (CUDA 12.x) needs a separate install command because the wheels are hosted on PyTorch's own index, not PyPI:

python -m venv .venv
# Windows
.venv\Scripts\activate
# macOS / Linux
source .venv/bin/activate

# 1. PyTorch + torchvision (CUDA 12.8 wheels)
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128

# 2. The rest of the project + the "ml" extras (diffusers, peft, etc.)
pip install -e ".[ml]"

Why two steps for pip: this project pins CUDA-12.8 PyTorch via a custom index in pyproject.toml under [tool.uv.sources] — that's a uv-only feature, so pip users must specify the index by hand. uv sync handles both in one step.

Download the pa_v2 LoRA

The 1.4 GB fine-tuned LoRA is published on HuggingFace (see release notes for the upload URL — TODO when uploaded). Place it under checkpoints/sdxl_lora_pa_v2/final/.

Until then, the LoRA must be trained locally — see § Training.

Launch the web UI

Windows: double-click run_web_ui.bat.

Cross-platform:

PYTHONPATH=. python scripts/web_ui.py

The browser will open at http://localhost:7860. First model load takes ~30 s.

Or generate from CLI

PYTHONPATH=. python scripts/generate_skin_pa.py \
    "anime catboy with brown ears and orange hoodie" \
    --seeds 0 42 1234 \
    --reference path/to/character_art.png \
    --reference-scale 0.6

Outputs land in generated/.

How it works

Pipeline at a glance

                           ┌──────────────────────┐
                           │ SDXL base (1024 ckpt) │
                           └──────────┬───────────┘
                                      ▼
                    fuse: nerijs/pixel-art-xl  (locks the pixel-art style)
                                      ▼
                    attach: pa_v2 LoRA  (taught the Minecraft skin UV layout +
                                          short skin captions on 27.7K examples)
                                      ▼
                    optional: IP-Adapter-plus  (color/style guidance from
                                                 a reference image)
                                      ▼
                    sample at 768×768 (28 steps, DPM-Solver++)
                                      ▼
                    NEAREST downscale → 64×64
                                      ▼
                    UV mask + symmetry enforcement + N-color quantization
                                      ▼
                    final skin.png (Minecraft 1.8+ format)

Why this stack?

  • SDXL has strong general visual priors, better than SD 1.5 for stylized character art.
  • PixelArt-XL (nerijs/pixel-art-xl) is fused at load time so every output starts with a pixel-art bias — no anti-aliased blur.
  • pa_v2 LoRA (rank 64, 18 K steps, 27.7K curated skins) teaches the model what Minecraft skin UV layouts look like, since SDXL has no prior for that specific format.
  • IP-Adapter-plus lets a user supply a reference image (anime art, animal photo, another skin) whose palette/style transfers to the output without retraining.
  • Post-processing (src/inference/) enforces the Minecraft format invariants: hard pixel edges, left/right symmetry, limited palette, valid alpha mask.

Why a LoRA, not a full fine-tune?

We only have one consumer GPU. Full SDXL fine-tuning needs 40 GB+ VRAM. LoRA (rank 64 on 8 attention/projection modules) keeps trainable params at ~170 M (6.2 % of the UNet), trainable in bf16 + 8-bit Adam on a 16 GB card.

Project structure

minecraft-skin-generator/
├── scripts/                    # User-facing CLIs
│   ├── web_ui.py                  # Gradio web app (Generate / Edit / 3D viewer)
│   ├── generate_skin_pa.py        # CLI: text → skin (pa_v2 + IP-Adapter)
│   ├── generate_skin_v4.py        # Legacy SD 1.5 generator
│   ├── train_sdxl.py              # LoRA training entrypoint
│   ├── curate_v16.py              # Heuristic dataset curation
│   ├── caption_v17_qwen.py        # Re-captioning with Qwen2.5-VL
│   ├── eval_grid.py               # Build comparison grids
│   ├── composite_eval.py          # LoRA weighted-merge experiment
│   ├── uv_composite_eval.py       # UV-region composite experiment
│   ├── ip_adapter_test_pa_v2.py   # IP-Adapter probe sweep
│   └── pa_v2_knob_sweep.py        # Inference knob sweep
├── src/
│   ├── data/                   # Skin format, UV layout, loader, dedup, render
│   ├── inference/              # SDXL/SD1.5 pipelines, quantize, symmetry
│   └── training/               # Dataset, configs, training loops
├── examples/
│   └── showcase_skins/         # 38 sample outputs + contact_sheet.png
├── run_web_ui.bat              # Windows launcher
├── pyproject.toml              # Python dependencies (uv-managed)
└── README.md                   # this file

Hidden from version control (see .gitignore):

  • checkpoints/ — model weights (download from releases)
  • data/raw/, data/*.jsonl — training data (see § Data sources)
  • logs/, generated/, eval/ — runtime outputs

Training experiments — what we tried

This is the abbreviated record. Multiple LoRAs were trained over ~3 weeks in search of the best skin quality. Findings, in chronological order:

Tag Base Data Captions Steps Verdict
v9 SD 1.5 15 K filtered (metadata_v3) short 18 K OK; v1 production for SD 1.5
pa_v1 SDXL + PixelArt-XL 8 K bootstrap hybrid 18 K First SDXL win; data ceiling at 8 K
pa_v2 ⭐ SDXL + PixelArt-XL 27.7 K (metadata_v13) hybrid (short+long) 18 K Current production. Loss 0.0251. Generalizes to anime / mecha / pirate / catboy / maid / demon.
pa_v3 same 32 K + synth (metadata_v15) random short/long 18 K Quality regression; self-distilled synthesis hurt diversity.
pa_v4 same 12.8 K strict curation + Qwen2.5-VL structured captions short 14 K Clothing structure improved on some prompts; face quality regressed. Overall: worse than pa_v2.
pa_v5 same, warm-start from pa_v2 same as pa_v4 (metadata_v17) short 10 K Closer to pa_v2 (warm-start preserved face) but new captions didn't push past it.

Key lessons

  1. Data variety > data curation tightness. The pa_v4 attempt aggressively curated from 32 K → 12.8 K skins (palette 16-65, outer layers required, classic model only). This lost too much face/style diversity, and structured captions alone couldn't compensate. pa_v2's 27.7 K mixed dataset beat both curated runs.

  2. Caption quality matters, but isn't a silver bullet. Replacing the verbose meta-commentary captions from summykai/minecraft-skins-captioned-900k ("This image depicts a blocky character typical of Minecraft...") with structured spatial captions (pirate: head red hat, torso black coat, arms bare, legs blue pants) helped clothing readability but not overall quality.

  3. Naive LoRA composition destroys quality. Weighted-merging pa_v2 + pa_v4 adapters via add_weighted_adapter produced muddier outputs than either alone — the two LoRAs were trained on different caption distributions, and their weight updates point in incompatible directions.

  4. UV-region image-level compositing is a small win. Cropping pa_v2's head strip (y=0..16 of the UV layout) onto pa_v4's body did give "best of both" in some prompts, but boundaries can look awkward.

  5. Inference-time tuning helps only marginally. Sweeping guidance 5–11, steps 28–48, and stronger negative prompts gives subtle improvements but doesn't address the underlying "structural noise" issue.

  6. IP-Adapter is the most useful inference-only addition. Adding h94/IP-Adapter (plus variant, ViT-H/14) gives users style/color control via reference images without any retraining. Doesn't fix structural issues but unlocks an AI-assistance workflow.

Known limitations

I want to be direct about these — they're the reason this is published as "experimental" not "v1.0".

Output quality

  • Structural noise. Bodies and clothing often look like colored noise patterns shaped like a skin, rather than designed character outfits. Faces tend to be more coherent (small UV region, strong prior in data) but bodies are weak.
  • Specific characters fail. Prompts like "Deku from My Hero Academia" or "Korean school boy" produce muddy/random outputs. The model only generalizes within the rough categories it saw during training (knight, wizard, anime girl, pirate, etc.).
  • Fine details are impossible. At 64×64, a human face is ~8×8 pixels. "Freckles", "round green eyes", "stitched seams" are below the model's resolution budget.

What the model is OK at

  • Short, archetypal prompts: knight in red plate armor with golden trim
  • Color palette guided by reference images (IP-Adapter at 0.4–0.7)
  • Variety: same prompt with different seeds gives different valid designs

What the model is bad at

  • Specific real characters / IP (Deku, Naruto, etc.)
  • Tiny facial features (freckles, specific eye shapes, scars)
  • Logos, text, brand marks
  • Complex multi-part outfits described in long prompts
  • Non-humanoid mobs (animal-shaped skins are weak)

Resource limits

  • Requires ~12 GB VRAM at inference, ~14 GB during training.
  • One generation: ~10 s.
  • Training pa_v2 (18 K steps) takes ~12–16 hours on an RTX 5070 Ti.

Roadmap & ideas

Things I'd try next, in roughly priority order. Contributions on any of these would be very welcome.

Short-term (without retraining)

  • Better prompt templates — curated library shipped with the UI for one-click testing
  • PNG metadata — embed prompt/seed/timestamp in skin PNGs for reproducibility
  • Multiple references — IP-Adapter supports multi-image embedding
  • Inpainting workflow — let users re-generate only the head, only the torso, etc.
  • Color picker / eyedropper in the pixel editor
  • HuggingFace Space deployment — hosted demo

Medium-term (training changes)

  • Mask-weighted loss — weight UV regions (face, body) higher during training
  • UV-template ControlNet — train a ControlNet on per-skin color-block hints to enforce region structure
  • Curated NameMC scrape — augment training data with top-voted community skins (license permitting)
  • DPO / preference tuning — generate pairs, hand-label, fine-tune on preferences

Long-term (architecture changes)

  • Flux base model with pixel-art LoRA — community results suggest substantially higher quality than SDXL for pixel art
  • Native 64×64 pixel diffusion — abandon SDXL's 768→64 downscale entirely (we tried this once as v12; it failed for lack of data, but is worth revisiting)
  • Two-stage generation — silhouette/segmentation map first, then color fill

Help wanted

If you find this project interesting, the most useful contributions are:

  1. Try it and report. What worked? What didn't? Which prompts fail? Open an issue with screenshots.
  2. Better prompts. Add to examples/showcase_skins/index.tsv and submit a PR.
  3. UI polish. The Gradio UI is minimal — see the Roadmap.
  4. Training experiments. All training infrastructure is in place — try a new curation strategy, new caption style, or warm-starting from a different base. src/training/train_sdxl_lora.py accepts --metadata and --init-lora-from for easy experimentation.
  5. Evaluation methodology. We currently have no automated "is this skin good?" metric. Visual eval is too subjective. A learned aesthetic scorer would unlock a lot.
  6. Honest negative results. "I tried X, it didn't work because Y." Those are often more valuable than positive results.

Data sources & licenses

This project was trained on publicly available Minecraft skin datasets:

Dataset Records used License Link
summykai/minecraft-skins-captioned-900k ~50 K sampled MIT HuggingFace
neurlang/Minecraft-Skins-Captioned-1M ~7 K added in v14+ MIT HuggingFace

Each metadata record includes the original source, source_url, license, and scraped_date so attribution can be traced. The raw skin PNGs themselves are NOT redistributed in this repo — re-download from the source datasets above using scripts/ingest_hf_rare.py (or your own ingestion script).

The pre-trained models we depend on:

Model Purpose License
stabilityai/stable-diffusion-xl-base-1.0 Base diffusion model CreativeML Open RAIL++-M
nerijs/pixel-art-xl Pixel-art style LoRA, fused into base (check repo)
h94/IP-Adapter (sdxl_models/plus) Reference-image conditioning Apache 2.0
madebyollin/sdxl-vae-fp16-fix bf16/fp16-friendly VAE MIT
Qwen/Qwen2.5-VL-7B-Instruct Re-captioning pipeline (training only) Apache 2.0

Our pa_v2 LoRA weights inherit from the SDXL base license. If you re-publish generations commercially, please review the SDXL license terms.

Acknowledgments

  • Diffusers, PEFT, transformers, accelerate by HuggingFace
  • The skinview3d library for the in-browser 3D viewer
  • The Minecraft skin dataset contributors at summykai and neurlang
  • The nerijs/pixel-art-xl LoRA author
  • IP-Adapter authors (h94)

This project was developed iteratively with Claude (Anthropic) as a coding collaborator — many of the design decisions, postmortem analyses, and training scripts were built through dialogue. See the commit history.

License

Code: MIT

Generated outputs: inherit the licenses of the underlying pretrained models (SDXL, PixelArt-XL, IP-Adapter, etc.). See NOTICE for the full list. Most notably, SDXL outputs fall under CreativeML Open RAIL++-M.

Korean / 한국어 요약

이 프로젝트는 텍스트 설명 또는 참고 이미지로 마인크래프트 스킨을 생성하는 실험적 AI 도구입니다. SDXL + PixelArt-XL + 자체 학습한 LoRA(pa_v2) + IP-Adapter 조합.

3주간의 학습 실험 (pa_v1 ~ pa_v5) 결과 pa_v2가 가장 좋은 품질을 보였으나, 시중 마인크래프트 스킨 수준은 아직 도달 못함. 솔직하게:

  • ✅ 일반 카테고리 (기사/마법사/해적/애니메이션 등) 적당히 됨
  • ✅ 참고 이미지로 색감/스타일 전이 가능
  • ❌ 특정 캐릭터 (예: 데쿠) 재현 불가
  • ❌ 작은 디테일 (주근깨, 눈동자 모양) 64×64 해상도 한계
  • ❌ "전체적으로 노이즈 패턴" 느낌이 여전

다음 단계 후보: Flux 모델 전환, mask-weighted loss, UV ControlNet, DPO 등. 자세한 내용은 위 Roadmap & Ideas 참조.

피드백 / 아이디어 / 개선 PR 환영합니다.

About

AI Minecraft skin generator (SDXL + LoRA + IP-Adapter)

Topics

aigradioimage-generationip-adapterloraminecraftpixel-artsdxlskinskin-generatorstable-diffusiontext-to-image

Resources

Readme
MIT license

Contributing

Contributing
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages