scepter 0.0.4

No project description provided

🪄SCEPTER

📖 Table of Contents

• News
• Introduction
• Installation
• Getting Started
• SCEPTER Studio
• Gallery
• Features
• Learn More
• License
• Acknowledgement

🎉 News

• [2024.03]: We optimize the training UI and checkpoint management. New LAR-Gen model has been added on SCEPTER Studio, supporting zoom-out, virtual try on, inpainting.
• [2024.02]: We release new SCEdit controllable image synthesis models for SD v2.1 and SD XL. Multiple strategies applied to accelerate inference time for SCEPTER Studio.
• [2024.01]: We release SCEPTER Studio, an integrated toolkit for data management, model training and inference based on Gradio.
• [2024.01]: SCEdit support controllable image synthesis for training and inference.
• [2023.12]: We propose SCEdit, an efficient and controllable generation framework.
• [2023.12]: We release 🪄SCEPTER library.

📝 Introduction

SCEPTER is an open-source code repository dedicated to generative training, fine-tuning, and inference, encompassing a suite of downstream tasks such as image generation, transfer, editing. It integrates popular community-driven implementations as well as proprietary methods by Tongyi Lab of Alibaba Group, offering a comprehensive toolkit for researchers and practitioners in the field of AIGC. This versatile library is designed to facilitate innovation and accelerate development in the rapidly evolving domain of generative models.

Main Feature:

• Task:
  • Text-to-image generation
  • Controllable image synthesis
  • Image editing
• Training / Inference:
  • Distribute: DDP / FSDP / FairScale / Xformers
  • File system: Local / Http / OSS / Modelscope
• Deploy:
  • Data management
  • Training
  • Inference

Currently supported approaches (and counting):

1. SD Series: Stable Diffusion v1.5 / Stable Diffusion v2.1 / Stable Diffusion XL
2. SCEdit(CVPR2024): SCEdit: Efficient and Controllable Image Diffusion Generation via Skip Connection Editing
3. Res-Tuning(NeurIPS2023 TODO): Res-Tuning: A Flexible and Efficient Tuning Paradigm via Unbinding Tuner from Backbone
4. LAR-Gen: Locate, Assign, Refine: Taming Customized Image Inpainting with Text-Subject Guidance

🛠️ Installation

• Create new environment

conda env create -f environment.yaml
conda activate scepter

• We recommend installing the specific version of PyTorch and accelerate toolbox xFormers. You can install these recommended version by pip:

pip install -r requirements/recommended.txt

• Install SCEPTER by the pip command:

pip install scepter

🚀 Getting Started

Dataset

Modelscope Format

We use a custom-stylized dataset, which included classes 3D, anime, flat illustration, oil painting, sketch, and watercolor, each with 30 image-text pairs.

# pip install modelscope
from modelscope.msdatasets import MsDataset
ms_train_dataset = MsDataset.load('style_custom_dataset', namespace='damo', subset_name='3D', split='train_short')
print(next(iter(ms_train_dataset)))

CSV Format

For the data format used by SCEPTER Studio, please refer to 3D_example_csv.zip.

TXT Format

To facilitate starting training in command-line mode, you can use a dataset in text format, please refer to 3D_example_txt.zip

mkdir -p cache/datasets/ && wget 'https://modelscope.cn/api/v1/models/damo/scepter_scedit/repo?Revision=master&FilePath=dataset/3D_example_txt.zip' -O cache/datasets/3D_example_txt.zip && unzip cache/datasets/3D_example_txt.zip -d cache/datasets/ && rm cache/datasets/3D_example_txt.zip

Training

We provide a framework for training and inference, so the script below is just for illustration purposes. To achieve better results, you can modify the corresponding parameters as needed.

Text-to-Image Generation

• SCEdit

python scepter/tools/run_train.py --cfg scepter/methods/scedit/t2i/sd15_512_sce_t2i.yaml  # SD v1.5
python scepter/tools/run_train.py --cfg scepter/methods/scedit/t2i/sd21_768_sce_t2i.yaml  # SD v2.1
python scepter/tools/run_train.py --cfg scepter/methods/scedit/t2i/sdxl_1024_sce_t2i.yaml  # SD XL

• Existing Tuning Strategies

python scepter/tools/run_train.py --cfg scepter/methods/examples/generation/stable_diffusion_1.5_512.yaml  # fully-tuning on SD v1.5
python scepter/tools/run_train.py --cfg scepter/methods/examples/generation/stable_diffusion_2.1_768_lora.yaml  # lora-tuning on SD v2.1

• Data Text Format

# Download the 3D_example_txt.zip as previously mentioned
python scepter/tools/run_train.py --cfg scepter/methods/scedit/t2i/sdxl_1024_sce_t2i_datatxt.yaml

Controllable Image Synthesis

• SCEdit

The YAML configuration can be modified to combine different base models and conditions. The following is provided as an example.

python scepter/tools/run_train.py --cfg scepter/methods/scedit/ctr/sd15_512_sce_ctr_hed.yaml  # SD v1.5 + hed
python scepter/tools/run_train.py --cfg scepter/methods/scedit/ctr/sd21_768_sce_ctr_canny.yaml  # SD v2.1 + canny
python scepter/tools/run_train.py --cfg scepter/methods/scedit/ctr/sd21_768_sce_ctr_pose.yaml  # SD v2.1 + pose
python scepter/tools/run_train.py --cfg scepter/methods/scedit/ctr/sdxl_1024_sce_ctr_depth.yaml  # SD XL + depth
python scepter/tools/run_train.py --cfg scepter/methods/scedit/ctr/sdxl_1024_sce_ctr_color.yaml  # SD XL + color

• Data Text Format

# Download the 3D_example_txt.zip as previously mentioned
python scepter/tools/run_train.py --cfg scepter/methods/scedit/ctr/sdxl_1024_sce_ctr_color_datatxt.yaml

Inference

Base Model Inference

python scepter/tools/run_inference.py --cfg scepter/methods/examples/generation/stable_diffusion_1.5_512.yaml --prompt 'a cute dog' --save_folder 'inference'  # generation on SD v1.5
python scepter/tools/run_inference.py --cfg scepter/methods/examples/generation/stable_diffusion_2.1_768.yaml --prompt 'a cute dog' --save_folder 'inference'  # generation on SD v2.1
python scepter/tools/run_inference.py --cfg scepter/methods/examples/generation/stable_diffusion_xl_1024.yaml --prompt 'a cute dog' --save_folder 'inference'  # generation on SD XL

Fine-tuned Model Inference

python scepter/tools/run_inference.py --cfg scepter/methods/scedit/t2i/sd15_512_sce_t2i_swift.yaml --pretrained_model 'cache/save_data/sd15_512_sce_t2i_swift/checkpoints/ldm_step-100.pth' --prompt 'A close up of a small rabbit wearing a hat and scarf' --save_folder 'trained_test_prompt_rabbit'

Controllable Image Synthesis Inference

• SCEdit

python scepter/tools/run_inference.py --cfg scepter/methods/scedit/ctr/sd21_768_sce_ctr_canny.yaml --num_samples 1 --prompt 'a single flower is shown in front of a tree' --save_folder 'test_flower_canny' --image_size 768 --task control --image 'asset/images/flower.jpg' --control_mode canny --pretrained_model ms://damo/scepter_scedit@controllable_model/SD2.1/canny_control/0_SwiftSCETuning/pytorch_model.bin   # canny
python scepter/tools/run_inference.py --cfg scepter/methods/scedit/ctr/sd21_768_sce_ctr_pose.yaml --num_samples 1 --prompt 'super mario' --save_folder 'test_mario_pose' --image_size 768 --task control --image 'asset/images/pose_source.png' --control_mode source --pretrained_model ms://damo/scepter_scedit@controllable_model/SD2.1/pose_control/0_SwiftSCETuning/pytorch_model.bin   # pose

Customize Modules

Refer to example, build the modules of your task in example/{task}.

cd example/classifier
python run.py --cfg classifier.yaml

🖥️ SCEPTER Studio

Launch

To fully experience SCEPTER Studio, you can launch the following command line:

pip install scepter
python -m scepter.tools.webui

or run after clone repo code

git clone https://github.com/modelscope/scepter.git
PYTHONPATH=. python scepter/tools/webui.py --cfg scepter/methods/studio/scepter_ui.yaml

The startup of SCEPTER Studio eliminates the need for manual downloading and organizing of models; it will automatically load the corresponding models and store them in a local directory. Depending on the network and hardware situation, the initial startup usually requires 15-60 minutes, primarily involving the download and processing of SDv1.5, SDv2.1, and SDXL models. Therefore, subsequent startups will become much faster (about one minute) as downloading is no longer required.

• LAR-Gen: we release zoom-out, virtual try on, inpainting(text guided), inpainting(text + reference image guided) image editing capabilities. Please note that the Data Preprocess button must be clicked before clicking the Generate button.

Modelscope Studio

We deploy a work studio on Modelscope that includes only the inference tab, please refer to ms_scepter_studio

🖼️ Gallery

LAR-Gen: Zoom Out

Origin Image Prompt: a temple on fire	Zoom-Out CenterAround:0.75	Zoom-Out CenterAround:0.75	Zoom-Out CenterAround:0.75	Zoom-Out CenterAround:0.75

LAR-Gen: Virtual Try-on

Model Image	Model Mask	Clothing Image	Clothing Mask	Try-on Output

LAR-Gen: Inpainting (Text guided)

Origin Image Prompt: a blue and white porcelain	Inpainting Mask1	Inpainting Output1	Inpainting Mask2 Prompt: a clock	Inpainting Output2

LAR-Gen: Inpainting (Text and Subject guided)

Origin Image Prompt: a dog wearing sunglasses	Origin Mask	Reference Image	Reference Mask	Inpainting Output

Dragon Year Special: Dragon Tuner

Gold Dragon Tuner	Sloppy Dragon Tuner	Red Dragon Tuner + Papercraft Mantra	Azure Dragon Tuner + Pose Control

Text Effect Image

Conditional Image	Midas Control "Race track, top view"	Midas Control + Watercolor Mantra "white lilies"	Midas Control + Dragon Tuner "Spring Festival, Chinese dragon"

✨ Features

Text-to-Image Generation

Model	SCEdit	Full	LoRA
SD 1.5	🪄	✅	✅
SD 2.1	🪄	✅	✅
SD XL	🪄	✅	✅

Controllable Image Synthesis

• SCEdit

Model	Canny	HED	Depth	Pose	Color
SD 1.5	✅	✅	✅	✅	✅
SD 2.1	🪄	🪄	🪄	🪄	🪄
SD XL	🪄	🪄	🪄	🪄	🪄

Image Editing

• LAR-Gen

Model	Locate	Assign	Refine
SD XL	🪄	🪄	⏳

Model URL

• ✅ indicates support for both training and inference.
• 🪄 denotes that the model has been published.
• ⏳ denotes that the module has not been integrated currently.
• More models will be released in the future.

Model	URL
SCEdit	ModelScope HuggingFace
LAR-Gen	ModelScope

PS: Scripts running within the SCEPTER framework will automatically fetch and load models based on the required dependency files, eliminating the need for manual downloads.

🔍 Learn More

• Alibaba TongYi Vision Intelligence Lab

  Discover more about open-source projects on image generation, video generation, and editing tasks.

• ModelScope library

  ModelScope Library is the model library of ModelScope project, which contains a large number of popular models.

• SWIFT library

  SWIFT (Scalable lightWeight Infrastructure for Fine-Tuning) is an extensible framwork designed to faciliate lightweight model fine-tuning and inference.

BibTeX

If our work is useful for your research, please consider citing:

@misc{scepter,
    title = {SCEPTER, https://github.com/modelscope/scepter},
    author = {SCEPTER},
    year = {2023}
}

License

This project is licensed under the Apache License (Version 2.0).

Acknowledgement

Thanks to Stability-AI, SWIFT library and Fooocus for their awesome work.