diff --git a/README.md b/README.md index b4ab2f5ab..1337c5b92 100644 --- a/README.md +++ b/README.md @@ -48,6 +48,7 @@ API and command-line option may change frequently.*** - [FLUX.2-dev/FLUX.2-klein](./docs/flux2.md) - [FLUX.3 Image Klein / Klein Base](./docs/flux3.md) - [MOSS-Image-Base / MOSS-Image-SFT](./docs/moss_image.md) + - [Fibo 2.0 Dev](./docs/fibo2.md) - [Lens](./docs/lens.md) - [Chroma](./docs/chroma.md) - [Chroma1-Radiance](./docs/chroma_radiance.md) @@ -79,6 +80,7 @@ API and command-line option may change frequently.*** - [Boogu Image Edit](./docs/boogu_image.md) - [Mage-Flow-Edit](./docs/mage_flow.md#image-editing) - [LLaDA-Image Edit](./docs/llada_image.md#image-editing) + - [Fibo 2.0 Dev / Edit Turbo](./docs/fibo2.md#editing-and-turbo) - Video Models - [Wan2.1/Wan2.2](./docs/wan.md) - [MiniMax-H3](./docs/minimax_h3.md) diff --git a/assets/fibo2/dev_edit.png b/assets/fibo2/dev_edit.png new file mode 100644 index 000000000..fd6e06be1 Binary files /dev/null and b/assets/fibo2/dev_edit.png differ diff --git a/assets/fibo2/t2i.png b/assets/fibo2/t2i.png new file mode 100644 index 000000000..688ffb36a Binary files /dev/null and b/assets/fibo2/t2i.png differ diff --git a/docs/adetailer.md b/docs/adetailer.md index 2bc64df2c..f42831823 100644 --- a/docs/adetailer.md +++ b/docs/adetailer.md @@ -1,9 +1,10 @@ # ADetailer `sd-cli` can run a YOLOv8 object detector on an existing or newly generated -image and perform a cropped inpaint pass for every detected object. The first -implementation supports YOLOv8 detection checkpoints. YOLOv8 segmentation and -MediaPipe models are not supported yet. +image, and `sd-server` can run it after image generation. Both perform a cropped +inpaint pass for every detected object. The implementation supports YOLOv8 +detection checkpoints. YOLOv8 segmentation and MediaPipe models are not +supported yet. ## Convert a detector @@ -68,6 +69,36 @@ ADetailer can also run automatically after normal image generation: --extra-ad-args "confidence=0.3,denoising_strength=0.4,inpaint_width=512,inpaint_height=512" ``` +The same options can be supplied when starting `sd-server`. When `--ad-model` is set, its detector runs after OpenAI-compatible and native asynchronous image generation. SDAPI requests enable ADetailer through `alwayson_scripts`, as described below. If no object is detected, the original image is returned unchanged. RGBA output keeps its original alpha channel, including when detected regions are repaired. Detector initialization or detail-pass failures are reported as request failures. + +Server image-generation requests limit base, hires, and detail-pass dimensions to 4096 pixels per side. ADetailer step overrides are limited to 1–100; `steps=0` in `extra_ad_args` keeps inheriting the main generation steps. These limits also apply to inherited dimensions and `extra_ad_args` overrides. Requests exceeding the limits return a parameter error before generation. CLI limits are unchanged. + +### SDAPI requests + +`/sdapi/v1/txt2img` and `/sdapi/v1/img2img` accept the WebUI ADetailer argument format: + +```json +{ + "prompt": "portrait photo", + "alwayson_scripts": { + "ADetailer": { + "args": [true, false, { + "ad_model": "face_yolov8n.pt", + "ad_prompt": "[PROMPT], detailed face", + "ad_use_steps": true, + "ad_steps": 20 + }] + } + } +} +``` + +The detector must be configured at server startup with `--ad-model`. `ad_model` must match that file's basename; the original `.pt` basename is also accepted for a converted detector. Requests cannot load arbitrary detector paths. Omitting the script, setting its first argument to `false`, selecting `"None"`, or setting `ad_tab_enable` to `false` disables detailing. Dictionary-only argument arrays are also accepted. Multiple enabled dictionaries run in order using the configured detector. + +Supported settings include prompts, confidence, mask filtering by area or confidence, mask count and area ratios, dilation/erosion, offsets, mask merging/inversion, blur, denoising strength, masked or full-image inpainting, padding, and optional overrides for inpaint dimensions, steps, CFG, sampler/scheduler, and CLIP skip. The corresponding `ad_use_*` flags control whether overrides apply. Otherwise dimensions, steps, CFG, sampler, scheduler, and CLIP skip inherit the main request. Denoising strength defaults to `0.4`. Pixel mask settings are limited to the larger output dimension, with a minimum limit of 32 and a maximum of 16384 pixels. + +Model/VAE switching, non-default noise multipliers, face restoration, ControlNet, and skipping the initial img2img pass are not supported; active requests for these features return an error. Unknown settings and invalid active values also return errors. During detailing, `/sdapi/v1/progress` remains active with an `adetailer` job; its percentage and preview stay fixed until detailing finishes. + An empty ADetailer prompt inherits the main prompt. `[PROMPT]` inserts the main prompt, `[SEP]` assigns different prompts to consecutive masks, and `[SKIP]` skips the corresponding mask. @@ -82,6 +113,7 @@ All settings other than the detector path and prompts are passed through | `nms` | `0.45` | NMS IoU threshold | | `max_detections` | `100` | Maximum detections retained after NMS | | `mask_k_largest` | `0` | Keep only the largest K masks; zero keeps all | +| `mask_filter_method` | `area` | Rank retained masks by `area` or `confidence` | | `mask_min_ratio` | `0` | Minimum bbox area relative to the image | | `mask_max_ratio` | `1` | Maximum bbox area relative to the image | | `dilate_erode` | `4` | Positive values dilate; negative values erode | @@ -90,6 +122,7 @@ All settings other than the detector path and prompts are passed through | `merge_masks`, `invert_mask` | `false` | Boolean alternatives to `mask_mode` | | `mask_blur` | `4` | Final composite feather radius | | `inpaint_padding` | `32` | Padding around the detected region | +| `inpaint_only_masked` | `true` | Crop to the detected region; `false` uses the full image with the detection mask | | `inpaint_width`, `inpaint_height` | mode-specific | `512x512` after generation; input/output size in `adetailer` mode | | `denoising_strength` | mode-specific | `0.4` after generation; inherits `--strength` in `adetailer` mode | | `steps` | `0` | Detail steps; zero inherits the main generation | diff --git a/docs/edit.md b/docs/edit.md index 62ecff2af..9f33862d2 100644 --- a/docs/edit.md +++ b/docs/edit.md @@ -19,6 +19,7 @@ Depending on the architecture, different models handle reference images differen | **Krea2 (Community Edit LoRAs)** | `krea2_ostris_edit` | | [**Mage-Flow-Edit**](./mage_flow.md#image-editing) | `mage_flow` | | [**LLaDA-Image**](./llada_image.md#image-editing) | `llada_image` | +| [**Fibo 2.0 Dev / Edit Turbo**](./fibo2.md#examples) | `fibo2` | | **Anima (Community Edit LoRAs)** | `cosmos_reference` | Stable-diffusion.spp also supports basic Unet-based editing models like instruct-pix2pix or CosXL-Edit. This document is not about those. @@ -50,6 +51,7 @@ The `--ref-image-args` argument accepts a comma-separated list of key-value pair | `flux_kontext` | FLUX.1 Kontext | | `longcat` | LongCat Image Edit | | `flux2` | FLUX.2 models | +| `fibo2` | Fibo 2.0 Dev / Edit Turbo | | `qwen` | Qwen Image Edit | | `qwen_layered` | Qwen Image Layered | | `z_image_omni` | Boogu, Z-Image Omni | @@ -73,7 +75,12 @@ If presets are insufficient, you can manually configure the following parameters | `ref_index_mode` | string | Behavior of the RoPE index. | `fixed`, `increase`, `decrease` | | `force_ref_timestep_zero` | bool | Forces timestep=0 for reference tokens. | `true`, `false` (Krea2 only) | | `resize_before_vae` | bool | Whether reference images are resized before VAE encoding. | `true`, `false` | -| `vae_input_max_pixels` | int | Maximum pixel area for VAE reference inputs. | Integer | +| `resize_vae_to_target` | bool | Use output dimensions for every reference, overriding area-based sizing. | `true`, `false` | +| `resize_single_ref_to_target` | bool | Use output dimensions when exactly one reference is supplied. | `true`, `false` | +| `vae_size_buckets` | bool | Choose aligned dimensions within ±10% of the requested area, minimizing aspect-ratio cropping. Ignored when using output dimensions. | `true`, `false` | +| `vae_center_crop` | bool | Scale to cover the selected dimensions with Lanczos, then center-crop. Otherwise resize directly. | `true`, `false` | +| `vae_input_max_pixels` | int | VAE area budget, also capped by output area in ordinary resizing. With size buckets, this is the target area and output area does not cap it. | Integer | +| `vlm_use_vae_input` | bool | Reuse the resized/cropped VAE input for the VLM, before the VLM's own resize. Has no effect when `resize_before_vae=false`. | `true`, `false` | | `vlm_resize_mode` | string | How to resize VLM reference inputs. | `longest_side`, `area`, `none` | | `vlm_max_size` | int | Maximum VLM input size; interpreted according to `vlm_resize_mode`. | Integer | | `vlm_min_size` | int | Minimum VLM input size; interpreted according to `vlm_resize_mode`. | Integer | @@ -88,6 +95,7 @@ For a technical overview of how each preset is configured, see the table below. | `flux_kontext` | No | `fixed` | `none` | | | `longcat` | Yes | `fixed` | `area` | | | `flux2` | No | `increase` | `none` | | +| `fibo2` | Yes | `increase` | `longest` | Shared reference crops; clean reference timesteps; see [Fibo 2.0](./fibo2.md) | | `qwen` | Yes | `increase` | `area` | | | `qwen_layered` | Yes | `decrease` | `area` | | | `mage_flow` | Yes | `increase` | `longest` | `vlm_max_size = 384`, VAE input resized to target | @@ -99,3 +107,5 @@ For a technical overview of how each preset is configured, see the table below. **Additional Default Notes:** - **VLM Input Sizes:** For most presets, `vlm_max_size` and `vlm_min_size` are set to `-1`, meaning the values are model-dependent and handled automatically. In `area` mode they represent pixel area; in `longest_side` mode they represent a side length in pixels. - **VAE Input Size:** `vae_input_max_pixels` defaults to $1024 \times 1024$ pixels (`1048576`). +- **VAE Geometry:** `resize_vae_to_target` defaults to `true` for Mage-Flow and LLaDA-Image. Fibo enables `resize_single_ref_to_target`, `vae_size_buckets`, and `vae_center_crop`. These options are otherwise `false`. Bucket dimensions use the model's alignment (normally 16, Qwen Image 32), with a minimum side of four alignment units; if no bucket fits, output dimensions are used. +- **Shared Encoder Input:** `vlm_use_vae_input` defaults to `true` for Fibo and Qwen Image 2.1, otherwise `false`. All these options can be overridden individually after selecting a preset. diff --git a/docs/fibo2.md b/docs/fibo2.md new file mode 100644 index 000000000..064d80089 --- /dev/null +++ b/docs/fibo2.md @@ -0,0 +1,53 @@ +# How to Use + +Fibo 2.0 uses a Fibo diffusion transformer, the standard FLUX.2 VAE, and Qwen3-VL 4B as the LLM text encoder. The Qwen tokenizer is built in. + +## Download weights + +- Download Fibo 2.0 Dev + - safetensors: https://huggingface.co/Comfy-Org/Fibo-2.0/tree/main/diffusion_models +- Download Fibo 2.0 Edit Turbo + - safetensors: https://huggingface.co/Comfy-Org/Fibo-2.0/tree/main/diffusion_models +- Download vae + - safetensors: https://huggingface.co/black-forest-labs/FLUX.2-dev/tree/main +- Download Qwen3-VL 4B + - safetensors: https://huggingface.co/Comfy-Org/Fibo-2.0/tree/main/text_encoders + - gguf: https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct-GGUF/tree/main + +The original Diffusers weights from [briaai/Fibo-2.0-dev](https://huggingface.co/briaai/Fibo-2.0-dev) and [briaai/Fibo-2.0-edit-turbo](https://huggingface.co/briaai/Fibo-2.0-edit-turbo) are also supported. Pass the transformer's `diffusion_pytorch_model.safetensors.index.json` to `--diffusion-model` and keep its shards alongside it. + +INT8 convrot weights load directly; see [INT8 convrot](./int8_convrot.md) for backend requirements. + +## Examples + +### Fibo 2.0 Dev + +Dev supports both text-to-image generation and reference-image editing. + +``` +.\bin\Release\sd-cli.exe --diffusion-model ..\models\diffusion_models\fibo2_dev_int8_convrot.safetensors --llm ..\models\text_encoders\Qwen3-VL-4B-Instruct-Q4_K_M.gguf --llm_vision ..\models\text_encoders\Qwen3-VL-4B-Instruct-mmproj-BF16.gguf --vae ..\models\vae\flux2_ae.safetensors -p "a lovely cat holding a sign says 'fibo2.cpp'" --cfg-scale 5.0 --sampling-method euler -v --offload-to-cpu +``` + + + +``` +.\bin\Release\sd-cli.exe --diffusion-model ..\models\diffusion_models\fibo2_dev_int8_convrot.safetensors --llm ..\models\text_encoders\Qwen3-VL-4B-Instruct-Q4_K_M.gguf --llm_vision ..\models\text_encoders\Qwen3-VL-4B-Instruct-mmproj-BF16.gguf --vae ..\models\vae\flux2_ae.safetensors -r ..\assets\flux\flux1-dev-q8_0.png -p "change 'flux.cpp' to 'fibo2.cpp'" --cfg-scale 5.0 --sampling-method euler -v --offload-to-cpu +``` + + + +### Fibo 2.0 Edit Turbo + +Edit Turbo is distilled for reference-image editing in 4 steps with CFG 1. + +``` +.\bin\Release\sd-cli.exe --diffusion-model ..\models\diffusion_models\fibo2_edit_turbo_int8_convrot.safetensors --llm ..\models\text_encoders\Qwen3-VL-4B-Instruct-Q4_K_M.gguf --llm_vision ..\models\text_encoders\Qwen3-VL-4B-Instruct-mmproj-BF16.gguf --vae ..\models\vae\flux2_ae.safetensors -r ..\assets\flux\flux1-dev-q8_0.png -p "change 'flux.cpp' to 'fibo2.cpp'" --cfg-scale 1 --steps 4 --sampling-method euler -v --offload-to-cpu +``` + +The BF16 files accept the same options. Use dimensions divisible by 16. Repeat `-r` to use multiple reference images; instructions can refer to ``, ``, etc. See [Image Editing](./edit.md) for reference-image preprocessing options. Separate GGUF text encoders need their vision projector passed with `--llm_vision` for editing. + +The automatic `fibo2` reference preset combines `resize_single_ref_to_target=true`, `vae_size_buckets=true`, `vae_center_crop=true`, and `vlm_use_vae_input=true`. A single reference is scaled to cover the output dimensions and center-cropped. Multiple references use dimensions divisible by 16, chosen within ±10% of one megapixel to minimize cropping for each image's aspect ratio; `vae_input_max_pixels` overrides that target area. The VAE and vision encoder receive the same crops, with smaller copies used by the vision encoder. These are generic reference-image options and can be overridden individually: for example, `--ref-image-args vae_center_crop=false` uses direct resizing, while `resize_before_vae=false` skips this automatic crop and resize step entirely. + +The vision encoder defaults to `vlm_resize_mode=longest_side,vlm_max_size=768`. Override these with `--ref-image-args`, for example `vlm_max_size=512`, or use `vlm_resize_mode=none` to skip size-based scaling. Vision inputs are still aligned to the encoder's patch grid. + +Plain prompts and structured JSON captions are accepted. This integration uses the caption-only text-rendering path (`render_text_images=False`); it does not generate auxiliary glyph images or expand prompts with Bria VGL. Native Fibo mask preprocessing is not connected to `--mask`. diff --git a/docs/rpc.md b/docs/rpc.md index ea9edf1c3..cb28e256f 100644 --- a/docs/rpc.md +++ b/docs/rpc.md @@ -131,7 +131,7 @@ If you want to allow connections from other machines on the network: > **Drivers & Hardware:** Ensure the Server machine has the necessary drivers installed and functional (e.g., Nvidia Drivers for CUDA, Vulkan SDK, or Metal). If no devices are found, the server will simply fallback to CPU usage. - +``` + +The options can be given in either order; no model is required. The client exits +with an error if RPC support is unavailable or a server cannot provide devices. -### Step B: Run with RPC device +### Step C: Run with RPC device If everything is working correctly, you can now run the client while offloading some or all of the work to the RPC server. @@ -217,4 +217,4 @@ The client will map these servers to sequential device IDs (e.g., RPC0 from the ## 6. Performance Considerations -RPC performance is heavily dependent on network bandwidth, as large weights and activations must be transferred back and forth over the network, especially for large models, or when using high resolutions. For best results, ensure your network connection is stable and has sufficient bandwidth (>1Gbps recommended). This shoumd not be a concern if you are running the server and client on the same machine, as the data transfer will happen over the loopback interface. \ No newline at end of file +RPC performance is heavily dependent on network bandwidth, as large weights and activations must be transferred back and forth over the network, especially for large models, or when using high resolutions. For best results, ensure your network connection is stable and has sufficient bandwidth (>1Gbps recommended). This shoumd not be a concern if you are running the server and client on the same machine, as the data transfer will happen over the loopback interface. diff --git a/examples/cli/main.cpp b/examples/cli/main.cpp index ca8ce4e79..8ee51865f 100644 --- a/examples/cli/main.cpp +++ b/examples/cli/main.cpp @@ -301,6 +301,10 @@ void parse_args(int argc, const char** argv, SDCliParams& cli_params, SDContextP log_level = cli_params.log_level; log_color = cli_params.color; + if (ctx_params.list_devices) { + exit(ctx_params.print_devices() ? 0 : 1); + } + bool valid = cli_params.resolve_and_validate(); if (valid && cli_params.mode != METADATA) { valid = ctx_params.resolve_and_validate(cli_params.mode) && diff --git a/examples/common/common.cpp b/examples/common/common.cpp index 9e7851fef..7f65edac2 100644 --- a/examples/common/common.cpp +++ b/examples/common/common.cpp @@ -596,6 +596,11 @@ ArgOptions SDContextParams::get_options() { }; options.bool_options = { + {"", + "--list-devices", + "list available ggml backend devices, including --rpc-servers (one 'namedescription' per line), and exit; " + "the names are the device names accepted by --backend and --params-backend", + true, &list_devices}, {"", "--disable-prefetch", "disable asynchronous next-segment weight prefetch (defaults to false)", @@ -819,23 +824,22 @@ ArgOptions SDContextParams::get_options() { "but it usually offers faster inference speed and, in some cases, lower memory usage. " "The at_runtime mode, on the other hand, is exactly the opposite.", on_lora_apply_mode_arg}, - {"", - "--list-devices", - "list available ggml backend devices (one 'namedescription' per line) and exit; " - "the names are the device names accepted by --backend and --params-backend", - [](int /*argc*/, const char** /*argv*/, int /*index*/) { - size_t device_list_size = sd_list_devices(nullptr, 0); - std::vector devices(device_list_size + 1); - sd_list_devices(devices.data(), devices.size()); - fputs(devices.data(), stdout); - std::exit(0); - return 0; - }}, }; return options; } +bool SDContextParams::print_devices() const { + if (!sd_add_rpc_servers(rpc_servers.c_str())) { + return false; + } + size_t device_list_size = sd_list_devices(nullptr, 0); + std::vector devices(device_list_size + 1); + sd_list_devices(devices.data(), devices.size()); + fputs(devices.data(), stdout); + return true; +} + void SDContextParams::build_embedding_map() { static const std::vector valid_ext = {".gguf", ".safetensors", ".pt", ".ckpt"}; @@ -2219,7 +2223,7 @@ bool SDGenerationParams::from_json_str( if (target_custom_sigmas != nullptr && sample_json.contains("custom_sigmas") && sample_json["custom_sigmas"].is_array() && - !sample_json["custom_sigmas"].empty()) { // empty [] = "not specified", keep server/CLI default + !sample_json["custom_sigmas"].empty()) { // empty [] = "not specified", keep server/CLI default *target_custom_sigmas = sample_json["custom_sigmas"].get>(); } if (sample_json.contains("sample_method") && sample_json["sample_method"].is_string()) { diff --git a/examples/common/common.h b/examples/common/common.h index 91e75c685..992f74c17 100644 --- a/examples/common/common.h +++ b/examples/common/common.h @@ -168,6 +168,7 @@ struct SDContextParams { std::string model_args; bool auto_fit = true; std::string rpc_servers; + bool list_devices = false; std::string effective_backend; std::string effective_params_backend; bool enable_mmap = false; @@ -189,6 +190,7 @@ struct SDContextParams { float flow_shift = INFINITY; ArgOptions get_options(); + bool print_devices() const; void build_embedding_map(); void prepare_backend_assignments(); bool resolve(SDMode mode); diff --git a/examples/server/api.md b/examples/server/api.md index bb4252366..bfa98f6f9 100644 --- a/examples/server/api.md +++ b/examples/server/api.md @@ -280,7 +280,8 @@ TAE slot. | `model` | `checkpoints` | | `diffusion_model`, `high_noise_diffusion_model`, `uncond_diffusion_model` | `diffusion_models` | | `clip_l`, `clip_g`, `t5xxl`, `llm`, `audio_encoder`, `embeddings_connectors` | `text_encoders` | -| `clip_vision`, `llm_vision` | `clip_vision` | +| `clip_vision` | `clip_vision` | +| `llm_vision` | `text_encoders` or `clip_vision` | | `vae`, `audio_vae` | `vae` | | `tae` (legacy alias: `taesd`) | `vae` or `vae_approx` | | `tokenizer` | `tokenizers` | diff --git a/examples/server/async_jobs.cpp b/examples/server/async_jobs.cpp index 7f3c8a1bf..edd93593c 100644 --- a/examples/server/async_jobs.cpp +++ b/examples/server/async_jobs.cpp @@ -244,6 +244,10 @@ bool execute_img_gen_job(ServerRuntime& runtime, } clear_preview_callback(); results.adopt(raw_results, num_results); + if (!results.empty() && !apply_server_adetailer(runtime, job.img_gen, params, results)) { + error_message = "ADetailer failed"; + return false; + } } const int num_results = results.count(); diff --git a/examples/server/main.cpp b/examples/server/main.cpp index 170877686..d79c6e2f3 100644 --- a/examples/server/main.cpp +++ b/examples/server/main.cpp @@ -48,6 +48,10 @@ static void parse_args(int argc, log_level = svr_params.log_level; log_color = svr_params.color; + if (ctx_params.list_devices) { + exit(ctx_params.print_devices() ? 0 : 1); + } + const bool random_seed_requested = default_gen_params.seed < 0; if (!svr_params.resolve_and_validate() || diff --git a/examples/server/model_catalog.cpp b/examples/server/model_catalog.cpp index a3c59206d..e8a6414c6 100644 --- a/examples/server/model_catalog.cpp +++ b/examples/server/model_catalog.cpp @@ -32,7 +32,7 @@ static const ModelSlot k_model_slots[] = { {"clip_vision", "CLIP Vision", {"clip_vision"}, &SDContextParams::clip_vision_path}, {"t5xxl", "T5", {"text_encoders"}, &SDContextParams::t5xxl_path}, {"llm", "LLM Text Encoder", {"text_encoders"}, &SDContextParams::llm_path}, - {"llm_vision", "LLM Vision Encoder", {"clip_vision"}, &SDContextParams::llm_vision_path}, + {"llm_vision", "LLM Vision Encoder", {"text_encoders", "clip_vision"}, &SDContextParams::llm_vision_path}, {"tokenizer", "Tokenizer", {"tokenizers"}, &SDContextParams::tokenizer}, {"vae", "VAE", {"vae"}, &SDContextParams::vae_path}, {"audio_vae", "Audio VAE", {"vae"}, &SDContextParams::audio_vae_path}, diff --git a/examples/server/recipes/fibo2-dev.json b/examples/server/recipes/fibo2-dev.json new file mode 100644 index 000000000..aa7e81460 --- /dev/null +++ b/examples/server/recipes/fibo2-dev.json @@ -0,0 +1,39 @@ +{ + "schema_version": 1, + "id": "fibo2-dev", + "name": "Fibo 2.0 Dev", + "description": "Text-to-image generation and native reference-image editing. Select a separate vision projector for editing when it is not bundled in the text encoder.", + "model_url": "https://huggingface.co/briaai/Fibo-2.0-dev", + "slots": [ + { + "id": "diffusion_model", + "required": true + }, + { + "id": "llm", + "required": true, + "label": "Qwen 3 VL 4B Text Encoder" + }, + { + "id": "llm_vision", + "required": false, + "label": "Vision projector (required for separate GGUF text encoders)" + }, + { + "id": "vae", + "required": true, + "label": "FLUX.2 VAE" + } + ], + "defaults": { + "sampling_method": "euler", + "cfg_scale": 5, + "steps": 20, + "width": 1024, + "height": 1024 + }, + "features": [ + "flow_shift", + "ref_images" + ] +} diff --git a/examples/server/recipes/fibo2-edit-turbo.json b/examples/server/recipes/fibo2-edit-turbo.json new file mode 100644 index 000000000..b84e1cce4 --- /dev/null +++ b/examples/server/recipes/fibo2-edit-turbo.json @@ -0,0 +1,39 @@ +{ + "schema_version": 1, + "id": "fibo2-edit-turbo", + "name": "Fibo 2.0 Edit Turbo", + "description": "Dedicated reference-image editing checkpoint; supply reference images. Select a separate vision projector when it is not bundled in the text encoder.", + "model_url": "https://huggingface.co/briaai/Fibo-2.0-edit-turbo", + "slots": [ + { + "id": "diffusion_model", + "required": true + }, + { + "id": "llm", + "required": true, + "label": "Qwen 3 VL 4B Text Encoder" + }, + { + "id": "llm_vision", + "required": false, + "label": "Vision projector (required for separate GGUF text encoders)" + }, + { + "id": "vae", + "required": true, + "label": "FLUX.2 VAE" + } + ], + "defaults": { + "sampling_method": "euler", + "cfg_scale": 1, + "steps": 4, + "width": 1024, + "height": 1024 + }, + "features": [ + "flow_shift", + "ref_images" + ] +} diff --git a/examples/server/routes_openai.cpp b/examples/server/routes_openai.cpp index 29dfe313a..27a7e18f9 100644 --- a/examples/server/routes_openai.cpp +++ b/examples/server/routes_openai.cpp @@ -75,7 +75,7 @@ static bool build_openai_generation_request(const httplib::Request& req, error_message = "invalid params"; return false; } - return true; + return validate_server_img_gen_request(runtime, request, error_message); } static bool build_openai_edit_request(const httplib::Request& req, @@ -158,22 +158,22 @@ static bool build_openai_edit_request(const httplib::Request& req, request.gen_params.height = height; request.gen_params.batch_count = n; - std::string sd_cpp_extra_args_str = extract_and_remove_sd_cpp_extra_args(request.gen_params.prompt); - const json extra_args = sd_cpp_extra_args_str.empty() - ? json::object() - : json::parse(sd_cpp_extra_args_str, nullptr, false); - const bool has_extra_init_image_override = extra_args.is_object() && extra_args.contains("init_image"); - const bool has_extra_ref_images_override = extra_args.is_object() && extra_args.contains("ref_images"); - const bool has_extra_strength_override = extra_args.is_object() && - extra_args.contains("strength") && - extra_args["strength"].is_number(); - const bool has_extra_mask_image = extra_args.is_object() && - extra_args.contains("mask_image") && - !extra_args["mask_image"].is_null(); + std::string sd_cpp_extra_args_str = extract_and_remove_sd_cpp_extra_args(request.gen_params.prompt); + const json extra_args = sd_cpp_extra_args_str.empty() + ? json::object() + : json::parse(sd_cpp_extra_args_str, nullptr, false); + const bool has_extra_init_image_override = extra_args.is_object() && extra_args.contains("init_image"); + const bool has_extra_ref_images_override = extra_args.is_object() && extra_args.contains("ref_images"); + const bool has_extra_strength_override = extra_args.is_object() && + extra_args.contains("strength") && + extra_args["strength"].is_number(); + const bool has_extra_mask_image = extra_args.is_object() && + extra_args.contains("mask_image") && + !extra_args["mask_image"].is_null(); const bool supports_ref_image_conditioning = sd_ctx_supports_ref_image_conditioning(runtime.sd_ctx); const bool may_need_uploaded_init_image = !has_extra_init_image_override && - (!supports_ref_image_conditioning || !mask_bytes.empty() || - has_extra_mask_image || has_extra_strength_override || has_extra_ref_images_override); + (!supports_ref_image_conditioning || !mask_bytes.empty() || + has_extra_mask_image || has_extra_strength_override || has_extra_ref_images_override); SDImageOwner uploaded_image_for_init; bool uploaded_image_was_first_reference = false; for (auto& bytes : images_bytes) { @@ -252,7 +252,7 @@ static bool build_openai_edit_request(const httplib::Request& req, return false; } - return true; + return validate_server_img_gen_request(runtime, request, error_message); } static bool execute_sync_img_gen_request(ServerRuntime& runtime, @@ -270,6 +270,10 @@ static bool execute_sync_img_gen_request(ServerRuntime& runtime, num_results = 0; } results.adopt(raw_results, num_results); + if (!results.empty() && !apply_server_adetailer(runtime, request, img_gen_params, results)) { + error_message = "ADetailer failed"; + return false; + } } if (results.empty()) { diff --git a/examples/server/routes_sdapi.cpp b/examples/server/routes_sdapi.cpp index 4b8e7a5b1..014b0b5f8 100644 --- a/examples/server/routes_sdapi.cpp +++ b/examples/server/routes_sdapi.cpp @@ -3,9 +3,12 @@ #include #include #include +#include #include +#include #include #include +#include #include "common/common.h" #include "common/media_io.h" @@ -81,6 +84,202 @@ static enum sample_method_t get_sdapi_sample_method(std::string name) { return it != hardcoded.end() ? it->second : SAMPLE_METHOD_COUNT; } +static bool parse_sdapi_adetailer(const json& body, + ServerRuntime& runtime, + bool img2img, + ImgGenJobRequest& request, + std::string& error_message) { + request.adetailer_enabled = false; + request.adetailer_passes.clear(); + if (!body.contains("alwayson_scripts")) { + return true; + } + try { + const auto& scripts = body.at("alwayson_scripts"); + if (!scripts.is_object()) { + throw std::invalid_argument("alwayson_scripts must be an object"); + } + if (!scripts.contains("ADetailer")) { + return true; + } + const auto& args = scripts.at("ADetailer").at("args"); + if (!args.is_array()) { + throw std::invalid_argument("ADetailer.args must be an array"); + } + if (args.empty() || (args[0].is_boolean() && !args[0].get())) { + return true; + } + static const std::unordered_set keys = { + "ad_model", "ad_model_classes", "ad_tab_enable", "ad_prompt", "ad_negative_prompt", + "ad_confidence", "ad_mask_filter_method", "ad_mask_k", "ad_mask_min_ratio", "ad_mask_max_ratio", + "ad_dilate_erode", "ad_x_offset", "ad_y_offset", "ad_mask_merge_invert", "ad_mask_blur", + "ad_denoising_strength", "ad_inpaint_only_masked", "ad_inpaint_only_masked_padding", + "ad_use_inpaint_width_height", "ad_inpaint_width", "ad_inpaint_height", "ad_use_steps", "ad_steps", + "ad_use_cfg_scale", "ad_cfg_scale", "ad_use_sampler", "ad_sampler", "ad_scheduler", + "ad_use_checkpoint", "ad_checkpoint", "ad_use_vae", "ad_vae", "ad_use_noise_multiplier", + "ad_noise_multiplier", "ad_use_clip_skip", "ad_clip_skip", "ad_restore_face", "ad_controlnet_model", + "ad_controlnet_module", "ad_controlnet_weight", "ad_controlnet_guidance_start", "ad_controlnet_guidance_end", "is_api"}; + for (const auto& config : args) { + if (!config.is_object()) { + continue; + } + auto boolean = [&](const char* key, bool fallback) { + if (!config.contains(key)) + return fallback; + if (!config.at(key).is_boolean()) + throw std::invalid_argument(std::string(key) + " must be boolean"); + return config.at(key).get(); + }; + auto text = [&](const char* key, const char* fallback) { + if (!config.contains(key)) + return std::string(fallback); + if (!config.at(key).is_string()) + throw std::invalid_argument(std::string(key) + " must be a string"); + return config.at(key).get(); + }; + auto number = [&](const char* key, double fallback, double minimum, double maximum) { + double value = fallback; + if (config.contains(key)) { + if (!config.at(key).is_number()) + throw std::invalid_argument(std::string(key) + " must be numeric"); + value = config.at(key).get(); + } + if (!std::isfinite(value) || value < minimum || value > maximum) + throw std::invalid_argument(std::string(key) + " is out of range"); + return value; + }; + auto integer = [&](const char* key, int fallback, int minimum, int maximum = std::numeric_limits::max()) { + if (config.contains(key) && !config.at(key).is_number_integer()) + throw std::invalid_argument(std::string(key) + " must be an integer"); + return static_cast(number(key, fallback, minimum, maximum)); + }; + const std::string model = text("ad_model", "None"); + if (model == "None" || !boolean("ad_tab_enable", true)) { + continue; + } + for (const auto& item : config.items()) { + if (keys.count(item.key()) == 0) + throw std::invalid_argument("unsupported ADetailer setting: " + item.key()); + } + const auto& configured = runtime.default_gen_params->ad_model_path; + if (configured.empty()) + throw std::invalid_argument("ADetailer requires a server-configured --ad-model"); + const auto path = fs::u8path(configured); + if (model.find_first_of("/\\") != std::string::npos || + (lower_ascii(model) != lower_ascii(path.filename().u8string()) && lower_ascii(model) != lower_ascii(path.stem().u8string() + ".pt"))) { + throw std::invalid_argument("ADetailer model must match the server-configured detector"); + } + if (!text("ad_model_classes", "").empty()) + throw std::invalid_argument("ad_model_classes is not supported"); + if (boolean("ad_use_checkpoint", false)) { + const auto checkpoint = config.contains("ad_checkpoint") && config.at("ad_checkpoint").is_null() ? "" : text("ad_checkpoint", "Use same checkpoint"); + if (!checkpoint.empty() && checkpoint != "None" && checkpoint != "Use same checkpoint") + throw std::invalid_argument("ADetailer checkpoint switching is not supported"); + } + if (boolean("ad_use_vae", false)) { + const auto vae = config.contains("ad_vae") && config.at("ad_vae").is_null() ? "" : text("ad_vae", "Use same VAE"); + if (!vae.empty() && vae != "None" && vae != "Use same VAE") + throw std::invalid_argument("ADetailer VAE switching is not supported"); + } + if (boolean("ad_use_noise_multiplier", false) && number("ad_noise_multiplier", 1, 0.5, 1.5) != 1) + throw std::invalid_argument("ADetailer noise multiplier is not supported"); + if (boolean("ad_restore_face", false)) + throw std::invalid_argument("ADetailer face restoration is not supported"); + if (text("ad_controlnet_model", "None") != "None") + throw std::invalid_argument("ADetailer ControlNet is not supported"); + + ServerADetailerPass pass; + pass.prompt = text("ad_prompt", ""); + pass.negative_prompt = text("ad_negative_prompt", ""); + pass.width = request.gen_params.get_resolved_width(); + pass.height = request.gen_params.get_resolved_height(); + pass.strength = static_cast(number("ad_denoising_strength", 0.4, 0, 1)); + auto append = [&](const char* key, const std::string& value) { + if (!pass.extra_args.empty()) + pass.extra_args += ","; + pass.extra_args += std::string(key) + "=" + value; + }; + auto append_number = [&](const char* target, const char* key, double fallback, double minimum, double maximum) { + append(target, json(number(key, fallback, minimum, maximum)).dump()); + }; + auto append_integer = [&](const char* target, const char* key, int fallback, int minimum, int maximum = std::numeric_limits::max()) { + append(target, std::to_string(integer(key, fallback, minimum, maximum))); + }; + append_number("confidence", "ad_confidence", 0.3, 0, 1); + append_integer("mask_k_largest", "ad_mask_k", 0, 0); + const auto filter = text("ad_mask_filter_method", "Area"); + if (filter != "Area" && filter != "Confidence") + throw std::invalid_argument("invalid ad_mask_filter_method"); + append("mask_filter_method", lower_ascii(filter)); + const auto min_ratio = number("ad_mask_min_ratio", 0, 0, 1); + const auto max_ratio = number("ad_mask_max_ratio", 1, 0, 1); + if (min_ratio > max_ratio) + throw std::invalid_argument("ad_mask_min_ratio exceeds ad_mask_max_ratio"); + append("mask_min_ratio", json(min_ratio).dump()); + append("mask_max_ratio", json(max_ratio).dump()); + // Bound pixel operations before converting them to the native mask parameters. + const int mask_limit = std::clamp(std::max(pass.width, pass.height), 32, 16384); + append_integer("dilate_erode", "ad_dilate_erode", 4, -mask_limit, mask_limit); + append_integer("x_offset", "ad_x_offset", 0, -mask_limit, mask_limit); + append_integer("y_offset", "ad_y_offset", 0, -mask_limit, mask_limit); + append_integer("mask_blur", "ad_mask_blur", 4, 0, mask_limit); + append_integer("inpaint_padding", "ad_inpaint_only_masked_padding", 32, 0, mask_limit); + append("inpaint_only_masked", boolean("ad_inpaint_only_masked", true) ? "true" : "false"); + const auto merge = text("ad_mask_merge_invert", "None"); + if (merge != "None" && merge != "Merge" && merge != "Merge and Invert") + throw std::invalid_argument("invalid ad_mask_merge_invert"); + append("mask_mode", merge == "None" ? "none" : merge == "Merge" ? "merge" + : "merge_invert"); + if (boolean("ad_use_inpaint_width_height", false)) { + pass.width = integer("ad_inpaint_width", 512, 1, k_max_generation_dimension); + pass.height = integer("ad_inpaint_height", 512, 1, k_max_generation_dimension); + } + if (boolean("ad_use_steps", false)) + append_integer("steps", "ad_steps", 28, 1, k_max_adetailer_steps); + if (boolean("ad_use_cfg_scale", false)) + append_number("cfg_scale", "ad_cfg_scale", 7, 0, std::numeric_limits::max()); + if (boolean("ad_use_clip_skip", false)) { + pass.clip_skip = integer("ad_clip_skip", 1, 1); + if (pass.clip_skip > 12) + throw std::invalid_argument("ad_clip_skip is out of range"); + } + if (boolean("ad_use_sampler", false)) { + std::string sampler = text("ad_sampler", "DPM++ 2M Karras"); + if (sampler != "Use same sampler") { + const std::string suffix = " karras"; + const bool legacy_karras = sampler.size() > suffix.size() && lower_ascii(sampler.substr(sampler.size() - suffix.size())) == suffix; + if (legacy_karras) + sampler.resize(sampler.size() - suffix.size()); + const auto method = get_sdapi_sample_method(sampler); + if (method == SAMPLE_METHOD_COUNT) + throw std::invalid_argument("unsupported ADetailer sampler"); + append("sample_method", sd_sample_method_name(method)); + if (legacy_karras && text("ad_scheduler", "Use same scheduler") == "Use same scheduler") + append("scheduler", sd_scheduler_name(KARRAS_SCHEDULER)); + } + const auto scheduler = text("ad_scheduler", "Use same scheduler"); + if (scheduler == "Automatic") { + pass.automatic_scheduler = true; + } else if (scheduler != "Use same scheduler") { + const auto value = str_to_scheduler(scheduler.c_str()); + if (value == SCHEDULER_COUNT) + throw std::invalid_argument("unsupported ADetailer scheduler"); + append("scheduler", sd_scheduler_name(value)); + } + } + request.adetailer_passes.push_back(std::move(pass)); + } + if (!request.adetailer_passes.empty() && img2img && args.size() > 1 && args[1].is_boolean() && args[1].get()) { + throw std::invalid_argument("ADetailer skip img2img is not supported"); + } + request.adetailer_enabled = !request.adetailer_passes.empty(); + return true; + } catch (const std::exception& error) { + error_message = std::string("invalid ADetailer args: ") + error.what(); + return false; + } +} + static bool build_sdapi_img_gen_request(const json& j, ServerRuntime& runtime, bool img2img, @@ -259,7 +458,8 @@ static bool build_sdapi_img_gen_request(const json& j, return false; } - return true; + return parse_sdapi_adetailer(j, runtime, img2img, request, error_message) && + validate_server_img_gen_request(runtime, request, error_message); } static nlohmann::json prepare_info_field(const SDContextParams& ctx_params, @@ -349,7 +549,7 @@ static void preview_callback(int step, int frame_count, sd_image_t* frames, bool } struct SDAPIPreviewGuard { - SDAPIPreviewGuard(const std::string& id_task, const sd_img_gen_params_t& params, const std::string& job_type) { + SDAPIPreviewGuard(const std::string& id_task, const sd_img_gen_params_t& params, const std::string& job_type, bool detailing = false) { std::lock_guard lock(g_preview.mutex); g_preview.id_task = id_task; g_preview.total = 0; @@ -358,11 +558,20 @@ struct SDAPIPreviewGuard { g_preview.step = 0; g_preview.active = true; // Each batch item has a base pass and, when enabled, a hires pass. - g_preview.job_count = static_cast(std::max(1, params.batch_count)) * (params.hires.enabled ? 2 : 1); + g_preview.job_count = static_cast(std::max(1, params.batch_count)) * (params.hires.enabled ? 2 : 1) + (detailing ? 1 : 0); g_preview.job_no = 0; sd_set_preview_callback(preview_callback, PREVIEW_PROJ, 1, true, false, nullptr); } + void begin_detailing() { + sd_set_preview_callback(nullptr, PREVIEW_NONE, 1, false, false, nullptr); + std::lock_guard lock(g_preview.mutex); + g_preview.job_type = "adetailer"; + g_preview.job_no = static_cast(g_preview.job_count - 1); + g_preview.step = 0; + g_preview.total = 0; + } + ~SDAPIPreviewGuard() { sd_set_preview_callback(nullptr, PREVIEW_NONE, 1, false, false, nullptr); std::lock_guard lock(g_preview.mutex); @@ -410,13 +619,21 @@ void register_sdapi_endpoints(httplib::Server& svr, ServerRuntime& rt) { { std::lock_guard lock(*runtime->sd_ctx_mutex); - SDAPIPreviewGuard preview_guard(id_task, img_gen_params, img2img ? "img2img" : "txt2img"); + const bool detailing = request.adetailer_enabled && !runtime->default_gen_params->ad_model_path.empty(); + SDAPIPreviewGuard preview_guard(id_task, img_gen_params, img2img ? "img2img" : "txt2img", detailing); sd_image_t* raw_results = nullptr; if (!generate_image(runtime->sd_ctx, &img_gen_params, &raw_results, &num_results)) { raw_results = nullptr; num_results = 0; } results.adopt(raw_results, num_results); + if (detailing && !results.empty()) + preview_guard.begin_detailing(); + if (!results.empty() && !apply_server_adetailer(*runtime, request, img_gen_params, results)) { + res.status = 500; + res.set_content(R"({"error":"ADetailer failed"})", "application/json"); + return; + } } if (results.empty()) { diff --git a/examples/server/routes_sdcpp.cpp b/examples/server/routes_sdcpp.cpp index 7e93e5edb..8eb9c9ab3 100644 --- a/examples/server/routes_sdcpp.cpp +++ b/examples/server/routes_sdcpp.cpp @@ -410,9 +410,9 @@ static json make_capabilities_json(ServerRuntime& runtime) { result["defaults_by_mode"] = defaults_by_mode; result["limits"] = { {"min_width", 64}, - {"max_width", 4096}, + {"max_width", k_max_generation_dimension}, {"min_height", 64}, - {"max_height", 4096}, + {"max_height", k_max_generation_dimension}, {"max_batch_count", 8}, {"max_queue_size", manager.max_pending_jobs}, {"max_upscale_width", k_max_upscale_dimension}, @@ -457,7 +457,7 @@ static bool parse_img_gen_request(const json& body, error_message = "invalid generation parameters"; return false; } - return true; + return validate_server_img_gen_request(runtime, request, error_message); } static bool parse_vid_gen_request(const json& body, diff --git a/examples/server/runtime.cpp b/examples/server/runtime.cpp index 34140ea35..5b33b338a 100644 --- a/examples/server/runtime.cpp +++ b/examples/server/runtime.cpp @@ -3,6 +3,7 @@ #include #include #include +#include #include #include #include @@ -18,6 +19,157 @@ namespace fs = std::filesystem; +bool validate_server_img_gen_request(const ServerRuntime& runtime, + const ImgGenJobRequest& request, + std::string& error_message) { + auto dimensions_valid = [&](double width, double height) { + if (!std::isfinite(width) || !std::isfinite(height) || + width < 1 || height < 1 || width > k_max_generation_dimension || height > k_max_generation_dimension) { + error_message = "generation width and height must be between 1 and " + std::to_string(k_max_generation_dimension); + return false; + } + return true; + }; + const auto& params = request.gen_params; + const int width = params.get_resolved_width(); + const int height = params.get_resolved_height(); + if (!dimensions_valid(width, height)) { + return false; + } + if (params.hires_enabled) { + const double hires_width = params.hires_width > 0 ? params.hires_width : params.hires_height > 0 ? params.hires_height + : std::round(width * params.hires_scale); + const double hires_height = params.hires_height > 0 ? params.hires_height : params.hires_width > 0 ? params.hires_width + : std::round(height * params.hires_scale); + if (!dimensions_valid(hires_width, hires_height)) { + return false; + } + } + if (!request.adetailer_enabled || runtime.default_gen_params->ad_model_path.empty()) { + return true; + } + auto pass_valid = [&](const ServerADetailerPass& pass) { + if (!dimensions_valid(pass.width, pass.height)) { + return false; + } + // Match the native key normalization, but leave full option parsing to ADetailer. + auto trim = [](const std::string& text) { + const auto first = text.find_first_not_of(" \t\r\n"); + return first == std::string::npos ? std::string() : text.substr(first, text.find_last_not_of(" \t\r\n") - first + 1); + }; + std::stringstream stream(pass.extra_args); + std::string item; + while (std::getline(stream, item, ',')) { + const auto equals = item.find('='); + if (equals == std::string::npos) { + continue; + } + auto key = trim(item.substr(0, equals)); + std::transform(key.begin(), key.end(), key.begin(), [](unsigned char c) { return static_cast(std::tolower(c)); }); + if (key != "inpaint_width" && key != "inpaint_height" && key != "steps") { + continue; + } + const auto value = trim(item.substr(equals + 1)); + const int minimum = key == "steps" ? 0 : 1; + const int maximum = key == "steps" ? k_max_adetailer_steps : k_max_generation_dimension; + try { + size_t used = 0; + const int parsed = std::stoi(value, &used); + if (used == value.size() && parsed >= minimum && parsed <= maximum) { + continue; + } + } catch (const std::exception&) { + } + error_message = "ADetailer " + key + " must be between " + std::to_string(minimum) + " and " + std::to_string(maximum); + return false; + } + return true; + }; + if (request.adetailer_passes.empty()) { + ServerADetailerPass pass; + pass.extra_args = params.extra_ad_args; + return pass_valid(pass); + } + return std::all_of(request.adetailer_passes.begin(), request.adetailer_passes.end(), pass_valid); +} + +bool apply_server_adetailer(ServerRuntime& runtime, + const ImgGenJobRequest& request, + const sd_img_gen_params_t& params, + SDImageVec& results) { + const auto& gen_params = request.gen_params; + const auto& ad_model_path = runtime.default_gen_params->ad_model_path; + if (!request.adetailer_enabled || ad_model_path.empty()) { + return true; + } + std::string error_message; + if (!validate_server_img_gen_request(runtime, request, error_message)) { + LOG_ERROR("%s", error_message.c_str()); + return false; + } + + std::vector passes = request.adetailer_passes; + if (passes.empty()) { + ServerADetailerPass pass; + pass.prompt = gen_params.ad_prompt; + pass.negative_prompt = gen_params.ad_negative_prompt; + pass.extra_args = gen_params.extra_ad_args; + passes.push_back(std::move(pass)); + } + + ADetailerCtxPtr ad_ctx(new_adetailer_ctx(ad_model_path.c_str(), + runtime.ctx_params->n_threads, + runtime.ctx_params->effective_backend.c_str(), + runtime.ctx_params->effective_params_backend.c_str())); + if (ad_ctx == nullptr) { + LOG_ERROR("new_adetailer_ctx failed"); + return false; + } + + for (const auto& pass : passes) { + sd_adetailer_params_t ad_params{}; + ad_params.prompt = pass.prompt.empty() ? nullptr : pass.prompt.c_str(); + ad_params.negative_prompt = pass.negative_prompt.empty() ? nullptr : pass.negative_prompt.c_str(); + ad_params.extra_ad_args = pass.extra_args.c_str(); + for (int i = 0; i < results.count(); ++i) { + if (results[i].data == nullptr) { + continue; + } + sd_img_gen_params_t ad_generation_params = params; + ad_generation_params.seed = params.seed + i; + ad_generation_params.width = pass.width; + ad_generation_params.height = pass.height; + ad_generation_params.strength = pass.strength; + if (pass.clip_skip > 0) { + ad_generation_params.clip_skip = pass.clip_skip; + } + if (pass.automatic_scheduler) { + ad_generation_params.sample_params.scheduler = SCHEDULER_COUNT; + } + + sd_image_t* detailed_images = nullptr; + int detailed_count = 0; + if (!adetail_image(ad_ctx.get(), + runtime.sd_ctx, + results[i], + &ad_params, + &ad_generation_params, + &detailed_images, + &detailed_count) || + detailed_count <= 0 || detailed_images == nullptr || detailed_images[0].data == nullptr) { + free_sd_images(detailed_images, detailed_count); + LOG_ERROR("ADetailer failed for image %d", i + 1); + return false; + } + free(results[i].data); + results[i] = detailed_images[0]; + detailed_images[0] = {0, 0, 0, nullptr}; + free_sd_images(detailed_images, detailed_count); + } + } + return true; +} + static std::string lower_ascii(std::string value) { std::transform(value.begin(), value.end(), value.begin(), [](unsigned char c) { return static_cast(std::tolower(c)); diff --git a/examples/server/runtime.h b/examples/server/runtime.h index d9272f6c2..8986f3827 100644 --- a/examples/server/runtime.h +++ b/examples/server/runtime.h @@ -67,18 +67,38 @@ struct ServerRuntime { ServerModels* models; }; +struct ServerADetailerPass { + std::string prompt; + std::string negative_prompt; + std::string extra_args; + int width = 512; + int height = 512; + float strength = 0.4f; + int clip_skip = -1; + bool automatic_scheduler = false; +}; + struct ImgGenJobRequest { SDGenerationParams gen_params; std::string output_format = "png"; int output_compression = 100; std::string preview_mode = "none"; int preview_interval = 1; + bool adetailer_enabled = true; + std::vector adetailer_passes; sd_img_gen_params_t to_sd_img_gen_params_t() { return gen_params.to_sd_img_gen_params_t(); } }; +constexpr int k_max_generation_dimension = 4096; +constexpr int k_max_adetailer_steps = 100; + +bool validate_server_img_gen_request(const ServerRuntime& runtime, + const ImgGenJobRequest& request, + std::string& error_message); + struct VidGenJobRequest { SDGenerationParams gen_params; std::string output_format = "webm"; @@ -111,3 +131,7 @@ void refresh_lora_cache(ServerRuntime& rt); std::string get_lora_full_path(ServerRuntime& rt, const std::string& path); void refresh_upscaler_cache(ServerRuntime& rt); int64_t unix_timestamp_now_ms(); +bool apply_server_adetailer(ServerRuntime& runtime, + const ImgGenJobRequest& request, + const sd_img_gen_params_t& params, + SDImageVec& results); diff --git a/include/stable-diffusion.h b/include/stable-diffusion.h index 404d8cbdd..85fe51d46 100644 --- a/include/stable-diffusion.h +++ b/include/stable-diffusion.h @@ -640,6 +640,11 @@ SD_API const char* sd_version(void); // terminator. Passing nullptr or buffer_size 0 only queries the required size. SD_API size_t sd_list_devices(char* buffer, size_t buffer_size); +// Register comma-separated RPC endpoints (host:port) before listing devices. +// Repeated endpoints are safe; nullptr or an empty string is a no-op. +// Returns false if RPC is unavailable or an endpoint has no usable devices. +SD_API bool sd_add_rpc_servers(const char* servers); + // for C API, caller needs to call free_sd_images to free the memory after use // This helps avoid CRT problems on Windows when memory is allocated in the library but freed in the caller, which may use a different CRT. SD_API void free_sd_images(sd_image_t* result_images, int num_images); diff --git a/src/conditioning/conditioner.hpp b/src/conditioning/conditioner.hpp index 7bbc38348..c0e0c03b1 100644 --- a/src/conditioning/conditioner.hpp +++ b/src/conditioning/conditioner.hpp @@ -2008,7 +2008,8 @@ struct LLMEmbedder : public Conditioner { arch = LLM::LLMArch::GPT_OSS_20B; } else if (sd_version_is_pid(version)) { arch = LLM::LLMArch::GEMMA2_2B; - } else if (version == VERSION_QWEN_IMAGE_2_1 || + } else if (version == VERSION_FIBO2 || + version == VERSION_QWEN_IMAGE_2_1 || sd_version_is_flux3(version) || sd_version_is_lingbot_video(version) || sd_version_is_ideogram4(version) || @@ -2367,6 +2368,53 @@ struct LLMEmbedder : public Conditioner { } } + bool expand_vision_marker(std::string& prompt, + size_t& marker_start, + std::vector> outputs, + int grid_h, + int grid_w, + std::vector>>& image_embeds, + LLM::DeepStackImageEmbeds& deepstack_image_embeds, + std::vector& image_grids, + int max_length = std::numeric_limits::max()) { + if (outputs.empty()) { + return false; + } + const std::string placeholder = "<|image_pad|>"; + size_t position = prompt.find(placeholder, marker_start); + if (position == std::string::npos) { + LOG_ERROR("Reference image marker is missing"); + return false; + } + std::vector prefix_tokens; + if (!tokenizer->encode(prompt.substr(0, position), prefix_tokens, nullptr)) { + return false; + } + int index = static_cast(prefix_tokens.size()); + int count = static_cast(outputs[0].shape()[1]); + if (index > max_length || count > max_length - index) { + LOG_ERROR("Reference images exceed the %d-token context", max_length); + return false; + } + if (deepstack_image_embeds.empty()) { + deepstack_image_embeds.resize(outputs.size() - 1); + } + GGML_ASSERT(deepstack_image_embeds.size() == outputs.size() - 1); + image_embeds.emplace_back(index, std::move(outputs[0])); + for (size_t layer = 1; layer < outputs.size(); ++layer) { + deepstack_image_embeds[layer - 1].emplace_back(index, std::move(outputs[layer])); + } + image_grids.push_back({index, count, grid_h, grid_w}); + std::string markers; + markers.reserve(static_cast(count) * placeholder.size()); + for (int i = 0; i < count; ++i) { + markers += placeholder; + } + prompt.replace(position, placeholder.size(), markers); + marker_start = position + markers.size(); + return true; + } + SDCondition get_learned_condition(int n_threads, const ConditionerParams& conditioner_params) override { std::string prompt; @@ -2394,9 +2442,8 @@ struct LLMEmbedder : public Conditioner { prompt_attn_range = {0, 0}; if (llm->enable_vision) { - const std::string placeholder = "<|image_pad|>"; - const int patch_size = llm->config.vision.patch_size; - const int factor = patch_size * llm->config.vision.spatial_merge_size; + const int patch_size = llm->config.vision.patch_size; + const int factor = patch_size * llm->config.vision.spatial_merge_size; auto resize_for_vision = [&](const sd::Tensor& image) { int height = static_cast(image.shape()[1]); @@ -2424,29 +2471,11 @@ struct LLMEmbedder : public Conditioner { int grid_h, int grid_w) { GGML_ASSERT(image_outputs.size() == 4); - auto image_embed = std::move(image_outputs[0]); prompt += "<|vision_start|>"; - std::vector prefix_tokens; - if (!tokenizer->encode(prompt, prefix_tokens, nullptr)) { - return false; - } - int image_embed_idx = static_cast(prefix_tokens.size()); - image_embeds.emplace_back(image_embed_idx, image_embed); - if (deepstack_image_embeds.empty()) { - deepstack_image_embeds.resize(image_outputs.size() - 1); - } - for (size_t layer = 0; layer < deepstack_image_embeds.size(); ++layer) { - deepstack_image_embeds[layer].emplace_back(image_embed_idx, std::move(image_outputs[layer + 1])); - } - image_grids.push_back({image_embed_idx, - static_cast(image_embed.shape()[1]), - grid_h, - grid_w}); - for (int64_t i = 0; i < image_embed.shape()[1]; ++i) { - prompt += placeholder; - } - prompt += "<|vision_end|>"; - return true; + size_t marker_start = prompt.size(); + prompt += "<|image_pad|><|vision_end|>"; + return expand_vision_marker(prompt, marker_start, std::move(image_outputs), grid_h, grid_w, + image_embeds, deepstack_image_embeds, image_grids); }; const auto* references = conditioner_params.minimax_h3_references; @@ -2660,26 +2689,14 @@ struct LLMEmbedder : public Conditioner { return {}; } prompt += (i == 0 ? "" : " ") + std::string("<|vision_start|>"; - std::vector prefix_tokens; - if (!tokenizer->encode(prompt, prefix_tokens, nullptr)) { + size_t marker_start = prompt.size(); + prompt += "<|image_pad|><|vision_end|>"; + if (!expand_vision_marker(prompt, marker_start, std::move(outputs), + static_cast(height) / llm->config.vision.patch_size, + static_cast(width) / llm->config.vision.patch_size, + image_embeds, deepstack_image_embeds, image_grids)) { return {}; } - int index = static_cast(prefix_tokens.size()); - int count = static_cast(outputs[0].shape()[1]); - image_embeds.emplace_back(index, std::move(outputs[0])); - if (deepstack_image_embeds.empty()) { - deepstack_image_embeds.resize(outputs.size() - 1); - } - for (size_t layer = 1; layer < outputs.size(); ++layer) { - deepstack_image_embeds[layer - 1].emplace_back(index, std::move(outputs[layer])); - } - image_grids.push_back({index, count, - static_cast(height) / llm->config.vision.patch_size, - static_cast(width) / llm->config.vision.patch_size}); - for (int j = 0; j < count; ++j) { - prompt += "<|image_pad|>"; - } - prompt += "<|vision_end|>"; } } prompt_attn_range.first = static_cast(prompt.size()); diff --git a/src/conditioning/fibo2.hpp b/src/conditioning/fibo2.hpp new file mode 100644 index 000000000..43b2ee084 --- /dev/null +++ b/src/conditioning/fibo2.hpp @@ -0,0 +1,182 @@ +#ifndef __SD_CONDITIONING_FIBO2_HPP__ +#define __SD_CONDITIONING_FIBO2_HPP__ + +#include "conditioning/conditioner.hpp" + +struct Fibo2Conditioner : public LLMEmbedder { + using JSON = nlohmann::ordered_json; + + Fibo2Conditioner(ggml_backend_t backend, const String2TensorStorage& weights, std::shared_ptr weight_manager, const TokenizerConfig& tokenizers) + : LLMEmbedder(backend, weights, VERSION_FIBO2, "", true, weight_manager, tokenizers) { + if (llm->config.hidden_size != 2560 || llm->config.num_layers != 36) { + throw std::runtime_error("Fibo 2.0 requires its Qwen3-VL-4B text encoder"); + } + } + + static bool empty_value(const JSON& value) { + return value.is_null() || + (value.is_string() && value.get_ref().empty()) || + ((value.is_array() || value.is_object()) && value.empty()); + } + + static JSON clean_caption(const JSON& value) { + if (value.is_object()) { + auto result = JSON::object(); + for (auto it = value.begin(); it != value.end(); ++it) { + auto cleaned = clean_caption(it.value()); + if (!empty_value(cleaned)) { + result[it.key()] = std::move(cleaned); + } + } + return result; + } + if (value.is_array()) { + auto result = JSON::array(); + for (const auto& item : value) { + auto cleaned = clean_caption(item); + if (!empty_value(cleaned)) { + result.push_back(std::move(cleaned)); + } + } + return result; + } + return value; + } + + static std::pair parse_prompt(const std::string& prompt) { + auto caption = JSON::parse(prompt, nullptr, false); + if (!caption.is_object()) { + return {prompt, nullptr}; + } + JSON instruction; + if (caption.contains("edit_instruction")) { + instruction = caption["edit_instruction"]; + caption.erase("edit_instruction"); + } + if (caption.contains("structured_caption")) { + caption = JSON(caption["structured_caption"]); + if (!caption.is_object()) { + return {caption, instruction}; + } + } + bool short_form = caption.size() == 2 && caption.contains("short_description") && caption.contains("text_render"); + caption = clean_caption(caption); + for (const char* score : {"aesthetic_score", "preference_score"}) { + caption.erase(score); + for (auto& value : caption) { + if (value.is_object()) { + value.erase(score); + } + } + } + caption = clean_caption(caption); + if (short_form) { + caption = JSON{{"short_description", caption.value("short_description", JSON(""))}, + {"text_render", caption.value("text_render", JSON::array())}}; + } + return {caption, instruction}; + } + + static std::string format_prompt(const std::string& text, int num_images) { + auto [caption, instruction] = parse_prompt(text); + if (num_images > 0) { + if (instruction.is_null()) { + caption = nullptr; + instruction = text; + } + auto edit = JSON::object(); + for (int i = 0; i < num_images; ++i) { + edit[num_images == 1 ? "image" : "image_" + std::to_string(i + 1)] = "<|vision_start|><|image_pad|><|vision_end|>"; + } + if (!empty_value(caption)) { + edit["structured_caption"] = caption; + } + edit["edit_instruction"] = instruction; + return edit.dump(); + } + if (text.empty()) { + return R"({"short_description":"","objects":[],"background_setting":"","lighting":{},"aesthetics":{},"photographic_characteristics":{},"style_medium":"","context":"","artistic_style":""})"; + } + if (caption.is_string()) { + return caption.get(); + } + return empty_value(caption) ? "{}" : caption.dump(); + } + + static sd::Tensor make_bundles(sd::Tensor states, const std::set& layers) { + constexpr int width = 2560; + constexpr int bundles[6][3] = {{9, 20, 31}, {5, 16, 27}, {7, 18, 29}, {9, 20, 31}, {11, 22, 33}, {13, 24, 35}}; + for (int64_t token = 0; token < states.shape()[1]; ++token) { + for (size_t layer = 0; layer < layers.size(); ++layer) { + float sum = 0.f; + int64_t offset = token * states.shape()[0] + layer * width; + for (int i = 0; i < width; ++i) { + sum += states[offset + i] * states[offset + i]; + } + float scale = 1.f / std::sqrt(sum / width + 1e-6f); + for (int i = 0; i < width; ++i) { + states[offset + i] *= scale; + } + } + } + sd::Tensor result({6 * 3 * width, states.shape()[1]}); + for (int b = 0; b < 6; ++b) { + for (int j = 0; j < 3; ++j) { + int index = static_cast(std::distance(layers.begin(), layers.find(bundles[b][j]))); + for (int64_t token = 0; token < states.shape()[1]; ++token) { + std::copy_n(states.data() + token * states.shape()[0] + index * width, width, + result.data() + token * result.shape()[0] + (b * 3 + j) * width); + } + } + } + return result; + } + + SDCondition get_learned_condition(int n_threads, const ConditionerParams& p) override { + int image_count = p.ref_images ? static_cast(p.ref_images->size()) : 0; + if (image_count && !llm->enable_vision) { + LOG_ERROR("Fibo 2.0 editing requires vision weights"); + return {}; + } + std::string prompt = "<|im_start|>user\n" + format_prompt(p.text, image_count) + "<|im_end|>\n<|im_start|>assistant\n"; + std::vector>> images; + LLM::DeepStackImageEmbeds deepstack; + std::vector grids; + size_t marker_start = 0; + for (int i = 0; i < image_count; ++i) { + const auto& image = (*p.ref_images)[i]; + int factor = llm->config.vision.patch_size * llm->config.vision.spatial_merge_size; + int height = static_cast(image.shape()[1]), width = static_cast(image.shape()[0]); + int h = std::max(factor, static_cast(std::round(static_cast(height) / factor)) * factor); + int w = std::max(factor, static_cast(std::round(static_cast(width) / factor)) * factor); + resize_image_dims(height, width, h, w, factor, + p.ref_image_params.vlm_min_size, + p.ref_image_params.vlm_max_size, + p.ref_image_params.vlm_resize_mode); + auto pixels = sd::ops::interpolate(image, {w, h, image.shape()[2], image.shape()[3]}, sd::ops::InterpolateMode::Bicubic, false, true); + auto rgb = sd::Tensor({w, h, 3, 1}); + for (int64_t pixel = 0; pixel < int64_t(w) * h; ++pixel) { + float alpha = pixels.shape()[2] == 4 ? pixels[pixel + 3 * w * h] : 1.f; + for (int channel = 0; channel < 3; ++channel) { + rgb[pixel + channel * w * h] = 2.f * std::clamp(pixels[pixel + channel * w * h] * alpha + 1.f - alpha, 0.f, 1.f) - 1.f; + } + } + auto outputs = llm->encode_image_outputs(n_threads, rgb, false); + if (!expand_vision_marker(prompt, marker_start, std::move(outputs), + h / llm->config.vision.patch_size, w / llm->config.vision.patch_size, + images, deepstack, grids, 4096)) { + return {}; + } + } + const std::set layers = {5, 7, 9, 11, 13, 16, 18, 20, 22, 24, 27, 29, 31, 33, 35}; + auto states = encode_prompt(n_threads, prompt, {0, 0}, 0, 0, images, layers, 0, false, 4096, deepstack, grids); + if (states.empty()) { + return {}; + } + SDCondition result; + result.c_crossattn = make_bundles(std::move(states), layers); + return result; + } +}; + +#endif // __SD_CONDITIONING_FIBO2_HPP__ diff --git a/src/core/ggml_extend_backend.cpp b/src/core/ggml_extend_backend.cpp index 57b1e8dea..9e57efaed 100644 --- a/src/core/ggml_extend_backend.cpp +++ b/src/core/ggml_extend_backend.cpp @@ -294,6 +294,10 @@ bool add_rpc_devices(const std::string& servers) { return false; } ggml_backend_reg_t rpc_reg = ggml_backend_reg_by_name("RPC"); + if (!rpc_reg) { + ggml_backend_load_all(); + rpc_reg = ggml_backend_reg_by_name("RPC"); + } if (!rpc_reg) { LOG_ERROR("RPC backend not found, cannot add RPC servers"); return false; @@ -307,12 +311,19 @@ bool add_rpc_devices(const std::string& servers) { for (const auto& server : rpc_servers) { LOG_INFO("Adding RPC server: %s", server.c_str()); auto reg = ggml_backend_rpc_add_server_fn(server.c_str()); - // no return value to check for success but should print errors from the RPC backend if it fails to add the server + if (!reg) { + LOG_ERROR("failed to add RPC server: %s", server.c_str()); + return false; + } ggml_backend_register(reg); } return true; } +bool sd_add_rpc_servers(const char* servers) { + return add_rpc_devices(servers ? servers : ""); +} + static void ggml_backend_load_all_once() { // If the registry already has devices and the CPU backend is present, // assume either static registration or explicit host-side preloading has diff --git a/src/core/util.cpp b/src/core/util.cpp index c1d463e4d..7bd5c0bd1 100644 --- a/src/core/util.cpp +++ b/src/core/util.cpp @@ -568,50 +568,59 @@ bool parse_strict_bool(const std::string& text, bool& value) { } static std::string build_progress_bar(int step, int steps, char progress_char = '=', bool show_head = true) { - std::string progress = " |"; - int max_progress = 50; - int32_t current = 0; + std::string progress = " |"; + constexpr int max_progress = 50; + int32_t current = 0; if (steps > 0) { - current = (int32_t)(step * 1.f * max_progress / steps); + current = (int32_t)(std::clamp(step, 0, steps) * 1.f * max_progress / steps); } - for (int i = 0; i < 50; i++) { - if (i > current) { - progress += " "; - } else if (show_head && i == current && i != max_progress - 1) { - progress += ">"; - } else { - progress += progress_char; - } - } - progress += "|"; + int head_length = (current == max_progress) ? 0 : 1; + progress += std::string(current, progress_char) + std::string(head_length, show_head ? '>' : progress_char) + std::string(max_progress - current - head_length, ' ') + "|"; return progress; } -static void print_progress_line(int step, int steps, const std::string& speed_text, char progress_char = '=', bool show_head = true) { +static void print_progress_line(int step, int steps, const std::string& speed_text, char progress_char = '=', bool show_head = true, const std::string& eta_text = "") { if (step == 0) { return; } std::string progress = build_progress_bar(step, steps, progress_char, show_head); + int step_width = static_cast(std::to_string(steps).size()); const char* lf = (step == steps ? "\n" : ""); - printf("\r%s %i/%i - %s\033[K%s", progress.c_str(), step, steps, speed_text.c_str(), lf); + printf("\r%s %*i/%i - %10s%s%s\033[K%s", progress.c_str(), step_width, step, steps, speed_text.c_str(), + eta_text.empty() ? "" : " - ETA ", eta_text.c_str(), lf); fflush(stdout); // for linux } +static int show_eta = -1; // -1 - estimate duration, 0 - hide, 1 - show +static float passed_time = 0.f; + void pretty_progress(int step, int steps, float time) { if (sd_progress_cb) { sd_progress_cb(step, steps, time, sd_progress_cb_data); return; } if (step == 0) { + show_eta = -1; // reinitialize for batch generations + passed_time = 0.f; return; } + std::string remainder_text = ""; + if (show_eta && step < steps) { // if timer is not disabled and bar is not filled, + if (show_eta == -1 && step) // estimate the total time once, then + show_eta = (std::abs(time * steps / step) < 15.f) ? 0 : 1; // hide the timer if total is less than 15 seconds + if (show_eta) { + unsigned remainder = (passed_time += time) / step * (steps - step); // average speed; time*(steps-step) for current + remainder_text = remainder < 3600 ? sd_format("%02u:%02u", remainder / 60, remainder % 60) + : sd_format("%u:%02u:%02u", remainder / 3600, (remainder / 60) % 60, remainder % 60); + } + } const char* unit = "s/it"; float speed = time; if (speed < 1.0f && speed > 0.f) { speed = 1.0f / speed; unit = "it/s"; } - print_progress_line(step, steps, sd_format("%.2f%s", speed, unit)); + print_progress_line(step, steps, sd_format("%.2f%s", speed, unit), '=', true, remainder_text); } void pretty_bytes_progress(int step, int steps, uint64_t bytes_processed, float elapsed_seconds) { diff --git a/src/detailer.cpp b/src/detailer.cpp index 72d01dc6f..ec4d6065a 100644 --- a/src/detailer.cpp +++ b/src/detailer.cpp @@ -291,7 +291,9 @@ static void filter_and_sort_detections(std::vector* detectio std::partial_sort(detections->begin(), detections->begin() + params.mask_k_largest, detections->end(), - [](const auto& a, const auto& b) { return detection_area(a) > detection_area(b); }); + [&](const auto& a, const auto& b) { + return params.mask_filter_by_confidence ? a.confidence > b.confidence : detection_area(a) > detection_area(b); + }); detections->resize(params.mask_k_largest); } @@ -545,7 +547,16 @@ static sd_image_t resize_crop_mask(const Mask& source, const CropRegion& crop, i return output; } -static sd_image_t copy_as_rgb(const sd_image_t& source) { +static sd_image_t copy_for_detailing(const sd_image_t& source) { + if (source.channel == 4) { + sd_image_t output = source; + size_t bytes = static_cast(source.width) * source.height * source.channel; + output.data = static_cast(malloc(bytes)); + if (output.data != nullptr) { + memcpy(output.data, source.data, bytes); + } + return output; + } CropRegion full{0, 0, static_cast(source.width), static_cast(source.height)}; return resize_crop_image(source, full, source.width, source.height); } @@ -566,7 +577,7 @@ static void composite_crop(sd_image_t* destination, continue; } float source_x = (x - crop.x1 + 0.5f) * scale_x - 0.5f; - size_t destination_index = (static_cast(y) * destination->width + x) * 3; + size_t destination_index = (static_cast(y) * destination->width + x) * destination->channel; for (int c = 0; c < 3; ++c) { float generated_value = bilinear_channel(generated, source_x, source_y, c) * 255.f; float original_value = destination->data[destination_index + c]; @@ -787,6 +798,10 @@ static bool parse_adetailer_params(const sd_adetailer_params_t& public_params, ok = parse_int(value, ¶ms->max_detections); } else if (key == "mask_k_largest") { ok = parse_int(value, ¶ms->mask_k_largest); + } else if (key == "mask_filter_method") { + const std::string method = lower_copy(value); + ok = method == "area" || method == "confidence"; + params->mask_filter_by_confidence = method == "confidence"; } else if (key == "mask_min_ratio") { ok = parse_float(value, ¶ms->mask_min_ratio); } else if (key == "mask_max_ratio") { @@ -805,6 +820,8 @@ static bool parse_adetailer_params(const sd_adetailer_params_t& public_params, ok = parse_int(value, ¶ms->mask_blur); } else if (key == "inpaint_padding") { ok = parse_int(value, ¶ms->inpaint_padding); + } else if (key == "inpaint_only_masked") { + ok = parse_bool(value, ¶ms->inpaint_only_masked); } else if (key == "inpaint_width") { ok = parse_int(value, ¶ms->inpaint_width); } else if (key == "inpaint_height") { @@ -920,7 +937,7 @@ bool adetail_image(adetailer_ctx_t* context, filter_and_sort_detections(&detections, input_image.width, input_image.height, params); auto masks = make_masks(detections, input_image.width, input_image.height, params); - sd_image_t current = copy_as_rgb(input_image); + sd_image_t current = copy_for_detailing(input_image); if (current.data == nullptr) { return false; } @@ -939,12 +956,13 @@ bool adetail_image(adetailer_ctx_t* context, if (!mask_bbox(masks[i], &crop)) { continue; } - crop = expand_crop(crop, - current.width, - current.height, - params.inpaint_padding, - params.inpaint_width, - params.inpaint_height); + crop = params.inpaint_only_masked ? expand_crop(crop, + current.width, + current.height, + params.inpaint_padding, + params.inpaint_width, + params.inpaint_height) + : CropRegion{0, 0, static_cast(current.width), static_cast(current.height)}; sd_image_t local_image = resize_crop_image(current, crop, params.inpaint_width, params.inpaint_height); sd_image_t local_mask = resize_crop_mask(masks[i], crop, params.inpaint_width, params.inpaint_height); if (local_image.data == nullptr || local_mask.data == nullptr) { diff --git a/src/detailer.h b/src/detailer.h index 17a81a1dd..0e3ae50f5 100644 --- a/src/detailer.h +++ b/src/detailer.h @@ -27,30 +27,32 @@ enum ADetailerSort { }; struct ADetailerParams { - const char* prompt = nullptr; - const char* negative_prompt = nullptr; - int input_size = 640; - float confidence = 0.3f; - float nms_threshold = 0.45f; - int max_detections = 100; - int mask_k_largest = 0; - float mask_min_ratio = 0.f; - float mask_max_ratio = 1.f; - int dilate_erode = 4; - int x_offset = 0; - int y_offset = 0; - bool merge_masks = false; - bool invert_mask = false; - int mask_blur = 4; - int inpaint_padding = 32; - int inpaint_width = 512; - int inpaint_height = 512; - float denoising_strength = 0.4f; - int steps = 0; - float cfg_scale = -1.f; - sample_method_t sample_method = SAMPLE_METHOD_COUNT; - scheduler_t scheduler = SCHEDULER_COUNT; - ADetailerSort sort_by = ADETAILER_SORT_NONE; + const char* prompt = nullptr; + const char* negative_prompt = nullptr; + int input_size = 640; + float confidence = 0.3f; + float nms_threshold = 0.45f; + int max_detections = 100; + int mask_k_largest = 0; + bool mask_filter_by_confidence = false; + float mask_min_ratio = 0.f; + float mask_max_ratio = 1.f; + int dilate_erode = 4; + int x_offset = 0; + int y_offset = 0; + bool merge_masks = false; + bool invert_mask = false; + int mask_blur = 4; + int inpaint_padding = 32; + bool inpaint_only_masked = true; + int inpaint_width = 512; + int inpaint_height = 512; + float denoising_strength = 0.4f; + int steps = 0; + float cfg_scale = -1.f; + sample_method_t sample_method = SAMPLE_METHOD_COUNT; + scheduler_t scheduler = SCHEDULER_COUNT; + ADetailerSort sort_by = ADETAILER_SORT_NONE; }; struct ADetailerGGML { diff --git a/src/model.h b/src/model.h index 0c1a37a66..73a369301 100644 --- a/src/model.h +++ b/src/model.h @@ -68,6 +68,7 @@ enum SDVersion { VERSION_IRIS, VERSION_FLUX3, VERSION_MOSS_IMAGE, + VERSION_FIBO2, VERSION_COUNT, }; @@ -293,7 +294,8 @@ static inline bool sd_version_uses_flux2_vae(SDVersion version) { sd_version_is_ideogram4(version) || sd_version_is_sefi_image(version) || sd_version_is_llada_image(version) || - version == VERSION_MOSS_IMAGE) { + version == VERSION_MOSS_IMAGE || + version == VERSION_FIBO2) { return true; } return false; @@ -351,7 +353,8 @@ static inline bool sd_version_is_dit(SDVersion version) { sd_version_is_sensenova_u1(version) || sd_version_is_pixart(version) || version == VERSION_IRIS || - version == VERSION_MOSS_IMAGE) { + version == VERSION_MOSS_IMAGE || + version == VERSION_FIBO2) { return true; } return false; @@ -385,7 +388,8 @@ static inline bool sd_version_supports_ref_image_conditioning(SDVersion version) sd_version_is_mage_flow(version) || sd_version_is_anima(version) || sd_version_is_sefi_image(version) || - sd_version_is_pid(version); + sd_version_is_pid(version) || + version == VERSION_FIBO2; } static bool sd_version_is_inpaint_or_unet_edit(SDVersion version) { diff --git a/src/model/diffusion/fibo2.hpp b/src/model/diffusion/fibo2.hpp new file mode 100644 index 000000000..f53b6ecd1 --- /dev/null +++ b/src/model/diffusion/fibo2.hpp @@ -0,0 +1,395 @@ +#ifndef __SD_MODEL_DIFFUSION_FIBO2_HPP__ +#define __SD_MODEL_DIFFUSION_FIBO2_HPP__ + +#include "model/diffusion/z_image.hpp" + +// https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/transformers/transformer_bria_fibo2.py +namespace Fibo2 { + struct Fibo2Config { + int64_t dim = 3584; + int64_t channels = 128; + int64_t text_dim = 7680; + int64_t head_dim = 128; + int64_t depth = 44; + int64_t refiner_depth = 2; + int64_t perceiver_depth = 2; + int64_t max_gist = 512; + int64_t ffn_dim = 9536; + bool fused_qkv = false; + bool fused_kv = false; + bool fused_ffn = false; + std::vector injection_layers = {6, 14, 22, 30, 38}; + + static Fibo2Config detect_from_weights(const String2TensorStorage& weights, const std::string& prefix) { + Fibo2Config config; + int64_t depth = 0, refiner_depth = 0, perceiver_depth = 0; + std::set injection_layers; + for (const auto& [name, tensor] : weights) { + if (!starts_with(name, prefix + ".")) { + continue; + } + auto key = name.substr(prefix.size() + 1); + if (key == "x_embedder.weight") { + config.channels = tensor.ne[0]; + config.dim = tensor.ne[1]; + } else if (key == "perceiver.latent_queries") { + config.max_gist = tensor.ne[1]; + } else if (key == "perceiver.layers.0.cross_attention.to_k.weight" || key == "perceiver.layers.0.cross_attention.to_kv.weight") { + config.text_dim = tensor.ne[0]; + config.fused_kv = key == "perceiver.layers.0.cross_attention.to_kv.weight"; + } else if (key == "layers.0.attention.norm_q.weight") { + config.head_dim = tensor.ne[0]; + } else if (key == "layers.0.attention.qkv.weight") { + config.fused_qkv = true; + } else if (key == "layers.0.feed_forward.w1.weight") { + config.ffn_dim = tensor.ne[1]; + } else if (key == "layers.0.feed_forward.w13.weight") { + config.ffn_dim = tensor.ne[1] / 2; + config.fused_ffn = true; + } + if (starts_with(key, "layers.")) { + int index = std::stoi(key.substr(7)); + depth = std::max(depth, index + 1); + if (ends_with(key, ".cross_attn_gate")) { + injection_layers.insert(index); + } + } else if (starts_with(key, "noise_refiner.")) { + refiner_depth = std::max(refiner_depth, std::stoi(key.substr(14)) + 1); + } else if (starts_with(key, "perceiver.layers.")) { + perceiver_depth = std::max(perceiver_depth, std::stoi(key.substr(17)) + 1); + } + } + if (depth) { + config.depth = depth; + config.refiner_depth = refiner_depth; + config.perceiver_depth = perceiver_depth; + config.injection_layers.assign(injection_layers.begin(), injection_layers.end()); + LOG_VERBOSE("fibo2: dim = %" PRId64 ", depth = %" PRId64 ", head_dim = %" PRId64 ", text_dim = %" PRId64, + config.dim, config.depth, config.head_dim, config.text_dim); + } + return config; + } + + int gist_count(int text_length) const { + double ratio = std::clamp((text_length - 256.0) / (2048.0 - 256.0), 0.0, 1.0); + return std::min(static_cast(max_gist), static_cast(std::ceil((256 + ratio * (max_gist - 256)) / 64)) * 64); + } + }; + + struct Attention : public GGMLBlock { + int64_t head_dim; + int64_t heads; + bool fused_qkv; + bool fused_kv; + + Attention(const Fibo2Config& config, int64_t kv_dim, bool joint = false) + : head_dim(config.head_dim), heads(config.dim / config.head_dim), fused_qkv(joint && config.fused_qkv), fused_kv(!joint && config.fused_kv) { + if (fused_qkv) { + blocks["qkv"] = std::make_shared(config.dim, 3 * config.dim, false); + } else { + blocks["to_q"] = std::make_shared(config.dim, config.dim, false); + if (fused_kv) { + blocks["to_kv"] = std::make_shared(kv_dim, 2 * config.dim, false); + } else { + blocks["to_k"] = std::make_shared(kv_dim, config.dim, false); + blocks["to_v"] = std::make_shared(kv_dim, config.dim, false); + } + } + blocks["to_out.0"] = std::make_shared(config.dim, config.dim, false); + blocks["norm_q"] = std::make_shared(head_dim, 1e-5f); + blocks["norm_k"] = std::make_shared(head_dim, 1e-5f); + } + + ggml_tensor* forward(GGMLRunnerContext* ctx, ggml_tensor* x, ggml_tensor* kv, ggml_tensor* pe = nullptr) { + auto c = ctx->ggml_ctx; + auto project = [&](const char* name, ggml_tensor* h) { + return std::dynamic_pointer_cast(blocks[name])->forward(ctx, h); + }; + ggml_tensor *q, *k, *v; + if (fused_qkv) { + auto parts = ggml_ext_chunk(c, project("qkv", x), 3, 0); + q = parts[0]; + k = parts[1]; + v = parts[2]; + } else { + q = project("to_q", x); + if (fused_kv) { + auto parts = ggml_ext_chunk(c, project("to_kv", kv), 2, 0); + k = parts[0]; + v = parts[1]; + } else { + k = project("to_k", kv); + v = project("to_v", kv); + } + } + auto reshape = [&](ggml_tensor* h) { + return ggml_reshape_4d(c, h, head_dim, heads, h->ne[1], h->ne[2]); + }; + q = std::dynamic_pointer_cast(blocks["norm_q"])->forward(ctx, reshape(q)); + k = std::dynamic_pointer_cast(blocks["norm_k"])->forward(ctx, reshape(k)); + v = reshape(v); + auto prepare = [&](ggml_tensor* h) { + if (pe) { + return Rope::apply_rope(ctx, h, pe); + } + h = ggml_cont(c, ggml_permute(c, h, 0, 2, 1, 3)); + return ggml_reshape_3d(c, h, head_dim, h->ne[1], heads * h->ne[3]); + }; + auto y = ggml_ext_attention_ext(ctx, prepare(q), prepare(k), v, heads, nullptr, true, ctx->flash_attn_enabled); + return std::dynamic_pointer_cast(blocks["to_out.0"])->forward(ctx, y); + } + }; + + struct FeedForward : public ZImage::FeedForward { + bool fused; + + FeedForward(const Fibo2Config& config) + : ZImage::FeedForward(config.dim, config.ffn_dim, 1), fused(config.fused_ffn) { + if (fused) { + blocks.erase("w1"); + blocks.erase("w3"); + blocks["w13"] = std::make_shared(config.dim, 2 * config.ffn_dim, false); + } + } + + ggml_tensor* forward(GGMLRunnerContext* ctx, ggml_tensor* x) { + if (!fused) { + return ZImage::FeedForward::forward(ctx, x); + } + auto w2 = std::dynamic_pointer_cast(blocks["w2"]); + if (sd_backend_is(ctx->backend, "Vulkan") || sd_backend_is(ctx->backend, "ROCm")) { + w2->set_force_prec_f32(true); + } + x = std::dynamic_pointer_cast(blocks["w13"])->forward(ctx, x); + return w2->forward(ctx, ggml_swiglu(ctx->ggml_ctx, x)); + } + }; + + struct PerceiverLayer : public GGMLBlock { + PerceiverLayer(const Fibo2Config& config) { + for (const char* name : {"cross_attn_norm", "self_attn_norm", "ffn_norm"}) { + blocks[name] = std::make_shared(config.dim, 1e-5f); + } + blocks["cross_attention"] = std::make_shared(config, config.text_dim); + blocks["self_attention"] = std::make_shared(config, config.dim); + blocks["ffn"] = std::make_shared(config); + } + + ggml_tensor* forward(GGMLRunnerContext* ctx, ggml_tensor* x, ggml_tensor* text) { + auto norm = [&](const char* name, ggml_tensor* h) { + return std::dynamic_pointer_cast(blocks[name])->forward(ctx, h); + }; + auto h = std::dynamic_pointer_cast(blocks["cross_attention"])->forward(ctx, norm("cross_attn_norm", x), text); + x = ggml_add(ctx->ggml_ctx, x, h); + h = norm("self_attn_norm", x); + h = std::dynamic_pointer_cast(blocks["self_attention"])->forward(ctx, h, h); + x = ggml_add(ctx->ggml_ctx, x, h); + h = std::dynamic_pointer_cast(blocks["ffn"])->forward(ctx, norm("ffn_norm", x)); + return ggml_add(ctx->ggml_ctx, x, h); + } + }; + + struct Perceiver : public GGMLBlock { + Fibo2Config config; + + Perceiver(const Fibo2Config& config) + : config(config) { + for (int i = 0; i < config.perceiver_depth; ++i) { + blocks["layers." + std::to_string(i)] = std::make_shared(config); + } + blocks["norm_out"] = std::make_shared(config.dim, 1e-5f); + } + + void init_params(ggml_context* ctx, const String2TensorStorage& weights = {}, const std::string prefix = "") override { + params["latent_queries"] = ggml_new_tensor_2d(ctx, GGML_TYPE_F32, config.dim, config.max_gist); + } + + ggml_tensor* forward(GGMLRunnerContext* ctx, ggml_tensor* text, int gist_count) { + auto x = ggml_ext_slice(ctx->ggml_ctx, params["latent_queries"], 1, 0, gist_count); + for (int i = 0; i < config.perceiver_depth; ++i) { + x = std::dynamic_pointer_cast(blocks["layers." + std::to_string(i)])->forward(ctx, x, text); + } + return std::dynamic_pointer_cast(blocks["norm_out"])->forward(ctx, x); + } + }; + + struct TransformerBlock : public GGMLBlock { + bool injection; + + TransformerBlock(const Fibo2Config& config, bool injection) + : injection(injection) { + blocks["attention"] = std::make_shared(config, config.dim, true); + blocks["feed_forward"] = std::make_shared(config); + for (const char* name : {"attention_norm1", "attention_norm2", "ffn_norm1", "ffn_norm2"}) { + blocks[name] = std::make_shared(config.dim, 1e-5f); + } + blocks["adaLN_modulation.0"] = std::make_shared(std::min(config.dim, 256), 4 * config.dim); + if (injection) { + blocks["cross_attention"] = std::make_shared(config, config.text_dim); + blocks["cross_attn_norm1"] = std::make_shared(config.dim, 1e-5f); + blocks["cross_attn_norm2"] = std::make_shared(config.dim, 1e-5f); + } + } + + void init_params(ggml_context* ctx, const String2TensorStorage& weights = {}, const std::string prefix = "") override { + if (injection) { + params["cross_attn_gate"] = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 1); + } + } + + ggml_tensor* forward(GGMLRunnerContext* ctx, ggml_tensor* x, ggml_tensor* pe, ggml_tensor* noisy, ggml_tensor* clean, int image_start, int image_count, ggml_tensor* text = nullptr) { + auto c = ctx->ggml_ctx; + auto norm = [&](const char* name, ggml_tensor* h) { + return std::dynamic_pointer_cast(blocks[name])->forward(ctx, h); + }; + auto modulation = std::dynamic_pointer_cast(blocks["adaLN_modulation.0"]); + auto mn = ggml_ext_chunk(c, modulation->forward(ctx, noisy), 4, 0); + auto mc = ggml_ext_chunk(c, modulation->forward(ctx, clean), 4, 0); + auto select = [&](int index, bool gate) { + auto n = gate ? ggml_tanh(c, mn[index]) : mn[index]; + auto k = gate ? ggml_tanh(c, mc[index]) : mc[index]; + auto repeat = [&](ggml_tensor* h, int64_t count) { + return ggml_repeat(c, h, ggml_new_tensor_2d(c, GGML_TYPE_F32, h->ne[0], count)); + }; + auto m = repeat(n, image_count); + if (image_start) { + m = ggml_concat(c, repeat(k, image_start), m, 1); + } + int64_t tail = x->ne[1] - image_start - image_count; + if (tail) { + m = ggml_concat(c, m, repeat(k, tail), 1); + } + return m; + }; + auto h = ZImage::modulate(c, norm("attention_norm1", x), select(0, false)); + h = std::dynamic_pointer_cast(blocks["attention"])->forward(ctx, h, h, pe); + x = ggml_add(c, x, ggml_mul(c, norm("attention_norm2", h), select(1, true))); + if (injection) { + h = std::dynamic_pointer_cast(blocks["cross_attention"])->forward(ctx, norm("cross_attn_norm1", x), text); + h = ggml_mul(c, norm("cross_attn_norm2", h), ggml_tanh(c, params["cross_attn_gate"])); + x = ggml_add(c, x, h); + } + h = ZImage::modulate(c, norm("ffn_norm1", x), select(2, false)); + h = std::dynamic_pointer_cast(blocks["feed_forward"])->forward(ctx, h); + return ggml_add(c, x, ggml_mul(c, norm("ffn_norm2", h), select(3, true))); + } + }; + + struct Fibo2Model : public GGMLBlock { + Fibo2Config config; + + Fibo2Model(const Fibo2Config& config) + : config(config) { + blocks["x_embedder"] = std::make_shared(config.channels, config.dim); + blocks["t_embedder"] = std::make_shared(1024, 256, std::min(config.dim, 256)); + blocks["perceiver"] = std::make_shared(config); + for (int i = 0; i < config.refiner_depth; ++i) { + blocks["noise_refiner." + std::to_string(i)] = std::make_shared(config, false); + } + for (int i = 0; i < config.depth; ++i) { + bool injection = std::find(config.injection_layers.begin(), config.injection_layers.end(), i) != config.injection_layers.end(); + blocks["layers." + std::to_string(i)] = std::make_shared(config, injection); + } + blocks["final_layer"] = std::make_shared(config.dim, 1, config.channels); + } + + ggml_tensor* forward(GGMLRunnerContext* ctx, ggml_tensor* x, ggml_tensor* t, ggml_tensor* text, ggml_tensor* pe, int gist_count, const std::vector& refs) { + auto c = ctx->ggml_ctx; + int64_t width = x->ne[0], height = x->ne[1]; + int image_count = static_cast(width * height); + // The shared FLUX.2 VAE already packs (channel, dy, dx) into 128 channels. + auto tokens = [&](ggml_tensor* h) { + h = ggml_reshape_3d(c, h, h->ne[0] * h->ne[1], config.channels, 1); + return ggml_cont(c, ggml_permute(c, h, 1, 0, 2, 3)); + }; + x = tokens(x); + for (auto ref : refs) { + x = ggml_concat(c, x, tokens(ref), 1); + } + x = std::dynamic_pointer_cast(blocks["x_embedder"])->forward(ctx, x); + auto embedder = std::dynamic_pointer_cast(blocks["t_embedder"]); + auto noisy = embedder->forward(ctx, t); + auto clean = embedder->forward(ctx, ggml_scale(c, t, 0.f)); + auto image_pe = ggml_ext_slice(c, pe, 3, gist_count, pe->ne[3]); + for (int i = 0; i < config.refiner_depth; ++i) { + x = std::dynamic_pointer_cast(blocks["noise_refiner." + std::to_string(i)])->forward(ctx, x, image_pe, noisy, clean, 0, image_count); + sd::ggml_graph_cut::mark_graph_cut(x, "fibo2.noise_refiner." + std::to_string(i), "x"); + } + auto bundle = [&](int i) { return ggml_ext_slice(c, text, 0, i * config.text_dim, (i + 1) * config.text_dim); }; + auto gist = std::dynamic_pointer_cast(blocks["perceiver"])->forward(ctx, bundle(0), gist_count); + x = ggml_concat(c, gist, x, 1); + int bundle_index = 1; + for (int i = 0; i < config.depth; ++i) { + auto block = std::dynamic_pointer_cast(blocks["layers." + std::to_string(i)]); + auto context = block->injection ? bundle(bundle_index++) : nullptr; + x = block->forward(ctx, x, pe, noisy, clean, gist_count, image_count, context); + sd::ggml_graph_cut::mark_graph_cut(x, "fibo2.layers." + std::to_string(i), "x"); + } + x = ggml_ext_slice(c, x, 1, gist_count, gist_count + image_count); + x = std::dynamic_pointer_cast(blocks["final_layer"])->forward(ctx, x, noisy); + x = ggml_cont(c, ggml_permute(c, x, 1, 0, 2, 3)); + return ggml_reshape_4d(c, x, width, height, config.channels, 1); + } + }; + + struct Fibo2Runner : public DiffusionModelRunner { + Fibo2Config config; + Fibo2Model model; + std::vector pe_values; + + Fibo2Runner(ggml_backend_t backend, const String2TensorStorage& weights, const std::string& prefix, std::shared_ptr weight_manager = nullptr) + : DiffusionModelRunner(backend, prefix, weight_manager), + config(Fibo2Config::detect_from_weights(weights, prefix)), + model(config) { + model.init(params_ctx, weights, prefix); + } + + std::string get_desc() override { return "fibo2"; } + + void get_param_tensors(std::map& tensors, const std::string& prefix) override { + model.get_param_tensors(tensors, prefix); + } + + sd::Tensor compute(int n_threads, const DiffusionParams& p) override { + if (!p.x || !p.timesteps || !p.context || p.context->shape()[0] != config.text_dim * 6) { + LOG_ERROR("Fibo 2.0 requires six text feature bundles"); + return {}; + } + auto build = [&]() { + auto graph = new_graph_custom(65536); + auto x = make_input(*p.x); + GGML_ASSERT(x->ne[3] == 1 && x->ne[2] == config.channels); + int count = config.gist_count(static_cast(p.context->shape()[1])); + Rope::Embedding embedding; + for (int i = 0; i < count; ++i) { + embedding.ids.push_back({static_cast(i + 1), 0.f, 0.f}); + } + embedding.positions.append_tokens(count); + auto append_image = [&](ggml_tensor* image, int plane) { + auto ids = Rope::gen_flux_img_ids(static_cast(image->ne[1]), static_cast(image->ne[0]), + 1, 1, 3, plane, 0, 0, false, &embedding.positions); + embedding.ids.insert(embedding.ids.end(), ids.begin(), ids.end()); + }; + append_image(x, static_cast(config.max_gist + 1)); + std::vector refs; + if (p.ref_latents && p.ref_image_params.pass_to_dit) { + for (const auto& ref : *p.ref_latents) { + refs.push_back(make_input(ref)); + append_image(refs.back(), static_cast(config.max_gist + 1 + refs.size())); + } + } + embedding.values = Rope::embed_nd(embedding.ids, 1, 256.f, {32, 48, 48}, embedding.layout, &embedding.frequencies); + pe_values = finish_rope_pe(std::move(embedding)); + auto pe = ggml_new_tensor_4d(compute_ctx, GGML_TYPE_F32, 2, 2, 64, static_cast(pe_values.size() / 256)); + set_backend_tensor_data(pe, pe_values.data()); + auto ctx = get_context(); + auto out = model.forward(&ctx, x, make_input(*p.timesteps), make_input(*p.context), pe, count, refs); + ggml_build_forward_expand(graph, out); + return graph; + }; + return restore_trailing_singleton_dims(GGMLRunner::compute(build, n_threads, false), p.x->dim()); + } + }; +} + +#endif // __SD_MODEL_DIFFUSION_FIBO2_HPP__ diff --git a/src/model/diffusion/model.hpp b/src/model/diffusion/model.hpp index 955c5277a..7c14d9d1f 100644 --- a/src/model/diffusion/model.hpp +++ b/src/model/diffusion/model.hpp @@ -27,9 +27,25 @@ struct RefImageParams { int vlm_min_size = -1; int vlm_max_size = -1; bool resize_vae_to_target = false; + bool resize_single_ref_to_target = false; + bool vae_size_buckets = false; + bool vae_center_crop = false; + bool vlm_use_vae_input = false; }; const std::unordered_map REF_IMAGE_PRESETS = { + {"fibo2", [] { + RefImageParams params; + params.pass_to_vlm = true; + params.ref_index_mode = Rope::RefIndexMode::INCREASE; + params.vlm_resize_mode = RefImageResizeMode::LONGEST_SIDE; + params.vlm_max_size = 768; + params.resize_single_ref_to_target = true; + params.vae_size_buckets = true; + params.vae_center_crop = true; + params.vlm_use_vae_input = true; + return params; + }()}, {"flux_kontext", {false, true, Rope::RefIndexMode::FIXED, false, true, -1, RefImageResizeMode::NONE, -1, -1}}, {"longcat", {true, true, Rope::RefIndexMode::FIXED, false, true, -1, RefImageResizeMode::AREA, -1, -1}}, {"flux2", {false, true, Rope::RefIndexMode::INCREASE, false, true, -1, RefImageResizeMode::NONE, -1, -1}}, diff --git a/src/model_loader.cpp b/src/model_loader.cpp index 8822b2c3f..a0f500819 100644 --- a/src/model_loader.cpp +++ b/src/model_loader.cpp @@ -439,6 +439,10 @@ bool ModelLoader::init_from_diffusers_file(const std::string& file_path, const s SDVersion ModelLoader::get_sd_version() const { TensorStorage token_embedding_weight, input_block_weight, context_ebedding_weight; + if (tensor_storage_map.count("model.diffusion_model.perceiver.latent_queries")) { + return VERSION_FIBO2; + } + if (tensor_storage_map.find("model.diffusion_model.emb_in.image.weight") != tensor_storage_map.end() && tensor_storage_map.find("model.diffusion_model.txt_mode_blocks.0.q_proj.weight") != tensor_storage_map.end()) { return VERSION_FLUX3; diff --git a/src/model_manager.cpp b/src/model_manager.cpp index 4562ebdfd..921989df7 100644 --- a/src/model_manager.cpp +++ b/src/model_manager.cpp @@ -1916,12 +1916,12 @@ bool ModelManager::ensure_compute_backend_capacity( }; size_t last_eviction_count = 0; auto release_eviction_states = [&]() { - if (eviction_states.size() == last_eviction_count) { - return false; + if (eviction_states.size() != last_eviction_count) { + last_eviction_count = eviction_states.size(); + release_compute_staging_blocks(false, &eviction_states); + release_params_storage_blocks(false, &eviction_states); } - last_eviction_count = eviction_states.size(); - release_compute_staging_blocks(false, &eviction_states); - release_params_storage_blocks(false, &eviction_states); + // Device memory reports can lag behind releases even when no new candidates are added. return fits(); }; diff --git a/src/name_conversion.cpp b/src/name_conversion.cpp index 6a519ba21..f37c4ddbe 100644 --- a/src/name_conversion.cpp +++ b/src/name_conversion.cpp @@ -197,7 +197,11 @@ std::string convert_cond_stage_model_name(std::string name, std::string prefix, replace_with_name_map(name, llm_vision_name_map); } else { replace_with_name_map(name, llm_name_map); - name = convert_llada2_moe_te_name(name); + name = convert_llada2_moe_te_name(name); + const std::string language_prefix = "llm.model.language_model."; + if (starts_with(name, language_prefix)) { + name = "llm.model." + name.substr(language_prefix.size()); + } } } else { name = convert_open_clip_to_hf_clip_name(name); @@ -1034,6 +1038,18 @@ static std::string convert_ming_image_dit_name(std::string name) { return name; } +static std::string convert_fibo2_dit_name(std::string name) { + static const std::vector> name_map = { + {".attention.q_norm.", ".attention.norm_q."}, + {".attention.k_norm.", ".attention.norm_k."}, + {".attention.out.", ".attention.to_out.0."}, + {".cross_attention.to_out.weight", ".cross_attention.to_out.0.weight"}, + {".self_attention.to_out.weight", ".self_attention.to_out.0.weight"}, + }; + replace_with_name_map(name, name_map); + return name; +} + std::string convert_diffusers_dit_to_original_minimax_h3(std::string name) { static const std::vector> prefix_map = { {"transformer_blocks.", "blocks."}, @@ -1067,6 +1083,8 @@ std::string convert_diffusion_model_name(std::string name, std::string prefix, S name = convert_moss_image_dit_name(name); } else if (version == VERSION_MING_IMAGE) { name = convert_ming_image_dit_name(name); + } else if (version == VERSION_FIBO2) { + name = convert_fibo2_dit_name(name); } else if (sd_version_is_z_image(version) || sd_version_is_z_image_l2p(version)) { name = convert_diffusers_dit_to_original_lumina2(name); } else if (sd_version_is_llada_image(version)) { @@ -1715,7 +1733,7 @@ std::string convert_tensor_name(std::string name, SDVersion version) { } } - if (version == VERSION_QWEN_IMAGE_2_1 || sd_version_is_boogu_image(version) || sd_version_is_krea2(version) || sd_version_is_mage_flow(version) || sd_version_is_minimax_h3(version)) { + if (version == VERSION_FIBO2 || version == VERSION_QWEN_IMAGE_2_1 || sd_version_is_boogu_image(version) || sd_version_is_krea2(version) || sd_version_is_mage_flow(version) || sd_version_is_minimax_h3(version)) { const std::string hf_vision_prefix = "text_encoders.llm.model.visual."; if (starts_with(name, hf_vision_prefix)) { name = "text_encoders.llm.visual." + name.substr(hf_vision_prefix.size()); diff --git a/src/pipeline/diffusion_engine.cpp b/src/pipeline/diffusion_engine.cpp index 22ef3ac45..1e3c2e24b 100644 --- a/src/pipeline/diffusion_engine.cpp +++ b/src/pipeline/diffusion_engine.cpp @@ -113,6 +113,7 @@ const char* model_version_to_str[] = { "Iris-3B", "Flux.3 Image Klein", "MOSS-Image", + "Fibo-2.0", }; static_assert(VERSION_COUNT == sizeof(model_version_to_str) / sizeof(model_version_to_str[0]), @@ -1499,7 +1500,8 @@ bool StableDiffusionGGML::build_denoiser() { sd_version_is_pid(version) || sd_version_is_ideogram4(version) || version == VERSION_IRIS || - version == VERSION_MOSS_IMAGE) { + version == VERSION_MOSS_IMAGE || + version == VERSION_FIBO2) { pred_type = FLOW_PRED; if (sd_version_is_wan(version) || version == VERSION_MOSS_IMAGE) { default_flow_shift = 5.f; @@ -3205,7 +3207,9 @@ bool StableDiffusionGGML::is_flow_denoiser() { } std::string StableDiffusionGGML::get_default_ref_image_preset(SDVersion version) const { - if (sd_version_is_longcat(version)) { + if (version == VERSION_FIBO2) { + return "fibo2"; + } else if (sd_version_is_longcat(version)) { return "longcat"; } else if (sd_version_is_flux(version)) { return "flux_kontext"; @@ -3257,6 +3261,9 @@ RefImageParams StableDiffusionGGML::resolve_ref_image_params(const char* ref_ima LOG_INFO("Using '%s' preset for reference images", preset_name.c_str()); params = REF_IMAGE_PRESETS.at(preset_name); } + if (version == VERSION_QWEN_IMAGE_2_1) { + params.vlm_use_vae_input = true; + } for (const auto& [key, value] : parse_key_value_args(ref_image_args, "reference image args")) { if (key == "pass_to_vlm") { @@ -3285,6 +3292,26 @@ RefImageParams StableDiffusionGGML::resolve_ref_image_params(const char* ref_ima if (!parse_strict_bool(value, params.resize_before_vae)) { LOG_WARN("ignoring invalid reference image arg '%s=%s'", key.c_str(), value.c_str()); } + } else if (key == "resize_vae_to_target") { + if (!parse_strict_bool(value, params.resize_vae_to_target)) { + LOG_WARN("ignoring invalid reference image arg '%s=%s'", key.c_str(), value.c_str()); + } + } else if (key == "resize_single_ref_to_target") { + if (!parse_strict_bool(value, params.resize_single_ref_to_target)) { + LOG_WARN("ignoring invalid reference image arg '%s=%s'", key.c_str(), value.c_str()); + } + } else if (key == "vae_size_buckets") { + if (!parse_strict_bool(value, params.vae_size_buckets)) { + LOG_WARN("ignoring invalid reference image arg '%s=%s'", key.c_str(), value.c_str()); + } + } else if (key == "vae_center_crop") { + if (!parse_strict_bool(value, params.vae_center_crop)) { + LOG_WARN("ignoring invalid reference image arg '%s=%s'", key.c_str(), value.c_str()); + } + } else if (key == "vlm_use_vae_input") { + if (!parse_strict_bool(value, params.vlm_use_vae_input)) { + LOG_WARN("ignoring invalid reference image arg '%s=%s'", key.c_str(), value.c_str()); + } } else if (key == "vae_input_max_pixels") { if (!parse_strict_int(value, params.vae_input_max_pixels)) { LOG_WARN("ignoring invalid reference image arg '%s=%s'", key.c_str(), value.c_str()); diff --git a/src/pipeline/image.cpp b/src/pipeline/image.cpp index d3d25c8af..c683b53ff 100644 --- a/src/pipeline/image.cpp +++ b/src/pipeline/image.cpp @@ -284,26 +284,34 @@ namespace sd::pipeline { LOG_VERBOSE("auto resize ref images"); double vae_width; double vae_height; - if (ref_image_params.resize_vae_to_target) { + int factor = sd_version_is_qwen_image(sd->version) ? 32 : 16; + if (ref_image_params.resize_vae_to_target || (ref_image_params.resize_single_ref_to_target && ref_images.size() == 1)) { vae_width = request->width; vae_height = request->height; } else { int default_pixels = sd->version == VERSION_QWEN_IMAGE_2_1 ? request->width * request->height : 1024 * 1024; int target_pixels = ref_image_params.vae_input_max_pixels > 0 ? ref_image_params.vae_input_max_pixels : default_pixels; - int vae_image_size = std::min(target_pixels, request->width * request->height); - vae_width = sqrt(vae_image_size * ref_images[i].shape()[0] / ref_images[i].shape()[1]); - vae_height = vae_width * ref_images[i].shape()[1] / ref_images[i].shape()[0]; + if (ref_image_params.vae_size_buckets) { + auto size = sd::select_image_size_bucket(static_cast(ref_images[i].shape()[0]), static_cast(ref_images[i].shape()[1]), target_pixels, factor, request->width, request->height); + vae_width = size.first; + vae_height = size.second; + } else { + int vae_image_size = std::min(target_pixels, request->width * request->height); + vae_width = sqrt(vae_image_size * ref_images[i].shape()[0] / ref_images[i].shape()[1]); + vae_height = vae_width * ref_images[i].shape()[1] / ref_images[i].shape()[0]; + } } - int factor = sd_version_is_qwen_image(sd->version) ? 32 : 16; vae_height = round(vae_height / factor) * factor; vae_width = round(vae_width / factor) * factor; - auto resized_ref_img = sd::ops::interpolate(ref_images[i], - {static_cast(vae_width), - static_cast(vae_height), - ref_images[i].shape()[2], - ref_images[i].shape()[3]}); + auto resized_ref_img = ref_image_params.vae_center_crop + ? sd::resize_and_center_crop(ref_images[i], static_cast(vae_width), static_cast(vae_height)) + : sd::ops::interpolate(ref_images[i], + {static_cast(vae_width), + static_cast(vae_height), + ref_images[i].shape()[2], + ref_images[i].shape()[3]}); LOG_VERBOSE("resize vae ref image %d from %" PRId64 "x%" PRId64 " to %" PRId64 "x%" PRId64, static_cast(i), @@ -313,7 +321,7 @@ namespace sd::pipeline { resized_ref_img.shape()[0]); ref_latent = sd->encode_first_stage(resized_ref_img); - if (sd->version == VERSION_QWEN_IMAGE_2_1) { + if (ref_image_params.vlm_use_vae_input) { ref_images[i] = std::move(resized_ref_img); } } else { @@ -482,8 +490,8 @@ namespace sd::pipeline { } condition_params.text = request->negative_prompt; condition_params.zero_out_masked = zero_out_masked; - if (sd_version_is_llada_image(sd->version)) { - // LLaDA-Image CFG keeps the source latent but drops its SigVQ features. + if (sd_version_is_llada_image(sd->version) || sd->version == VERSION_FIBO2) { + // Keep source latents in CFG, but omit source images from the text encoder. condition_params.ref_images = nullptr; } uncond = sd->get_learned_condition(condition_params); diff --git a/src/pipeline/model_builders.cpp b/src/pipeline/model_builders.cpp index 4b435170f..ca4d1998b 100644 --- a/src/pipeline/model_builders.cpp +++ b/src/pipeline/model_builders.cpp @@ -5,6 +5,7 @@ #include #include "conditioning/conditioner.hpp" +#include "conditioning/fibo2.hpp" #include "core/ggml_extend_backend.h" #include "core/util.h" #include "extensions/generation_extension.h" @@ -14,6 +15,7 @@ #include "model/diffusion/boogu.hpp" #include "model/diffusion/control.hpp" #include "model/diffusion/ernie_image.hpp" +#include "model/diffusion/fibo2.hpp" #include "model/diffusion/flux.hpp" #include "model/diffusion/flux3.hpp" #include "model/diffusion/hidream_o1.hpp" @@ -94,7 +96,12 @@ namespace sd::model_builders { return false; } - if (sd_version_is_sd3(version)) { + if (version == VERSION_FIBO2) { + result.conditioner = std::make_shared(ctx.backends.runtime_backend(SDBackendModule::TE), + tensor_storage_map, weight_manager, tokenizers); + result.diffusion = std::make_shared(ctx.backends.runtime_backend(SDBackendModule::DIFFUSION), + tensor_storage_map, "model.diffusion_model", weight_manager); + } else if (sd_version_is_sd3(version)) { result.conditioner = std::make_shared(ctx.backends.runtime_backend(SDBackendModule::TE), tensor_storage_map, weight_manager, diff --git a/src/pipeline/request.cpp b/src/pipeline/request.cpp index 69ebe0b35..1df3e9f19 100644 --- a/src/pipeline/request.cpp +++ b/src/pipeline/request.cpp @@ -65,7 +65,7 @@ namespace sd::pipeline { return LCM_SCHEDULER; } else if (sample_method == DDIM_TRAILING_SAMPLE_METHOD) { return SIMPLE_SCHEDULER; - } else if (sd != nullptr && (sd_version_is_flux(sd->version) || sd->version == VERSION_QWEN_IMAGE_2_1)) { + } else if (sd != nullptr && (sd_version_is_flux(sd->version) || sd->version == VERSION_QWEN_IMAGE_2_1 || sd->version == VERSION_FIBO2)) { return FLUX_SCHEDULER; } else if (sd != nullptr && sd_version_is_flux2(sd->version)) { return FLUX2_SCHEDULER; diff --git a/src/runtime/image_preprocess.cpp b/src/runtime/image_preprocess.cpp index a8df7c3b5..ec649390b 100644 --- a/src/runtime/image_preprocess.cpp +++ b/src/runtime/image_preprocess.cpp @@ -6,6 +6,42 @@ namespace sd { + std::pair select_image_size_bucket(int source_width, int source_height, int area, int factor, int fallback_width, int fallback_height) { + int width = fallback_width; + int height = fallback_height; + double ratio = static_cast(source_width) / source_height; + double best_crop = INFINITY; + int64_t best_area_error = INT64_MAX; + int min_size = 4 * factor; + for (int w = min_size; w < 2LL * area / min_size; w += factor) { + for (int h = min_size; h < 2LL * area / w; h += factor) { + int64_t pixels = int64_t(w) * h; + if (pixels < 0.9 * area || pixels > 1.1 * area) { + continue; + } + double candidate_ratio = static_cast(w) / h; + double crop = std::round((1.0 - std::min(candidate_ratio / ratio, ratio / candidate_ratio)) * 1e9); + int64_t area_error = std::abs(pixels - area); + if (crop < best_crop || (crop == best_crop && (area_error < best_area_error || (area_error == best_area_error && w > width)))) { + width = w; + height = h; + best_crop = crop; + best_area_error = area_error; + } + } + } + return {width, height}; + } + + Tensor resize_and_center_crop(const Tensor& image, int width, int height, ops::InterpolateMode filter) { + double scale = std::max(static_cast(width) / image.shape()[0], static_cast(height) / image.shape()[1]); + int w = static_cast(std::ceil(image.shape()[0] * scale)); + int h = static_cast(std::ceil(image.shape()[1] * scale)); + auto resized = ops::interpolate(image, {w, h, image.shape()[2], image.shape()[3]}, filter); + resized = ops::slice(resized, 0, (w - width) / 2, (w - width) / 2 + width); + return ops::slice(resized, 1, (h - height) / 2, (h - height) / 2 + height); + } + static constexpr std::pair image_targets[] = { {"init", ImageTarget::Init}, {"end", ImageTarget::End}, diff --git a/src/runtime/image_preprocess.h b/src/runtime/image_preprocess.h index b75c4e0e5..7dfc45899 100644 --- a/src/runtime/image_preprocess.h +++ b/src/runtime/image_preprocess.h @@ -3,6 +3,7 @@ #include #include +#include #include #include "core/tensor.hpp" @@ -10,6 +11,9 @@ namespace sd { + std::pair select_image_size_bucket(int source_width, int source_height, int area, int factor, int fallback_width, int fallback_height); + Tensor resize_and_center_crop(const Tensor& image, int width, int height, ops::InterpolateMode filter = ops::InterpolateMode::Lanczos); + enum class ImageTarget { Init, End,