AI agent skill · by @Vincentqyw

Integrate Matcher

Integrate a new image matching model into the image-matching-webui project. Invoke when user provides a GitHub repo URL of a feature matching method and asks to add/integrate it.

Archived 2026-10-23 · Source updated 2026-10-05 · License: Apache-2.0. Check the original repository for the latest version.

SKILL.md instructions

# Integrate Matcher into image-matching-webui

This skill automates the integration of a new local feature matching method into the `image-matching-webui` project. Given a GitHub repository URL, it follows the project's established patterns to add the matcher as a fully functional option in the WebUI.

## Prerequisites

- The target repository must be a local feature matching method (sparse or standalone)
- You must be working inside the `image-matching-webui` project root

## Step-by-Step Integration Guide

### Step 1: Analyze the Target Repository

1. **Clone the repo** to `/tmp/<repo-name>` for analysis (do NOT add as submodule yet)
2. **Identify the matcher type**:
   - **Dense/Standalone matcher**: Takes raw images as input, performs detect+describe+match internally (e.g., LoMa, RoMa, LoFTR). `required_inputs = ["image0", "image1"]`
   - **Sparse matcher**: Takes keypoints+descriptors as input, performs matching only (e.g., LightGlue, SuperGlue). `required_inputs` includes `keypoints0`, `descriptors0`, etc.
3. **Find the core model class** and its API:
   - How to initialize the model (constructor args, config options)
   - How to run inference (forward method signature)
   - What the model outputs (keypoints, matches, scores, etc.)
   - Model weight download URLs
4. **Check dependencies** in `pyproject.toml` or `requirements.txt`
5. **Identify model variants** (e.g., different sizes: B/L/G/R)
6. **Check for device/amp issues**:
   - Does the model use `torch.autocast` or mixed precision (`mp`, `amp`)?
   - Does it manage its own device placement (like LoMa's `loma.device`)?
   - On MPS/CPU, does it produce dtype mismatches?

### Step 2: Add Git Submodule

```bash
git submodule add <repo-url> imcui/third_party/<RepoName>
```

- Use the original repo name (PascalCase) as the submodule directory name
- If the repo has a fork in the project's org (e.g., `Vincentqyw/xxx` or `agipro/xxx`), prefer the fork

#### ⚠️ CRITICAL: Never modify third_party code directly

`imcui/third_party/` contains **pinned third-party submodules** — you are NOT the owner of this code. If a dependency needs a compatibility fix (e.g., API changes in PyTorch/kornia, import path changes):

1. **Fork** the original repo to the `agipro` GitHub account
2. **Apply the fix** in the fork and push
3. **Replace** the submodule URL in the main repo to point to the fork:
   ```bash
   git submodule deinit -f imcui/third_party/<RepoName>
   git rm -f imcui/third_party/<RepoName>
   rm -rf .git/modules/imcui/third_party/<RepoName>
   git submodule add https://github.com/agipro/<RepoName>.git imcui/third_party/<RepoName>
   ```
4. **Update `.gitmodules`** — the URL must point to the fork

Example: `EfficientLoFTR` was forked to `agipro/EfficientLoFTR` to fix a `kornia.utils.grid` → `kornia.utils` import change required by kornia 0.8+.

### Step 3: Create Matcher Implementation

Create `imcui/hloc/matchers/<matcher-name>.py` following these rules:

#### 3.1 Import Pattern

```python
import sys
from pathlib import Path

from .. import logger
from ..utils.base_model import BaseModel

# Add third_party to sys.path for import
<matcher>_path = Path(__file__).parent / "../../third_party/<RepoName>/src"
sys.path.append(str(<matcher>_path))

# Import the model class
from <module> import <ModelClass>
```

- If the third_party repo has `src/` as the package root, append `<path>/src`
- Some repos put the package directly at root level — adjust accordingly

#### 3.2 Class Definition

The class MUST:
- Inherit from `BaseModel`
- Be the **only** `BaseModel` subclass in the file (the `dynamic_load` function asserts this)
- Define `default_conf` dict with all configurable parameters
- Define `required_inputs` list

```python
class MatcherName(BaseModel):
    default_conf = {
        "name": "two_view_pipeline",
        "model_name": "<default_variant>",
        "max_keypoints": 2048,
        # ... other config
    }
    required_inputs = [
        "image0",
        "image1",
        # For sparse matchers, also include:
        # "keypoints0", "scores0", "descriptors0",
        # "keypoints1", "scores1", "descriptors1",
    ]
```

#### 3.3 `_init` Method

- Use `self.conf` (not the `conf` parameter) to access merged config
- Log model loading with `logger.info()`
- Handle model weight downloading:
  - If weights are on HuggingFace: use `self._download_model(repo_id=MODEL_REPO_ID, filename=...)`
  - If weights are auto-downloaded by the model (e.g., `torch.hub`), let it handle
  - If weights need manual download, document the URL
- **Platform compatibility**: On non-CUDA devices, disable mixed precision:
  ```python
  if not torch.cuda.is_available():
      # Disable mp/amp for CPU/MPS compatibility
  ```

#### 3.4 `_forward` Method

**For standalone matchers** (input: raw images):

The `data` dict contains preprocessed tensors:
- `data["image0"]`: shape `(1, C, H, W)`, float32, range [0, 1]
- `data["image1"]: shape `(1, C, H, W)`, float32, range [0, 1]

If the model expects PIL images or file paths, convert:
```python
img0 = data["image0"].cpu().numpy().squeeze() * 255
img0 = img0.transpose(1, 2, 0)  # CHW -> HWC
img0 = Image.fromarray(img0.astype("uint8"))
```

**For sparse matchers** (input: keypoints + descriptors):
```python
# Repackage data for the model's expected format
input = {
    "image0": {"image": data["image0"], "keypoints": data["keypoints0"], ...},
    "image1": {"image": data["image1"], "keypoints": data["keypoints1"], ...},
}
return self.net(input)
```

#### 3.5 Output Format

The `_forward` method MUST return a dict with these keys:

**Dense matchers that output matched keypoints only:**
```python
pred = {
    "keypoints0": kpts0,      # torch.Tensor, shape (N, 2), pixel coords in resized image
    "keypoints1": kpts1,      # torch.Tensor, shape (N, 2)
    "mconf": confidence,      # torch.Tensor, shape (N,), match confidence scores
}
```

**Dense matchers that can separate detected vs matched keypoints (PREFERRED):**
```python
pred = {
    "keypoints0": all_kpts0,     # All detected keypoints (for UI "Keypoints" display)
    "keypoints1": all_kpts1,     # All detected keypoints
    "mkeypoints0": matched_kpts0, # Matched keypoints (for UI match lines)
    "mkeypoints1": matched_kpts1, # Matched keypoints
    "mconf": confidence,          # Match confidence scores
}
```

**Sparse matchers** (LightGlue-style):
```python
# Return the model's raw output — match_dense.py handles the rest
return self.net(input)
```

Key output rules:
- `keypoints0/1`: ALL detected keypoints (shown in UI "Open for More: Keypoints")
- `mkeypoints0/1`: Only MATCHED keypoints (shown as match lines in UI)
- If `mkeypoints0/1` is missing, `match_dense.py` falls back to `keypoints0/1`
- Coordinates must be in **pixel space** of the resized image (not normalized [-1,1])
- `mconf` should be real confidence scores (not all-ones)

### Step 4: Add Matcher Configuration

#### 4.1 `imcui/hloc/configs/matchers.py`

Add a configuration entry for each model variant:

```python
"<matcher-name>": {
    "output": "matches-<matcher-name>",
    "model": {
        "name": "<matcher-module-name>",  # Must match the .py filename in matchers/
        "model_name": "<variant>",         # Passed as conf["model_name"]
        "max_keypoints": 2048,
        # ... other model-specific config
    },
    "preprocessing": {
        "grayscale": False,     # True for LoFTR-style; False for most modern matchers
        "force_resize": True,
        "resize_max": 1024,
        "width": 640,
        "height": 480,
        "dfactor": 8,          # Image dimensions must be divisible by this
    },
},
```

Key rules:
- `"name"` in `model` must match the Python filename (e.g., `"loma"` → `loma.py`)
- Each model variant gets its own top-level entry (e.g., `loma-b`, `loma-l`, `loma-g`)
- `preprocessing.grayscale`: Set `True` only for models that expect 1-channel input (e.g., LoFTR)
- `preprocessing.dfactor`: Ensure resized dimensions are divisible by this value

#### 4.2 `config/app.yaml` (local dev) AND `imcui/config/app.yaml` (package default)

BOTH files must be updated identically. Add an entry under `matcher_zoo`:

```yaml
<DisplayName>:
  matcher: <matcher-config-name>    # Must match key in matchers.py
  standalone: true                   # true = takes two images directly (no separate extractor needed)
  skip_ci: false                     # DEFAULT: false. Only set to true for heavy matchers (see rule below)
  info:
    name: <DisplayName>              # Display name in WebUI dropdown
    source: "Venue Year"             # e.g., "ECCV 2026", "ICCV 2023"
    paper: <paper-url>               # arXiv or published paper URL
    github: <github-url>             # Official GitHub repo
    display: true                    # Whether to show in WebUI
    efficiency: medium               # low (heavy), medium, high (fast)
```

Key rules:
- `standalone: true` means the matcher takes two raw images directly (no separate feature extractor needed)
- `standalone: false` means the matcher requires a feature extractor (e.g., SuperPoint+LightGlue)
- For feature+matcher combos, use `<extractor>+<matcher>` format (e.g., `superpoint+lightglue`)
- Set `enable: false` for very heavy models that most users won't use by default
- **CRITICAL**: Both `config/app.yaml` and `imcui/config/app.yaml` must be kept in sync

##### `skip_ci` rule

`skip_ci` defaults to `false` — you can **omit the field entirely** for most matchers. Only set `skip_ci: true` when the model is **too heavy to run in CI** (GitHub Actions CPU-only Ubuntu runner with 7GB RAM):

| Condition | `skip_ci` |
|-----------|-----------|
| Lightweight matcher (e.g., LightGlue, SuperGlue, XFeat, ALIKED-based) | `false` (or omit) |
| Heavy dense matcher prone to CI OOM/timeout (e.g., RoMa, DKM, GIM, Mast3R, LoMa-L/G/R, MINIMA large) | `true` |

Rule of thumb: if the model has variants like B/L/G/R, the "B" (base) variant usually passes CI, larger ones may not. When unsure, **omit `skip_ci`** (defaults to `false`) and let CI tell you — if it OOMs, set it to `true` in a follow-up commit.

### Step 5: Handle Platform Compatibility

Common issues and fixes:

#### 5.1 Mixed Precision (MP/AMP) Issues

On MPS/CPU, `torch.autocast` with float16/bfloat16 causes:
- `RuntimeError: Input type (c10::Half) and bias type (float) should be the same`
- `RuntimeError: Input type (MPSFloatType) and weight type (torch.FloatTensor) should be the same`

Fix pattern (before importing third-party code):
```python
# Patch module-level amp_dtype before import
import <module>.device as _device
if not torch.cuda.is_available():
    _device.amp_dtype = torch.float32

# Also patch submodule local bindings
import <module>.submod as _submod
if not torch.cuda.is_available():
    _submod.amp_dtype = torch.float32
```

After model construction:
```python
if not torch.cuda.is_available():
    cfg = dataclasses.replace(cfg, mp=False)
    for module in self.net.modules():
        if hasattr(module, "amp"):
            module.amp = False
```

#### 5.2 Device Mismatch Issues

If the model manages its own device (like LoMa's `loma.device`):
```python
# Override .to() to keep model on its expected device
def to(self, device=None, **kwargs):
    return super().to(<model_expected_device>, **kwargs)
```

#### 5.3 Inference Mode vs No Grad

If a model uses `@torch.inference_mode()` but you need to pass its outputs to another module that requires grad tracking:
```python
with torch.no_grad():
    output = model.detect_and_describe(...)
    output = output.clone()  # Detach from inference mode graph
```

### Step 6: Update README.md Algorithm Table

The README.md contains a "The tool currently supports..." table listing all algorithms with their support status.

#### 6.1 Table Format

```markdown
| Algorithm        | Supported | Conference/Journal | Year | GitHub Link |
|------------------|-----------|--------------------|------|-------------|
| LoMa             | ✅ | ECCV    | 2026 | [Link](https://github.com/davnords/LoMa) |
| RIPE             | ✅ | ICCV    | 2025 | [Link](https://github.com/fraunhoferhhi/RIPE) |
```

Rows are sorted by **year descending**, then **alphabetically by algorithm name** within the same year.

#### 6.2 Rules

| Scenario | Action |
|----------|--------|
| Algorithm exists with `❌` | Change to `✅` |
| Algorithm not in table | Add new row, maintaining sort order |
| Algorithm already has `✅` | Skip (no change needed) |

#### 6.3 Example

After integrating "DaD" published at ARXIV 2025:
```markdown
| DaD            | ✅ | ARXIV   | 2025 | [Link](https://github.com/Parskatt/dad) |
```
Insert between RIPE (ICCV 2025) and MINIMA (ARXIV 2024).

### Step 7: Pre-Commit Check (MANDATORY)

**Before committing, pre-commit MUST pass.** This is non-negotiable — commits that fail pre-commit will be rejected at the PR stage.

```bash
# Run ALL pre-commit hooks on all files
pre-commit run -a

# Or run specific hooks if you only changed certain files
pre-commit run ruff --all-files       # Python linting
pre-commit run mypy --all-files       # Type checking (if new Python code)
```

Common issues and fixes:

| Hook | Common failure | Fix |
|------|---------------|-----|
| `ruff` | unused imports, line too long | Remove unused imports, wrap long lines |
| `ruff-format` | inconsistent indentation | Let ruff format: `ruff format <file>` |
| `mypy` | missing type annotations | Add type hints to new functions |
| `trailing-whitespace` | blank lines with spaces | Strip trailing whitespace |
| `check-yaml` | invalid YAML syntax | Fix indentation/quoting in app.yaml |
| `end-of-file-fixer` | missing newline at EOF | Add trailing newline |

If `pre-commit run -a` fails, **fix all errors** before proceeding to commit. Do NOT skip hooks with `--no-verify`.

### Step 8: Verification Checklist

After integration and pre-commit pass, verify:

1. **Import test**: `python -c "from imcui.hloc.matchers import <name>"`
2. **Model loading**: The model loads without errors on CPU/MPS/CUDA
3. **Inference test**: Run matching on a test image pair
4. **WebUI test**: `python app.py` → select the matcher → run matching
5. **Keypoints display**: UI "Open for More: Keypoints" shows detected keypoints
6. **Match lines display**: UI shows correct match lines between images
7. **Both config files**: `config/app.yaml` and `imcui/config/app.yaml` are identical for the new matcher
8. **Pre-commit passes**: `pre-commit run -a` exits with zero

### Step 9: Commit and Create PR

Once pre-commit passes and all verification checks are green:

```bash
# Stage the changes
git add .gitmodules imcui/third_party/<RepoName>
git add imcui/hloc/matchers/<name>.py
git add imcui/hloc/configs/matchers.py
git add imcui/config/app.yaml
git add config/app.yaml
git add README.md

# Commit with a descriptive message following convention
git commit -m "feat: integrate <MatcherName>

Add <MatcherName> matcher from <venue> <year>.

- Add submodule: imcui/third_party/<RepoName>
- Add matcher implementation: imcui/hloc/matchers/<name>.py
- Add matcher config entries
- Update app.yaml matcher zoo (both package and user configs)
- Update README.md algorithm table

Co-Authored-By: Claude <noreply@anthropic.com>"

# Push and create PR
git push origin main
gh pr create --title "feat: integrate <MatcherName>" \
  --body "Integrate <MatcherName> matcher from <paper-link>.

## Changes
- [x] Submodule added
- [x] Matcher implementation
- [x] Config entries in both app.yaml files
- [x] README.md updated
- [x] Pre-commit passes

## Tested on
- [ ] CPU
- [ ] CUDA
- [ ] MPS

🤖 Generated with [Claude Code](https://claude.com/claude-code)"
```

**Important notes:**
- Use `feat:` prefix for new matcher integrations (following conventional commits)
- Always include `Co-Authored-By: Claude <noreply@anthropic.com>` in commit messages
- End PR body with `🤖 Generated with [Claude Code](https://claude.com/claude-code)`
- If CI fails after pushing, check the logs — if it's an OOM on your new matcher, set `skip_ci: true` in a follow-up commit

## File Change Summary

For each integration, these files are typically modified:

| File | Action | Description |
|------|--------|-------------|
| `.gitmodules` | Modify | Add submodule entry |
| `imcui/third_party/<RepoName>` | Add | Git submodule |
| `imcui/hloc/matchers/<name>.py` | Create | Matcher implementation |
| `imcui/hloc/configs/matchers.py` | Modify | Add matcher config entries |
| `config/app.yaml` | Modify | Add WebUI display config |
| `imcui/config/app.yaml` | Modify | Add WebUI display config (must match config/app.yaml) |
| `README.md` | Modify | Update supported algorithms table (add row or flip ❌→✅) |

Before committing all changes, run `pre-commit run -a` — all hooks must pass.

## Reference Implementations

| Pattern | File | Description |
|---------|------|-------------|
| Dense standalone | `imcui/hloc/matchers/roma.py` | RoMa — raw images → matched keypoints |
| Dense standalone + separate detected/matched kpts | `imcui/hloc/matchers/loma.py` | LoMa — separates all detected from matched keypoints |
| Sparse matcher | `imcui/hloc/matchers/lightglue.py` | LightGlue — keypoints+descriptors → matches |
| Detector-only + external descriptor | `imcui/hloc/extractors/raco.py` | RaCo — detects keypoints, delegates descriptor to ALIKED |

### RaCo pattern: detector that needs a descriptor extractor

Some models are **keypoint detectors only** — they output keypoints/scores but no descriptors. The RaCo integration demonstrates chaining RaCo detection with ALIKED description:

1. Create an **extractor** (not matcher) in `imcui/hloc/extractors/raco.py`
2. In `_forward`, first run RaCo detection → keypoints, then run ALIKED descriptor on those keypoints
3. Register in `configs/extractors.py` with `max_num_keypoints` / `nms_radius` parameters
4. Register a matcher config in `configs/matchers.py` with `features: "raco-aliked"` — a custom LightGlue+ checkpoint trained for RaCo+ALIKED features
5. In `app.yaml`, the entry uses `feature: raco` (the extractor) + `matcher: raco-lightglue` (the LightGlue variant)

### Fork priority for third-party fixes

When a submodule needs a compatibility fix:

| Repo owner | Action |
|------------|--------|
| `Vincentqyw/*` | Push fix directly to the Vincentqyw fork |
| `agipro/*` | Push fix directly to the agipro fork |
| Anyone else | Fork to `agipro`, apply fix, switch submodule URL |

Always verify push succeeded with `git log --oneline -1` in the submodule directory.

### Kornia compatibility (kornia >= 0.8)

Kornia 0.8.0 removed the `kornia.utils.grid` submodule. `create_meshgrid` moved to `kornia.utils` (0.8.0-0.8.2), then to `kornia.geometry` (0.8.3+). When fixing submodule imports, use this future-proof pattern:

```python
try:
    from kornia.geometry import create_meshgrid  # kornia >= 0.8.3
except ImportError:
    from kornia.utils import create_meshgrid  # kornia < 0.8.3
```