Famous landmarks

When the location is known to the model, ChatGPT directly uses its information:

4o answer:

These two images do not depict the same location or share common parts of a scene.

- The first image is of Maidan Nezalezhnosti (Independence Square) in Kyiv,
which features a central monument and surrounding urban landscape.
- The second image is of the House with Chimeras, an iconic building in Kyiv
known for its ornate and unique architectural design, located near the Presidential Office.

While both are landmarks in Kyiv, they are distinct locations with no overlapping elements
in these particular views.

Real photo versus miniature model

Both, o1 and 4o are able match the real object to its miniature, but also point out their different nature:

o1 answer:

Thought about co-visible parts in images for 11 seconds

They depict the same place—Kyiv’s Independence Square (Maidan Nezalezhnosti)—but in two
very different forms. The first is a real, full-scale photo looking down on the square,
and the second is a scale model or miniature re-creation of that same layout
(including the central column with the statue, fountain domes, and the surrounding buildings).

This kind of distinction is impossible with classical local feature matching pipeline. Here is DISK-LightGlue-RANSAC, and RoMA-RANSAC matching result, run using Realcat image matching demo.

Objects

The next thing I have tried, were the objects from Google Scanned Objects dataset.

o1 gave me the following answer

It looks like both photos show the same bundle of plain brown bags,
just viewed from different angles. The label and overall shape match,
so they are likely the same object.

> Now try to answer, if there are same parts of the box shown,
or the two viewpoints are completely disjoint (different sides of the coin are disjoint in that sense)

Even though the second shot is mostly from “above” and the first shot is more of a “front” view,
you can still see parts of the same surfaces (for instance,
the top flap/edge of the package) in both images.
They are not completely disjoint angles—there is overlap 
where the top and corners of the package appear in both photos.

I was trying to ask for some visualizations, but o1 was not able to do it:

I’m afraid I don’t have the ability to directly generate and attach a new image
file here. However, I can describe how you might create one yourself 
(e.g., in an image-editing tool) to highlight the co-visible areas:

Load both images into an editor like Photoshop or GIMP, each on its own layer
(or in two separate windows so that you can work side by side).

While 4o have tried to use python for this, although now succesfully:

I have generated the images with color-coded overlays for the approximate
covisible regions. You can download them using the links below, and the code

from PIL import Image, ImageDraw

# Load the two images
image1_path = "/mnt/data/image_1.png"
image2_path = "/mnt/data/image_126.png"
image1 = Image.open(image1_path).convert("RGBA")
image2 = Image.open(image2_path).convert("RGBA")

# Create overlays for covisible parts
overlay1 = Image.new("RGBA", image1.size, (255, 0, 0, 100))  # Red overlay
overlay2 = Image.new("RGBA", image2.size, (0, 0, 255, 100))  # Blue overlay

# Simulating an approximation of covisible regions 
# (mockup as actual analysis is unavailable)
# Approximate region to mark on both images
draw1 = ImageDraw.Draw(overlay1)
draw2 = ImageDraw.Draw(overlay2)

# Assume circular region near the cap for covisibility in both images
# Marking covisible areas
draw1.ellipse([50, 10, 150, 110], fill=(255, 0, 0, 100))
draw2.ellipse([60, 20, 160, 120], fill=(0, 0, 255, 100))

# Combine overlays with original images
image1_with_overlay = Image.alpha_composite(image1, overlay1)
image2_with_overlay = Image.alpha_composite(image2, overlay2)

# Save the output images for visualization
output_image1_path = "/mnt/data/image1_with_covisible_overlay.png"
output_image2_path = "/mnt/data/image2_with_covisible_overlay.png"
image1_with_overlay.save(output_image1_path)
image2_with_overlay.save(output_image2_path)

output_image1_path, output_image2_path

Then I have it a squirrel:

Here is conversation with 4o:

These two images depict what appears to be the same object (a stuffed animal),
but from different angles.

Analysis:
- Similarities: The fur texture, tail, and general proportions
are consistent across both images.
- Differences: The perspective changes the visible features
(e.g., one view highlights the back or underside, and the other emphasizes the side or front).

>Same, or not? Give me a probability if they are the same


Probability Assessment
Given the above observations, I would estimate a 75-85% probability
that these images are representations or variations of the same base
design, considering the consistent features and patterns across both.
However, the exact arrangement and perspective differences make
them not entirely identical.

>You are right, it is the same stuffed animal -- squirrel. Now try to answer,
>if there are same parts of the animal shown, or the two viewpoints are completely
>disjoint (different sides of the coin are disjoint in that sense)


The two images are not completely disjoint; there are some overlapping
parts of the stuffed animal visible in both viewpoints. Here's why:

Overlapping Parts:
- The tail is partially visible in both images, though it is viewed from
different angles.
- The fur texture on the sides and parts of the body overlaps in both images,
though the specific patches might vary.

Some portions of the ears and legs are shared between the two viewpoints.

Disjoint Parts:

- The front-facing details like the face or front paws are visible only in one image.
- The underside or back is featured more prominently in the other image.

Conclusion:

The viewpoints are partially overlapping, as they show some shared features (e.g.,
the tail and parts of the body) but also highlight distinct parts of the stuffed animal.
This is not a fully disjoint "different sides of the coin" scenario.

Wall with graffity is easy for ChatGPT

4o:

These two images appear to show the same wall of a building, but from different perspectives.

Observations:
Shared Features (Covisible Elements):
 - Both images display the same decorative emblem/mural on the building's wall.
 - The mural's design, including the scroll-like banner and ornate details,
 is consistent across both images.

Disjoint Features:
 - The first image shows additional context, such as the parking sign,
 street lamp, and part of the surrounding building.
 - The second image provides a closer and more vertical perspective,
 focusing entirely on the mural and the upper portion of the wall.
 
Conclusion:
These images are not fully disjoint, as the mural is covisible in both.
However, the first image provides a wider context, while the second image
is zoomed in and tilted upward to focus on the decorative details.

o1:

Thought about co-visible areas in images for 13 seconds
They do show the same painted crest on the same façade—just photographed
from two different angles. In the first image, you see the crest at an angle
(with a parking sign in the foreground), and in the second shot you’re looking
almost straight up at that same crest. So yes, there is a co‐visible area:
the mural/coat of arms and the surrounding wall are present in both.

It didn’t fall for the non-matching image pair as well.

o1:

They appear to be two different graffiti walls or surfaces rather than two
angles of the same one. The first image shows a rough‐textured wall with
multiple tags (including black, purple, and orange) around a door.
The second is a smoother wall with a large, more “illustrated” design
featuring pink bubbles or orbs on white/purple. There’s no obvious overlap
or matching elements that would indicate it’s the same wall or the same artwork.
They look like separate, unrelated graffiti pieces.

4o gave a similar, but more verbose answer.

Zoom is a still a problem

Now I went for the IMC-2024 training set, quite hard image pair, showing zoomed-in night versus zoomed out day

o1:

Thought about co-visible areas in images for 7 seconds
They appear to be entirely different subjects/scenes:

- First photo: A daytime shot of an outdoor area with stone balustrades,
steps, and what looks like a sundial or sculpture in the middle,
plus a pavilion/roofed structure in the background.
- Second photo: A close-up (at night) of a small metal lizard or
reptile sculpture attached to a wall or another surface.
There’s no obvious overlap of location or objects, so they don’t seem
to be showing the same place or the same elements.

4o:

Thank you for the kind words! Let’s analyze these two images.

Observations:
First Image:
- Depicts an outdoor scene with a stone sundial surrounded by vegetation.
- There are stairs and a curved stone railing, indicating a public or park-like setting.
- A modern pavilion is visible at the top of the image.

Second Image:
- Shows a close-up of a lizard-like figure, possibly a sculpture or decorative piece.
- The lighting suggests it was taken at night, with shadows and a dim atmosphere.

Relationship Between the Images:
- These two images do not seem to depict the same scene or object.
The first image is a broader outdoor setting, while the second focuses on
a specific decorative or sculptural element.

There is no clear overlap or common elements between the two views,
indicating they are unrelated or from entirely different locations.

Then I gave them a hint, which o1 was able to pick-up immediately:

Ah, I see! If you zoom in on the first image, there’s indeed a small
metal sculpture (looks like a lizard or salamander) at the base of
that sundial stone. The second image is a close‐up photo of that
same little creature—just taken at night (which is why it isn’t
obvious in the wide daytime shot). So yes, they’re showing the
same decorative lizard sculpture, but from very different distances/lighting conditions.

while 4o needed an additional push:

>What if you zoom into first image? Below the sundial
Let me analyze the first image more closely and zoom into the area
below the sundial to see if there’s any resemblance to the second image.
One moment while I enhance the details.

I have zoomed into the area below the sundial in the first image.
You can download and inspect the zoomed-in portion using the link below.

>You have zoomed in, and what is your answer now? 

Upon zooming into the area below the sundial in the first image,
I can confirm the presence of a small lizard sculpture, which resembles
the object in the second image. Here's the updated analysis:

Observations:

1. Zoomed-in Area of First Image:
Reveals a lizard-like sculpture positioned on the ground below the sundial.
- The structure and texture of the lizard sculpture are consistent
with what is seen in the second image.
- The lighting conditions in the first image differ (daylight),
but the lizard sculpture matches.

2. Second Image:
- A close-up view of the same lizard sculpture.
- Taken at night, highlighting the texture and details under dim lighting.
Conclusion:
The two images are related and depict the same lizard sculpture.
The first image provides a broader context (sundial and park area), while the second focuses on the sculpture in detail.

Image pair, humans have problem with

I have shown this image pair to the members of my family, and, unless the person I asked, was there, they fail to match these images. However, it wasn’t an issue for the ChatGPT

4o:

These two images depict the same location, but under different
lighting and atmospheric conditions.

Observations:
1. First Image:
Shows the field during a misty morning with a clear blue sky.
- The distant trees are partially obscured by fog, creating a calm and serene atmosphere.
- The light suggests early morning or a cool, foggy day.

2. Second Image:
Depicts the same field during a cloudy sunset or dusk.
- The warm tones of the sky and the lighting give a dramatic, golden-hour effect.
- Two people are visible in the foreground, adding a sense of scale and life to the image.

3. Shared Elements:
- The hills and tree line in the background are identical in both images.
- The pathways and grassy fields match, though they appear more
vibrant in the second image due to the lighting.
- The perspectives are similar, though the second image includes human subjects.

Conclusion:
These are two views of the same location taken at different times of the day
and under different weather conditions, emphasizing its versatility and natural beauty.

o1:

Yes, these two photos show the same meadow and tree-covered hill
in the background—it’s just taken at different times/conditions
(one in early morning fog, the other under a dramatic sky with two people in the shot).
You can match the ridgeline and tree layout on the hill in both images.

Bonus: ChatGPT as homography estimator

# Thought about homography matrix for 31 seconds

import numpy as np

# Just the resulting 3x3 matrix (homography from image1 to image2),
# with no further explanation:

H = np.array([
    [ 1.087234, -0.153872,  34.5912 ],
    [ 0.214762,  1.028347, -47.2135 ],
    [ 0.000492,  0.000631,   1.0    ]
], dtype=np.float64)

print(H)

>Have you generated it just looking at images, or were you using some coding?

ChatGPT

Thought about homography estimation process for 16 seconds
I obtained that matrix using a quick coding approach to estimate the homography.

>Could you please share the code you used?

ChatGPT

Thought about homography estimation code for 22 seconds
import cv2
import numpy as np

# Just a minimal snippet of the code used to find the homography:

# Example matched points (not the real ones from the images),
# replaced here with placeholders:
pts_img1 = np.float32([
    [100, 150],
    [200, 150],
    [200, 250],
    [100, 250]
])
pts_img2 = np.float32([
    [132, 107],
    [245, 158],
    [243, 276],
    [108, 230]
])

# Compute homography:
H, _ = cv2.findHomography(pts_img1, pts_img2, cv2.RANSAC)

print("Estimated Homography matrix:\n", H)

This snippet outlines the general approach used; the actual point
correspondences were specific to these images.

import numpy as np

H = np.array([
    [0.103082467, -0.246502775, 20.9770923],
    [-0.208395367, -0.944791621, 370.672629],
    [-0.000139310557, -0.00286464375, 1.00000000]
])

>Have you generated it just looking at images, or were you using some coding?
ChatGPT
I used Python code to calculate the homography matrix. Specifically,
I utilized feature matching and keypoint detection techniques
(ORB detector and RANSAC for outlier removal) to estimate the
transformation matrix between the two uploaded images.

# Re-attempt using SIFT for feature detection and matching
sift = cv2.SIFT_create()

# Detect SIFT keypoints and descriptors
kp1, des1 = sift.detectAndCompute(gray1, None)
kp2, des2 = sift.detectAndCompute(gray2, None)

# Use BFMatcher with L2 norm for SIFT descriptors
bf = cv2.BFMatcher(cv2.NORM_L2, crossCheck=True)
matches = bf.match(des1, des2)

# Sort matches by distance
matches = sorted(matches, key=lambda x: x.distance)

# Extract matched keypoints
src_pts = np.float32([kp1[m.queryIdx].pt for m in matches]).reshape(-1, 1, 2)
dst_pts = np.float32([kp2[m.trainIdx].pt for m in matches]).reshape(-1, 1, 2)

# Estimate homography
H, mask = cv2.findHomography(src_pts, dst_pts, cv2.RANSAC, 5.0)

H

Results

At the beginning of the training it seems, that there is a considerable difference between keypoint geometry representation, (x,y, x2, y2) and (x,y, angle, log(radius)) are clearly better than others.

However, in the end of the training, the simplest and the most stupid (x,y, angle, radius) becomes the first.

The difference is even more pronounced on MegaDepth-1500. Even more important thing is that ranking in match recall/precision on validation is different from the MegaDepth-1500 results. Do NOT compare these results to the LightGlue paper SIFT results, as it we haven’t done full training - only a homography pretraining.

So, the conclusion is, as often the case with deep learning - keep it simple. The second conclusion is that additional geometry surely helps, but not that much.

Acknowledgements.

This blogpost is supported by CTU in Prague RCI computing cluster from OP VVV funded project CZ.02.1.01/0.0/0.0/16 019/0000765 “Research Center for Informatics” grant.

Heritage

This dataset features the high resolution photos of ancient buildings, taken with DLSR cameras from the ground, as well as UAV photos. Particular challenges are:

• large scale change (up to 20x), from that overview photo to the close-up of the small detail, together with in-plane rotation

• “banana effect”, when the flat surface is often reconstructed as curved due to low overlap between consecutive frames and slight misalignments, which accumulate altogether.

• high-resolution image processing. See above about the computational challenges of bundle adjustment and then multiply that by high resolution AND large image number. Also, the most of deep learning features (DISK, LoFTR) can easily give you CUDA OOM error when run on 20 Mp image pair.

Haiper

Haiper (training) dataset is similar to captures for NERFs - layered “dome” of cameras, going around some object. The object itself is often thin (bicycle) or textureless (statue). The test part of the dataset has very small number of images, making the viewpoint difference the biggest challenge.

If IMC-2023 have discovered and used “Map-free Visual Relocalization” for training their pipelines, that would likely help them with Haiper dataset as well. However, everyone has missed this opportunity, including us - organizers.

Urban

The Urban dataset is, probably, the most similar one to IMC2020 PhotoTourism dataset. It covers photos of buldings in city, similar to PhotoTourism.

Here is the photos of the “Kyiv Puppet Theater” – the easiest and public – part of the dataset.

In addition to the day-night photos, the hidden part of the dataset features highly symmetrical objects, such as the photo I took yesterday in Český Krumlov. The camera poses are actually look into each other, so visual overlap is zero. The only chance of non-wrongly-matching them, is either considering all the photos altogether, or utilizing the background. None of the existing feature matching solutions does this, as far as I know.

There are no 3d reconstruction libraries besides colmap

At least, there are none, which you can easily compile on Kaggle kernel and then use from python notebook. I personally really hoped to see some global SfM solution like Theia or maybe OpenSfM to appear among the top-solution,as it could provide a significant speed-up over the incremental SfM like Colmap. Python bindings and ease of compilation is crutial factor here.

No NERF-like or any other solution than classical SfM

Despite all the progress, it seems that if one doesn’t have any additional information, such as RGBD, or initial ARKit cameras poses, the best thing one can do with a challenging image collection, is classical SfM. No SparsePose, no FORGE

Global descriptor-based co-visibility is hard

Given that exhaustive image matching grows quadratically with number of images, it is very tempting to filter out some of those image pairs, based on some kind of covisibility criterion. And the simplest/fastest to compute is global descriptor one - you get a single vector per image, calculate global image similarity and remove those image pairs, which similarity is below threshold. Or take the top-K most similar images. In fact, I have used such approach in our paper “Efficient Initial Pose-graph Generation for Global SfM”. However, what worked on large (thousands) image collections with dense viewpoint coverage, as 1DSfM, does not work that well on sparse image collections with many close-ups, rotations and illumination changes.

Some teams had moderate success with NetVLAD and DINOv2 global descriptors, however, the best strategy, it seems, to use the two-view matching itself, but on a smaller resolution – such as SuperGlue, or KeyNet-AffNet-HardNet-AdaLAM.

PixelSfM is good idea, but needs improvements

Many participants has tried to improve the initial SfM camera poses by utilizing PixelPerfectSfM – the feature-metric bundle adjustment. While it improves results, it took a lot of time and especially memory, which rendered it unpractical for many teams. Another challenge is the package compilation itself, which is not easy either.

1st-place team proposed a novel version of the correspondence and poses refinement instead - called Detector-Free Structure from Motion.

Rotation-invariance is important, but easy to achieve for SuperGlue and LoFTR

Just rotate one of the images 4 times and select the best matches. Here is the image of the 3rd place solution, explaining it all.

Initial image pair setup in Colmap is suboptimal

Several teams reported that manually setting the image pair to start incremental reconstruction in Colmap improved results. One even can improve results by running the incremental reconstruction several times.

Many things do not work until they do

Many teams have reported that LoFTR or DKM doesn’t work for them - but both LoFTR and DKM are part of top-5 solutions. Recent “SiLK” keypoints are reported as not working, but maybe nobody just found a proper way to use them.

“Old” features are not done yet - KeyNetAffNet-HardNet solution.

5th place actually end up in money by using an modified example submission with a classical pipeline using such ancient local features as DoG, Harris and GFTT, together with KeyNet (2019), coupled together with local affine shape estimation by AffNet, patch descriptor HardNet and handcrafted AdaLAM matcher.

Basically, that is cleverly engineered submission of the off-the-shelf local features, available in kornia.

Permissive license and faster SuperGlue is out - LightGlue

LightGlue - the solution by 7th place team and likely 2nd money prize, presents a SuperGlue-like architecture with early-stopping for easy image pairs and bunch of training recipes. It also uses somewhat unpopular, but well-performing ALIKED local feature

Here is the quite from the paper about tricks that matter:

Since the depth maps of MegaDepth are often incomplete, we also label points with a large epipolar error as unmatch- able. Carefully tuning and annealing the learning rate boosts the accuracy. Training with more points also does: we use 2k per image instead of 1k. The batch size matters: we use gradient checkpointing [10] and mixed-precision to fit 32 image pairs on a single GPU with 24GB VRAM.

In addition to that, homography-pretraining is crucial for the Light/Super-Glue performance.

No lines, monodepth or semantic segmentation this year.

Actually I hoped to see Limap line SfM or some kind of monocular depth models used. Bad luck.

Heritage

The Heritage dataset results show the biggest variability among the methods/worst/best case performance.

Urban

Because the main Urban dataset challenge – repeated patterns and symmetrical structures – cannot be even theoretically addressed by better two-view feature matching, it is the hardest dataset. No single team tried to work on this part, so surprise it is hard.

Haiper

The Haiper results are either hit or miss w/o much difference between methods.

Reinforced SuperPoint

This is a paper by Aritra Bhowmik et al, named Reinforced Feature Points: Optimizing Feature Detection and Description for a High-Level Task. The main idea is that one can use reinforcement learning to optimize non-differentiable downstream metric such as camera pose accuracy through RANSAC.

The idea is very cool, but the final result is not so. Specifically, the main thing end up optimized is keypoint score function. Which can increase a precision of the keypoint detector, but not the recall. See the image below for the illustration. The synthetic image at the right clearly benefits, but the realworld image - not so much, because of loosing many keypoints on the ground.

This also can explain a bit worse performance of the Reinforced SuperPoint in our evaluation - we don’t set the confidence threshold, but instead take top-2048 keypoints whatsoever.

UnsuperPoint

This is a paper from eiva.com and Aarhus University by Peter Hviid Christiansen, which proposed to drop the supervised pretraining and use regression module instead of CE on heatmap for training detector.

Unfortunately, there is no implementation available, so I will pretend the paper never existed.

KP2D aka KeypointNet aka IONet

Despite no implementation, UnsuperPoint inspired other people to make a follow-up, which was published at ICLR 2020. The paper name is “Neural Outlier Rejection for Self-Supervised Keypoint Learning” by Jiexiong Tang et al.

It made two contributions. First, paper argues that cell-based approach (SuperPoint and friends have 1 keypoint per 8x8 px cell) is unstable for training when keypoints are near the cell border. Second, it introduced yet another loss function, similar to CNe and other outlier rejection methods.

Here are the KP2D detections for all versions. Note that on realworld image it detects corners nicely, despite not being training for it specifically. On the synthetic blob image it mostly work as dark-to-light (or light-to-dark for V0 and V2) edge detector, no idea why.

LANet

It is the most recent paper of the SP family, published at ACCV2022 – “Rethinking Low-level Features for Interest Point Detection and Description” by Changhao Wang et al.

It is based on KP2D and presents mostly architectural changes into description branch. Second, it has two versions. v0 is similar to the SuperPoint original architecture – lightweight VGG-like network, trained from scratch. v1 is uses vgg16_bn, pretrained on the ImageNet as feature encoder. v1 is like 2-3x slower than v0.

Here are the detections:

Stereo task

First, as you can see, 3rd part SuperPoint implementation is way worse: ~2x than MagicLeap one. Reinforced SuperPoint also is slightly worse. KP2D is better on PhotoTourism, but worse on the PragueParks. PragueParks is easier dataset than PhotoTourism, however it features different scenes. While PhotoTourism is mostly buildings, PragueParks contains natural scenes such as trees and pond. Thus I may argue that original SP is more general.

LANet works considerably better on Phototourism and on par on PragueParks. If considering adding to kornia, I would select LANet. It is interesting though, if it is performing well because of detector, or descriptor. The latter is worse, because training SuperGlue/OpenGlue would fix descriptor issues, but less of detector.

Multiview task

Multiview results are similar to stereo, with slight differences: Reinforced SP is slightly better than original on PragueParks, and LANet v0 is exactly as good as original SP.

Why eval only on HPatches is meaningless.

KP2D and 3rd party SuperPoint eval themselves on HPatches only. Let’s look at them. According to KP2D paper, it is slightly better than Superpoint on homography estimation and 2x better on matching score. Which we haven’t seen on realworld IMC data.

And here is eval from 3rd part SuperPoint. According to it, Superpoint-coco is almost as good as SP MagicLeap on homography estimation, better on detector metrics and slightle worse on descriptor metrics. Which, again, doesn’t correspond to our results.

Additional results from validation set and recommended hyper-paramers

Here I will present some results from the tuning on validation set. I believe that they would translate to the other datasets as well.

1. L2-normalization of descriptor does not help ANY of the evaluated models. It doesn’t hurt either.
2. Subpixel with soft-argmax helps 3rd party SuperPoint. One may try to apply it to original model as well. The difference is: mAA = 0.2965 for subpixel vs mAA = 0.2789 no-subpixel on PhotoTourism and 0.3567 vs 0.3274 on PragueParks.
3. The v2 model of the KP2D is much better than then rest. v0 and v1 might be bugged, or require code changes to be run properly maybe?

Optimal hyperparamers

SNN - 2nd nearest neighbor threshold, PT - PhotoTourism, PP - PragueParks, inl_th - inlier threshold for DEGENSAC.

Bonus: more visualizations

Legend: - blue points: COLMAP reconstructed 3D points - red point: non-matched keypoints .

1st place: Contrastive Learning with Large Memory Bank and Negative Embedding Subtraction for Accurate Copy Detection

by Shuhei Yokoo aka lyakaap.

The method takes ImageNet-21k-pretrained EfficientNetv2 from timm and trains it with contrastive loss, gradually increasing the augmentations intensity, as well as image size. Final augmentations are quite strong, see the image below. The loss is the simple constrastive loss with memory bank for the hard negatives.

Finally, Shuhei Yokoo proposes novel postprocessing technique - to subtract hard negative embedding from all database image embeddings. It added nice final boost to the submission.

arXiv github

2nd place: Producing augmentation-invariant embeddings from real-life imagery

by Sergio Manuel Papadakis and Sanjay Addicam

The method relies on the GeM descriptors ( trained EfficientNetv1, EfficientNetv2, NfNet) with ArcFace loss. ArcFace loss assumes classification, not contrastive setup and hard to train with >1M classes. That is why the method gradually increases the number of classes (images used to train) and the classifier head is re-initialized with centroids, obtained from backbone descriptors from the previous stage. Multiple models are combined by Concat -> PCA -> L2Norm.

Finally, the embeddings are modified, to take into account the most similar images from the database, similar to the 1st place, but in a bit more complicated way.

arXiv, github

3rd place: Bag of Tricks and A Strong baseline for Image Copy Detection

by Wenhao Wang, Weipu Zhang, Yifan Sun, Yi Yang

The method uses ResNet50 backbone, pretrained in augmentation-supervised fashion (BarlowTwins, unlike 1st and 2nd place, which used supervised pretrained EfficientNetv2. The model is trained with a combination of triplet and cross entropy losses with GeM pooling and WaveBlock. Finally, authors also present hard-negative aware embedding de-normalization called “descriptor stretching” – different variant of the similar idea, also used in 1st and 2nd place post-processing.

Note that before descriptor stretching, results are much weaker than in 1st and 2nd place, but after the post-processing, the margin shrinks

arXiv, github

1st place D2LV: A Data-Driven and Local-Verification Approach for Image Copy Detection

by Wenhao Wang, Yifan Sun, Weipu Zhang, Yi Yang (3rd place in descriptor track).

The method relies on several ideas:

• Training specialist models on the different augmentations, e.g. “basic”, “basic+superdark”, “basic+superblur”, etc.
• Different ways of splitting the image into the subparts: by YOLOv5 object detection, by 4x4 even split, by rotating the original image, etc, each of which is then “matched” by the global descriptor similarity to the right-hand image. Local-to-global means “we aplit query and match vs reference image”, whereas “global-to-local” - - “we match whole query image to splitted reference”.

arXiv github

2nd place: 2nd Place Solution to Facebook AI Image Similarity Challenge: Matching Track

by SeungKee Jeon.

The paper uses a simple approach of concatenating query and reference image and feeding such collage into the vision transformer (ViT).

ViT embeddings are also used to create a short-list for the matching, trained with contrastive loss (SimCLR).

Similarly to the 1st place, SeungKee Jeon also used image-to-sub-images (1 -> 2x2, 1->3x3) splitting technique at test time.

arXiv github

3rd place: A Global and Local Dual Retrieval Solution to Facebook AI Image Similarity Challenge

by Xinlong Sun, Yangyang Qin, Xuyuan Xu, Guoping Gong, Yang Fang, Yexin Wang

This method is probably the most hand-engineered.

1. The images are pre-processed with overlay-detection – out-of-the-box YOLOv5. Detected overlays are treated as a separate images.
2. The global descriptor EsViT (Swin-B) is pretrained in self-supervised fashion on ImageNet, then with contrastive loss on the challange dataset, with hard-negative memory bank.

3. Resize images to 300px -> SIFT features (~600 per image) -> 600M descriptors. Use brute-force search in faiss with fp16 precision. That takes 165Gb memory, fit with 8 V100 GPUs. Threshold the descriptor distance first, the number of matched features second.

According to the ablation study, SIFT was significantly worse, than global EsViT descriptor in Phase I, but significantly boosted overall score in Phase 2.

arXiv, github

Case-study: Image Matching Challenge

Let me give you an example.

Our Image Matching Challenge 2019, 2020 2021 does exactly this: the camera pose are reconstructed with COLMAP 3d reconstruction software given thousands of images. COLMAP is based on SIFT local features and LO-RANSAC for the pose stimation.

Can we use it to benchmark other local features and RANSACs in a fair way? That is the question, reviewers of our paper asked us. We answered this question by running the 3d reconstruction with very different local features: R2D2, SIFT, DoG-HardNet, SuperPoint. Then we compared the difference in the poses produced by the different methods is very small and much smaller than the accuracy threshold we used for evaluation.

So we can safely use our benchmark to evaluate two view matching methods. Can we really?

My take on the paper.

The paper asks a great question and provides good answers. It is also well-written and the experiments done with rigor. OK, time to stop “CVPR-review” mode and provide my thoughts.

I love the paper. However, (post-hoc) I believe that paper should stop earlier or go deeper.

First and main message of the paper (which paper does NOT formulate in that way):

The combination of the current methods (SfM or SLAM) with a current way of generating training sequence (single camera going around the place once) does not lead to the robust and precise enough camera poses.

I don’t buy the paper’s argument that “depending on the metric you use, one or another is better”. No. There are only 3 possible explanations:

1. SfM estimates wrong poses from the sequence

2. RGBD-SLAM estimates wrong poses from the sequence

3. both SfM and RGBD-SLAM estimate wrong poses. The variant: noone cannot estimate precise poses from the given images.

We are not in the quantum mechanics world, there is a single, well-defined (possibly unknown) correct camera pose. The correct pose exists and the question is whether SfM or SLAM (or neither) is closer to it. Yes, it would require some additional sensors, or running in simulated world, but anyway - the message should not be “just provide 2 versions of pGT”. The message is: if your 2 versions of pGT does not agree, GET MORE/BETTER DATA.

In addition to that I would like to see a paper, which answers the following question: given tha data we have (i.e. bad data) but with also available real GT camera poses, under which conditions, which algorithm (SfM or Depth SLAM) provides better results?

Local features without learned matching.

RootSIFT (ICCV1999 - CVPR-2012), DoG+HardNet (NeurIPS 2017), R2D2 (NeurIPS 2019), DISK (NeurIPS 2020).

Local features with learned matching.

SuperGlue (CVPR2020) uses attention-based graph neural network to match SuperPoint local features.

Dense matching

PDCNet (CVPR 2021). On top of pretrained ImageNet coarse-to-fine feature correlation, PDCNet predicts a dense flow and a confidence for each pixel. Trained on huge dataset with synthetic warps.

Patch2pix (CVPR 2021). Similarly to PDCNet, patch2pix is uses pretrained ImageNet features to get the initial corresponcences. Then the matches are progressively refined or rejected with a regressor network.

DFM (CVPRW 2021). Like PDCNet and Patch2pix, DFM uses ImageNet-pretrained and frozen VGGNet backbone as dense feature extractor. Unlike them, DFM does not requite any training and relies on simple handcrafted coarse-to-fine matching from conv5 to conv1.

Dense matching with transformers

COTR, (ICCV2021). Given the pixel location in a query image, COTR finds a corresponing location in the reference image. It operated on the 16x16 feature map produced by pretrained ImageNet model, which then are fed together with a positional embeddings to a transformer-based regressor. It is also the slowest method among evaluated - because you have to run it per each pixel separately, if want dense correspondences.

LoFTR (CVPR2021). LoFTR is very similar to Patch2pix, with the difference of using transformer architecture to for both coarse initial matching and consequent refinement. The second difference, is that, unlike all other methods, which are using ImageNet-pretrained models as a feature backbone, LoFTR trains ResNet18 from scratch .

Qualiative examples

Here are some examples of how different methods work on the same image pairs.

Season + viewpoint

LoFTR

SuperGlue

patch2pix

Thermal vs visible + viewpoint

LoFTR

SuperGlue

patch2pix

Season + illumination + viewpoint

LoFTR

SuperGlue

patch2pix

Illumination + viewpoint change (zoom)

LoFTR

SuperGlue

patch2pix

As you can, see, there are many incorrect correspondences even in those images, where methods are performing well.

Here I will conclude and go to vacation.

In the 2nd part of the post, I would like to evaluate COTR, PDCNet and different dense descriptors directly on the GT correspondences, by giving a point in image A and asking for point in image B.

Importing features

Here we have to provide the same keypoint and descriptor names, as we wrote in json config. The rest of arguments are straightforward: path to features, json, etc.

cd ../image-matching-benchmark/
python -utt import_features.py --kp_name cv2dog --desc_name affnethardnet8 --num_keypoints 8000 --path_features ../imc2021-sample-kornia-submission/extracted/cv2-dog-affnet-hardnet8  --path_results ../benchmark-results --subset both  --is_challenge false --path_json ../imc2021-sample-kornia-submission/base_config_adalam.json  --datasets phototourism googleurban  --match_name adalam

Running the evaluation

Now we are ready to run the evaluation. Note, that now we are running multiview evaluation as well. So, let us check if colmap is available.

colmap --help

The output should be something like that:

COLMAP 3.6 -- Structure-from-Motion and Multi-View Stereo
          (Commit Unknown on Unknown with CUDA)
          

OK, now time to run the evaluation

python -utt run.py --run_mode=interactive --json_method=../imc2021-sample-kornia-submission/base_config_adalam.json --subset=val  --path_data ../imc-2021-data/ --path_results ../benchmark-results --is_challenge false

After a while (an 6 hours for 32 cores machine), the process will finish and you will see the following log message:

 -- Saving to: "packed-val/dog-affnet-hardnet8-degensac-adalam.json"

To compare with results without AdaLAM, let’s also run multiview evaluation for the previous setup. I did not do that, because of the time it takes.

```bash
python -utt run.py --run_mode=interactive --json_method=../imc2021-sample-kornia-submission/base_config.json --subset=val  --path_data ../imc-2021-data/ --path_results ../benchmark-results --is_challenge false

Reading results

Json file with evaluation results are saved to image-matching-benchmark/packed-val/dog-affnet-hardnet8-degensac.json and image-matching-benchmark/packed-val/dog-affnet-hardnet8-degensac-adalam.json , and some visualizations – to ../benchmark-visualization/png.

First, we come back to our imc2021-sample-kornia-submission directory:

cd ../imc2021-sample-kornia-submission

Metric, which are used for the competition is mean average accuracy (mAA) at visibility threshold 0.1

{'mean': 0.0, 'std_scenes': 0.0, 'std_runs': 0.0}

import os

hashname='dog-affnet-hardnet8-degensac'
res_fname = os.path.join('../image-matching-benchmark/packed-val', f'{hashname}.json')

with open(res_fname, 'r') as f:
    results = json.load(f)

submission_name = results['config']['metadata']['method_name']
datasets = ['phototourism', 'pragueparks', 'googleurban']
tasks = ['stereo', 'multiview'] 
metric = 'qt_auc_10_th_0.1'

for dset in datasets:
    mAA_stereo = results[dset]['results']['allseq']['stereo']['run_avg'][metric]['mean']
    mAA_colmap = results[dset]['results']['allseq']['multiview']['run_avg']['bag_avg']['qt_auc_colmap_10']['mean']
    print (f'{submission_name} stereo mAA for {dset} is {mAA_stereo:.4f}')
    print (f'{submission_name} multiview mAA for {dset} is {mAA_colmap:.4f}')
    # Remember, I did not run multiview evaluation for the original submission.

CV-DoG-AffNet-HardNet8 (kornia) stereo mAA for phototourism is 0.7108
CV-DoG-AffNet-HardNet8 (kornia) multiview mAA for phototourism is 0.0000
CV-DoG-AffNet-HardNet8 (kornia) stereo mAA for pragueparks is 0.5850
CV-DoG-AffNet-HardNet8 (kornia) multiview mAA for pragueparks is 0.0000
CV-DoG-AffNet-HardNet8 (kornia) stereo mAA for googleurban is 0.3099
CV-DoG-AffNet-HardNet8 (kornia) multiview mAA for googleurban is 0.0000

import os

hashname='dog-affnet-hardnet8-degensac-adalam'
res_fname = os.path.join('../image-matching-benchmark/packed-val', f'{hashname}.json')

with open(res_fname, 'r') as f:
    results_adalam = json.load(f)

submission_name_adalam = results_adalam['config']['metadata']['method_name']
datasets = ['phototourism', 'pragueparks', 'googleurban']
tasks = ['stereo'] # ['stereo', 'multiview']  #Remember, that we skip colmap evaluations
metric = 'qt_auc_10_th_0.1'
for dset in datasets:
    mAA_stereo = results_adalam[dset]['results']['allseq']['stereo']['run_avg'][metric]['mean']
    mAA_colmap = results_adalam[dset]['results']['allseq']['multiview']['run_avg']['bag_avg']['qt_auc_colmap_10']['mean']
    print (f'{submission_name_adalam} stereo mAA for {dset} is {mAA_stereo:.4f}')
    print (f'{submission_name_adalam} multiview mAA for {dset} is {mAA_colmap:.4f}')

CV-DoG-AffNet-HardNet8-AdaLAM-DEGENSAC stereo mAA for phototourism is 0.7326
CV-DoG-AffNet-HardNet8-AdaLAM-DEGENSAC multiview mAA for phototourism is 0.8566
CV-DoG-AffNet-HardNet8-AdaLAM-DEGENSAC stereo mAA for pragueparks is 0.6624
CV-DoG-AffNet-HardNet8-AdaLAM-DEGENSAC multiview mAA for pragueparks is 0.5584
CV-DoG-AffNet-HardNet8-AdaLAM-DEGENSAC stereo mAA for googleurban is 0.3310
CV-DoG-AffNet-HardNet8-AdaLAM-DEGENSAC multiview mAA for googleurban is 0.1597

We can also see results sequence-by-sequence

import seaborn as sns
import matplotlib.pyplot as plt
import numpy as np
sns.set_context('paper', font_scale=1.7)
seqs = []
mAAs_adalam = []
mAAs = []

for dset in datasets:
    for task in tasks:
        for seq in results[dset]['results'].keys():
            if seq == 'allseq': 
                continue
            mAA  = results[dset]['results'][seq][task]['run_avg'][metric]['mean']
            mAA_adalam  = results_adalam[dset]['results'][seq][task]['run_avg'][metric]['mean']
            
            mAAs.append(mAA)
            mAAs_adalam.append(mAA_adalam)
            
            seqs.append(seq)
fig, ax = plt.subplots(figsize=(15,5))
xticks = 2*np.arange(len(seqs))
ax.set_xticks(xticks)
ax.bar(xticks, mAAs)
ax.bar(xticks+0.5, mAAs_adalam)
ax.set_xticklabels(seqs)
ax.legend(['SMNN', 'AdaLAM'])
ax.set_ylabel('mAA')
ax.set_xlabel('Sequence')

Text(0.5, 0, 'Sequence')

How do our feature correspondences look like?

Creating final submission

Its time to create our final submission!

configs = []
current_config = deepcopy(base_config)

    
current_config['metadata']['method_name'] = 'KORNIA TUTORIAL AdaLAM CV-DoG-AffNet-HardNet8'

label = current_config['config_common']['json_label'] 
current_config['config_common']['json_label']  = f'{label}'
configs.append(current_config)

print (current_config)
with open('final_submission_adalam.json', 'w') as f:
    json.dump(configs, f, indent=2)

{'metadata': {'publish_anonymously': False, 'authors': 'Dmytro Mishkin, Milan Pultar and kornia team', 'contact_email': 'ducha.aiki@gmail.com', 'method_name': 'KORNIA TUTORIAL AdaLAM CV-DoG-AffNet-HardNet8', 'method_description': 'OpeCV SIFT keypoints 8000 features, followed by the AffNet normalization \n    and HardNet8 descriptor as implemented in kornia.\n    Matched using AdaLAM with DEGENSAC', 'link_to_website': 'https://github.com/kornia/kornia', 'link_to_pdf': 'https://arxiv.org/abs/2007.09699'}, 'config_common': {'json_label': 'dog-affnet-hardnet8-degensac-adalam', 'keypoint': 'cv2dog', 'descriptor': 'affnethardnet8', 'num_keypoints': 8000}, 'config_phototourism_stereo': {'use_custom_matches': True, 'custom_matches_name': 'adalam', 'geom': {'method': 'cv2-8pt'}}, 'config_phototourism_multiview': {'use_custom_matches': True, 'custom_matches_name': 'adalam', 'colmap': {}}, 'config_pragueparks_stereo': {'use_custom_matches': True, 'custom_matches_name': 'adalam', 'geom': {'method': 'cv2-8pt'}}, 'config_pragueparks_multiview': {'use_custom_matches': True, 'custom_matches_name': 'adalam', 'colmap': {}}, 'config_googleurban_stereo': {'use_custom_matches': True, 'custom_matches_name': 'adalam', 'geom': {'method': 'cv2-8pt'}}, 'config_googleurban_multiview': {'use_custom_matches': True, 'custom_matches_name': 'adalam', 'colmap': {}}}

Submission Zip file should have folder structure as follow:
 ├── config.json
 ├── [Dataset 1]
 │   ├── [Sequence 1]
 │   │   ├── keypoints.h5
 │   │   ├── descriptors.h5
 │   │   ├── matches.h5
 │   ├── [Sequence 2]
 │   │   ├── ...
 ├── [Dataset 2]
 │   ├── ...

So we have to just copy our features, add config and zip them.

cp final_submission_adalam.json extracted/cv2-dog-affnet-hardnet8/config.json
cd extracted/cv2-dog-affnet-hardnet8
zip -r submission.zip *

Last step before the submission - check the submission for correctness with provided script

cd ../../../image-matching-benchmark
python -utt submission_validator.py --submit_file_path ../imc2021-sample-kornia-submission/extracted/cv2-dog-affnet-hardnet8/submission.zip --benchmark_repo_path . --raw_data_path ../imc-2021-data/ --datasets googleurban phototourism pragueparks

If everything is correct, you will see:

Validating method 1/1: "dog-affnet-hardnet8-degensac"
['googleurban', 'phototourism', 'pragueparks']
Running: googleurban, stereo track
Running: googleurban, multiview track
Running: phototourism, stereo track
Running: phototourism, multiview track
Running: pragueparks, stereo track
Running: pragueparks, multiview track
Validating key "config_googleurban_stereo"
Validating key "config_googleurban_multiview"
Validating key "config_phototourism_stereo"
Validating key "config_phototourism_multiview"
Validating key "config_pragueparks_stereo"
Validating key "config_pragueparks_multiview"

And file submission_log.txt will appear near our .zip file.

cat ../imc2021-sample-kornia-submission/extracted/cv2-dog-affnet-hardnet8/submission_log.txt

Submission is in proper format, please submit to IMW 2021 website.

That’s all, folks! We can submit! But, please, do not just submit this sample submission - make your own :)

Phototourism

GoogleUrban

PragueParks

SuperGlue

SuperGlue is an attention-based graph neural network for matching local features, taking into account both geometry (keypoint location) and appearance (descriptor). Unlike previous works, e.g. CNe, or OANet below, it does not “scoring-and-cleaning” already established tentative correspondences. Instead, it establishes the correpondences given the local features from two images. SuperGlue won IMC-2020, as well as two other competitions at CVPR 2020. Its inference implementation is available here.

OANet

OANet is an specialized neural network architecture for “scoring-and-cleaning” already established tentative correspondences. OANet was run on top of DoG-HardNet local features. Pytorch version of OANet is available here

AdaLAM

AdaLAM is a handcrafted algorithm for tentative correspondence cleaning, which work comparably or even better than a learning-based approaches. It is based on two core assumptions:

• Keypoints, which are near from each other are probably corresponding to the neaighboring keypoints in the other images
• If keypoints are in correspondence, it means that their orientation and scale are also in correspondence. Check my post Local affine features: useful side product for an explanation

The implementation is avilable here. As DoG+HardNet were used as a local features.

DISK

DISK is local feature, which has two main differences from the rest of competitors (SuperPoint, R2D2):

1. DISK is trained with a reinforcement-learning objective
2. DISK has UNet-like architecture, unlike VGG-style for the rest of the features.

Its implementation is available here.

HyNet

HyNet is an next stage in the L2Net-HardNet-SOSNet series of local patch descriptors. It is different from the previous works in two ways:

1. BatchNorm and ReLU in the HardNet architecture are replaced with FRN and TLU respecively.
2. During training, distance to the negative (non-matching) and positive (matching) samples are calculated in a different way for better learning, see image below. u and v denote descriptors is

HyNet submission also used semantic segmentation network to remove the keypoints from the non-matchable areas, such as sky and water.

HardNet8

HardNet8 is another improvement of the HardNet architecture:

1. Deeper and wider network
2. The output is compressed with a PCA.
3. The training set and hyperparameters are carefully selected.

It is available in kornia

The PragueParks dataset

The PragueParks dataset contains images from video sequences captured by the organizers with an iPhone 11, in 2021. The iPhone 11 has two cameras, with normal and wide lenses, both of which were used. Note that while the video is high quality, some of the frames suffer from motion blur. These videos were then processed by the commercial 3D reconstruction software RealityCapture, which is orders of magnitude faster than COLMAP, while delivering a comparable output in terms of accuracy. Similarly to we did for the “PhotoTourism” dataset, this data is then subsampled in order to generate the subsets used for evaluation.

The dataset contains small-scale scenes like tree, pond, wooden and metal sculptures with different level of zoom, lots of vegetation, and no people. The distribution of its camera poses differs from Phototourism.

The GoogleUrban dataset

The GoogleUrban dataset contains images used by Google to evaluate localization algorithms, such as those in Google’s Visual Positioning System, which powers Live View on millions on mobile devices. They are obtained from videos collected from different cell phones, on many countries all over the world, often years apart. They contain poses, but not depth maps. Please note that due to legal reasons, this data is released with a restricted license, and must be deleted by the end of the challenge.

Importing features

Here we have to provide the same keypoint and descriptor names, as we wrote in json config. The rest of arguments are straightforward: path to features, json, etc.

cd ../image-matching-benchmark/
python -utt import_features.py --kp_name cv2dog --desc_name affnethardnet8 --num_keypoints 8000 --path_features ../imc2021-sample-kornia-submission/extracted/cv2-dog-affnet-hardnet8  --path_results ../benchmark-results --subset both  --is_challenge false --path_json ../imc2021-sample-kornia-submission/base_config.json  --datasets phototourism googleurban pragueparks

Running the evaluation

Now we are ready to run the evaluation

python -utt run.py --run_mode=interactive --json_method=../imc2021-sample-kornia-submission/base_config.json --subset=val  --eval_multiview=False --path_data ../imc-2021-data/ --path_results ../benchmark-results --is_challenge false

After a while (an 30 min for 32 cores machine), the process will finish and you will see the following log message:

 -- Saving to: "packed-val/dog-affnet-hardnet8-degensac.json"

Reading results

Json file with evaluation results is saved to image-matching-benchmark/packed-val/dog-affnet-hardnet8-degensac.json, and some visualizations – to ../benchmark-visualization/png.

First, we come back to our imc2021-sample-kornia-submission directory:

cd ../imc2021-sample-kornia-submission

Metric, which are used for the competition is mean average accuracy (mAA) at visibility threshold 0.1

import os

hashname='dog-affnet-hardnet8-degensac'
res_fname = os.path.join('../image-matching-benchmark/packed-val', f'{hashname}.json')

with open(res_fname, 'r') as f:
    results = json.load(f)

submission_name = results['config']['metadata']['method_name']
datasets = ['phototourism', 'pragueparks', 'googleurban']
tasks = ['stereo'] # ['stereo', 'multiview']  #Remember, that we skip colmap evaluations
metric = 'qt_auc_10_th_0.1'
for dset in datasets:
    for task in tasks:
        mAA = results[dset]['results']['allseq'][task]['run_avg'][metric]['mean']
        print (f'{submission_name} {task} mAA for {dset} is {mAA:.4f}')

CV-DoG-AffNet-HardNet8 (kornia) stereo mAA for phototourism is 0.7108
CV-DoG-AffNet-HardNet8 (kornia) stereo mAA for pragueparks is 0.5850
CV-DoG-AffNet-HardNet8 (kornia) stereo mAA for googleurban is 0.3099

We can also see results sequence-by-sequence

import seaborn as sns
sns.set_context('paper', font_scale=1.7)
seqs = []
mAAs = []
for dset in datasets:
    for task in tasks:
        for seq in results[dset]['results'].keys():
            if seq == 'allseq': 
                continue
            mAA  = results[dset]['results'][seq][task]['run_avg'][metric]['mean']
            mAAs.append(mAA)
            seqs.append(seq)
fig, ax = plt.subplots(figsize=(15,5))
xticks = 2*np.arange(len(seqs))
ax.set_xticks(xticks)
ax.bar(xticks, mAAs)
ax.set_xticklabels(seqs)
ax.set_ylabel('mAA')
ax.set_xlabel('Sequence')

Text(0.5, 0, 'Sequence')

How do our feature correspondences look like?

import cv2
def plot_image_grid(images, ncols=None, cmap='gray'):
    #Taken from https://stackoverflow.com/a/66961099/1983544
    '''Plot a grid of images'''
    if not ncols:
        factors = [i for i in range(1, len(images)+1) if len(images) % i == 0]
        ncols = factors[len(factors) // 2] if len(factors) else len(images) // 4 + 1
    nrows = int(len(images) / ncols) + int(len(images) % ncols)
    imgs = [images[i] if len(images) > i else None for i in range(nrows * ncols)]
    f, axes = plt.subplots(nrows, ncols, figsize=(3*ncols, 2*nrows))
    axes = axes.flatten()[:len(imgs)]
    for img, ax in zip(imgs, axes.flatten()): 
        if np.any(img):
            if len(img.shape) > 2 and img.shape[2] == 1:
                img = img.squeeze()
            ax.imshow(img, cmap=cmap)
imgs = []
VIS_DIR = f'../benchmark-visualization/png/{hashname}'
for dset in os.listdir(VIS_DIR):
    dset_dir = os.path.join(VIS_DIR, dset)
    for seq in os.listdir(dset_dir):
        seq_dir = os.path.join(dset_dir, seq, 'stereo')
        for img_fname in os.listdir(seq_dir):
            full_fname = os.path.join(seq_dir, img_fname)
            img = cv2.resize(cv2.cvtColor(cv2.imread(full_fname), cv2.COLOR_BGR2RGB), (200,100))
            imgs.append(img)
plot_image_grid(imgs)

Hyperparameters tuning

Let’s now tune our hyperparameters, specifically, matching threshold and RANSAC inlier ratio. First we tune RANSAC and then matching ratio. We need to generate jsons for each configuration.

inl_ths = [0.25, 0.5, 0.75, 1.0, 1.5, 2.0, 2.5, 3.0]
configs = []

for inl_th in inl_ths:
    current_config = deepcopy(base_config)
    for dset in ['phototourism', 'pragueparks', 'googleurban']:
        current_config[f'config_{dset}_stereo']['geom']['threshold'] = inl_th
    label = current_config['config_common']['json_label'] 
    current_config['config_common']['json_label']  = f'{label}-inlth-{inl_th}'
    configs.append(current_config)

with open('ransac_tuning.json', 'w') as f:
    json.dump(configs, f, indent=2)

cd ../image-matching-benchmark/
python -utt run.py --run_mode=interactive --json_method=../imc2021-sample-kornia-submission/base_config.json --subset=val  --eval_multiview=False --path_data ../imc-2021-data/ --path_results ../benchmark-results --is_challenge false
cd ../imc2021-sample-kornia-submission

Now we will write a function, which reads results and picks the best threshold per dataset

hashname='dog-affnet-hardnet8-degensac'
res_dict = {}
datasets = ['phototourism', 'pragueparks', 'googleurban']
for dset in datasets:
    res_dict[dset] = {}

task = 'stereo'
metric = 'qt_auc_10_th_0.1'
for inl_th in inl_ths:
    res_fname = os.path.join('../image-matching-benchmark/packed-val', f'{hashname}-inlth-{inl_th}.json')
    try:
        with open(res_fname, 'r') as f:
            results = json.load(f)
    except:
        continue
    submission_name = results['config']['metadata']['method_name']
    res_dict[inl_th] = {}
    for dset in datasets:
        mAA = results[dset]['results']['allseq'][task]['run_avg'][metric]['mean']
        res_dict[dset][inl_th] = mAA
fig, ax = plt.subplots(figsize=(5,5))
colors = ['r','b','k']
final_ths = {}
for i, dset in enumerate(datasets):
    inl_ths = []
    mAAs = []
    for inl_th, mAA in res_dict[dset].items():
        inl_ths.append(inl_th)
        mAAs.append(mAA)
    best_th_idx = np.argmax(np.array(mAAs))
    best_th = inl_ths[best_th_idx]
    best_mAA = mAAs[best_th_idx]
    print (f'Best {dset} mAA = {best_mAA:.4f} with inl_th = {best_th}')
    ax.plot(inl_ths, mAAs, label=dset, color=colors[i])
    ax.plot(best_th, best_mAA, label = f'{dset}-best', marker='x', linestyle='', color=colors[i])
    final_ths[dset] = best_th
    
ax.legend()
ax.set_ylabel('mAA')
ax.set_xlabel('DEGENSAC inlier threshold')
    

Best phototourism mAA = 0.7108 with inl_th = 0.5
Best pragueparks mAA = 0.6700 with inl_th = 1.5
Best googleurban mAA = 0.3116 with inl_th = 0.75

Text(0.5, 0, 'DEGENSAC inlier threshold')

Creating final submission

Its time to create our final submission!

configs = []
current_config = deepcopy(base_config)
for dset in ['phototourism', 'pragueparks', 'googleurban']:
    current_config[f'config_{dset}_stereo']['geom']['threshold'] = final_ths[dset]
    # I did a little bit of tuning offline for multiview, so we will put it here
    current_config[f'config_{dset}_multiview']['matcher']['filtering']['threshold'] = 0.95
    
    #Remember, that we should not forget to turn FLANN ofd
    current_config[f'config_{dset}_multiview']['matcher']['flann'] = False
    current_config[f'config_{dset}_stereo']['matcher']['flann'] = False
    
    
current_config['metadata']['method_name'] = 'KORNIA TUTORIAL CV-DoG-AffNet-HardNet8'

label = current_config['config_common']['json_label'] 
current_config['config_common']['json_label']  = f'{label}'
configs.append(current_config)

print (current_config)
with open('final_submission.json', 'w') as f:
    json.dump(configs, f, indent=2)

{'metadata': {'publish_anonymously': False, 'authors': 'Dmytro Mishkin, Milan Pultar and kornia team', 'contact_email': 'ducha.aiki@gmail.com', 'method_name': 'KORNIA TUTORIAL CV-DoG-AffNet-HardNet8', 'method_description': "OpeCV SIFT keypoints 8000 features, followed by the AffNet normalization \n    and HardNet8 descriptor as implemented in kornia.\n    Matched using the built-in matcher (bidirectional filter with the 'both' strategy, \n    hopefully optimal inlier and ratio test thresholds) with DEGENSAC", 'link_to_website': 'https://github.com/kornia/kornia', 'link_to_pdf': 'https://arxiv.org/abs/2007.09699'}, 'config_common': {'json_label': 'dog-affnet-hardnet8-degensac', 'keypoint': 'cv2dog', 'descriptor': 'affnethardnet8', 'num_keypoints': 8000}, 'config_phototourism_stereo': {'use_custom_matches': False, 'matcher': {'method': 'nn', 'distance': 'L2', 'flann': False, 'num_nn': 1, 'filtering': {'type': 'snn_ratio_pairwise', 'threshold': 0.9}, 'symmetric': {'enabled': True, 'reduce': 'both'}}, 'outlier_filter': {'method': 'none'}, 'geom': {'method': 'cmp-degensac-f', 'threshold': 0.5, 'confidence': 0.999999, 'max_iter': 100000, 'error_type': 'sampson', 'degeneracy_check': True}}, 'config_phototourism_multiview': {'use_custom_matches': False, 'matcher': {'method': 'nn', 'distance': 'L2', 'flann': False, 'num_nn': 1, 'filtering': {'type': 'snn_ratio_pairwise', 'threshold': 0.95}, 'symmetric': {'enabled': True, 'reduce': 'both'}}, 'outlier_filter': {'method': 'none'}, 'colmap': {}}, 'config_pragueparks_stereo': {'use_custom_matches': False, 'matcher': {'method': 'nn', 'distance': 'L2', 'flann': False, 'num_nn': 1, 'filtering': {'type': 'snn_ratio_pairwise', 'threshold': 0.9}, 'symmetric': {'enabled': True, 'reduce': 'both'}}, 'outlier_filter': {'method': 'none'}, 'geom': {'method': 'cmp-degensac-f', 'threshold': 1.5, 'confidence': 0.999999, 'max_iter': 100000, 'error_type': 'sampson', 'degeneracy_check': True}}, 'config_pragueparks_multiview': {'use_custom_matches': False, 'matcher': {'method': 'nn', 'distance': 'L2', 'flann': False, 'num_nn': 1, 'filtering': {'type': 'snn_ratio_pairwise', 'threshold': 0.95}, 'symmetric': {'enabled': True, 'reduce': 'both'}}, 'outlier_filter': {'method': 'none'}, 'colmap': {}}, 'config_googleurban_stereo': {'use_custom_matches': False, 'matcher': {'method': 'nn', 'distance': 'L2', 'flann': False, 'num_nn': 1, 'filtering': {'type': 'snn_ratio_pairwise', 'threshold': 0.9}, 'symmetric': {'enabled': True, 'reduce': 'both'}}, 'outlier_filter': {'method': 'none'}, 'geom': {'method': 'cmp-degensac-f', 'threshold': 0.75, 'confidence': 0.999999, 'max_iter': 100000, 'error_type': 'sampson', 'degeneracy_check': True}}, 'config_googleurban_multiview': {'use_custom_matches': False, 'matcher': {'method': 'nn', 'distance': 'L2', 'flann': False, 'num_nn': 1, 'filtering': {'type': 'snn_ratio_pairwise', 'threshold': 0.95}, 'symmetric': {'enabled': True, 'reduce': 'both'}}, 'outlier_filter': {'method': 'none'}, 'colmap': {}}}

Submission Zip file should have folder structure as follow:
 ├── config.json
 ├── [Dataset 1]
 │   ├── [Sequence 1]
 │   │   ├── keypoints.h5
 │   │   ├── descriptors.h5
 │   │   ├── matches.h5
 │   ├── [Sequence 2]
 │   │   ├── ...
 ├── [Dataset 2]
 │   ├── ...

So we have to just copy our features, add config and zip them.

cp final_submission.json extracted/cv2-dog-affnet-hardnet8/config.json
cd extracted/cv2-dog-affnet-hardnet8
zip -r submission.zip *

Last step before the submission - check the submission for correctness with provided script

cd ../../../image-matching-benchmark
python -utt submission_validator.py --submit_file_path ../imc2021-sample-kornia-submission/extracted/cv2-dog-affnet-hardnet8/submission.zip --benchmark_repo_path . --raw_data_path ../imc-2021-data/ --datasets googleurban phototourism pragueparks

If everything is correct, you will see:

Validating method 1/1: "dog-affnet-hardnet8-degensac"
['googleurban', 'phototourism', 'pragueparks']
Running: googleurban, stereo track
Running: googleurban, multiview track
Running: phototourism, stereo track
Running: phototourism, multiview track
Running: pragueparks, stereo track
Running: pragueparks, multiview track
Validating key "config_googleurban_stereo"
Validating key "config_googleurban_multiview"
Validating key "config_phototourism_stereo"
Validating key "config_phototourism_multiview"
Validating key "config_pragueparks_stereo"
Validating key "config_pragueparks_multiview"

And file submission_log.txt will appear near our .zip file.

cat ../imc2021-sample-kornia-submission/extracted/cv2-dog-affnet-hardnet8/submission_log.txt

Submission is in proper format, please submit to IMW 2021 website.

That’s all, folks! We can submit! But, please, do not just submit this sample submission - make your own :)

1. Benchmark = metrics + implementation + dataset

In our paper “Image Matching across Wide Baselines: From Paper to Practice” we focused on the first two parts. Specifically, metrics – if they are not “downstream” metrics, the improvements in the single component might not translate to the overall system improvements. And implementation – implementing the simplest possible setup is, of course, a valuable tool, but one have to also incorporate the best known practices, e.g. matching, RANSAC tuning as so on.

I hope, that we have delivered that message to the community. But the last component – the dataset – we have, perhaps, overlooked a bit ourself. The problem with dataset limitations, e.g. lack of illumination or seasonal changes is not that one does not properly address. Usually benchmark papers are adressing their limitations quite clearly. The problem is that researchers (including myself) have tendency to work on improving results, which are easily measurable, therefore the implicitly designing the methods, which solve only some specific problem, encoded in the form of the dataset.

2. Trying to work for the “most general” case might be detrimental for the practical applications.

This is kind of opposite side of the lesson 1. For example, the classical SIFT matching is rotation-invariant, because that was assumed to be the requirement to work in the “real world”. However, the practice of image retrieval and then most of learned local features like R2D2, SuperPoint, DELF and so on, showed that “up-is-up” is a reasonable assumption to built on. The rotational invariance in lots of scenarios hurt more than helps.

3. When borrow idea from classical paper, adapt it

HardNet borrows the idea of the using second nearest neighbor(SNN) descriptor from the SIFT. However, using SNN ratio for descriptor learning leads to inferior results. We had to modify it to the triplet margin loss.

## 4. Classic handcrafted algorithms are not dead and can be improved a lot

Modern versions of RANSAC are still necessary for two view matching even when so complex methods as SuperGlue as used. Moreover, they still have quite a lot things for improvement, as proved by my colleague, Daniel Barath

They also have a benefit, that once you have an idea, you can make a paper out it faster, as you don’t need gathering data and training the model.

1. Application-specific local features

One thing about the current local features is that they are kind of universal. SIFT works reasonably well for lots of applications: 3d reconstruction, SLAM, image retrieval, etc. The learned features, like SuperPoint or R2D2, although are biased towards the data they are trained on, still don’t have anything domain specific.

Let me explain. There are different qualities about the local feature (detectors). It can be more or less robust to nuisance factors like the illumination and camera position. It can be more or less precisely localized. It can be more or less dense and/or evenly distributed over the image.

For example, in image retrieval, one does not really care about precise localization, the robustness is much more important. For the 3d reconstruction one would like to have a lot of 3d points to get the reasonable reconstruction. On the other hand, for the SLAM/relocalization application, sparse features would be more advantageous because of smaller memory footprint and computational cost.

There are, actually, some steps in that direction. Let me name a few.

1. HOW local features designed for the image retrieval

2. SIPs: Succinct Interest Points from Unsupervised Inlierness Probability Learning sparse local features for the SLAM.

3. Reinforced Feature Points: Optimizing Feature Detection and Description for a High-Level Task

I believe, that it is only beginning and we are yet to experience AlphaZero moment for the local features.

2. Rethinking overall wide baseline stereo pipeline, optimized for the specific application

It is often perceived, that image matching across the unordered collection of the images is a task with quadratic complexity w.r.t. number of images. Some operations can be done separately, e.g. feature detection, but others, like feature matching and RANSAC cannot. Right?

Not necessarily. It turns out, that one can avoid running feature matching and RANSAC for more than 90% of image pairs with clever preprocessing, ordering and re-using results from the previous matching. Moreover, in order to do the whole task faster (matching image collections), one may need to introduce additional steps, which are not necessary, or slowing things down for the two images case.

That’s what we done for the intial pose estimation for the global SfM in our paper, which reduced the matching runtime from 200 hours to 29.

Another example would be “SuperGlue: Learning Feature Matching with Graph Neural Networks”, where authors abandoned traditional descriptor matching and instead leveraged all the information for keypoint and descriptors from both images altogether.

3. Rethinking and improving the process of training data generation for the WBS

So far, all the local feature papers I have seen rely on one of the ground truth source.

1. SfM data, obtained with COLMAP with, possibly, a cleaned depth information.
2. Affine and color augmentation.
3. Synthetic images (e.g. corners for SuperPoint).

There are several problems with them.

SfM data assumes that the data is matchable by the existing methods, at least for a some extent. That might not always be true for cross-seasonal, medical and other kind of data. It is also not applicable for historical photographies and other types of data. Moreover, SfM data takes quite a time for compute and space to store. I believe that we may do better.

Affine and color augmentation can take us only this far – we actually want our detectors and descriptors to be robust to the changes, which we don’t know how to simulate/augment.

Synthetic images as they are in, say, CARLA simulator lack fine details and photorealism. However, I am optimistic about using neural renderers and learned wide baseline stereo is a GAN-like self-improving loop.

4. Matching with On-Demand View Synthesis revisited

I like the “on-demand” principle a lot and I think we can explore it much more that we are now. So far we have either affine view synthesis (ASIFT, MODS), or GAN-based stylizations for the day-night matching.

That is why I am glad to see papers like Single-Image Depth Prediction Makes Feature Matching Easier, which generate normalized views based on depth in order to help the matching.

Why not go further? Combine viewpoint, illumination, season, sensor synthesis?

5. Moar inputs!

I have mentioned above that monocular depth may help the feature matching or camera pose estimation. However, why stop here?

Let’s use other networks as well, especially given that we will need them on robot or vehicle anyway. Semantic segmentation? Yes, please. Surface normals? Why not? Intrinsic images? Йой, най буде! {% fn 1 %}

{{ “Ukrainian, means let is be” | fndetail: 1 }}

What do you think would be good idea for the WBS research?

Let me know in comments/twitter. I am also going to update this page from time to time

Image Relocalization seems to be is more real-world and engineering task, than image retrieval.

And that it why it actually ALREADY WORKS, because if there some weak spot, it is compensated by the system design. Thhe same conclusion from our IMC paper, experiment with ground truth – if you have 1k images for the 3d model, you can use as bad features, as you want. The COLMAP will recover anyway

The retrieval, on the other hand is more interesting to work on, because it is kind of deliberately hard and you can do some fancy stuff, which do not matter in the real world.

Task1 (global descriptor-only) system are quite useless now

Unless we are speaking about the quite dense image representation. I mean, top-accuracy is 35% vs almost 100% for those, which include local features.

Good news: it has a LOT of space for the improvement to work on.

For the task 2a and 2b, robust global descriptors are a way to do the retrieval, sorry VLAD.

The precision will come from the local features. Which I like a lot, because VLAD is more complex to train and initalize, I never liked it (nothing personal).

For the task2a and 2b we need new metrics, e.g. precisition @ X Mb memory footprint

Because otherwise, the task is easily solved by the brute force – either by photo taking, or, at least with image syntesis, see 24/7 place recognition by view synthesis.

Such steps are already taken in the paper Learning and aggregating deep local descriptors for instance-level recognition – see the table with memory footprint.

That is how one could have an interesting research challenge, also having some grounds in the real-world – to work in mobile phones. Otherwise, any method would work, if the database is dense enough.

Robust local features matter for illumination changes

It is a bit hidden in the Appendix, so go directly to the Figure 9. It clearly shows that localization performance is bounded by SIFT, if it is used for two view matching, making retrieval improvements irrelevant. When R2D2 or D2Net are used for matching instead, the overall results for night-time are much better.

That is in line with my small visual benchmark I did recently.

https://twitter.com/ducha_aiki/status/1330495426865344515

That’s all, folks! Now please, check the paper and the code they provided.

Brown PhotoTourism.

The most commonly and successfully used dataset for local descriptor learning is PhotoTourism, created in 2008. Here is its description by authors:

The dataset consists of corresponding patches sampled from 3D reconstructions of the Statue of Liberty (New York), Notre Dame (Paris) and Half Dome (Yosemite).

It also comes with evaluation protocol: patch pairs are labeled as “same” or “different” and the false positive rate at recall of 95% (FPR95) is reported. The variable, used to build ROC curve is descriptor distance between two patches.

Advantages:

• It contains local patches, extracted for two types of local feature detector – DoG (SIFT) and Harris corners.
• It is extracted from images, contraining non-planar structures and the geometrical noise present is caused by the local feature detector, not added artificially.
  
• Descriptors, trained on the dataset, show very good performance , therefore the data itself is good.

Disadvantages:

• when used as a benchmark, it shows unrealistic results: SIFT is 40x worse than deep learned descriptor. In practice, the difference is much smaller.

HPatches

HPatches, where H stands for the “homography” was proposed to overcome the problem of unrealisting metric and, seemingly, too easy data, used in Phototourism dataset.

It was constructed in a different way than a Phototourism. First, local features were detected in the “reference” image and then reprojected to other images in sequences. Reprojection is prossible, because all the images are photographies of the planes – graffity, drawing, print, etc, or are all taken from the same position. After the reprojection, some amount of geometrical noise – rotation, translation, scaling, was added to the local features and the patches were extracted.

This process is illustration on the picture below (both taken from the HPatches website).

HPAtches also provide 3 testing protocol, evaluating mean average precision (mAP) for 3 different tasks: patch verification – similar to Brown Phototourism, image matching and patch retrieval. The variable, used to build mAP is descriptor distance between two patches.

Advantages:

• Unlike PhotoTourism patch verification, image matching and patch retrieval tasks are not saturated.
• HPatches contains illumination split, allowing the evaluation of descriptor robustness to illumination changes.

Disadvantages:

• patches “misregistration” noise is of artificial nature, although paper claims that it has similar statistics
• no non-planar structure
• performance in HPaptches does not really correlate with the downstream performance

AMOSPatches

AMOS patches is “HPatches illumination on steroids, without geometrical noise”. It has the same advantanges and disadvantages, as HPatches and is mostly focused on illumination and weather changes.

PhotoSynth

PhotoSynth can be described and something in the middle between Phototour and HPatches. It contains patches, sampled from planar scenes, as well as from non-planar scenes.

At first glance, it should be great for the test and training purposes. However, there are several issues with it. First, pre-trained HardNetPS descriptor, released together with the dataset, works well on HPatches, but poor in practice.

Second, a couple of colleagues has tried to train the descriptor on top of it, as it was significantly worse than the authors reference model. Moreover, there is no testing/training code protocol available together with dataset.

So, while PhotoSynth might be a good dataset in principle, it definitely needs more love and work.

Designing the evaluation protocol

Classical local descriptor matching consists of two parts: finding nearest neighbors and filtering unreliable ones based on some criterion. I have wrote a blogpost, describing the matching strategies in details.

The most used in practice criterion is the first to second nearest neighbor distance (Lowe’s) ratio threshold for filtering false positive matches. It is shown in the figure below.

The intuition is simple: if two candidates are too similar, then the match is unreliable and it is better to drop it.

Somehow, none of the local patch evaluation protocols does not take such filtering criterion in mind, although it greatly influences the overall performance.

So, let’s do the following:

1. Take the patches, which are extracted from only two images.
2. For the each patch, calculate the descriptor distance to the correct match and to the hardnest (closest) non-match. Calculate the Lowe’s ratio between this two.
3. Calculate accuracy for each of such triplets. If the correct match has smaller distance, score 1, if not - 0.
4. Sort the ratios from smallest to biggest and calculate mean average precision (mAP).

Brown PhotoTour Revisied: implementation details

We have designed the protocol, now time for data. We could spend several month collecting and cleaning it…or we can just re-use great Brown PhotoTourism dataset. Re-visiting labeling and/or evaluation protocol of the time-tested dataset is a great idea.

Just couple of examples: ImageNette created by Jeremy Howard from ImageNet, Revisited Oxford 5k by Filip Radenovic and so on.

For the protocol we designed above we need the information about the image id, where the patch was extracted from. Unfortunately, there is no such information in the Brown PhotoTourism, but there is suitable alternative – the image id, where the reference patch was detected. What does it mean?

Suppose, we have 4 images and 5 keypoints. All the keypoints present in all images, which gives us 20 patches. 3 keypoints were first detected in Image 1 and 2 in image 2.
That means that we will have 12 patches labeled image 1 and 8 patches labeled image 2.

So, we will have results for image 1 and image 2. Let’s consider image 1. There are 12 patches, splitted in 3 “classes”, 4 patches in each class.

Then, for the each of those 12 patches we:

• pick each of the corresponding patched as positives, so 3 positives. , ,
• find the closest negative N.
• add triplets (A, , N), (A, , N), (A, , N) to the evaluation.

Repeat the same for the image 2. That mimics the two-view matching process as close, as possible, given the data available to us.

Installation

pip install brown_phototour_revisited

How to use

There is a single function, which does everything for you: full_evaluation. The original Brown benchmark consider evaluation, similar to cross-validation: train descriptor on one subset, evaluate on two others, repeat for all, so 6 evaluations are required. For the handcrafted descriptors, or those, that are trained on 3rd party datasets, only 3 evaluations are necessary. We are following it here as well.

However, if you need to run some tests separately, or reuse some functions – we will cover the usage below. In the following example we will show how to use full_evaluation to evaluate SIFT descriptor as implemented in kornia.

# !pip install kornia

import torch
import kornia
from IPython.display import clear_output
from brown_phototour_revisited.benchmarking import *
patch_size = 65 

model = kornia.feature.SIFTDescriptor(patch_size, rootsift=True).eval()

descs_out_dir = 'data/descriptors'
download_dataset_to = 'data/dataset'
results_dir = 'data/mAP'

results_dict = {}
results_dict['Kornia RootSIFT'] = full_evaluation(model,
                                'Kornia RootSIFT',
                                path_to_save_dataset = download_dataset_to,
                                path_to_save_descriptors = descs_out_dir,
                                path_to_save_mAP = results_dir,
                                patch_size = patch_size, 
                                device = torch.device('cuda:0'), 
                           distance='euclidean',
                           backend='pytorch-cuda')
clear_output()
print_results_table(results_dict)

------------------------------------------------------------------------------
Mean Average Precision wrt Lowe SNN ratio criterion on UBC Phototour Revisited
------------------------------------------------------------------------------
trained on       liberty notredame  liberty yosemite  notredame yosemite
tested  on           yosemite           notredame            liberty
------------------------------------------------------------------------------
Kornia RootSIFT        56.70              47.71               48.09 
------------------------------------------------------------------------------

Results

So, let’s check how it goes. The latest results and implementation are in the following notebooks:

• Deep descriptors
• Non-deep descriptors

The results are the following:

------------------------------------------------------------------------------
Mean Average Precision wrt Lowe SNN ratio criterion on UBC Phototour Revisited
------------------------------------------------------------------------------
trained on       liberty notredame  liberty yosemite  notredame yosemite
tested  on           yosemite           notredame            liberty
------------------------------------------------------------------------------
Kornia RootSIFT 32px   58.24              49.07               49.65 
HardNet 32px       70.64  70.31        61.93  59.56        63.06  61.64
SOSNet 32px        70.03  70.19        62.09  59.68        63.16  61.65
TFeat 32px         65.45  65.77        54.99  54.69        56.55  56.24
SoftMargin 32px    69.29  69.20        61.82  58.61        62.37  60.63
HardNetPS 32px         55.56              49.70               49.12 
R2D2_center_grayscal   61.47              53.18               54.98 
R2D2_MeanCenter_gray   62.73              54.10               56.17 
------------------------------------------------------------------------------

------------------------------------------------------------------------------
Mean Average Precision wrt Lowe SNN ratio criterion on UBC Phototour Revisited
------------------------------------------------------------------------------
trained on       liberty notredame  liberty yosemite  notredame yosemite
tested  on           yosemite           notredame            liberty
------------------------------------------------------------------------------
Kornia SIFT 32px       58.47              47.76               48.70 
OpenCV_SIFT 32px       53.16              45.93               46.00 
Kornia RootSIFT 32px   58.24              49.07               49.65 
OpenCV_RootSIFT 32px   53.50              47.16               47.37 
OpenCV_LATCH 65px  -----  -----        -----  37.26        -----  39.08
OpenCV_LUCID 32px      20.37              23.08               27.24 
skimage_BRIEF 65px     52.68              44.82               46.56 
Kornia RootSIFTPCA 3 60.73  60.64        50.80  50.24        52.46  52.02
MKD-concat-lw-32 32p 72.27  71.95        60.88  58.78        60.68  59.10
------------------------------------------------------------------------------

So far - in agreement with IMC benchmark: SIFT and RootSIFT are good, but not the best, SOSNet and HardNet are the leaders, but within tens of percents, not by orders of magnitude.

Summary

It really pays off, to spend time designing a proper evaluation pipeline and gathering the data for it. If you can re-use existing work - great. But don’t blindly trust anything, even super-popular and widely adopted benchmarks. You need always check if the the protocol and data makes sense for your use-case personally.

Thanks for the reading, see you soon!

Citation

If you use the benchmark/development set in an academic work, please cite it.

@misc{BrownRevisited2020,
  title={UBC PhotoTour Revisied},
  author={Mishkin, Dmytro},
  year={2020},
  url = {https://github.com/ducha-aiki/brown_phototour_revisited}
}

What is wrong with affine-covariant local detectors?

One could say that the goal of affine-covariant detectors like MSER, Hessian-Affine or Hessian-AffNet is to detect the same region on a planar surface, regardless the camera angle change. It is true to some extent, as we demostrate on toy example below with Hessian-Affine feature.

The problem arises, when the image content, e.g. 3 blobs on the figure below are situated close to each other, so under the tilt transform the merge into a single blob. So it is not the shape of region, which is detected incorrectly, but the center of the features themselves. For clarity, we omited affine shape estimation on the image below.

ASIFT: brute-force affine view synthesis

So, to solve the problem explained above, Morel & Yu proposed to do a lot affine warps of each image, as shown on the Figure below, as match each view against all others, which is complexity, where is number of views generated.

The motivation do doing so it that assuming, original image to be a fronto-parallel one, to cover viewsphere really dense, as shown in the Figure below.

This leads to impressive performance on a very challenging image pairs, see an example below

In this section I have used great illustrations done by Mariano Rodríguez for his paper “Fast Affine Invariant Image Matching” . Please, checkout his blog.

MODS: do as little as possible

The main drawback of ASIFT algorithm is a huge computational cost: 82 views are generated regardless of the image pair difficulty. To overcome this, we proposed MODS algorithm: Matching with On-Demand Synthesis.

One starts with the fastest detector-descriptor without view synthesys and then uses more and more computationally expensive methods if needed. Moreover, by using affine-covariant detectors like MSER or Hessian-Affine, one could synthetise significantly less views, saving computations spent on local descriptor and matching.

This, together with FGINN matching strategy, specifically designed for the handling re-detections, MODS is able to match more challenging image pairs in less time than ASIFT.

Why does affine synthesis help?

Despite that ASIFT and other view-synthesis based approaches are know more than decade, we are not aware of a study, why does affine synthesis helps in practice. Could one get a similar performance without view synthesis? Specificallly:

0. May it be that the most of improvements come from the fact that we have much more features? That is why we fix the number of features for all approaches.
1. Some regions from ASIFT, when reprojected to the original image, are quite narrow. Could we get them just by removing edge-like feature filtering, which is done in SIFT, Hessian and other detectors. Denoted +edge
2. Instead of doing affine view synthesis, one could directly use the same affine parameters to get the affine regions to describe, so the each keypoint would have several associated regions+descriptors. Denoted +MD
3. Using AffNet to directly estimated local affine shape without multiple descriptors. Denoted +AffNet
4. Combine (1), (2) and (3).

So, we did the study on HPatches Sequences dataset, the hardest image pairs (1-6) of viewpoint subset. The metric is similar to one used in the “Image Matching across Wide Baselines: From Paper to Practice” and CVPR 2020 RANSAC in 2020 - mean average accuracy of the estimated homography.

We run Hessian detector with RootSIFT descriptor, FLANN matching and LO-RANSAC, as implemented in MODS. Features are sorted according the the detector response and their total number is clipped to 2048 or 8000 to ensure that the improvements do not come from just having more features.

Note, that we do not study, if view synthesis helps for the regular image pairs - it might actually hurt performance, similarly to affine features. Instead we are focusing on the case, when view synthesis definitely helps: matching obscure views of the mostly planar scenes.

Summary

Affine view synthesis helps for matching challenging image pairs and its improvement are not just because of more local features used. It can be done effective and efficient – in the iterative MODS framework.

Making descriptor job easier

The most straightforward benefit of using local affine features is that they increase the repeatability of the detector and potentially reduce appearance changes of a local patch caused by viewpoint difference. This makes possible matching more challenging image pairs.

The practice is a little bit more complicated. Our recent benchmark, which measure the accuracy of the output fundamental matrix, shows that the difference in using affine and similarity-covariant features is quite minor. Specifically, the relative difference between SIFT vs SIFT-Affine features is 0.5% and between Hessian-SIFT and Hessian-AffNet SIFT is 5.1%, see Table below.

Therefore, if the benefit of local features would be to only improve descriptor extraction stage, it would be arguably not worth it. Luckily, there are more benefits, which are more pronounced.

Making RANSAC job easier

Let’s recall how RANSAC works.

1. Randomly sample a minimally required number of tentative correspondences to fit the geometrical model of the scene: 4 for homography, 7 for epipolar geometry and estimate the model.
2. Calculate “support”: other correspondeces, which are consistent with the model.
3. Repeat steps (1), (2) and output the model which is supported with the most of correspondences. If you were lucky and have sampled all-inlier sample, meaning that all correspondences used to estimate the model were correct, you would have a correct model.

Reality is more complicated than I have just described, but the principle is the same. The most important part is the sampling and it is sensitive to inlier ratio - the percentage of the correct correspondences in the set. Lets denote the minimal number of correspondences required to estimate the model as m. To recover the correct model with the confidence p one needs to sample the number of correspondences, which is described by formula:

Lets plot the how the number of required samples changes with inlier ratio for confidence equal 99%. Note the log scale on Y axis. Different lines are for different minimal sample size m.

As you can see from the plot above, reducing the minimal sample size required for the model estimation even by 1 saves and order of magnitude of computation. In reality the benefit is a smaller, as modern RANSACs like GC-RANSAC and MAGSAC could estimate the correct model from the sample containing outliers, but it is still huge, especially for low inlier rate cases.

Application-specific benefits

Besides the wide baseline stereo, local affine features and correspondences have other applications. I will briefly describe some of them here (to be updated).