Image feature point correspondence

Oct 17, 2014 at 2:56 AM
Hi Christoph,

Firstly, I would like to thank you for your effort and making this project opensouce!

For each 3D point in the point cloud, I was wondering if there was a way to which images it came from? So, what I can already do is that I can project a 3D point onto all images using the camera parameters. However, since not all images see this 3D point, some of these projections are wrong. Did I miss something obvious?

I would appreciate if you could throw any light on my question.

Thanks a lot in advance,
Best regards,
Arjun
Coordinator
Oct 17, 2014 at 10:17 AM
Arjun, this information exists although SynthExport currently does not make it available for export in any form. Can you elaborate a bit on what you intend to do with the data?
Oct 18, 2014 at 6:34 AM
Thank you very much for your email!! I am training a machine learning algorithm which would perform image feature point matching, and I want to use this as training data.

I am sure you are very busy, but can you maybe point me in code code or tell me a bit about how to do this? I will try to write some temporary code myself to this? I will be very grateful to you, and I am happy to acknowledge SynthExport/You in any resulting publication. Thanks a lot in advance!
Coordinator
Oct 18, 2014 at 10:17 AM
Edited Oct 18, 2014 at 10:19 AM
Thanks Arjun. Take a look at PointCloud.cs, in particular PointCloud.LoadBinFile. What you need is the data in the bin file that comes after the number of cameras and before the actual point cloud data. For each camera the corresponding points are encoded as a number of index ranges (offset and length in the code). The indices reference the points that follow in the file.

All values in the bin file are encoded in a compressed big-endian format, see BigEndianExtensions in PointCloud.cs for how that works. To get the bin file itself, the easiest way is probably to fetch it locally from %TEMP%\Photosynther while Photosynth is uploading the synth.
Oct 18, 2014 at 10:36 PM
Fantastic! Thank you! I should be able to do thanks.

To get the bin files for synths I create I can get it from the tmp Dir, can I get the same for others created also from the same tmp Dir while browsing synths?
Coordinator
Oct 19, 2014 at 10:22 AM
No, unfortunately it's more complicated. In order to get to the URLs of the bin files you will have to query the Photosynth web service at http://photosynth.net/photosynthws/PhotosynthService.asmx?op=GetCollectionData and download the files yourself. You pass the collection ID to GetCollectionData and use the returned CollectionRoot to build the URLs of the bin files.
Oct 23, 2014 at 5:18 AM
Thanks a lot Christoph. Correct me if I am wrong, but I think I dont need to create the url for the bin specifically, as this gets done when I click export, and I can simply add the additional things I want to save out.

About the offset/length in code, can you tell me how to use it? You also say they reference points which follow later in file. Is it immediately after the 3D point cloud XYZ and RGB? I have been trying to figure this out but I havent had success. If you could give me some rough pseudocode to add to the loadBinFile, it would be super helpful. I would understand if you do not have the time to be able to do that.
Oct 23, 2014 at 11:36 AM
So, like I said, when I export point clouds, I hit the function loadBinFile. And all I think I need to is figure how the correspondence data is organized in the .bin file and read it in variables there and later write it out the a file. BTW, I am super happy you have written this tool, thank you!
Coordinator
Oct 23, 2014 at 2:11 PM
Edited Oct 23, 2014 at 2:18 PM
stencilman wrote:
Correct me if I am wrong, but I think I dont need to create the url for the bin specifically, as this gets done when I click export, and I can simply add the additional things I want to save out.
Sorry, I misunderstood you.
stencilman wrote:
About the offset/length in code, can you tell me how to use it? You also say they reference points which follow later in file. Is it immediately after the 3D point cloud XYZ and RGB?
offset is an index into the collection of points that immediately follows in the bin file after all offsets/lengths. Those are the points that are described by XYZ and RGB. offset and length are currently simply skipped in the file, you will want to store them in some List<List<Tuple<int, int>>> pointsPerCamera or similar.
for (int i = 0; i < numCameras; i++)
{
    int numRanges = binaryReader.ReadCompressedInt();
    
    pointsPerCamera.Add(new List<Tuple<int, int>>());

    for (int j = 0; j < numRanges; j++)
    {
        int offset = binaryReader.ReadCompressedInt();
        int length = binaryReader.ReadCompressedInt();
        
        pointsPerCamera[i].Add(new Tuple<int, int>(offset, length));
    }
}
Then at the end of one of the ExportAs* methods you can write the ranges into a file, e.g.
using (StreamWriter streamWriter = new StreamWriter(path + ".pointspercamera"))
    for (int i = 0; i < pointsPerCamera.Length; i++)
        streamWriter.WriteLine("{0}: {1}", i, 
            string.Join(", ", pointsPerCamera[i].Select(range =>
                range.Item1 + "-" + (range.Item1 + range.Item2))));
This will give you a file with the structure
0: 0-5
1: 4-11, 14-20
i.e. points 0 to 5 belong to the first camera/photo, points 4 to 11 and 14 to 20 belong to the second camera, etc.

I haven't tested this out but I suspect there could be problems when you have multiple bin files to merge. The ranges in the bin files probably always start at zero, so when you merge bin files you will need to adjust the indices by + points.Count.

Hope this helps. Let me know if you have any more questions. I'm thinking about adding a feature like this to SynthExport in some form as part of the camera parameters, but probably not anytime soon.
Oct 23, 2014 at 6:19 PM
You are awesome Christoph! It works (a minor typo: pointsPerCamera.Length -> pointsPerCamera.Count). I havent taken care of the merge issue yet, but it gets me started with what I wanted. Thank you once again, I will keep you posted!
Nov 12, 2015 at 8:43 AM
I would also like to export common points but with XY for each image in image coordinates and resolution. This could be later used to stitch the photos together. Is it also possible?
Nov 12, 2015 at 10:28 AM
Format could be CSV or XML:

Sample XML:
<Photo>
                <Id>2</Id>
                <ImagePath>[fullpath]</ImagePath>                   
            </Photo>
<Photo>
                <Id>9</Id>
                <ImagePath>[fullpath]</ImagePath>                   
            </Photo>
<TiePoints>
        <TiePoint>
            <Name>User tie point #1</Name>
            <Type>User</Type>
            <Measurement>
                <PhotoId>2</PhotoId>
                <x>2061.8699999999999</x>
                <y>1939.8399999999999</y>
            </Measurement>
            <Measurement>
                <PhotoId>9</PhotoId>
                <x>1911.98</x>
                <y>1919.6700000000001</y>
            </Measurement>
            <Measurement>
                <PhotoId>0</PhotoId>
                <x>3109</x>
                <y>2303.2199999999998</y>
            </Measurement>
            <Measurement>
                <PhotoId>12</PhotoId>
                <x>2640.2600000000002</x>
                <y>1974.1099999999999</y>
            </Measurement>
        </TiePoint>
        <TiePoint>
            <Name>User tie point #2</Name>
            <Type>User</Type>
            <Measurement>
                <PhotoId>11</PhotoId>
                <x>3498.8699999999999</x>
                <y>1929.9400000000001</y>
            </Measurement>
            <Measurement>
                <PhotoId>9</PhotoId>
                <x>2844.7399999999998</x>
                <y>1942.0999999999999</y>
            </Measurement>
            <Measurement>
                <PhotoId>0</PhotoId>
                <x>3109.5999999999999</x>
                <y>2535.48</y>
            </Measurement>
        </TiePoint>                     
    </TiePoints>
Coordinator
Nov 12, 2015 at 2:58 PM
otoluk, I assume this is possible in theory but you'll probably need quite some heavy math to do the projections. This thread may be a good starting point. If you just need matching feature points between images, why not take software that implements SIFT or something similar and that can output the data you need directly? I am fairly sure you would get better results much faster than via Photosynth.
Nov 12, 2015 at 6:22 PM
Thanks for prompt answer. The idea is to get matching points which can be later used as additional control points in software I use (Acute3D) to reconstruct scene in 3D. It could speed up the reconstruction for incomplete historical greyscale image sets I have. Have not found better results than using Photosynth it matches more images than any other and also better handles camera properties.