Associative Video Memory - Frequently Asked Questions

Question: I noticed that algorithm is not rotation invariant, am I right?

Answer: Yes, you are right but object may be learned under different angles.

Question: Do you perform per pixel search of your template in the image, as well as in decreased size images, or you scale your template beforehand in order to obtain some scale invariance?

Answer: Image scan with 25% increase of scale on each step and start it from 75% of key image size (scaling: 75%, 100%, 125%, 150% ... until size biger than scan image).

Question: Unrecognized events are multiple. Is it possible to make only one event per stored object id per single recognition / tracking function call?

Answer: For single similarity estimation you can use "Read" method:


#include "stdafx.h"
#include "cv.h"
#include "highgui.h"
#include "AssociativeMemory.h"

void TestAVM()
{
  CvRect rect = cvRect(312, 312, 159, 159);
  CvRect rectFound;
  long id = 1;

  IplImage *img1 = cvLoadImage("4_i110.png", 0);
  IplImage *img2 = cvLoadImage("4_i120.png", 0);

  // Create
  CvAssociativeMemory32S *am = new CvAssociativeMemory32S;
  am->Create(cvSize(rect.width, rect.height));

  // Learn
  am->SetImage(img1);
  am->Write(rect, &id);

  // Recognize
  am->SetImage(img2);
  long* pId;
  double Similarity;
  if(am->Read(rect, &rectFound, &pId, NULL, NULL, &Similarity, true)) {
    printf("Object recognized, rectFound(%d, %d, %d, %d)\n",
      rectFound.x, rectFound.y, rectFound.width, rectFound.height);
  }
  printf("Id = %lu, Similarity = %f or %lu\n", *pId, Similarity, long(Similarity*10000));

  cvReleaseImage(&img1);
  cvReleaseImage(&img2);
  delete am;
}

int _tmain(int argc, _TCHAR* argv[])
{
  TestAVM();
  return 0;
}

Question: We have the following problem: when associative memory is stored and then is loaded similarity is always zero.


  ................

  // Learn and save
  am->SetImage(img1);
  am->Write(rect, &id);
  am->Save("test.model");

  ................

Answer: It happens because "Write" method was called only once. Associative tree optimization was performed before saving and it removed recognition data from associative tree because must be more three hitting into associative base. If number of hitting less then associative base remove during optimization process.


I modified interface of "Save" method:

  // Saving of recognition data
  bool Save(char* aFileName, bool aSaveWithoutOptimization = false);


The program works properly in this case:
  ..................
  // Learn and save
  am->SetImage(img1);
  am->Write(rect, &id);
  // Save without optimization
  am->Save("test.model", true);
  ..................

Question: If interest rectangle in the Write method has different aspect ratio from the key image rectangle, what happens, does it changes aspect ratio of input rectangle to fit the key image rectangle, and how this behavior affects recognition?

Answer: The key image size of associative video memory and size of interest area for "Write" method should be equal for effective learning process and it is necessary that the aspect ratio must be the same as in key image size otherwise it not will be able to give effective learning with different aspect ratio. For recognition in "Read" method the interest area can has any scaling (it is desirable from 50% and higher) and any aspect ratio.

Question: But in this case there will be difficulties with learning objects of different sizes as it is in the Amsterdam database. Of course I can use the biggest rectangle found between the training images, but how it will affects recognition quality of small objects?

Answer: I propose another method for evaluation associative video memory on image data base. The strategy conclude in fixing of key image size and using different interest area sizes for "Write" and "Read" method (interest area size for write/read is identical). You can use image key size parameter as tool for control of recognition capacity. The key image size and recognition accuracy is depended as the biger, the accurater. But a key image size must be less than an interest area size.


The key image size is depended on sequence 40, 80, 160, 320, 640, 1280... 2^n*10.
Use similar combinations for aspect ratio control:
80x80 - 1:1; 320x320 - 1:1; 80x160 - 1:2; 640x160 - 4:1; ...

Image key size Associative levels
80x803
160x1604
320x3205
640x6406
......
80x1603
640x1604
......

Object number Interest area size Optimal image key size
4314x327320x320
15500x380
18600x500
44500x380


Command line example for "Similarity test" application:

SimilarityTest.exe -DBPath(../../TestPhoto) -Obj(4) -Rect(223, 211, 314, 327)
SimilarityTest.exe -DBPath(../../TestPhoto) -Obj(15) -Rect(100,150,500,380)
SimilarityTest.exe -DBPath(../../TestPhoto) -Obj(18) -Rect(100, 50, 600, 500)
SimilarityTest.exe -DBPath(../../TestPhoto) -Obj(44) -Rect(100,150,500,380)

Modification in SimilarityTest.cpp file:

Previous text version:
  .....................
  sprintf(OutStr, "Object number: %lu, image key size: %dx%d pixels",
     ObjNumber, InterestArea.width, InterestArea.height);
  .....................
  CvAssociativeMemory32S* am = new CvAssociativeMemory32S;
  am->Create(cvSize(InterestArea.width, InterestArea.height));
  .....................

Text after modification:

  .....................
  sprintf(OutStr, "Object number: %lu, interest area size: %dx%d",
     ObjNumber, InterestArea.width, InterestArea.height);
  .....................
  CvAssociativeMemory32S* am = new CvAssociativeMemory32S;
  am->Create(cvSize(320, 320)); // Set a fixed key size
  .....................

I made some diagrams for comparing:

<- key size 640x640; <- key size 320x320;
<- key size 160x160; <- key size 80x80.

Question: I set the same quadratic region for several different object, however they are not recognized. Could you check what is the problem.


#include "stdafx.h"
#include "cv.h"
#include "highgui.h"
#include "AssociativeMemory.h"

void TestAVM()
{
  CvRect rect = cvRect(223, 211, 315, 315);
  CvRect rectFound;
  long id;

  IplImage *img1 = cvLoadImage("4_i110.png", 0);
  IplImage *img2 = cvLoadImage("4_i120.png", 0);

  IplImage *img1a = cvLoadImage("32_i110.png", 0);
  IplImage *img2a = cvLoadImage("32_i120.png", 0);

  IplImage *img1b = cvLoadImage("35_i110.png", 0);
  IplImage *img2b = cvLoadImage("35_i120.png", 0);


  // Create
  CvAssociativeMemory32S *am = new CvAssociativeMemory32S;
  am->Create(cvSize(315, 315));

  // Learn
  am->SetImage(img1);
  id = 0;
  am->Write(rect, &id);

  am->SetImage(img1a);
  id = 1;
  am->Write(rect, &id);

  am->SetImage(img1b);
  id = 2;
  am->Write(rect, &id);

  am->Save("test.model", true);

  // Destroy recreate and load
  delete am;
  am = new CvAssociativeMemory32S;
  bool res;
  if((res = am->Load("test.model"))) {
   printf("\nLoad associative bases from file: %s\n"
    "Associative bases ......... %-7lu\n"
    "Associative Levels ........ %-7u\n"
    "Current base index ........ %-7I64u\n"
    "Wr/Rd counter ............. %-7I64u\n\n",
    "test.model", am->GetTotalABases(),
    am->GetTotalLevels(), am->GetCurIndex(), am->GetWrRdCounter());
  }

  // Recognize

  am->SetImage(img2);
  long* pId;
  double Similarity;
  rect = cvRect(0, 0, img2->width, img2->height);
  if(am->Read(rect, &rectFound, &pId, NULL, NULL, &Similarity, true)) {
   printf("Object recognized, rectFound(%d, %d, %d, %d)\n",
    rectFound.x, rectFound.y, rectFound.width, rectFound.height);
  }
  printf("Id = %lu, Similarity = %f or %lu\n", *pId, Similarity, long(Similarity*10000));

  am->SetImage(img2a);
  rect = cvRect(0, 0, img2a->width, img2a->height);
  if(am->Read(rect, &rectFound, &pId, NULL, NULL, &Similarity, true)) {
   printf("Object recognized, rectFound(%d, %d, %d, %d)\n",
    rectFound.x, rectFound.y, rectFound.width, rectFound.height);
  }
  printf("Id = %lu, Similarity = %f or %lu\n", *pId, Similarity, long(Similarity*10000));

  am->SetImage(img2b);
  rect = cvRect(0, 0, img2a->width, img2a->height);
  if(am->Read(rect, &rectFound, &pId, NULL, NULL, &Similarity, true)) {
   printf("Object recognized, rectFound(%d, %d, %d, %d)\n",
    rectFound.x, rectFound.y, rectFound.width, rectFound.height);
  }
  printf("Id = %lu, Similarity = %f or %lu\n", *pId, Similarity, long(Similarity*10000));

  cvReleaseImage(&img1);
  cvReleaseImage(&img2);
  cvReleaseImage(&img1a);
  cvReleaseImage(&img2a);
  cvReleaseImage(&img1b);
  cvReleaseImage(&img2b);
  delete am;
}

int _tmain(int argc, _TCHAR* argv[])
{
  TestAVM();
  return 0;
}

Answer: The Interest area for Write/Read methods is used as key for data access. It is low level operation and "Read" method can give answer: Is it an object in interest area or not? But for searching in big image it must be performed by a multiple calling of "Read" method with different interest area value (or by calling "ObjectRecognition" method). I modified program text and now it works correctly.


#include "stdafx.h"
#include "cv.h"
#include "highgui.h"
#include "AssociativeMemory.h"

void doSearchObject(CvAM_State aState, long* apData, CvAM_Parameters* apParam, void* args)
{
  switch (aState)
  {

    case cProcessingIsBegun:
      printf("== Begin ==\n");
      break;

    case cUnknownObject:
      //printf("Unknown, Id = %lu, Similarity = %ld\n", *apData, long(apParam->Similarity*10000));
      break;

    case cRecognizedObject:
      printf("Recognized, Id = %lu, Similarity = %ld, rectFound(%d, %d, %d, %d)\n",
       *apData, long(apParam->Similarity*10000),
       apParam->ObjRect.x, apParam->ObjRect.y,
       apParam->ObjRect.width, apParam->ObjRect.height);
      break;

    case cGeneralizedObject:
    case cTrackedObject:
      printf("Tracked/Generalized, Id = %lu, Similarity = %ld, rectFound(%d, %d, %d, %d)\n",
       *apData, long(apParam->Similarity*10000),
       apParam->ObjRect.x, apParam->ObjRect.y,
       apParam->ObjRect.width, apParam->ObjRect.height);
      break;

    case cLearnThisObject:
      printf("Learning\n");
      break;

    case cProcessingIsFinished:
      printf("-- Finish --\n");
      break;
  }
}

void SimplyRecognition(CvAssociativeMemory32S* pAM, CvSize aInputImgSize, CvRcgAMFunc32S RcgAMFunc) {

  if(pAM && RcgAMFunc) {
    CvRect sRect;int i;
    printf("<***> Simply recognition -> begin\n");
    for(i=0;;i++) {
      float Rate = 0.75f + 0.25f*float(i);
      int SearchStep = int(15.0f*Rate);
      CvSize KeyImgSize = pAM->GetKeyImageSize();
      sRect.width = int(KeyImgSize.width*Rate);
      sRect.height = int(KeyImgSize.height*Rate);
      if((aInputImgSize.width < sRect.width) || (aInputImgSize.height < sRect.height)) break;
      int eX = aInputImgSize.width - sRect.width;
      int eY = aInputImgSize.height - sRect.height;
      for(int y=0; y < eY; y += SearchStep) {
        for(int x=0; x < eX; x += SearchStep) {
          sRect.x = x;
          sRect.y = y;
          long* pData;
          CvAM_Parameters Param;
          Param.ppUserTrackingInfo = NULL;
          Param.TotalABases = pAM->GetTotalABases();
          if(pAM->Read(sRect, &Param.ObjRect, &pData, NULL, NULL, &Param.Similarity)) {
            RcgAMFunc(cRecognizedObject, pData, &Param, NULL);
          }
        }
      }
    }
    printf("<***> Simply recognition -> finish\n\n");
  }
}

void TestAVM()
{
  CvRect rect = cvRect(223, 211, 315, 315);
  CvRect rectFound;
  long id;

  IplImage *img1 = cvLoadImage("4_i110.png", 0);
  IplImage *img2 = cvLoadImage("4_i120.png", 0);

  IplImage *img1a = cvLoadImage("32_i110.png", 0);
  IplImage *img2a = cvLoadImage("32_i120.png", 0);

  IplImage *img1b = cvLoadImage("35_i110.png", 0);
  IplImage *img2b = cvLoadImage("35_i120.png", 0);


  // Create
  CvAssociativeMemory32S *am = new CvAssociativeMemory32S;
  am->Create(cvSize(315, 315));

  // Learn
  am->SetImage(img1);
  id = 0;
  am->Write(rect, &id);

  am->SetImage(img1a);
  id = 1;
  am->Write(rect, &id);

  am->SetImage(img1b);
  id = 2;
  am->Write(rect, &id);

  am->Save("test.model", true);

  // Destroy recreate and load
  delete am;
  am = new CvAssociativeMemory32S;
  bool res;
  if((res = am->Load("test.model"))) {
   printf("\nLoad associative bases from file: %s\n"
    "Associative bases ......... %-7lu\n"
    "Associative Levels ........ %-7u\n"
    "Current base index ........ %-7I64u\n"
    "Wr/Rd counter ............. %-7I64u\n\n",
    "test.model", am->GetTotalABases(),
    am->GetTotalLevels(), am->GetCurIndex(), am->GetWrRdCounter());
  }

  // Recognize
  printf("Interest area rect(%d, %d, %d, %d)\n\n", rect.x, rect.y, rect.width, rect.height);
  am->SetImage(img2);
  long* pId;
  double Similarity;
  //!!! rect = cvRect(0, 0, img2->width, img2->height);
  if(am->Read(rect, &rectFound, &pId, NULL, NULL, &Similarity, true)) {
   printf("Object recognized, rectFound(%d, %d, %d, %d)\n",
    rectFound.x, rectFound.y, rectFound.width, rectFound.height);
  }
  printf("Id = %lu, Similarity = %f or %lu\n", *pId, Similarity, long(Similarity*10000));
  am->ObjectRecognition(doSearchObject , 0);
  SimplyRecognition(am, cvSize(img2->width, img2->height), doSearchObject);

  am->SetImage(img2a);
  //!!! rect = cvRect(0, 0, img2a->width, img2a->height);
  if(am->Read(rect, &rectFound, &pId, NULL, NULL, &Similarity, true)) {
   printf("Object recognized, rectFound(%d, %d, %d, %d)\n",
    rectFound.x, rectFound.y, rectFound.width, rectFound.height);
  }
  printf("Id = %lu, Similarity = %f or %lu\n", *pId, Similarity, long(Similarity*10000));
  am->ObjectRecognition(doSearchObject , 0);
  SimplyRecognition(am, cvSize(img2a->width, img2a->height), doSearchObject);

  am->SetImage(img2b);
  ///!!! rect = cvRect(0, 0, img2b->width, img2b->height);
  if(am->Read(rect, &rectFound, &pId, NULL, NULL, &Similarity, true)) {
   printf("Object recognized, rectFound(%d, %d, %d, %d)\n",
    rectFound.x, rectFound.y, rectFound.width, rectFound.height);
  }
  printf("Id = %lu, Similarity = %f or %lu\n", *pId, Similarity, long(Similarity*10000));
  am->ObjectRecognition(doSearchObject , 0);
  SimplyRecognition(am, cvSize(img2b->width, img2b->height), doSearchObject);

  cvReleaseImage(&img1);
  cvReleaseImage(&img2);
  cvReleaseImage(&img1a);
  cvReleaseImage(&img2a);
  cvReleaseImage(&img1b);
  cvReleaseImage(&img2b);
  delete am;
}

int _tmain(int argc, _TCHAR* argv[])
{
  TestAVM();
  return 0;
}
=====================================================================
Results:

Load associative bases from file: test.model
Associative bases ......... 12
Associative Levels ........ 4
Current base index ........ 12
Wr/Rd counter ............. 3

Interest area rect(223, 211, 315, 315)

Object recognized, rectFound(254, 240, 259, 259)
Id = 0, Similarity = 0.680402 or 6804
== Begin ==
Recognized, Id = 0, Similarity = 6630, rectFound(251, 239, 259, 259)
Tracked/Generalized, Id = 0, Similarity = 6630, rectFound(251, 239, 259, 259)
-- Finish --
<***> Simply recognition -> begin
Recognized, Id = 0, Similarity = 6402, rectFound(250, 240, 259, 259)
Recognized, Id = 0, Similarity = 6402, rectFound(255, 240, 259, 259)
Recognized, Id = 0, Similarity = 6630, rectFound(251, 239, 259, 259)
Recognized, Id = 0, Similarity = 6630, rectFound(256, 239, 259, 259)
Recognized, Id = 0, Similarity = 7236, rectFound(253, 240, 259, 259)
Recognized, Id = 0, Similarity = 7475, rectFound(256, 240, 259, 259)
Recognized, Id = 0, Similarity = 7861, rectFound(255, 240, 259, 259)
<***> Simply recognition -> finish

Object recognized, rectFound(254, 240, 259, 259)
Id = 1, Similarity = 0.673710 or 6737
== Begin ==
Recognized, Id = 1, Similarity = 6746, rectFound(251, 239, 259, 259)
Recognized, Id = 1, Similarity = 7351, rectFound(255, 239, 259, 259)
Tracked/Generalized, Id = 1, Similarity = 7048, rectFound(253, 239, 259, 259)
-- Finish --
<***> Simply recognition -> begin
Recognized, Id = 1, Similarity = 6437, rectFound(255, 240, 259, 259)
Recognized, Id = 1, Similarity = 6746, rectFound(251, 239, 259, 259)
Recognized, Id = 1, Similarity = 6746, rectFound(256, 239, 259, 259)
Recognized, Id = 1, Similarity = 7351, rectFound(255, 239, 259, 259)
Recognized, Id = 1, Similarity = 7445, rectFound(253, 240, 259, 259)
Recognized, Id = 1, Similarity = 6862, rectFound(252, 240, 259, 259)
Recognized, Id = 1, Similarity = 7702, rectFound(255, 240, 259, 259)
<***> Simply recognition -> finish

Id = 2, Similarity = 0.412393 or 4123
== Begin ==
Recognized, Id = 2, Similarity = 6666, rectFound(277, 239, 259, 259)
Tracked/Generalized, Id = 2, Similarity = 6666, rectFound(277, 239, 259, 259)
-- Finish --
<***> Simply recognition -> begin
Recognized, Id = 2, Similarity = 6970, rectFound(256, 239, 259, 259)
Recognized, Id = 2, Similarity = 6666, rectFound(277, 239, 259, 259)
<***> Simply recognition -> finish

Hosted by uCoz