Frequently Asked Questions

The desired method of communications with FRVT 2006 management is via this FAQ page.

All pertinent questions regarding the FRVT 2006 will be posted on this website. This policy ensures that all potential participants are guaranteed equal access to information concerning FRVT 2006. Exceptions will only be allowed for extenuating circumstances.

If you have any questions that have not already been answered in this section, please send your complete question to the FRVT 2006 Liaison

1.     What is the difference between the FRGC and the FRVT 2006?

A: The Face Recognition Grand Challenge (FRGC) is a still and 3D face recognition algorithm development project. It consists of a series of challenge problems that are progressively more difficult in nature. The main goal of the FRGC is to improve the performance of face recognition systems by an order of magnitude over the Face Recognition Vendor Test (FRVT) 2002. The FRGC began in May 2004 and is scheduled for completion in March 2006. The FRVT 2006 is an independent government evaluation of face recognition systems. The main goal of the FRVT 2006 is to measure performance progress since FRVT 2002 and to determine if the FRGC met its goal of improving performance by an order of magnitude. The FRVT 2006 began on 30 January 2006.

2.     Is the FRVT 2005 the same evaluation as the FRVT 2006?

A: Yes. The original start date for the FRVT 2005 was in September/October 2005. The start date was changed to 30 January 2006. Therefore, the name was changed to FRVT 2006. However, FRVT 2005 and FRVT 2006 refer to the same evaluation.

3.     If I registered to receive FRVT 2006 announcements, am I automatically registered to participate in the FRVT 2006 evaluation?

A: No. So far, organizations have only been able to register to receive information through FRVT 2005/6 announcements. There will be a separate registration process for actual participation in the FRVT 2006 evaluation beginning in early November 2005. Potential participants will need to sign a form agreeing to the rules for FRVT 2006 and sign-up for particular evaluation tasks and tracks. Everyone who wants to participate in the FRVT 2006 evaluation will have to complete this second registration process. Instructions for this second registration process are forthcoming in a future FRVT 2006 announcement.

4.     Do the organizations that have already registered to receive FTVT 2005 announcements have to re-register to receive FRVT 2006 announcements?

A: No. If you registered to receive FRVT 2005 announcements, we automatically transferred your registration for announcements to FRVT 2006. You will, however, still need to follow the instructions for participating in the FRVT 2006 evaluation. Those instructions will be distributed through an FRVT 2006 announcement by the end of October 2005. Registration will begin in early November 2005.

5.     What is the Biometric Experimentation Environment (BEE)?

A: The BEE provides the infrastructure for the FRVT 2006. It allows the participant to focus on the evaluation by simplifying test data management, evaluation configuration, and processing of results. A set of experiments will be specified by the government for the evaluation.

6.     Who is authorized to sign the data and software (BEE) licenses?

A: The appropriate person to execute the license is generally a lawyer in the organization's legal office. The key attribute of the person is that they be authorized to bind the organization legally. For example, faculty members at universities are generally not authorized to legally bind the university to a contract. Students and post-docs are definitely not able to do that. Likewise, most companies have legal offices/representatives who would perform the function.

7.     Is this the first time face recognition systems have been tested by the government?

A: The Face Recognition Vendor Test (FRVT) 2006 is the latest in a series of tests for face recognition systems. Previous tests were FERET, FRVT2000, and FRVT 2002 (see www.frvt.org).

8.     When will the FRVT 2006 take place?

A: The FRVT 2006 evaluation began on 30 January 2006. The evaluation report is in the review process and is due to be released in March 2007. When it is released, it will be posted on this website.

9.     What kind of data will be used in the FRVT 2006?

A: A standard dataset and test methodology are employed so that all participants are evenly evaluated. The government provides to participants both the test data and the extensive test environment. This environment is called the BEE or the Biometric Experimentation Environment.

10. Why is the FRVT 2006 being conducted?

A: The US Government needs unbiased and impartial R&D progress assessments and T&E of face recognition algorithms and prototype systems. These assessments and evaluations help the Government determine if these algorithms/systems can meet current and future requirements of Government agencies.

11. What is the goal of the FRVT 2006?

A: The goal of the FRVT 2006 is to measure state-of-the-art prototype systems/algorithms and commercial face recognition systems.

12. When the FRVT 2006 gets underway on 30 January 2006, where will the participant's test and evaluation efforts be located - at NIST in Washington, DC or at the participant's facilities?

A: FRVT 2006 will be an executable test. Participants will deliver their executables to NIST for testing. Since FRVT 2006 is an executable test, participants will not be present when the test is administered.

13. Are institutes outside the U.S. allowed to participate in the FRVT 2006?

A: Yes. Face recognition technology researchers and developers from companies, research institutions, and academia, both inside and outside the U.S., are eligible to participate.

14. I am not receiving FRVT 2006 updates even though I registered on this web site to receive them. Did I get dropped from the list?

A: Several people registered using a non-business email address (i.e., hotmail, charter.net, aol, etc.). We cannot send FRVT 2006 updates to non-business email addresses unless special arrangements have been made. If you have already registered and are not receiving FRVT 2006 updates, please send a message to frvt2006@nist.gov. If you have not registered, and would like to receive FRVT 2006 updates, please register with your business/organization email address.

15. What is the FRGC Supplemental Data Set and how can i gain access to it?

A: The FRGC supplemental data set contains additional facial images collected at the University of Notre Dame, including 3D images from a Qlonerator sensor. Participants will only be given access to the FRGC supplemental data set after they have 1) registered for FRGC (see FRGC website for more information at www.frvt.org/frgc), 2) been approved for access to FRGC v1.0 data and submitted results for at least one experiment, and 3) been approved for access to FRGC v2.0 data. Requests for FRGC Supplemental data prior to participants being approved for FRGC v2.0 data will be denied.

16. Can a participant's results remain anonymous?

A: No. All similarity matrices and performance scores submitted to NIST become the property of NIST. NIST does not conduct anonymous challenge problems and evaluations. NIST will, at its discretion, report attributed performance. If initial reports do not contain labeled or completely labeled results, this does not imply a participant's performance will not be labeled in future reports.

17. Why is there no score normalization option for the 1-Many Matching task? (Score normalization can improve identification performance provided the score for every pair of gallery signatures is available, in addition to the scores between the query signature and all gallery signatures.)

A: In similarity score normalization, the input is a set of similarity scores s(G,q) between a Gallery G a query q. The output is a new set of similarity scores s’(G,q). The new set of similarity scores is function of only the original set of similarity scores s(G,q). In the 1-many matching task, the input is a gallery G and a query q biometric sample; the output is a set of similarity scores s(G,q). In 1-many matching, computing the set of similarity scores is a function of a gallery G and a query q. By comparison, in the 1-1 matching task, a similarity score s(t,q) between a target t and query q is solely the function of a target t and query q. By the definition of 1-many matching, similarity score normalization can be included in the executable.

18. Can participants supply their own computer to take FRVT 2006?

A: No. FRVT 2006 participants have to provide executables that run on the FRVT 2006 computers at NIST.

19. Can you please provided additional information about controlled and uncontrolled still images.

A: Controlled stills are facial images nominally taken under studio lighting. Representative examples are all the images in the FERET dataset and FRGC images from Experiment 1. Uncontrolled stills are images that are not taken under studio lighting. Uncontrolled stills maybe taken in hallways and atriums with ambient lighting, and images taken outside.

20. How can we know if our 3D recognition system will work on the Minolta and Qlonerator data if sample data is not available?

A: Examples of the Minolta data is provided in the Face Recognition Grand Challenge (FRGC) versions 1 and 2, and FRGC supplemental data set. Examples of the Qlonerator data are available in the FRGC supplemental data set. Formal participation in FRGC is not required to obtain FRGC version 1. See http://frvt.org/FRGC for information on obtaining FRGC version 1. The formal procedure for gaining access to the FRGC supplemental data set requires participants to qualify for access to FRGC version 2. However, if potential FRVT 2006 participants, who are not formally participating in FRGC, would like to have access to the FRGC supplemental data set, they must first be granted access to FRGC version 1. If you have already been granted access to FRGC version 1 and would like to receive the FRGC Supplemental data set, please correspond with the FRVT 2006 liaison at frvt2006@nist.gov.

21. How do I gain access to Face Recognition Grand Challenge (FRGC) version 1 data without formally participating in FRGC?

A: Follow the procedure described on http://frvt.org/FRGC for gaining access to FRGC ver1.0a data. When access to FRGC ver1.0a is granted your organization is listed as a FRGC participant; however, there is not a requirement to submit FRGC results or attend FRGC meetings.

22. How do I gain access to the electronic bulletin board at http://bbs.bee-biometrics.org/

A: All FRGC participants can gain access to the electronic bulletin board. However, the FRGC registration policy will need to be followed (as stated on http://www.frvt.org/frgc).

23. What are the specifications for the computers that will run the executables?

A: The computers are Dell PowerEdge 850 servers with a single Intel Pentium 4 processor 660 at 3.6GHz and 2MB of 800Mhz cache. All systems have 4GB of DDR2 Ram at 533MHz.

24. What operating systems are supported by FRVT 2006?

A: As of 1 December 2005, Windows Server 2003 and Fedora 3.0 will be supported. Windows Server 2003 was chosen because WindowsXP is not supported by Dell on PowerEdge 850 servers. Experience by the FRVT 2006 evaluation team is that the Biometric Experiment Environment (BEE) and baseline PCA algorithm have been migrated from WindowsXP to Windows Server 2003 without any problems.

25. How does a participant sign up for multiple algorithms per task?

A: Send an e-mail to the FRVT 2006 liaison at frvt2006@nist.gov requesting to submit multiple algorithms per task.

26. Please elaborate on the meaning of "fully automatic" versus "partially automatic."

A: In "fully automatic" FRVT 2006 tasks, NO meta-data on the location of any facial features will be provided. In "partially automatic" FRVT 2006, meta-data for the location of the eyes is provided. For still images the location is the centers of the eyes. The locations of the eyes are manually marked by human operators, and reflect the accuracy of humans.

27. How does a participant obtain a copy of the Biometric Experimentation Environment (BEE)?

A: The BEE is included in the Face Recognition Grand Challenge (FRGC) v1.0a. See FAQ #21 about signing up to participant in FRGC.

28. What will be the format for the still images?

A: The format will be jpeg.

29. Will there be a test-run for submitted executables before the start of the actual test?

A: There will be a conformance test for submitted executables before the start of the actual test. The items the conformance test will check includes (but not limited to): 1) outputs are in the correct format, 2) the correct output was produced, and 3) the independence rules are followed. For each executable submitted, participants will be asked to submit the executables output on a specified set biometric signatures. The FRVT 2006 test team will then verify that the same answer was produced by the executable installed in the FRVT 2006 test facility.

30. Will multiple executables run at a time on a single computer?

A: Only a single executable performing a single experimental trail will run at a time on a computer.

31. What edition of Window 2003 system will be used?

A: Windows 2003 server standard edition will be used.

32. The preprocessing task description talks about "image set [in]" and one "image [out]". Is this indeed the case, or is it a typo, and the preprocessing task is N images in, N preprocessed images out?

A: This answer will clarify the protocol. For the preprocessing task, the input will be 1 image; the output is 1 image.

33. What is the maximum size for a target and query set for an experimental run in FRVT 2006?

A: The maximum number of biometric samples in a target or query set is 16,384. Therefore the largest similarity matrix that will be produced is 16,384 by 16,384. Participants should note that the number of biometric samples does not necessarily equal the number of images. A biometric sample can contain multiple images. One example is the Minolta 3D data that consists of both shape and texture channels. Another example is Experiment 2 from the Face Recognition Grand Challenge where each biometric sample consisted of four images.

34. Will biometrics samples in a target or query set always contain the same number of images?

A: No, biometrics samples in a target or query set could contain different number of images.

35. What will the image sizes be in FRVT 2006?

A: Image sizes will vary between a minimum of 150 by 100 pixels to a maximum of 3008 by 2000 pixels. The images can either be in portrait or landscape mode.

36. What is the range in the size of the face in the still images?

A: What is important is not the absolute size of the face, but rather the ratio of the size of the image to the size of the face. For still images, the size of the face will be measured in the number of pixels between the centers of the eyes. For calculations to this answer, the size of the image will be measured by the smaller of the two dimensions. For controlled still images, the largest faces could have a size ratio of 2.0, and the smallest could have a size ratio of 10.0. For uncontrolled still images, the largest faces could have a size ratio of 4.0, and the smallest could have a size ratio of 20.0.

37. What is the range of roll (in plane rotation) of the faces?

A: The images can be in either landscape or portrait mode. The roll (in plane rotation) is measured as the angle between the line through the centers of the eyes and the x-axis (horizontal axis). For controlled images, the mean angle is approximately 0 degrees, standard deviation of approximately 3.0 degrees, and limits of rotation of (approximately) plus/minus 20 degrees. For uncontrolled images, the mean angle is approximately 0 degrees, standard deviation of approximately 4.0 degrees, and limits of rotation of (approximately) plus/minus 20 degrees.

38. Your Windows server was specified with 4GBytes of memory. How much of this will be available to the test application? The default setting for 32bit windows is to assign a maximum of 2GBytes memory to a single application at a given time. This is extendable to 3GBytes but requires a configuration change.

A: The FRVT 2006 test system will be configured to allow for 3GBytes of memory to be assigned to a single process. Participants who plan to take advantage of the 3GGBytes of memory will need to compile their code to take advantage of this extra memory.

39. What libraries will be provided on the FRVT 2006 test system? Will standard libraries such as xalanc and xercesc be provided?

A: No libraries will be provided. All libraries that participants require will either need to be provided separately in the participant directory structure or incorporated into the executable.

40. Key information about experiments in FRVT 2006 is contained in xml description files. Do participants need to write routines to parse the description files?

A: Part of the Biometric Experimentation Environment (BEE) is a C++ class called XPathXMLParser.cpp. This class should make it easy for a participant to write a class or function to easily extract necessary information from the xml description files. On the FRGC/ICE bbs I will post an example for using XPathXMLParser.cpp to read a xml parameter file. In the BEE distribution, there is also a java version of XPathXMLParser.cpp.

41. Are there any fees associated with the FRVT or FRGC programs?

A: NIST does not charge a fee to participate in the FRVT 2006.

42. How will image quality results be reported?

A: Results will be reported using the Image Quality Receiver Operator Characteristic (IQROC) and Linear, Generalized Linear, and Generalized Linear Mixed Model frameworks. Information on IQROC can found in the ICE announcement section on bbs.bee-biometric.org website, in the program manager presentation at the first Iris Challenge Evaluation (ICE) workshop. For information on applying Linear, Generalized Linear, and Generalized Linear Mixed Model to face recognition see the following two papers: “How features of the human face affect recognition: a statistical comparison of three face recognition algorithms,” G. Givens, J. R, Beveridge, B. A. Draper, P. Grother, and P. J. Phillips, In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), Volume 2, 2004 Page(s):II-381 - II-388. “Repeated Measures GLMM Estimation of Subject-Related and False Positive Threshold Effects on Human Face Verification Performance,” G. H. Givens, J. R. Beveridge, B. A. Draper, and P. J. Phillips, Workshop on Empirical Evaluation Methods in Computer Vision in conjunction with CVPR 2005. Copies of these two papers are posted on the FRGC/ICE bbs under FRGC v2 Announcements.

43. What are the ranges of poses in the still images?

A: A pose flag will be added to experiment description file. Specifications will follow. If the pose flag is set to frontal, then all still images will nominally be frontal. If the pose flag is set to non-frontal, then the limits of right/left rotation are approximately 45 degrees left and right, and the limits of up/down rotations are normal head range. Combination of left/right and up/down are possible.

44. Your previous answer of face size in still images was expressed as a ratio of face size to image size. Can you please provide absolute numbers for the face size?

A: For controlled still images, the minimum size face will be 10 pixels, and the maximum will be 600 pixels. For uncontrolled still images, the minimum size face will be 10 pixels, and the maximum size face will be 350 pixels. The size of a face is measured as the number of pixels between the centers of the eyes.

45. What are the time constraints for FRVT 2006?

A: Participants are advised to follow the FRVT 2006 timing guidelines. Results for key experiments in FRVT 2006 will include execution times. Scheduling of FRVT 2006 executables will be based on completing the benchmark experiment in 36 hours on one of the processors/systems in the FRVT 2006 test facility. Executables must be able to complete the benchmark experiment with 72 hours. For still images, the benchmark experiment will be equivalent to the Face Recognition Grand Challenge (FRGC) ver2.0 Experiment 1. In the FRGC ver2.0 Experiment, the target and query set consisted of 16,028 still images of size 1704 X 2272 pixels. Execution time is from the start of an experiment through completion of an experiment. For example, for recognition this includes template generation, matching, and producing a complete similarity matrix. If running time is a critical factor, then participants can submit multiple executables with different running times. To request to submit multiple executables for a task, please contact the FRVT 2006 liaison at frvt2006@nist.gov. To aid in diagnosing the cause of an executable taking a long time to complete an experiment, it is recommended that executables write a log to standard out. Logical points to write execution information are: 1) after generating a template and 2) after completing all the matches between a query biometric sample and all the target biometric samples.

46. How much disk space may we use in the "temp" directory?

A: Participants will be given access to a 75Gbyte partition for the directory structure described in the FRVT 2006 protocol. The directory structure includes a “temp” directory.

47. What is the maximum number of three-dimensional (3D) biometric samples in a target or a query set?

A: The maximum number of biometric samples in a target or query set is 16,384. Therefore the largest similarity matrix that will be produced is 16,384 by 16,384. Participants should note that the number of biometric samples does not necessarily equal the number of images. A biometric sample can contain multiple images. One example is the Minolta 3D data that consists of both shape and texture channels. Another example is Experiment 2 from the Face Recognition Grand Challenge where each biometric sample consisted of four images.

48. Is it conceivable that a 3D sig-set will be presented to a still-only executable?

A: Only still images will be presented to still-only executables.

49. In the all-duplicate detection task, will the parameter file contain the max_recordings attribute?

A: In the all-duplicate detection task, the parameter file will contain the max_recording attribute. In the all-duplicate task, the sig-set parameters for the experiment will be contained in the Target element. (Note: in the all-duplicate detection task, there is only one sig-set.)

50. In reference to the similarity file header, you indicate that certain strings are to be terminated with 'eol'. However, the encoding conventions of 'eol' is platform dependent: In Unix systems, it is a single character '\n' (). In Windows text files or text streams, it is represented by two sequential characters '\r' '\n' (). Can you specify the exact character(s) to be used in similarity files as eol-separators?

A: The header in the similarity file is read as text and the similarity scores are read as binary. Since the header information is read as text, the FRVT score code can handle both Unix and Windows text files. I will post the C++ classes for reading and writing similarity files on bbs.bee-biometrics.org. I will also include an example of using the classes to write a similarity file [posted Mon 9 January 2006]. Code in post "Btoolsmatrix code examples for FRVT 2006" under FRGC v2 Announcements [11 Jan 06]

51. The example of a signature set (Figure C) in the FRVT test protocol document suggests, that a mixture of different types, like 3D and 2D, in a target or query sigset is possible. Do FRVT 2006 participants have to expect these mixtures or is there a rule that says: for one experiment all target sigsets are of a certain type A (say e.g. 3D) and all query sigsets are of type B (say 2D)? A and B may or may not be equal.

A: In FRVT 2006, all the biometrics samples in a target or a query set will either be three-dimensional (3D) or still (2D). However, there will be experiments that match 3D with 2D still.

52. According to the FRVT2006 Protocol document, the similarity normalization application accepts 5 arguments: 1) parameter_file, 2) similarity_file, 3) normalized_similarity_file, 4) probe_sigset, and 5) gallery_sigset. I do not quite understand why the last two arguments would be necessary (or helpful) for this purpose. Can you explain the purpose of the two arguments, 'probe_sigset' and 'gallery_sigset' ? (Do they perhaps specify a sub-matrix?)

A: The ‘gallery_sigset’ and ‘probe_sigset’ specify a sub-matrix of the input similarity matrix. The ‘gallery_sigset’ is a subset of the ‘target_sigset’, and the ‘probe_sigset’ is a subset of the ‘query_sigset.’ A ‘gallery_sigset’ contains one biometric sample per person, whereas, a ‘target_sigset’ may contain multiple biometric samples per person.

53. Can you please give examples of the types of sig-sets that we will encounter in FRVT 2006?

A: The Biometric Experimentation Environment (BEE) sig-set format is very flexible and was designed to accommodate complex, multi-modal biometric signatures. However, FRVT 2006 will use four versions of sig-sets. Type 1 is a simple list where each biometric sample consists of a single still image. Examples of this type of sig-sets are the target and query sets for FRGC experiment 1 (either ver1 or ver2). Type 2 is a multi-still sig-set. In this type of sig-set, each biometric sample will consist of multiple still images. The number of images in each biometric sample will vary. An example of this type of sig-set is the target and query sets in FRGC experiment 2. In FRVT 2006, there will be biometric samples where the number of stills in a sample is NOT 4. In Type 3 sig-sets, each biometric sample is a Minolta Vivid scan. An example is the target and query sets for FRGC Experiment 3. In Type 4 sig-sets, each biometric sample is a Qlonerator scan. The structure to this sig-set is similar to the Minolta Vivid scan, except the two recordings will be a *.wfm and a *.bmp file. I will place examples of all four types on bbs.bee-biometrics.org [10 Jan 06]. Examples in post "Example Sig-sets for FRVT 2006" under FRGC v2 Announcements [11 Jan 06]

54. For all-duplicate detections, what is maximum number of still (2D) images in an experiment? For the three-dimensional case (3D), what is the maximum number of 3D samples?

A: For the all-duplicate task, the maximum number of still images is 16,384. For the 3D case, the maximum number is 16,384.

55. In the 3D data, will the coordinates of the vertices represent absolute measurements? For example, will the eye distance be the same in the shape images of a given person, no matter how the distance of that person to the sensor is varied between the capturing sessions? If yes, will the measurement units be the same for both sensors (i.e. Minolta Vivid and Qlonerator)?

A: The Z measurements in Minolta data are relative to an origin corresponding to the front of the sensor. Likewise, the Qlonerator measurements are relative to a coordinate system established through a calibration procedure. There was no attempt made to standardize the position of any portion of the face. However, relative measurements on the face (e.g., intra-ocular distance) will be consistent within the measurement accuracy of the sensor. The units of Minolta measurements are mm. The measurements of Qlonerator images are cm.

56. In the benchmark timing experiment for still images, can a participant exploit the fact that target and query sets are identical, meaning only half as many templates have to be created?

A: No, participants cannot exploit the fact that the target and query sets are identical. In FRVT 2006, all image names will be randomized. So, even if (a big if) target and query sets in FRVT 2006 referenced identical sets of biometric samples, images in these target and query sets would have different randomized names. Thus, templates would have to be generated for both the target and query sets.

57. What is the timing benchmark experiment for 3D data?

A: For 3D data, the timing benchmark experiment is equivalent to the Face Recognition Grand Challenge (FRGC) ver2.0, Experiment 3. In the FRGC ver2.0, Experiment 3, the target and query set consists of 4,007 3D scans. As with the still images, scheduling of FRVT 2006 executables will be based on completing the benchmark experiment in 36 hours on one of the processors/systems in the FRVT 2006 test facility. Executables must be able to complete the benchmark experiment within 72 hours.

58. In the FRVT 2006 FAQ, number 45, it is unclear whether the time constraint is 36 or 72 hours? Please elaborate on this.

A: The time constraint is 72 hours.

59. In FAQ no. 45 you refer to the FRVT 2006 timing guidelines, where can I find these guidelines?

A: FAQ no. 45, along with FAQs 56, 57 and 58, are the FRVT 2006 timing guidelines. The answers to these three FAQs, the FRVT 2006 facility equipment descriptions, FRGC ver2.0 documentation, and FRGC ver2.0 data should be sufficient to determine if an executable meets the timing requirement.

60. Is there any required range for a participant distance metric or similarity measure?

A: There is no required range for distance metrics or similarity scores.

61. Is the evaluation solely qualitative, i.e., independent of the scale and possible non-linearity of a participant’s distance function or similarity score? Is any part of the evaluation dependent on a threshold value specified by a participant?

A: Performance in FRVT 2006 will be measured independently of the scale and possible non-linearity of a participant’s distance function or similarity score. Where appropriate, performance will be reported on a Receiver Operating Characteristic (ROC) and Cumulative Match Characteristic (CMC). For further details, participants should see the FRVT 2002 reports. No part of the evaluation is dependent on a threshold value specified by a participant.

62. In your recent "FRVT 2006 Update" e-mail, the deadline for FRVT 2006 is "Monday, 30 January 2006, at 5:00pm Eastern Standard Time". Would you please tell me if this deadline is determined by arrival or postmark?

A: This is arrival time.

63. The Face Recognition Vendor Test (FRVT) 2006 Protocol states that the format for 3D data will be the same as in the FRGC datasets. In the case of the Qlonerator sensor, what exactly does “same format” mean for the texture images? Does it mean the image format will be BMP, and the left and right side of the texture image will constitute a stereo image pair? If not, please provide a characterization of the texture image contents.

A: Qlonerator images are supplied as a VRML mesh accompanied by a Windows bitmap file.

64. Concerning the VRML format used to specify the shape representation and the texture mapping of a Qlonerator data sample: 1) Will the texture image be referenced through the basename of the texture image file (i.e. no path component)? 2) Will the shape representation be structurally equivalent to the shape representation used in the FRGC supplemental dataset?

A: The VRML file will contain a relative reference to the accompanying bitmap file. These files are created by a vendor-supplied conversion program. While we cannot guarantee a particular arrangement and orientation of the left and right images in the bitmap file, the arrangement described in the question has been consistently present in the files created. The VRML files in the FRVT 2006 Qlonerator data set are created by the same software used to create the VRML files in the supplemental data set.

65. Please provide an example of the eye-coordinate metafile.

A: An example is provide in the post "FRVT 2006 sample metadata file" under FRGC v2 Announcements on bbs.bee-biometrics.org.

66. Can we only participate in 2D still image evaluation?

A: Yes. In the protocol, a 2D image is referred to as a "still."

67. What is the file format for reporting quality scores?

A: Details on the file for are in the post "Quality score format for FRVT 2006" under FRGC v2 Announcements in the bbs.

68. The independence requirements for the 1-Many task (FRVT2006 Protocol) refer to probe signature and a gallery sigset. Is the 'Gallery' here identical with the 'target_sigset' that is passed as executable argument? Is the 'Probe' here identical with the 'query_sigset' that is passed as executable argument? If yes, does this mean that the 'target_sigset' always contains only one biometric sample per person in 1-Many tests?

A: For the call signature for 1-Many matching, the target_sigset will be the gallery, and the query_sigset will be the probe set. Yes, for 1-Many, the target_sigset will contain only one biometric sample per person.

69. Can you explicitly define the term 'biometric-sample' with respect to a 'Gallery'? I associate 'biometric-sample' with 'signature' (which could contain multiple images) but in older documents it seems to refer to a single image (e.g. Face Recognition Vendor Test 2002 Performance Metrics; Paragraph 1.1: ''the gallery .... contains only one image per person').

A: The terms biometric sample and biometric signature are synonymous. In an FRVT 2006 still image experiment, a biometric sample in a gallery can contain multi-still images. As noted in the question, this definition is different than in FRVT 2002.

70. If the probe and gallery sigsets specify the sub-matrix, how do I find the matrix values that correspond to the sub-matrix and are supposed to be changed? What parts of the signatures are required and sufficient to be compared in order to find these correspondences? If the 'signature-name' is sufficient, than all signature names in the target set must be unique for each signature. Can you clarify this?

A: Matching on the signature-name will be sufficient. In experiments where similarity score normalization will be performed, all the signature names will be unique in the target and query sets. For simple biometric samples, the name attribute of the element should be compared. In the above example, the appropriate value to compare is nd1S02463. For complex biometric samples, the name attribute of element should be compared. In the above example, the appropriate value to compare is nd1S02465.

71. In reference to FAQ no. 52, the calling signature specifies gallery and probe set sig-sets. The gallery and probe sets are subsets of target and query sets used in generating a similarity matrix. Without access to the original target and query sets, how can I determine the elements of the original similarity matrix to include in the normalized similarity matrix?

A: The target_sigset and query_sigset file names are contained in the header of the original similarity file (the similarity_file argument in the similarity file normalization calling signature).

72. In FAQ no. 33 you state that the maximum number of biometrics samples in a target or query set will be 16,384. However, multi-still biometric samples reference multiple still images. What will be the maximum number of still images that will be referenced by biometric samples in a target or query set?

A: The maximum number of images referenced in a target or query set will be 32,768.

73. Does the 72 hours maximum execution time in FAQ no. 45 apply to all tasks, including 1-1 Matching and 1-many Matching?

A: The time constraint of 72 hours applies to all tasks including 1-1 matching and 1-many matching.

74. : In FRVT 2006, will the range portion of the Minolta files be in the compressed abs.gz format, or will they be unpacked before the experiments?

A: The range portion of the Minolta files will be decompressed prior to the start of an experiment. The uncompressed range images will have the suffix .abs, NOT .abs.gz. If your executable requires the abs.gz compressed range images, please inform the FRVT2006 liaison at frvt2006@nist.gov.

75. There are only two allowed values for attribute "capture" (Controlled and Uncontrolled) of parameter file in the final Executable Calling Signatures document. Does this mean that the controlled and uncontrolled samples will not be in a target set or a query set simultaneously? And is it the same about the biometric samples with a different value of attribute "pose"?

A: If the attribute capture is set to Controlled, then all the biometrics samples in referenced sig-set will be taken under controlled conditions. The capture attribute value of Uncontrolled includes biometric samples taken under controlled conditions. If the attribute pose is set to Frontal, then all the biometrics samples in referenced sig-set will have a frontal pose. The pose attribute value of Nonfrontal includes frontal poses.

76. Where can I download the latest sig-set parser?

A: See the post "Latest SigSet parser for FRVT 2006" under FRGC v2 Announcements on the bbs.

77. Are the terms "Parameter File" and "Experiment Description File" synonymous?

A: The terms "Parameter File" and "Experiment Description File" are synonymous.

78. From FRGC, we know there are differences in the image properties among the Minolta Vivid 900, Minolta Vivid 910, and Qlonerator 3D scanners. In order to cope with the sensor specific effects we have the following questions: 1) Will the 3D sensor be specified prior to the start of an experiment? 2) Is it possible to have different executables for the various scanners? 3) Will experiments consist of a mixture of biometrics samples from different 3D scanners?

A: One of the primary goals of the 3D evaluations in FRVT 2006 is to measure the performance of general 3D face recognition algorithms, not to measure 3D face algorithms tuned to a particular sensor. To achieve this goal, 1) the 3D sensor will not be specified prior to the start of an experiment, 2) executables for 3D matching will have to be able to process 3D face scans from different sensors, and 3) experiments will require matching across 3D sensors.

79. Can you provide the format for the metadata for Qlonerator 3D scans?

A: An example is provided in the post “FRVT 2006 sample metadata file for Qlonerator 3D scans” under FRGC v2 Announcements on the bbs.

80. There is a file in the supplemental data set (FRGC-FRVT-transitional-recordings.xml) that contains eye coordinates. Will this type of file be used to provide eye coordinates be used in FRVT 2006?

A: The file FRGC-FRVT-transitional-recordings.xml is not related to FRVT 2006. Please see FAQ no. 79 for metadata for Qlonerator 3D scans.

81. Is it possible to put the dll files in the bin directory?

A: Yes.

82. Do we understand correctly that in the case of the 1-1 experiment there will be only one complex-presentation tag within each complex-biometric-signature tag? And in the case of the 1-M matching there may be many (i.e. up to M) complex-presentation tags within each complex-biometric-signature tag?

A: The main differences between 1-1 and 1-many matching are 1) the independence requirements for the similarity scores, and 2) the number of biometric samples per person in the target/gallery sig-set. The independence requirements for the similarity scores are discussed in the FRVT 2006 protocol document. In 1-1 matching, a target set can contain multiple biometric samples per person. In 1-many matching, the gallery (target sig-set in the executable calling document) contains one biometric sample per person. The difference between 1-1 and 1-many is NOT a function of the structure of the number of presentations in a complex-biometric-signature tag. For example, a gallery that is used in a 1-many matching experiment can be used as a target set in a 1-1 matching experiment. In this case, a comparison of the results between the 1-many and 1-1 experiments measures the effectiveness of 1-many matching over 1-1 matching.

83. I am unable to get the SigSet parsing classes to parse the “file-format” entry. Can I check for the “abs.gz” and “wrl” extensions of the “file-name” entry instead?

A: Please look at the new examples on the bbs. Deriving the file-format from the “abs.gz” and “wrl” extensions of the “file-name” entry should work, but it cannot be guaranteed to work.

84. What is the format for the log file?

A: There is not a prescribed format for the log file. The log file is to aid diagnosing problems in an executable. In deciding which information to write to the log file, participants should avoid making the log file too large. If too much information is reported, the file could be too large. If the file is too large it may not be possible for FRVT 2006 personnel to search the log file to provide you with useful information. In addition there is the amount of time to write the file and the possibility it could take up too much disk space. Logical points to write execution information are: 1) after generating a template, and 2) after completing all the matches between a query biometric sample and all the target biometric samples. FAQ no. 45 mentions writing a log to standard out. Participants can select to write the information to either standard out or the specified log file.

85. Can templates generated from a previous experiment be placed in the tmp directory and used in subsequent experiments? Will previously generated templates be erased?

A: Templates can be stored in the tmp directory (or tmp directory structure that an executable creates). Depending on the set of experiments being conducted, previously generated templates maybe deleted.

86. Your document states normalization as an option for 1:1, namely: 1-1 Matching. The basic biometric verification task is comparing two biometric samples and reporting a similarity. This task will measure performance when matching is based only on the two biometric samples being compared. Each similarity score between a target biometric sample t and query biometric sample q is independent of all other biometric samples in the target and query sets. The computation of image quality is an option for 1-1 matching. A single quality score is produced for each target and query image. The quality score for a biometric sample is an integer, with larger numbers indicating better quality. Another option available for this task is similarity score normalization. In similarity score normalization, the input is a set of similarity scores s(G,Q), the executable returns a new set of similarity scores s’(G,Q). Now, given, that we have 3D with shape and texture, we could generate shape similarities s_s and texture similarities s_t. In the plain 1-1 Matching without the normalization option, the system should do every calculation internally and the system could deliver a combined score s_c as a function f of s_s and s_t, say s_c=f(s_s,s_t). Choosing the option stated above, to my understanding the only choice would be to have all the combined scores s_c(G,q) produced by G and q (I follow the FAQ notation here to make clear, that only one query is used for normalization) as an input. From this input s_c’(G,q) shall be calculated. My question is: is it allowed to create first s_s’(G,q) and s_t’(G,q) based on s_s(G,q) and s_t(G,q), i.e. we have two similarity scores instead of just one that serve as an input for normalization.

A: No, the above-mentioned type of similarity score normalization is not permitted in the FRVT 2006 protocol. This type of normalization is permitted in the 1-many matching task.

87. What key do I use to relate the medadata coordinates with a recording in a sig-set?

A: This part of the question addresses still images. For the sig-set file, the name attribute of the presentation element is matched to the recording_id attribute in the Recording element of the metadata file. For example, the metadata for the image referenced in from the sig-set would be matched to in the metadata file. 3D scans consist of two presentations (recordings). The attribute name associated with either presentation can be matched with the recording_id attribute. Yes, this means that the meta-data for each 3D scan is listed twice in the metadata file, once mapped to the texture channel and once mapped to the shape channel.

88. Will you provide the metadata parser for the Qlonerator 3D scans?

A: A parser for 3D Qlonerator scans will not be provided. The metadata parser provided as part of the BEE system can be modified to parse the Qlonerator metadata.

89. In FAQ no. 73, you state that the 72-hour limit applies to all tasks. I read it that each participant is given a maximum time of 72 hours in which all FRVT tasks have to be completed. On the other hand, in FAQ 45, the 72 hours seem to refer only to the benchmark-test. Can you clarify this?

A: The 72 hours refers to only the benchmark experiment. Tasks refer to the type of executable, for example, 1-1 matching, 1-many matching, or preprocessing. Each executable submitted will need to complete the appropriate benchmark experiment in 72 hours.

90. Is the Linux OS Fedora 3 the 64-bit or 32-bit version?

A: The 32-bit version will be used.

91. The signup sheet for the tasks mentions "Image Quality" only under the 1-1 Matching task. However, the Executable Signature Document specifies quality sigsets for both the 1-1 and the 1-Many executables. Please clarify: the Image Quality subtask is only part of 1-1, or: is it part of 1-1 as well as 1-Many?

A: Image quality tasks will only be available under 1-1 matching. Paring quality with 1-1 matching was chosen because of similar independence requirements. To maintain consistency with the Executable Signature Document, the calling signature for 1-many will have filenames for quality_target_sigset and quality_query_sigset.

92. If we decide to store templates on disk, we'd associate each template with the image file name from which it was generated. Can we assume that image filenames (excluding sub-directory names) are unique in the sense that recurring filenames always refer to the same image throughout all experiments?

A: Yes.

93. You noted in FAQ 85 that the deletion of data in the 'temp' folder depends on the experiments being conducted. Will you leave the 'temp' folder intact for certain experiments where the re-using of previously stored templates could be of significant speed-advantage?

A: Yes.

94. Do you plan to provide a C++ or Java class for writing the output for the AllDuplicate task?

A: No, we do not plan to provide a C++ or Java class for writing the output for the AllDuplicate task.

95. Must the normalized similarity matrix be entirely normalized or just be normalized in the submatrix corresponding to the probe and gallery sigsets? How should we normalize scores for query samples not present in the probe sigset, while remaining in compliance with the independence constraints?

A: The output of the similarity score normalization task is a new similarity matrix. The new similarity matrix is the submatrix corresponding to the probe and gallery sigsets. The probe set is a subset of the query set.

96. The document "Executable_Calling_Signatures_for_FRVT2006.pdf", and in "Figure C: Example of a Signature Set" on page 7, has a different complex-presentation structure than posted on the BBS at http://bbs.bee-biometrics.org/phpBB2/viewtopic.php?t=139. Which one will be used?

A: For 3D sigsets, the structure on posted on the BBS at http://bbs.bee-biometrics.org/phpBB2/viewtopic.php?t=139 will used.

97. The Qlonerator 3D scans FRGC supplemental have measurement in millimeters (mm). You write in FAQ #55 that the measurements are in cm. Can you please clarity.

A: FAQ no. 55 was in error; the measurements are in millimeters (mm).

98. In the 'Executable Calling Signatures' document, in the first table defining the inputs and outputs, you state that the header and dimensions of the normalized similarity matrix should be the same as the dimensions of the input similarity matrix. In FAQ no. 95, the normalized similarity matrix is defined as a submatrix of the input similarity matrix (that is to say a matrix with new dimensions). Can you clarify this? <