This paper presents a horizontal and vertical 2D principal component analysis (2DPCA) based discriminant analysis (HVDA) method for face verification. The HVDA method, which derives features by applying 2DPCA horizontally and vertically on the image matrices (2D arrays), achieves high computational efficiency compared with the traditional PCA and/or LDA based methods that operate on high dimensional image vectors (ID arrays). The HVDA method further performs discriminant analysis to enhance the discriminating power of the horizontal and vertical 2DPCA features. Finally, the HVDA method takes advantage of the color information across two color spaces, namely, the YIQ and the YCbCr color spaces, to further improve its performance. Experiments using the Face Recognition Grand Challenge (FRGC) version 2 database, which contains 12,776 training images, 16,028 controlled target images, and 8,014 uncontrolled query images, show the effectiveness of the proposed method. In particular, the HVDA method achieves 78.24% face verification rate at 0.1% false accept rate on the most challenging FRGC experiment, i.e., the FRGC Experiment 4 (based on the ROC III curve).