We propose a new CGSFPHOG descriptor and perform image classification using a novel EFM-KNN classifier, which combines the Enhanced Fisher Model (EFM) and the K Nearest Neighbor (KNN) decision rule. We integrate the oRGB-SIFT descriptor with other color SIFT features to produce the Color SIFT Fusion (CSF) and the Color Grayscale SIFT Fusion (CGSF) descriptors. The CGSF is integrated to the PHOG to obtain the novel CGSFPHOG descriptor. The effectiveness of the proposed new descriptor and the classification method is evaluated using two grand challenge datasets: the Oxford flower database and the MIT scene database. The classification results using the EFM-KNN classifier show that (i) the CGSFPHOG descriptor improves recognition performance upon other descriptors; and (ii) the oRGB-SIFT, the CSF, and the CGSF perform better than the other color SIFT descriptors.