Short-term changes in atmospheric transmissivity caused by clouds can engender more severe fluctuations in photovoltaic (PV) outputs than those from traditional power plants. As PV energy continues to penetrate the U. S. National Energy Grid, such volatility increasingly lowers its reliability, efficiency, and value-added contribution. Therefore a model that can accurately predict the cloud motion and its affect on PV system's production is in a pressing demands. It can be used to mitigate the undesired behavior beforehand. In this paper we explore the use of Total Sky Images and the cloud estimation techniques based on such images. To further improve estimation quality of motion vector, we propose a novel hybrid algorithm taking the advantages of both correlation based and local feature based approaches. Our proposed hybrid approach significantly reduces the cloud motion prediction error rate by 25% on average, which can help to predict short term solar energy frustration in our later work.