Traffic safety is an important topic in the intelligent transportation system. One major factor that causes traffic accident is anomalous driving. This paper presents a novel anomalous driving detection method in videos, which can detect unsafe anomalous driving behaviors. The contributions of this paper are three-fold. First, a new multiple object tracking (MOT) method is proposed to extract the velocities and trajectories of moving foreground objects in video. The new MOT method is a motion based tracking method, which integrates the temporal and spatial features. Second, a novel Gaussian local velocity (GLV) modeling method is presented to model the normal moving behavior in traffic videos. The GLV model is built for every location in the video frame, and updated online. Third, a discrimination function is proposed to detect anomalous driving behaviors. Experimental results using the real traffic data from the New Jersey Department of Transportation (NJDOT) show that our proposed method can perform anomalous driving detection fast and accurately.