Green communications has received much attention in recent years. In cellular networks, base stations (BSs) account for more than 50 percent of the energy consumption. Reducing energy consumption of BSs is essential to realize green cellular networks. Utilizing green energy to power BSs is a promising way to reduce the on-grid energy consumption. Owing to the dynamics of both mobile traffic loads and green energy, the mismatch between the energy demands and green energy generation in a BS results in inefficient green energy utilization. Managing the battery in BSs can control the green energy usage in individual time slots, thus alleviating the inefficiency caused by the mismatch. In this paper, we propose an intelligent battery management mechanism to optimize the green energy utilization in BSs based on the Markov Decision Process (MDP). A large number of states in the Markov chain are required to model the dynamics of solar radiation and BS workload demands. Thus, the original MDP optimal policy iteration method incurs a high computational complexity. Therefore, we propose some heuristics to approximate the optimal energy dispatching strategy with low computational complexity, and validate the performance of the proposed algorithm through extensive simulations.