This paper proposes a novel scheme which can efficiently reduce the energy consumption of Optical Line Terminals (OLTs) in Time Division Multiplexing (TDM) Passive Optical Networks (PONs) such as EPON and GPON. Currently, OLTs consume a significant amount of energy in PON, which is one of the major FTTx technologies. To be environmental friendly, it is desirable to reduce energy consumption of OLT as much as possible; such requirement becomes even more urgent as OLT keeps increasing its provisioning data rate, and higher data rate provisioning usually implies larger energy consumption. In this paper, we propose a novel energy-efficient OLT structure which guarantees services of end users with the smallest number of power-on OLT line cards. More specifically, we adapt the number of power-on OLT line cards to the real-time incoming traffic. Also, in order to avoid service disruption resulted by powering off OLT line cards, proper optical switches are equipped in OLT to dynamically configure the communications between OLT line cards and ONUs.