Identifying the accurate location of small objects is a key element of Internet of Things (IoT). This paper investigates the feasibility of tracking the real-time location of a completely passive retroreflector using visible light for IoT applications. Existing ultra-low power visible light retroreflector systems modulate light with a liquid crystal display (LCD) shutter, which is powered by a solar cell. However, the solar cell costs additional space exclusively for energy harvesting, and it may not be able to supply enough power to the advanced power-hungry LCD shutter and its driver circuit. To eliminate the power supply, we design, implement and evaluate PassiveRETRO, an enhanced retroreflector-based visible light localization system. The PassiveRETRO system completely eliminates the necessity of any electronic component on the IoT devices. Polarization-based modulation and bandpass optical filters are adopted to identify the retroreflected optical signal and create multiple channels. Each IoT device operates on a specific range of the visible light spectrum. Optical rotatory dispersion is further applied to mitigate the mutual interference among different channels. Experimental results from our prototyped system show that PassiveRETRO is robust to environmental reflection and can still achieve centimeter-level location accuracy when multiple IoT devices are deployed.