"Always-on" smart speakers have raised privacy and security concerns, to address which vendors have introduced customizable privacy settings. But, does the act of customizing one's privacy preferences have any effects on user experience and trust? To address this question, we developed an app for Amazon Alexa and conducted a user study (N = 90). Our data show that the affordance to customize privacy settings enhances trust and usability for regular users, while it has adverse effects on power users. In addition, only enabling privacy-setting customization without allowing content customization negatively affects trust among users with higher privacy concerns. When they can customize both content and privacy settings, user trust is highest. That is, while privacy customization may cause reactance among power users, allowing privacy-concerned individuals to simultaneously customize content can help to alleviate the resultant negative effect on trust. These findings have implications for designing more privacy-sensitive and trustworthy smart speakers.