Unknown Signal Detection in Switching Linear Dynamical System Noise

A machine learning approach is presented for detecting unknown or anomalous signals in a complicated background of interfering signals and noise. The approach can be employed in RF spectrum monitoring applications to efficiently detect transmissions that deviate from a typical signal environment. For example, in the cognitive radio domain, the technique may be applied to learn the typical behavior of spectrum sharing secondary users and efficiently detect noncompliant transmissions. A switching linear dynamical system (SLDS) is trained to represent the interference and noise environment via a Bayesian nonparametric hierarchical Dirichlet process (HDP)-SLDS technique. An unknown signal is detected if the Viterbi hidden switching state path of the test data is sufficiently unlikely under the learned background SLDS. The detection scheme is derived as a generalized likelihood ratio test (GLRT) for an unknown deterministic signal in SLDS noise. The distribution of the Viterbi likelihood test statistic under the null hypothesis (signal absent) is analyzed and an asymptotic upper bound on the false alarm probability is derived as a function of the detection threshold. Numerical simulation and experimental results on a software-defined radio (SDR) testbed demonstrate that the empirical false alarm rate obeys the upper bound and that the SLDS detection approach substantially outperforms an earlier HMM-based scheme as well as a standard energy detector in a challenging interference and noise background.