SCIMITAR: Scalable stream-processing for sensor information brokering

Current sensor collection capabilities produce an incredible amount of data that needs to be processed, analyzed, and distributed in a timely and efficient manner. Information Management (IM) services supporting a publish-subscribe and query paradigm can be a powerful general purpose approach to enabling this information exchange between decoupled and dynamic information producers and consumers. These IM services will only be of value, however, if they can support operations in a manner that is responsive to the sheer quantity and frequency of data produced by surveillance platforms. Cloud computing is the technology of choice for providing the resources and services needed to enable and mange large-scale distributed computation. To date, there has been little work to develop highly scalable, dynamic IM processing and dissemination services in a cloud computing environment. In this paper we discuss our design, implementation and evaluation of a prototype cloud-based information broker which is a critical component of a highly scalable, distributed IM System. The brokering prototype is designed using a distributed stream processing framework and is shown to scale nearly linearly with the number of computing nodes as information load and subscription quantity increases.