CAREER: Streamlining Task Deployment on Crowdsourcing Platforms

Crowdsourcing leverages online infrastructure to tap an under-explored and richly heterogeneous pool of human knowledge and cognition for solving a variety of tasks that are otherwise considered hard for machines to solve alone. Crowdsourcing systems are built on private or public platforms and are a popular means of deploying a variety of tasks that require human intelligence. Task deployment on such platforms requires identifying appropriate deployment strategies to satisfy deployment parameters, provided by requesters as thresholds on quality, latency, and cost, and also requires analysis of the workforce that is available to undertake the deployed tasks. To date, task deployment remains a painstakingly manual process, as there is little to no help for requesters in deciding how to organize the workforce, in what style, and in what structure to satisfy deployment parameters. Consequently, requesters and workers are mostly confined to one platform, as there is no easy portability of deployment processes across platforms. This project investigates a middle layer that sits between multiple stakeholders in a crowdsourcing ecosystem to aid requesters in deploying crowdsourcing tasks by allowing easy and flexible specification of deployment constraints and goals, and then recommending deployment strategies based on those specifications. Development of this system thus enables the portability and reuse of deployment processes across platforms.

To achieve these goals, this project develops a middleware system called SLOAN (Scalable, decLarative, Optimization-driven, Adaptive, and uNified) with three integrated components: (1) The Deployment Strategy Recommendation Engine is optimized to accommodate multi-stakeholders in the ecosystem, and is responsible for modeling and recommending deployment strategies to a batch of requests. (2) The Workforce Analytics Engine analyzes the available workforce and feeds to the Recommendation Engine, as the deployed tasks are to be undertaken by the workers. The outputs of this engine are estimations of workers' preferences or human factors, such as availability of the workers, as precise (discrete), or imprecise (intervals or probability distribution functions) information. (3) The Result Aggregation Module estimates the quality of the deployed tasks, and then feeds to the other two engines for readjustment. It is empowered by fully automated or hybrid algorithms that sparingly involve human intelligence inside machine algorithms. The development plan of SLOAN involves principled modeling, rigorous algorithm design, declarative framework development, deployment and integration inside multiple real world platforms. Different components of SLOAN are empowered with multi-objective discrete optimization and computational geometric algorithms, as well as multi-faceted modeling techniques adapted from machine learning.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.