Top-1 Algorithms over Amazon Mechanical Turk
- I used to work on top-1 algorithms over Amazon Mechanical Turk. We made use of emperical experiments to try to figure out some interesting phenomena in crowdsourcing world. Specifically, we were interested in the factors which have impact on the performance of crowds and how these factors influence crowds.
- By using API provided by Amazon, I implemented a set of experiments to serve our purpose. You can find the gist in my talk given in a workshop. Here are the slides and poster.
- I also cooperated with some cool guys such as Petros, Hector Garcia-Molina and Alkis. Again, I took responsibility for the implementation work (experiments). Fortunately, our paper Max Algorithms in Crowdsourcing Environments was accepted for publication in WWW12.
- When I was doing top-1 algorithm implementation, I found some top-1 algorithms extremely useful and common, because many jobs in crowdsourcing environments are iterative. Thereby, I wrote a simple top-1 algorithm library based on Amazon Mechanical Turk for our team members or any other people to use.
SQLTurk
- Now, I am working on our project SQLTurk, which utilizes human interface to translate natural language statements into SQL queries.
Crowdsourcing Benchmark
- In this ongoing project, I am going to investigate some popular crowdsourcing platforms. Most of my previous work was built on Amazon Mechanical Turk, but due to the popularity of crowdsourcing, we are interested in how other platforms perform as well. After we get to know well about a bunch of different platforms, the benchmark will be very helpful for people to know the features of various platforms and have the sense that what kind of platform serve their purposes well and which to choose.
- Also, maybe I can write an uniform API for these different platforms in order to help people distribute their works across these platforms more easily. We will see if I can make it. :)
- Now, the platforms in our mind are: Amazon Mechanical Turk, CrowdFlower and oDesk.