###Upcoming: The economics of crowdsourcing
- Pricing HITs, incentivizing Turkers
- Ensuring high quality data, balancing cost and quality
####Reading
- "The Wisdom of Crowds" Chapter 10 - The Company
- "The Wisdom of Crowds" Chapter 11 - Markets
- Financial Incentives and the "Performance of Crowds"
- Cheap and Fast - But is it Good?, about the using non-expert workers to collect expert-quality data.
- Get Another Label? Improving Data Quality and Data Mining Using Multiple, Noisy Labelers, about using redundant labeling to improve data quality and how to balance cost and quality
###Upcoming: Crowdsourcing and machine learning
- Importance of labeled training data in ML, Computer Vision, Natural Language Processing
- Uses of crowdsourcing to reduce cost of ML, cost of expert annotations
- Uses of ML to reduce cost of crowdsourcing, EM algorithms for qualtiy estimation
####Reading
- "The Wisdom of Crowds" Chapter 6 - Society Does Exist
- "The Wisdom of Crowds" Chapter 7 - Traffic
- Maximum Likelihood Estimation of Observer Error-rates using the EM Algorithm, the theory which is implemented by Project Troia, which you will use in your assignment. This paper is math-heavy, but you all are a smart lot. Believe in yourself.
- CrowdFlow: Integrating Machine Learning with Mechanical Turk for Speed-Cost-Quality Flexibility, describes a framework for using both computers and humans to label data
- Games with a purpose, clever ideas for tricking people into having fun so you don't have to pay them to label your data.
###Upcoming: Crowdsourcing and social science
- Experiments in linguistics, cognitive science, and political science
- Ethics of crowdsourcing
- How to apply for Institutional Review Board approval
####Reading
- "The Wisdom of Crowds" Chapter 9 - Committees, Juries, and Teams
- "Running experiments on Amazon Mechanical Turk", an overview of the MTurk platform, and the associated benefits and challenges
- Crowdsourcing User Studies With Mechanical Turk, about best-practices for designing tasks for a crowdsourcing environment.
- Citizen Science: Can Volunteers Do Real Research?, discussion of the ability of non-experts to collect scientific data.
- Mechanical Turk is not Anonymous, discusses a privacy vunerability in MTurk, and the implications for research ethics.
- Dirty Deeds Done Dirt Cheap, discussion of some of the potentially malicious uses of crowdsourcing. Mostly included for the fantastically scathing title.
- "Gold Mine or Coal Mine?", another fantastically scathing title...
###Upcoming: Collective intelligence
- Surveying and polls, Prediction markets, A/B testing, Data mining
- Crowdfunding, product valuation, product reviews
####Reading
- "The Wisdom of Crowds" Chapter 12 - Democracy
- Harnessing the Wisdom of Crowds in Wikipedia: Quality through Coordination. I now feel guilty for complaining about group projects with a group size of 3.
- Opinion Mining Using Econometrics: A Case Study of Reputation Systems, about using online reviews to predict the price differentials for products sold online.