Defining crowdsourcing (CS) systems turns out to be surprisingly tricky. Since many view Wikipedia and Linux as well-known CS examples, as a natural starting point, we can say that a CS system enlists a crowd of users to explicitly collaborate to build a long-lasting artifact that is beneficial to the whole community.
Crowdsourcing occurs when businesses make use of the free, wide audience that the Internet gives them, and uses that audience to gain ideas, a product or service, or feedback - things that will ultimately lead that business to profit. It is typically opening up an idea that would normally be performed by one or a few people to a large group of people (or anyone on the Internet, as is becoming common). The obvious benefits of the concept are that you as a business have numerous volunteers (often into the hundreds, if not thousands) to help you solve a problem, you save money, and you get to pick and choose what you want out of many offers, all at your time and leisure. It is an efficient and quick way of sourcing talent.
Crowdsourcing systems enlist a multitude of humans to help solve a wide variety of problems. Over the past decade, numerous such systems have appeared on the World-Wide Web. Prime examples include Wikipedia, Linux, Yahoo! Answers, Mechanical Turk-based systems, and much effort is being directed toward developing many more. As is typical for an emerging area, this effort has appeared under many names, including peer production, user-powered systems, user-generated content, collaborative systems, community systems, social systems, social search, social media, collective intelligence, wikinomics, crowd wisdom, smart mobs, mass collaboration, and human computation. We define and classify such systems, then describe a broad sample of systems. The sample ranges from relatively simple well-established systems such as reviewing books to complex emerging systems that build structured knowledge bases to systems that "piggyback" onto other popular systems. We discuss fundamental challenges such as how to recruit and evaluate users, and to merge their contributions. Given the space limitation, we do not attempt to be exhaustive. Rather, we sketch only the most important aspects of the global picture, using real-world examples
Sample CS Systems on the Web
Building on this discussion of CS dimensions, we now focus on CS systems on the Web, first describing a set of basic system types, and then showing how deployed CS systems often combine multiple such types.
The accompanying table shows a set of basic CS system types. The set is not meant to be exhaustive; it shows only those types that have received most attention. From left to right, it is organized by collaboration, architecture, the need to recruit users, and then by the actions users can take. We now discuss the set, starting with explicit systems.
Explicit Systems: These standalone systems let users collaborate explicitly. In particular, users can evaluate, share, network, build artifacts, and execute tasks. We discuss these systems in turn
CS Systems on the Web
We now build on basic system types to discuss deployed CS systems on the Web. Founded on static HTML pages, the Web soon offered many interactive services. Some services serve machines (such as DNS servers, Google Map API server), but most serve humans. Many such services do not need to recruit users (in the sense that the more the better). Examples include pay-parking-ticket services (for city residents) and room-reservation services. (As noted, we call these crowd management systems). Many services, however, face CS challenges, including the need to grow large user bases. For example, online stores such as Amazon want a growing user base for their services, to maximize profits, and startups such as epinions.com grow their user bases for advertising. They started out as primitive CS systems, but quickly improved over time with additional CS features (such as reviewing, rating, networking). Then around 2003, aided by the proliferation of social software (for example, discussion groups, wiki, blog), many full-fledged CS systems (such as Wikipedia, Flickr, YouTube, Facebook, MySpace) appeared, marking the arrival of Web 2.0. This Web is growing rapidly, with many new CS systems being developed and non-CS systems adding CS features.
The goal is to further our collective understandingboth conceptual and practicalof this important emerging topic. It is also important to note that many crowdsourcing platforms have been built. Examples include Mechanical Turk, Turkit, Mob4hire, uTest, Freelancer, eLance, oDesk, Guru, Topcoder, Trada, 99design, Innocentive, CloudCrowd, and Cloud-Flower. Using these platforms, we can quickly build crowdsourcing systems in many domains. In this survey, we consider these systems (that is, applications), not the crowdsourcing platforms themselves.
It is also important to note that many crowdsourcing platforms have been built. Examples include Mechanical Turk, Turkit, Mob4hire, uTest, Freelancer, eLance, oDesk, Guru, Topcoder, Trada, 99design, Innocentive, CloudCrowd, and Cloud-Flower. Using these platforms, we can quickly build crowdsourcing systems in many domains. In this survey, we consider these systems (that is, applications), not the crowdsourcing platforms themselves.
Classifying CS systems:
CS systems can be classified along many dimensions. Here, we discuss nine dimensions we consider most important. The two that immediately come to mind are the nature of collaboration and type of target problem. As discussed previously, collaboration can be explicit or implicit, and the target problem can be any problem defined by the system owners (for example, building temporary or permanent artifacts, executing tasks).
The next four dimensions refer respectively to how a CS system solves the four fundamental challenges described earlier: how to recruit and retain users; what can users do; how to combine their inputs; and how to evaluate them. Later, we will discuss these challenges and the corresponding dimensions in detail. Here, we discuss the remaining three dimensions: degree of manual effort, role of human users, and standalone versus piggyback architectures.
Degree of manual effort: When building a CS system, we must decide how much manual effort is required to solve each of the four CS challenges. This can range from relatively little (for example, combining ratings) to substantial (for example, combining code), and clearly also depends on how much the system is automated. We must decide how to divide the manual effort between the users and the system owners. Some systems ask the users to do relatively little and the owners a great deal. For example, to detect malicious users, the users may simply click a button to report suspicious behaviors, whereas the owners must carefully examine all relevant evidence to determine if a user is indeed malicious. Some systems do the reverse. For example, most of the manual burden of merging Wikipedia edits falls on the users (who are currently editing), not the owners.
Role of human users: We consider four basic roles of humans in a CS system. Slaves:humans help solve the problem in a divide-and-conquer fashion, to minimize the resources (for example, time, effort) of the owners. Examples are ESP and finding a missing boat in satellite images using Mechanical Turk. Perspective providers: humans contribute different perspectives, which when combined often produce a better solution (than with a single human). Examples are reviewing books and aggregating user bets to make predictions. Content providers: humans contribute self-generated content (for example, videos on YouTube, images on Flickr). Component providers: humans function as components in the target artifact, such as a social network, or simply just a community of users (so that the owner can, say, sell ads). Humans often play multiple roles within a single CS system (for example, slaves, perspective providers, and content providers in Wikipedia). It is important to know these roles because that may determine how to recruit. For example, to use humans as perspective providers, it is important to recruit a diverse crowd where each human can make independent decisions, to avoid "group think."
