Should Algorithms Decide What We Write About Online?

There was a fascinating story in yesterday's New York Times about a company called Demand Media:

Demand uses a three-part formula of search terms, potential ad results and what competitors are doing to feed an algorithm that, with a human assist, comes up with headlines that are full of clickable, salient language that serves as bait for readers and search ads. (News is expensive to produce and not really a part of the formula because the company is looking for durable content, so “How to avoid a tiger attack” will have more value than, say, “Tiger’s not out of the woods, yet.”)

The topic is then fed into a central database where freelance writers sign up for the assignment. The articles they write are run through an automated plagiarism checker, an actual copy editor and posted on one of the company’s sites like eHow or LiveStrong.

Driven by search and video advertising, it’s a good business, with more than $200 million in revenue in 2009 and a current value for the company that has been estimated from $1 billion to $2 billion in various reports.

Based in Santa Monica, Demand Studios, the production arm of the company, has been described as a “content farm,” a place where many of the processes are automated and the writer and videographers serve as field hands, with pay to match. The average article pays $15 to $20 — videos pay about $30 — but the company has had no trouble signing up 7,000 steady contributors to bid for the work. (Copy editors make about $3.50 for editing a story.)

Contrast this with traditional media. At a newspaper, the assignment editor decides what stories his reporters cover (and what stories to take from the wire services) and, by extension, what the subscribers read about. Although alternate content is available in magazines, TV, and elsewhere, a substantial portion of any city's adults subscribe to whatever the local newspaper is and decide to read whatever that paper chooses to cover on any given day. This was even more true before the Internet.

The Internet has already revolutionized one side of the equation: what we choose to read. I don't have to read everything the Washington Post offers but can instead only browse the tech and business sections. I can keep up with the newspaper from my hometown from 3,000 miles away. For any niche interest, I can probably find several blogs. Or, when I have a particular topic in mind, I can find information about it via Google or Twitter. The Internet also has endless diversions outside of news, of course, whether it be shopping, games, pornography, or anything else.

So, the Internet has shaped what we read, but should it also shape what we write about? Can an algorithm have a better feel for what readers want than any human editor? Should content generation be guided by the "wisdom of crowds" that is lucrative Google search terms?

People produce content on blogs and other Web sources for a variety of reasons, but the primary one is profit. Profit is almost always directly proportional to how much traffic a site can attract. Blogs generate traffic both from loyal readers and from searches.

Some searches will generate traffic serendipitously, in ways that the author could not have anticipated. For example, my friend's blog has gotten several hits for the term "why maryland sucks" because of a flippant post title, yet it's doubtful that any of those searchers were interested in his discussion from 4 years ago about Wal-Mart's health benefits.

So, Demand Media's plan is to find these underrepresented keywords and generate something for users to click on. Under its business model, writing what interests you or what you think your readers will want is inefficient.

Cynically, I have to wonder if Demand Media has much incentive to produce quality content. The New York Times cites one example of less-than-stellar work:

But another query I randomly typed in at eHow — how to roof a house — yielded an article that started, “These are the basic instructions on how to roof a home wether (sic) you are a home owner trying to learn how to roof your home a (sic) wanting to get into the roofing buisness. (sic)”

If you end up on an article that is not exactly engrossing, it makes you more likely to leave the page but may also make you more likely to click an advertisement in search of more pertinent information, which generates revenue for the publisher. While content of a certain quality and quantity can generate loyal readership and increased prominence in Google searches (via more links to the site and a higher PageRank), maybe there is something to be said for quickly generating content to fit holes in Google's knowledge base in hopes of attracting a few random ad clicks, even if the content won't draw a long-term audience.