Archive for the ‘data and analytics’ Category

Can You Beat the Algorithm? Take the Retweet Quiz and Find Out

Monday, July 14th, 2014

retweetWhat’s it take to get retweeted on Twitter? Three computer scientists decided to find out. They created an algorithm that sorts through flows of social media data to determine which of paired tweets (i.e., “two tweets about the same link sent by the same person”) is more likely to be retweeted. The scientists wanted to find out if certain word patterns, phrase lengths, vocabulary choices and other content variables were predictive for which of two tweets on the same subject by the same writer would be retweeted more often.

After running through some 11,000 pairs of tweets, the algorithm got pretty good at predicting which tweet is more retweetable. Pretty good, but not outstanding. According to the New York Times, the algorithm “can guess which tweet gets retweeted about 67 percent of the time, beating humans, who on average get it right only 61 percent of the time.”
The Times developed a 25-question quiz Can You Tell What Makes a Good Tweet? to measure whether humans can beat big data analytics when it comes to guessing which tweets get retweeted. Take the quiz and see how you perform against the algorithm.

Tweet this: Can you beat an algorithm at picking top retweets? via @m_sendhil Take the quiz and find out

So, if an algorithm can predict retweeting patterns, can we use its insights to write better tweets (assuming that retweeted tweets were better, more engagingly written)? Well, not so much.

The study found that asking for what you want is a good strategy: People are very suggestible. Using the words “retweet” and “please” in tweets resulted in more retweets. Using unusual or novel words or phrases also seemed to be predictive of retweeting. However, once you start reusing attention-grabbing language, it quickly becomes less so: “Once an algorithm finds those things that draw attention and starts exploiting them, their value erodes. When few people do something, it catches the eye; when everyone does it, it is ho-hum.”

It seems that longer tweets are more likely to be retweeted than shorter tweets. Of course, given that this is Twitter, you can push length only so far. And don’t start maximizing tweet lengths with the expectation that you’ll automatically get retweeted more often. The upshot is that longer tweets have more content, and more content is more interesting than less content, so content-rich tweets will get retweeted more often. “So the lesson is not ‘make your tweets longer’ but ‘have more content,’ which is far harder to do.”

Turns out that there’s no secret formula for writing tweets that succeed in getting retweeted. Instead, write creatively about interesting content and you’ll get retweeted more often. That may seem apparent, but writing good tweets takes time, wit, and attention to detail to get right. If you want to win at the retweeting game, then you need to bring your A-game.

Let us know how you did on the quiz – and tell us if you have advice on writing tweets that get retweeted.

Big Data Analysis meets the Liberal Arts: Will Lit Crit Ever Be the Same?

Friday, May 23rd, 2014


As a tech B2B writer here at McBru, I write a good deal about Big Data, the enormous information flows derived from sensors, social media feeds, and device-to-device communications. For example, in an average airplane there are more than 50,000 sensors constantly monitoring everything from electrical flows to air quality. These sensors create between 5 to 6 petabytes of data per flight (a petabyte is a million gigabytes), and sifting through this information to detect patterns and anomalies is the job of big data analytics, a major new frontier for technical research.

But not so many years ago, while in university, I studied French and English literature and read hundreds of novels, scores of poems, and piles and piles of learned treatises on my way toward a graduate degree. We didn’t have the concept of big data back then, but I can see now that I was acting as my own big data analytics solution as I powered through all those pages in search of “actionable intelligence.”

So I shouldn’t be surprised that cutting-edge literary theory has embraced big data analytics to render new insights into the old-school study of literature. I have chanced across a number of articles recently that describe how the established literary canon has yielded new secrets after being processed through the algorithms of big data analysis.

  • In “Shakespeare’s Data,” in May’s hardcover edition of Wired magazine, Clive Thompson describes how two PhD students at the Stanford Literary Lab fed the content of 2,958 19th century novels through a series of big data analytics tools. One interesting pattern to emerge was that, as the century progressed, words describing action and body parts became more prevalent. The researchers concluded that increasing urbanization during the 19th century brought people closer together physically and people’s bodies and actions were increasingly difficult to ignore. Seemingly, after the industrial revolution, no one was far from the maddening crowd.
  • In The Data-Mining’s The Thing: Shakespeare Takes Center Stage In The Digital Age from Fast Company, Neil Ungerleider writes that the “same techniques used by businesses to analyze web content and by marketers to target audiences […] have big ramifications for Shakespeare–and have helped settle long-standing academic arguments.” Officials at the Folger Shakespeare Library fed portions of the Bard’s plays through rhetorical analysis tools and data-mining technics to discover distinct linguistic similarities between the tragedy Othello and Shakespeare’s comedy plays. In particular, the comedy Twelfth Night recycles a number of linguistic conventions and themes found in Othello.
  • Shakespeare, Herman Melville and today’s hip-hop artists were on the mind of data scientist Matt Daniels. He wanted to determine how the vocabulary of hip-hop artists stacked up against these two giants of literature. Using a research methodology called token analysis, Daniels compared 35,000-word data sets from the writings of Shakespeare, Melville, and 85 hip-hop performers (he used the first 5,000 words of seven of Shakespeare’s plays, the first 35,000 words of Moby Dick, and 35,000 words from the lyrics of published songs by the 85 performers in question). The biggest vocabulary? Somewhat surprisingly, the rapper Aesop Rock came out on top with 7,392 unique words used within his data set. Melville was certainly near the head of the class, with 6,022 unique words, while that slouch Shakespeare was closer to the middle of the pack with the use of 5,170 unique words in his data set.

Big data analysis will probably never dislodge more traditional literary theory from the classroom, but it can help tease out unexpected patterns and linguistic relationships and offer insights into language and themes that are invisible to more conventional critiques. And it’s kind of cool to realize that every book in your library is in fact, for better or worse, a big data flow.