Social analytics' questionable conclusions

IBM served up some unusual results with its latest volley of social media analytics at this year's  Australian Open. It looks like Victoria Azarenka didn't just pick up her second Australian Open title but was along the way crowned the tournament's most popular player.

That announcement would seem a tad surprising, given that  Azarenka's timeouts during her semifinal left a bemused crowd somewhat hot under the collar. 

As you may recall, Azarenka’s status took a tumble because of an incident during her match with up-and-coming US player Sloane Stephens. She's accused of abusing the injury time-out rule to regain her composure during the match. 

The incident sparked a wave of criticism of Azarenka from the Twittersphere and the mainstream media.

 

Yet despite the invectives, IBM’s social media analytics system found that she was the most popular player during the Open. She even outranked Stephens, whose surprising win against world champion Serena Williams saw her Twitter following triple overnight. She didn’t even make IBM’s top ten.

When pressed on the matter, the IBM said:

The most popular players were determined by the number of positive mentions they received across the full two weeks of the tournament on social media channels including Twitter, Facebook, news sites, blogs and videos. As an overall position for female players, Azarenka demonstrated her strong connection with fans, resulting in more than 97 per cent positive sentiment across the social media channels.

But can this really be right? How can a tennis player like Azarenka suffer such a fallout and still be crowned the most popular player?

The episode raises an interesting question about the accuracy of social media analytics. It’s easy to tell how many likes, tweets or clicks your posts are drawing, but when it comes to measuring sentiment or opinion from social media, things get decidedly murky.

It’s an important question to ask given that more and more companies are using social media to gauge and act on the attitude of their consumers. Social media marketing firms are leading this charge, but we’re also seeing the trend create business models. For instance, UK investment firm DCM Capital buys and sells shares based on the analytics they run on social media. Though the firm, perhaps in a testament to the viability of the idea, recently put itself up for sale

How measuring social media sentiment works

At its most basic level, all social media analysis is really just a keyword search across all the forms of social media across the net.

As marketing firm Mindshare’s head of social Mandi Bateson explains, marketers can use tools to apply filters to this keyword search to ensure that what is being measured is relevant to what they are looking for.

For instance, somebody searching for all Twitter mentions of Apple in Australia will use filters to ensure that all the results are from Twitter, were posted by users in Australia and are talking about Apple the company rather than Apple the fruit.

When it comes to measuring sentiment, marketers divide it into three categories: positive, neutral and negative. One of the more complex points about sentiment Bateson says,  is that  “each client has a different idea of what is positive”.

“For example if the mention said ‘I am drinking Brand X’ one marketer might deem that to be neutral because they haven’t said whether they liked it or not, while another marketer may believe the fact that they’re drinking the product and tweeting about it is positive.”

The need for a “human filter”

Perhaps the biggest limitation of social media analytics, is that unlike other forms of analytics, a machine can’t do it on its own.

Bateson says that because as things stands there will always be a need for a “human filter” to ensure that the insights that you draw from Twitter and Facebook are accurate.

“While the tools are evolving and improving their algorithms, there will always be a complexity to our language that only a human can determine,” Bateson says.

“Australians don’t make it easy – we’re fond of sarcasm, rhetoric understatements and double negatives. We say ‘not bad’ when we mean ‘good’.”

“Accurate automated sentiment analysis would require more intelligent text processing that we currently have at our disposal so for now it means someone needs to go through each mention and apply the correct sentiment category.”

Yet despite this flaw, Bateson says there are still uses for social media analysis in its current form. Her company, Mindshare finds it to be particularly helpful in monitoring social media campaigns and mitigating crisis situations.

When it comes to campaigns, Bateson says that social media analysis can help “evaluate what campaign elements got people talking and the honest, immediate reaction from the audience.”

Whereas in a crisis situation it can help “monitor where negative mentions are originating or spreading so we can resolve the issue and address any misinformation quickly and effectively,” Bateson says.

“We can also identify which social networking users are talking about us most often and whether they’re an advocate or a detractor,”

As things stand, social media analysis isn’t a broken tool, but it has its limits. And it’s this point that is perhaps missing amidst all the hype around this new untapped pool of data.

Social media analysis systems aren’t mass mind-readers, and therefore shouldn’t be used to draw targeted conclusions, not sweeping statements. While the data in its current form might hint at one outcome, real-life and common sense may point to another. 

More from Business Spectator