Predict trends hours beforehand
The algorithm is claimed to predict with 95 percent accuracy the topics that will show up on Twitter’s trending topics list.
It can make these predictions an average of an hour and a half before Twitter lists the topic as a trend, and can sometimes predict trends as much as four or five hours in advance.
Devavrat Shah, associate professor in the electrical engineering and computer science department at MIT, and MIT graduate student Stanislav Nikolov, conducted experiments using data from 200 Twitter topics that were listed as trends and 200 that were not.
How it works
The algorithm is a nonparametric machine-learning algorithm, meaning it makes no assumptions about the shape of patterns.
It compares changes over time in the number of tweets about a new topic to the changes over time seen in every sample in the training set.
Also, training set samples with statistics similar to the new topic are more heavily weighted when determining a prediction. Shah compared it to voting, where each sample gets a vote, but some votes count more than others.
This method is different from the standard approach to machine learning, where researchers create a model of the pattern whose specifics need to be inferred.
In theory, the new approach could apply to any quantity that varies over time (including the stock market), given the right subset of training data.
95% accuracy in forecasting
In addition to the algorithm’s 95 percent prediction rate, it also had only a four percent false-positive rate.
The accuracy of the system can increase with additional training sets, but the computing costs will also increase.
Shah said the algorithm had been designed to execute across separate machines, such as web servers. “It is perfectly suited to the modern computational framework,” said Shah.