Why Algorithms Alone Are Not Enough

In 2006 Netflix offered a million dollar prize to any individual or group that could improve their movie recommendation algorithm by more than 10%. After an entire year of effort one team,  Korbell (ATT Research), won an interim progress prize for improving the Cinematch algorithm by 8.34 %. This was achieved through the use of one hundred and seven (107) different algorithms used together in what is called an ‘ensemble’. Two of the algorithms, Singular Value Decomposition, and Restricted Boltzman Machines, performed well and were adopted to some extent by Netflix.

It took another two years for a combined team led by Korbell, to actually surpass the 10% improvement goal. But the technical approach used to finally win the contest proved too complex to actually implement within a production computing environment by Netflix. During the three year duration of the contest, there were over twenty thousand entrants, including some of the finest mathematical minds in the world.

Google, a company whose core line of business is intrinsically dependent on algorithms, uses the knowledge of people, in order to enhance the accuracy and completeness of their search results. As one example, information is taken from data repositories that are maintained by humans, such as Wikipedia, Freebase, and the CIA World Factbook, and then displayed alongside that of typical search results. In another instance people manually review two sets of search results that are produced through an ambiguous query, such as “what does king hold”. The individual is then asked to select the result set that makes more sense. As Scott Huffman, an engineering director in charge of search quality at Google states,  “There has been a shift in our thinking, a part of our resources are now more human curated.”

At Facebook, although algorithms manage to a degree what users see as trending news topics, people also play a crucial role in the selection as well. In 2014 Facebook assembled a team of ‘news curators’ in New York and tasked them with identifying the most commonly discussed topics on the social network. As SC Moatti, a former Facebook product leader stated, “Facebook’s news feed team needs a human touch because ranking based purely on algorithms would feel unnatural, the same way that robots do not appear quite human”.  What is very interesting to note is that very recently accusations have been made that these very same news editors have been politically biased in terms of the content they select. This led to Facebook to having to respond to an inquiry from the United States senate.

What conclusion can you reach when weighing all of the facts stated above? The first realization that one must make, is that we have just about reached the physical limits of what mathematics can accomplish when used by itself.  Simply put, algorithms have been maxed out. If one were to take the approach of having ‘human helpers’ such as what Google and Facebook have done, then the expected outcome is a somewhat better. But, we are still burdened by the fact that the human mind itself is imperfect, and is susceptible to what is known as ‘cognitive biases’, which directly affect rational decision making.

What is the best solution then? To use algorithms in conjunction with the software based simulation of human like thought processes, which can eliminate the unwanted cognitive bias, as well as other types of errors in human judgment. This is exactly what we are working on at nTeligence, systems that incorporate ‘hybrid models of cognitive behavior’, which leverage algorithms when needed, along with simulated human thought patterns, sans the biases, and errors in judgment. This purely digital, integrated approach, provides the best of both the art and science of decision making.

For Additional Information Please Contact:

nTeligence Sales Dept.

email: sales@nteligence.com

voice: 561-922-8054


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s