Who do you blame when AI projects fail? The technology? Your machine learning and data science team? Vendors? The data? Certainly you can put blame on solving the wrong problem with AI, or applying AI when you don’t need AI at all. But what happens when you have a very well-suited application for AI and the project still fails? Sometimes it comes down to a simple approach: don’t take so long.
At a recent Enterprise Data & AI event, a presenter shared that their AI projects take on average 18 to 24 months to go from concept to production. This is just way too long. There are many reasons why AI projects fail and one common reason is that your project is taking too long to go into production. AI projects shouldn’t be taking 18 or 24 months to go from pilot to production. Advocates of best-practices agile methodologies would tell you that’s the old-school “waterfall” way of doing things that’s ripe for all sorts of problems.
Yet, despite the desire to be “agile” with short, iterative sprints of AI projects, organizations often struggle to get their AI projects off the ground. They simply don’t know how to do short, iterative AI projects. This is because many organizations are running their AI projects as if they were research-style “proofs-of-concept”. When companies start with a proof of concept (POC) project, over a pilot, it sets them up for failure. Proof of concepts often lead to failures because they don’t aim to solve a problem in the real world, but rather focus on testing an idea using idealistic or simplistic data in a non-real world environment. As a result, these organizations are working with data that isn’t representative of the real world data, with users who aren’t heavily invested in the project, and potentially not working in systems where the model will actually live. Those who are successful with AI projects have one simple piece of advice: ditch the proof-of-concept.
AI Pilots vs. Proof of Concepts
A proof-of-concept is a project that is a trial or test run to illustrate if something is even possible and to prove your technology works. Proof of concepts (POCs) are run in very specific, controlled, limited environments instead of in real world environments and data. This is much the way that AI has been developed in research environments. Coincidentally, many AI project owners, data scientists, ML engineers and others come out of that research environment they are very comfortable and familiar with.
The problem with these POCs is they don’t actually prove if the specific AI solution will work in production. Rather, they only if it will work in these limited circumstances. Your technology may work great in your POC but then fall apart when put into production with real world scenarios. Also, if you run a proof of concept, you then might have to start over and run a pilot causing your project to run much longer than originally anticipated which might lead to staffing, resource, and budget issues. Andrew Ng ran into this exact problem when they tried to take their POC approach to medical image diagnosis to a real world environment.
Proof-of-Concept Failures Exposed
POCs fail for a variety reasons. The AI solution might only have been trained on good quality data that doesn’t exist in the real world . Indeed this was the reason cited by Andrew Ng for the failure of their medical imagery AI solution that didn’t work outside of the well-groomed data confines of Stanford hospitals. These POC AI solutions could also fail because the model hasn’t seen how real users as opposed to well-trained people will interact with it. Or, there is a problem with the real world environment. As a result, organizations that only run projects as a POC won’t get the opportunity to understand these issues until you’re too far along.
Another case in point with POC failure is with autonomous vehicles (AVs). AVs often work very well in controlled environments. There’s no distractions, no kids or animals running into the road, great weather, and other common issues drivers face. The AV performs very well in this hyper controlled environment. In many real-world scenarios, AVs don’t know how to handle many specific real-world issues.There’s a reason we don’t see level 5 autonomous vehicles on the road. They only work in these very controlled environments and don’t function like a pilot that can be scaled up.
Another example of AI POC systems failing is Softbank’s Pepper robot. Pepper, now discontinued as an AI project, was a collaborative robot intended to interact with customers at places such as museums, grocery stores and tourist areas. The robot worked very well in test environments but when rolled out to the real world it ran into issues. When deployed in a UK supermarket, which had much higher ceilings than US supermarkets where it was tested, Pepper was having difficulty understanding the customers. It turns out it was also scaring the customers. Not everyone was excited to have a robot approach them while shopping. Because Pepper wasn’t actually tested in a pilot, these issues were never properly discovered and addressed causing the whole release to be pulled. If only they had run a pilot where they rolled out the robot in one or two places first in a real world environment, they would have realized these issues before sinking time, money and resources into a failed project.
Building Pilots vs. Proofs-of-Concept
As opposed to a POC, a “pilot” project focuses on building a small test project in the real world, using real-world data in a controlled, limited environment. The idea is you’re going to test a real world problem, with real world data, on a real world system with users who may not have created the model. This way, if the pilot project works you can focus on scaling up the project versus applying a POC to an entirely different environment. As a result, a successful pilot project will save an organization time, money and other resources. And if it doesn’t work you find out what the real world issues are quickly and work to address those issues to make your model work. Just like a pilot that guides a plane to its final destination, a pilot project guides your AI solutions to a destination that is production. Why spend potentially millions on a project that may not work in the real world when you can spend that money and time on a pilot that then only has to be scaled up to a production level? Successful AI projects don’t start with proof of concepts, they start with pilots.
It is much better to run a very small pilot, solving a very small problem that can be scaled up with a high chance of success rather than trying to solve a big issue with a proof of concept that could fail. This approach to small, iterative successes focusing on pilots is a cornerstone of best-practice AI methodologies such as CRISP-DM or CPMAI that aim to give guidance on how to develop small pilots using short iterative steps to obtain quick results. Focusing on the highly iterative, real-world AI pilot will ground your project in that one simple method that many AI implementers are seeing with great success.
