Facebook (now Meta) popularized the Silicon Valley ethos with the saying “Move fast and break things”. This approach might have worked when disrupting the social media business, but it’s causing all sorts of problems for them as well as other major AI players. Breaking things and moving fast might be the reason why so many AI projects are failing. According to an MIT study, over 85% of AI projects fail to deliver their stated objectives, and 70% of data science projects never make it to fruition. Clearly moving fast and breaking things doesn’t work if you’re not getting closer to success.
There’s a difference between Iterating to Success and Breaking Things
The problem with these failures is that since AI is still nascent, despite over seven decades of development, organizations are tempted to give up altogether without ever reaching the success point. Part of the reason for this is that the move-fast-break-often ethos doesn’t work when data is the cornerstone of AI project success. Rather than focusing just on moving fast, organizations realize they need to focus on short iterative sprints that bring AI projects closer to meeting their desired objectives.
From this perspective, rather than just moving fast and breaking things, organizations need to “think big, start small, and iterate often”. This is a bit at odds with the Silicon Valley perception that all innovations need to be disruptive. Perhaps the AI solution might at the end of the day become disruptive, but to get to that point of disruption, AI projects need to first iterate through many stages of “small wins”. However, many organizations simply don’t have patience for those small wins.
The Real-World AI Disconnect
One of the major reasons why AI projects often fail is because in the rush to push things to production, organizations will take a small proof-of-concept done in a controlled environment and push that out to the much messier realities of the real-world. As a result, there’s little surprise that the AI project fails due to a mismatch of expectations and the real world.
There are two versions of this AI-meets-reality problem. The first version is that people are building models before they actually know how it's going to be used in the real world. That means ML engineers, AI project managers, and data scientists are operating on a limited understanding of the real world that faces challenges in the actual real world. This is part of the reason why so many autonomous vehicle projects face significant real-world challenges.
The second version of this problem is that the environment of how you're developing the model does not match the environment in the way that the model is being used. Usually this is because getting real-world data is difficult, messy, or requires a significant amount of time to clean and prepare. Since organizations push themselves on a limited timeframe, rather than dealing with the reality of the real-world messy data, they simply use the best-available subset and train models on that data.
Organizations are now realizing that jumping the gun to get from test to production harms projects more than it helps. The result is a more cautious approach to AI that aims to test a real project in the real world, even if it’s less ambitious and glamorous than a larger effort. This allows the organization to test to see how the real world will benefit from the AI model before implementing it on a larger and more risky project.
This approach to step-wise, highly iterative, low-risk AI projects even applies to use cases as simple as classifying documents using natural language processing. For example, organizations that aim to use NLP to automate a document classifier to separate PDFs into invoices, proposals and contracts using NLP, image recognition, or some other cognitive technology option. The objective is to remove a human from manually separating those documents and therefore speed up the process, reduce errors, lower costs, and eliminate bottlenecks. Upon deploying a first version of the model to perform a small part of that task, the organization may find that the model works but it isn’t actually saving any human labor. Your workers are still spending the same amount of time on classifying documents. At this point you might think the AI just isn’t working, but that’s not always the case. It could be that you built the wrong kind of model. Or, the documents used for training weren't representative of the real world documents. This step-wise iterative approach is popularized by methodologies such as Cross-Industry Standard Process for Data Mining (CRISP-DM) and the Cognitive Project Management for AI (CPMAI) methodologies.
“Think Big. Start Small. Iterate Often”
Even if an organization’s model does match the project expectations, it can still fail because the way in which the model is actually being implemented does not match the way that you assumed. Let’s say you build a model that uses facial recognition to unlock a phone. This model is on an edge device that requires an internet connection or access to the cloud to work. If you’re walking in a forest and are trying to view a map you had saved and don’t have good reception and can’t unlock your phone then you’re going to run into some issues. The system would then have to rely on non-AI backups such as passcodes or other methods for access. In this way, while the model works in some cases, it doesn’t work in many, thereby jeopardizing the project ROI.
One of the main goals for AI systems is to provide real-world functionality that leverages the capabilities of machine learning and other approaches to replace tasks that would have otherwise been difficult or required human labor or decision-making. In this way, the goal is not to just fail fast and break things often, but rather to iterate slowly towards success, even if the end result isn’t the sort of disruption that forms the cornerstone of the Silicon Valley ethos.
