Models and Beauty Contests

Organisations of all shapes and sizes are racing to build new applications and services, based on latest generation AI models. In November, OpenAI claimed that 92% of Fortune 500 companies, the largest businesses in the US, are using their tools already. As with other Keynesian Beauty Contests there are benefits to picking the same option as everyone else, particularly access to resources, tooling and developers. Closer to home, many of the UK-based companies we work with are developing prototypes and full products, often using OpenAI’s GPT-4. Amongst everyone we speak to, though, a question lingers: How can I build (or buy) when the tech is changing so fast we might be out of date before the project is halfway finished? 

‍

Whilst the media often treats ChatGPT as the totality of AI developments, the real transformation has come from proprietary systems which integrate the underlying models directly. And there are lots of them out there. At the very high end, OpenAI’s GPT-4 has topped general performance charts, but can’t be run on an organisation's own hardware, raising all manner of data & privacy challenges. Once unmatched, Google has now announced a rival product launching in the coming weeks. Beyond these two, there are hundreds of proprietary and Open Source models, many of them as good as the version of ChatGPT which made headlines a year ago. 

‍

Nobody can tell you what the best choice will be twelve months from now, because the rate of change and potential for specialisation makes such predictions idle speculation. This leaves organisations with three choices: 

‍

• Wait until the dust settles 
• Pick an option and hope that it wasn’t the wrong one down the line 
• Design and build for flexibility, allowing models to change later. 

‍

Unsurprisingly we recommend the latter. Whilst more costly in the short term, the flexibility is essential to insuring projects against sunk costs and redundancy a few months down the line. 

‍

Reasons to go with a model agnostic, modular & flexible design

‍

1. Model capabilities and costs vary. Whilst the most general systems achieve performance on a wide variety of tasks, smaller specialist models will be cheaper to run and may provide as good, if not  better performance. That is, if you can integrate them as they emerge. 
2. Open Source models are improving rapidly. The best models are currently pay-per-use and costs can quickly rack up. Open Source models can be hosted and run by anyone, which may prove significantly cheaper. Whilst they aren’t good enough yet to rival state of the art models, they are already sufficient for many tasks and improving quickly. 
3. OpenAI is still a start up. The longest lasting impact of November’s leadership coup, in which 90% of employees threatened to resign, is the realisation that, unlike industry stalwarts such as Google or Microsoft, OpenAI is still a rapidly evolving company. Many organisations are wondering quite how much reliance to place on such a young company, not to mention one with a highly atypical board structure. Sometimes, slow and steady pays dividends in the long run. 
4. The best is still to come. As technological capabilities shift and best practices emerge, avoiding getting stuck with technical debt (i.e. the negative impacts of previous tech decisions) is imperative. Imagine being forced to keep using fax machines as email emerged because all communications were designed to be paper based... It’s always preferable to design a system that can take advantage of technological advances readily. 

‍

In the face of a changing world, refusing to act until you can be sure of the outcome is a decision in and of itself. Sometimes wait and see is the correct decision. Other times, it is an invitation to competitors to steal a march. Uncertainty needn’t mean committing to an action and hoping for the best. A range of strategies, including scenario planning and the use of other Futures & Foresight tools, allow organisations to make decisions now which maximise the chance of getting it right in the future. For technical builds, avoiding being locked in to any one choice is a necessary starting point. 

‍

Paradigm Junction help organisations take decisions now, to prepare for uncertain futures and navigate technology-caused change. To find out more get in touch.