The AI chatbot ChatGPT, created by OpenAI, has captivated the public’s imagination with its ability to summarize complex topics and engage in long conversations. This has led to other AI companies racing to release their own large language models (LLMs), the technology behind chatbots like ChatGPT, which will be incorporated into various products, such as search engines.
ChatGPT-4’s Poor Performance on Wordle Puzzles
Despite ChatGPT-4 being trained on about 500 billion words, including all of Wikipedia, all public-domain books, huge volumes of scientific articles, and text from many websites, it struggled to solve simple word puzzles like Wordle. Wordle is a game in which players have six tries to guess a five-letter word, with the game indicating which letters, if any, are in the correct positions.
In testing ChatGPT-4 on a Wordle puzzle with the pattern “#E#L#”, where “#” represents unknown letters, and the answer was “mealy”, five out of its six responses failed to match the pattern. This poor performance provides insight into how LLMs represent and work with words and the limitations this brings.
The core of ChatGPT is a deep neural network that maps inputs to outputs, with both inputs and outputs being numbers. However, since ChatGPT-4 works with words, a computer program called a tokenizer is used to translate words into numbers for the neural network to work with them.
The tokenizer maintains a huge list of words and letter sequences, called “tokens,” which are identified by numbers. When the user enters a question, the words are translated into numbers before ChatGPT-4 processes the request. This process doesn’t capture the structure of letters within words, so ChatGPT-4 struggles with solving simple word puzzles or formulating palindromes.
However, LLMs are relatively good at generating other computer programs. For example, ChatGPT-4 was able to write a program for working out the identities of missing letters in Wordle, although the initial program it produced had a bug in it.
Possible Solutions for Future LLMs
There are two ways that future LLMs can overcome the limitations of working with words. First, the training data could be augmented to include mappings of every letter position within every word in its dictionary. Second, future LLMs could generate code to solve problems like word puzzles.
Although we are in the early days of these technologies, insights like this into current limitations can lead to even more impressive AI technologies in the future.
Leave a Reply