Boom and Bust
I might have mentioned that I love Alice in Wonderland. In fact I might have mentioned this several times. If you know the story (which I hope you do) it begins in Chapter 1 with Alice daydreaming, then seeing a white rabbit checking a pocket watch, a possibly slightly odd sight! She follows the rabbit, and as it disappears into a rabbit hole, she follows it, falling a long way down, ending up in Wonderland. It was a memorable image to introduce a story set in another and quite strange world. Today, the term ‘falling into a rabbit hole’ has been used in many ways, but it can still refer to ending up somewhere unlike where you had intended.
Every so often, I find myself falling down a rabbit hole, especially when thinking about topics for this blog. I can find myself reading more and more about colour blindness, the physical kind, when I meant to write about colour blindness in terms of relationship between people. I can find myself reading more and more about bird migration, and then find I’ve moved on to different sensory skills. I can find myself getting more involved in trying to understand gravity, which leads on to trying to understand the universe, which leads on to trying to think about the origin of the universe, which leads on to … usually, leads on to confusion, and a sense that much of what I’d read I simply didn’t understand.
However, another kind of rabbit hole which temps me is new technologies. I am a sucker for reading about a new device on the market, and developing a desire to acquire it, to be one of the early adopters. So it was with video recorders. Back in the 1970s, in 1975 to be precise, Sony announced the first digital video recorder, the Betamax. It wasn’t pushed hard in Australia for the first couple of years, but soon after that and with some money in my pocket, along with a state of excitement, I bought a Betamax recorder. This was at the cutting edge, a device that could record television programs, as well as give access to pre-recorded material you could buy, and play and play again. The recording medium was a ¾” magnetic tape cassette called U-matic, a now-obsolete format that was developed by Sony. It was the kind of technology designed to appeal to a moderately technically inclined person like me, and I loved the idea of being in the vanguard. Oh, and the recording system was brilliant: played back on a television, it was hard to believe this was only a copy.
Today, the Sony Betamax is lost to history, a great device that disappeared, almost completely forgotten. When working with groups at Cisco in California a few years ago, I had the opportunity to visit the Computer History Museum in Mountain View. There I could indulge my desire to look at examples of Babbage’s Differential Engine, parts of the first computers life EDVAC, and more. On one visit, I decided to look at objects from the recording side of the industry, like tape decks and massive disks. I couldn’t find a Betamax there, but there was a description of what had happened. I knew JVC had released the competing VHS format in 1976, a year after Sony’s Betamax, but I also remembered that, at the time, Betamax was higher quality than VHS, if also somewhat more expensive. VHS machines could record a full movie, while to begin with the Betamax was limited to only one hour. More to the point, Sony refused to license their system format to other manufacturers, but JVC built an ecosystem of partners and offered a greater selection of films.
Sony’s Betamax was in trouble. By 1980, VHS had proven favourable among consumers and was successful in controlling 60% of the North American market. By 1981, sales of Beta machines in the United States had sunk from 100% in 1976 to 25% of the VCR market. As movie studios, video studios, and video rental stores turned away from Betamax, the combination of lower market share and a lack of available titles further strengthened VHS’s position. What Sony hadn’t considered was what consumers wanted. While Betamax was believed to be superior format in the minds of technologists and press (due to excellent marketing by Sony), consumers preferred an affordable VCR (which often cost hundreds of dollars less than a Betamax player. By the middle of the 1980s, VHS had won the war. Sony saw what was happening and redirected its efforts. Building on the technologies they’d learnt with Betamax, they were to lead the field in camcorder development by the end of the 1980s.
So much for the Betamax. However, my interest in that technology was part of what me led, many years later, to teaching innovation. Indeed, that was the reason I was contributing to some courses at Cisco, encouraging staff to think carefully about innovation (at the time I was discovering the Betamax video recorder didn’t even rate a mention in the Computer History Museum – yes, I know because it wasn’t a computer!). My focus was linking innovation to entrepreneurship. Innovation is the task of finding something new, and while it can be rewarding to be an inventor, success comes from application. Entrepreneurs are the people who see there is a gap or an opening in a market that is being ignored or missed, and who are able to identify ways to exploit that gap. Unlike innovators, entrepreneurs are driven by business or organisational sense: take a promising and innovative idea and create a business. Most are in startups, but without getting too technical, there is an in-company variant of the entrepreneur, the intrapreneur, who does the same thing: sees opportunities and finds ways to meet those gaps but does so within the organisation.
This is an area full of rabbit holes. When Alice fell down the rabbit hole, she had an adventure, courted with disaster, and eventually returned back above ground in one piece. Multiply her experience a thousandfold, most new products fail or end in disaster, and that is a simple summary of the life of entrepreneurs. Leaving on one side innovations that never got off the ground (there are so many of them), history is full of innovations taken up by entrepreneurial individuals and organisations that have seen limited success. However, what makes this especially curious feature of this is that there are successful innovations that do build businesses, but many of them will still eventually collapse or disappear. The timescale for the Betamax might have been shorter than several others, but the path is familiar.
What are some major innovations in the 20th Century? We are well aware of successes. I could mention prescription drugs, cars, or plastic. Actually, that was a bit sneaky, because these are innovations that have lasted, but have brought with them all sorts of problems and challenges. Prescription drugs continue to see innovations, although now most new drugs are simply tweaking existing treatments, and often with unfortunate side-effects for users. We have become a drug-dependent society, and we live with the problems they cause in the rather odd hope that the drugs are good for us: some are, but many bring as many problems as they solve. Drugs exemplify the Betamax problem: when the innovative product doesn’t work as hoped, then a competitor will come up with one that is ‘better’. Remember, too, that VHS replaced U-Matic tapes … until it no longer did so. Now we record on solid-state interfaces, originally laser discs, and now thumb drives and the like.
Perhaps cars are a better example. The motor car has been around from some 140 years and must be seen as a successful innovation. Thank you, entrepreneurs, like Henry Ford, Alfred Sloan and more. I suppose it would be churlish of me to point out that the motor car has transformed how we live, where we live, and how we use our time, and not all of this has been positive, with suburbia, motorways, interstates, ubiquitous fuel outlets, drive-in stores and more, with the additional ‘downside’ that people continue to die in traffic accidents, and many more are permanently injured. It is easy to forget that the internal combustion engine wasn’t the only key factor, it was the creation of a massive private transport network. Although there is some small evidence the young are not as enamoured of cars as previous generations, it still seems the case people want to own their cars. I could make similar comments about the ubiquity of plastic, which has highlighted a problem at that motor cars also present, which is the challenge of getting rid of either when no longer wanted. Scrap metal yards full of squashed cars are obvious; millions of tons of plastic gathering in the oceans are less easily noticed.
Some innovations have been successful for a long time, and then suddenly recognised as dangerous. Some well-known examples of this ‘boom and bust’ process are leaded petrol, DDT, or chlorofluorocarbons. Now we know these were disastrous technologies, and only if we had known more at the time they should never have been adopted. Actually, I shouldn’t have included leaded petrol as an example, as we knew the dangers of lead right at the start, but leaded fuel allowed cars to run smoothly and efficiently. There are others where we are still having a challenge in recognising they are dangerous. Nuclear fission has to be the outstanding example here. It has been deployed commercially and does generate electricity, but most countries allow only it to provide a small share of energy, and just in case we forget its limitations, we suffer a different kind of ‘boom’ as a plant goes wrong.
All this is by way of an introduction to a current boom, that of artificial intelligence. Will Lockett, in Freemium this month, has given an important assessment of AI in his review AI Is Hitting a Hard Ceiling it Can’t Pass. A lot of what follows comes from that article, which he introduces with the observation; “There has been an insane amount of hype surrounding AI over the past few months. Supposedly, Teslas are going to entirely drive themselves in a year or two, AI will be smarter than humans next year, and an army of a billion AI-powered robots will replace human workers by 2040, and that is just the AI promises made by Elon Musk so far this year. The entire AI industry is awash with predictions and promises like this, and it feels like AI development is on an unstoppable exponential trajectory we humans simply can’t stop. However, that is far from the truth. You see, AI is starting to hit a development ceiling of diminishing returns, rendering these extravagant promises utterly hollow.”
To explain his perspective, he starts with looking at how AI works. Basically, despite all the hype, AI uses what are called ‘deep learning algorithms and artificial neural networks’. These software tools look at data, masses of data, to identify trends and links. To be clear, they are not intelligent in the way we usually think of the term, they are merely identifying that an item of data is frequently and significantly associated with another piece of data to identify trends in the data. Provide enough data, and what seems amazing becomes easy. Scan enough tagged photographs of people, and the AI system can identify each person in new photographs. Scan enough Google queries and answers, and the AI system can ‘answer’ a question from you, by finding examples of the question and the answers that have been provided. This is, of course, why many interrogations of AI systems lead to foolish or dangerous answers, because there are enough crazy people out there providing silly answers to questions on the internet, all of which will be scooped up and used by an AI system.
Lockett notes that AIs have become significantly more capable recently. He points out that this “has been partly due to better programming and algorithm development. But it is also 90% thanks to the fact that AIs have been trained on significantly larger datasets. … But there is a problem; we are seeing drastically diminishing returns in AI training, both in terms of data and computational power needed.” He provides an example: lets “build a simple computer vision AI designed to recognise dogs and cats, and we trained it using images and videos of 100 dogs and cats, and it can correctly identify them 60% of the time. If we doubled the number of training images and videos to 200, its recognition rate would improve, but only marginally to something like 65%. If we doubled the training images and videos again to 400, its improvement would be even more marginal, to something like 67.5%.” This is because as a dataset grows, “finding new and novel trends and connections that work for the entire dataset becomes harder and harder.”
It’s more than that. AI is very resource hungry. AIs are trained by comparing each individual point of data to every other data point in a set to find connections and trends, so that for each bit of data you add to an AI training database, the amount of computational work it takes to train that AI on that database increases exponentially, and with that the amount of physical computing power and energy required grows rapidly. Lockett suggests “there is evidence that we are at a stage where both the diminishing returns of training dataset growth and the exponential increase in computing power required to use said datasets are enforcing a hard ceiling on AI development. … Take OpenAI’s flagship AI ChatGPT4. Its improvement over ChatGPT3 was smaller than ChatGPT3’s improvement over ChatGPT2, and even though it was more accurate, it still had the same problems of hallucinating facts and lack of understanding as ChatGPT3 did.”
He reports investigations have shown ChatGPT3 used a training dataset about 78 times larger than ChatGPT2, and ChatGPT4 uses a dataset 571 times larger than ChatGPT3. Despite this, ChatGPT4 still has significant flaws that significantly limit its use cases. It can’t be trusted to write anything remotely fact-based, as it still makes up facts. “Some estimates put ChatGPT4’s raw training dataset at 45 TB of plaintext. This means that for the next iteration to be as big of an improvement as ChatGPT4 was over ChatGPT3, the training dataset would need to be tens of thousands of TBs.” Accessing and preparing that amount of plaintext data, together with using this dataset to train their AI could use so much energy that it’s unviable.
Guess what. I think we are on the edge of another boom to bust cycle. What AI does is collate a whole lot of information – unthinkingly. It would be like students writing their essays by simply quoting anything they see in books, with explanations or analysis. Oops, that’s what students do now! Seriously, AI is faster at retrieving data than a person can, even in a very good library. However, it can’t think about the data, which requires human skills, making judgements, considering evaluating, and even rejecting what is discovered.
Understanding AI’s ‘unthinking’ process also helps us see its limits. The danger is simple. Unless we are very careful, we are about to go down another rabbit hole, just like those that existed for DDT and chlorofluorocarbons, and should exist for motor cars. We will suffer from reading the results of unthinking AI collated data and believe what we are told. If humans are often foolish, this has the potential to make us foolish slaves to data.
Perhaps a final word should come from an educational perspective. Good teachers know that
real learning doesn’t come from lectures or essays. They can help put come building blocks in place. However, if you want students to gain understanding, insight and creativity, that comes from interaction. Send the students off to read, get them to put ideas on paper (knowing full well it might well be parents or AI that does the work). However, when that initial work is done, sitting round a table and talking about the topic, finding out what was learnt and what was challenging, that’s the process by which we acquire understanding and wisdom. Think of AI as a new and (possibly) better encyclopedia: print versions of those tomes went boom and then bust. I think AI may be heading in the same direction.
