My opinion it marketing. Look what our model can do when it "escapes". Great way to drum up government contracts. Also a way to make the market want AI to slow down. Aka hide it crap and can't do 5% of the hype. More on reddit.com

I will address the so-called existential threat of AI and what at least some prominent thinkers have argued. First, however, an existential threat is a threat with the ability to eradicate all of humanity. Some thinkers are more concerned with other possible harms of AI, such as instantiating sexist and racist beliefs within algorithms. Although we have good reasons to want to avoid that, it is hardly an existential threat. Central to many arguments about the existential threat of AI, is something called instrumental convergence . This is best illustrated with an absurd example... I get to the more realistic line of reasoning later. Key to an agent (that is an entity that can take acts, either human or non-human) is the objective. What is the agent trying to do. In many AIs (not all), the objective is fairly well defined sometimes as a function that we can numerically evaluate. A naive idea is to say that if we make the objective "nice" or "good" or "human friendly", then the problem is solved... the AI agent will act nice. And for the sake of argument, let us say an AI agent has been programmed with the objective to "put a smile on the faces of its human subjects". An AI agent that is powerful enough (key premise) may now pursue that objective in a somewhat too literal fashion. One way to put a smile on human faces is to force surgery on people, or perhaps convince the human subject to inhale lots of laughing gas etc. These are perverse instrumental acts of the AI that in a literal sense accomplishes the objective. One especially critical instrumental objective this very powerful AI converges to is that it should remain switched on. If the AI is switched off, then it is guaranteed to not be able to do its job. So although the objective isn't explicitly saying anything about the AI trying to stay switched on, that does follow instrumentally. This is the argument against the usual "well, if the AI misbehaves, we just switch it off". Make the AI sufficiently powerful, and the AI will resist being switched off. Think of the many dumb computer viruses that keep replicating in our digital infrastructure. Once outside certain confines, a piece of dumb logic can proliferate beyond devices we easily can switch off, assuming the piece of logic has converged to the instrumental objective of its own survival. This is related to the larger alignment problem, which you can find serious research about, even for non-existential harms. Let me mention a weak version of this issue of instrumental convergence. The recommender algorithms of Facebook and YouTube have the objective to keep you on their webpage and engaged with their ads. For many of us, we engage more with content that annoys us and angers us. So a recommender algorithm may instrumentally converge on serving you content about . This increased prevalence of idiocy in our feeds may make us alter our beliefs about our world -- everyone is an idiot, just look at this video of X saying Y. The point here is that in no recommender algorithm is this objective explicitly stated, it rather turns out we converge to it. To be clear, we can argue these specific concerns about social media are exaggerated. The point is that the mechanism that is suggested builds on that "nice" objectives has in theory the potential to become instrumentally pernicious. We do not have to assume evil intentions of parts of The Establishment (whatever they are). A key point in this whole argument is that the AI is powerful enough. Well, how powerful is powerful enough? This is tricky. Currently, ChatGPT interacts with us through text. Of course, some people send money to Nigerian Princes who write to them over e-mail, so let us agree the text interface has some power in the real world. Add image generation and a few images of a busty lady can instrumentally extract more cash. It does seem, however, that most of the existentially bad scenarios require more direct capabilities to reach out into the world by the AI agent. Say, if the AI agent becomes capable enough to hack and reprogram devices at a massive scale. So suddenly the AI agent starts to blackmail you because it has hacked your most private confessions. Ok, now the AI agent may be able to make you do stuff, the same way hackers and spies use honey pots and other tools of online extortion. Add the ability to reprogram robotics, and the powers to do harm increase. But ok, how do we even get there? These things of hackers, spies and seductive Nigerian princes are already with us and though they do harm, we build institutions and tools and procedures and surveillance to limit their harm. This is where "intelligence explosion" comes in. The idea is that once an AI agent is capable enough that it can enhance itself, that's when we have reached the point of no return. Suddenly this AI agent goes from being a doofus that can't do proper arithmetic or pass Gary Marcus' reasoning tests, to becoming an expert chemist, super perceptive psychologist and god-level hacker because it trains itself at an exponential rate. And boom, we are face-to-face with the most sci-fi-powered AI agent that can think of things we cannot design preventions again, and this Shiva is now running amok in our digital infrastructure because of instrumental convergence. In this reasoning, once we have an AI that can take acts towards its own improvement, then all that follows is a slippery slope to HAL 9000 (an interesting fictional example of an AI agent, which has taken its objective to preserve and help mankind to such extreme that the agent deliberately kills its human subject in order to prevent the meeting with the Jupiter monolith). And some see in GPT-4 that we are closer to that edge. Hence their worries. There is plenty one can argue against this reasoning. The case outlined above invites many counterarguments. However, it is not simply that some people have watched too many Schwarzenegger movies. There is a serious argument here. Still, reasonable people can disagree and I think there are good arguments against this bleak vision. Note also that this does not assume consciousness. We do not have to think of AI as consciously evil for the aforementioned scenarios... only as very powerful. A good book that collects these arguments is Superintelligence by Nick Bostrom from a few years ago. More on reddit.com

Practically speaking the most terrifying aspects are its military application and ability to concentrate wealth in even fewer hands. This, combined with the impending hype deflation, will invariably lead to resource shortages. In a more surrealistic vision, if it ever crosses the virtual-to-physical barrier, i.e. obtains a physical body, then the inevitable conflict with humanity. Nature has repeatedly taught us one thing - faster, smarter, stronger rarely has any patience for the weak. Maybe our grandchildren will some day live in a comfortable zoo as a curiosity from antiquity. More on reddit.com

Y2K is such a terrible example. We took the threat seriously, made the necessary changes, then (almost) nothing happened...BECAUSE WE TOOK IT SERIOUSLY. More on reddit.com