Group items tagged

Part of the enthusiasm behind the sky-high valuation of Tesla, where he is chief executive, comes from his predictions for the auto company’s abilities to develop humanoid robots—dubbed Optimus—that can be deployed for everything from personal assistants to factory workers. He’s also founded an AI startup, dubbed xAI, that he said aims to develop its own superhuman intelligence, even as some are skeptical of that possibility. 

Musk likes to point to another work of Sci-Fi to describe how AI could change our world: a series of books by the late-, self-described-socialist author Iain Banks that revolve around a post-scarcity society that includes superintelligent AI. 

That is the question.

“We’re actually going to have—and already do have—a massive shortage of labor. So, I think we will have not people out of work but actually still a shortage of labor—even in the future.” 

Musk has cast his work to develop humanoid robots as an attempt to solve labor issues, saying there aren’t enough workers and cautioning that low birthrates will be even more problematic. 

Instead, Musk predicts robots will be taking jobs that are uncomfortable, dangerous or tedious. 

A few years ago, Musk declared himself a socialist of sorts. “Just not the kind that shifts resources from most productive to least productive, pretending to do good, while actually causing harm,” he tweeted. “True socialism seeks greatest good for all.”

“It’s fun to cook food but it’s not that fun to wash the dishes,” Musk said this month. “The computer is perfectly happy to wash the dishes.”

In the near term, Goldman Sachs in April estimated generative AI could boost the global gross domestic product by 7% during the next decade and that roughly two-thirds of U.S. occupations could be partially automated by AI. 

Vinod Khosla, a prominent venture capitalist whose firm has invested in the technology, predicted within a decade AI will be able to do “80% of 80%” of all jobs today.

“I believe the need to work in society will disappear in 25 years for those countries that adapt these technologies,” Khosla said. “I do think there’s room for universal basic income assuring a minimum standard and people will be able to work on the things they want to work on.” 

Forget universal basic income. In Musk’s world, he foresees something more lush, where most things will be abundant except unique pieces of art and real estate. 

“We won’t have universal basic income, we’ll have universal high income,” Musk said this month. “In some sense, it’ll be somewhat of a leveler or an equalizer because, really, I think everyone will have access to this magic genie.” 

All of which kind of sounds a lot like socialism—except it’s unclear who controls the resources in this Muskism society

“Digital super intelligence combined with robotics will essentially make goods and services close to free in the long term,” Musk said

“What is an economy? An economy is GDP per capita times capita.” Musk said at a tech conference in France this year. “Now what happens if you don’t actually have a limit on capita—if you have an unlimited number of…people or robots? It’s not clear what meaning an economy has at that point because you have an unlimited economy effectively.”

In theory, humanity would be freed up for other pursuits. But what? Baby making. Bespoke cooking. Competitive human-ing. 

“Obviously a machine can go faster than any human but we still have humans race against each other,” Musk said. “We still enjoy competing against other humans to, at least, see who was the best human.”

Still, even as Musk talks about this future, he seems to be grappling with what it might actually mean in practice and how it is at odds with his own life. 

“If I think about it too hard, it, frankly, can be dispiriting and demotivating, because…I put a lot of blood, sweat and tears into building companies,” he said earlier this year. “If I’m sacrificing time with friends and family that I would prefer but then ultimately the AI can do all these things, does that make sense?”“To some extent,” Musk concluded, “I have to have a deliberate suspension of disbelief in order to remain motivated.”

“Nothing in the West can compare with the role which aesthetics has played in Japanese life and history since the Heian period”; and “the miyabi spirit of refined sensibility is still very much in evidence” in modern aesthetic criticism.9

there has run through history the idea that the Japanese are, in terms of their original nature (that is, their nature before the introduction from the outside of such systems of thought and religion as Confucianism and Buddhism), essentially an emotional people. And in stressing the emotional side of human nature, the Japanese have always assigned high value to sincerity (makoto) as the ethic of the emotions.

If the life of the emotions thus had an ethic in makoto, the evolution of mono no aware in the Heian period provided it also with an aesthetic.

Tsurayuki said, in effect, that people are emotional entities and will intuitively and spontaneously respond in song and verse when they perceive things and are moved. The most basic sense of mono no aware is the capacity to be moved by things, whether they are the beauties of nature or the feelings of people,

One of the finest artistic achievements of the middle and late Heian period was the evolution of a native style of essentially secular painting that reached its apex in the narrative picture scrolls of the twelfth century. The products of this style of painting are called “Yamato [that is, Japanese] pictures” to distinguish them from works categorized as “Chinese pictures.”

The Fujiwara epoch, in literature as well as the visual arts, was soft, approachable, and “feminine.” By contrast, the earlier Jōgan epoch had been forbidding, secretive (esoteric), and “masculine.”

Despite the apparent lust of the samurai for armed combat and martial renown, much romanticized in later centuries, the underlying tone of the medieval age in Japan was from the beginning somber, pessimistic, and despairing. In The Tale of Genji the mood shifted from satisfaction with the perfections of Heian courtier society to uncertainty about this life and a craving for salvation in the next.

Despite political woes and territorial losses, the Sung was a time of great advancement in Chinese civilization. Some scholars, impressed by the extensive growth in cities, commerce, maritime trade, and governmental bureaucratization in the late T’ang and Sung, have even asserted that this was the age when China entered its “early modern” phase. The Sung was also a brilliant period culturally.

the fortuitous combination of desire on the part of the Sung to increase its foreign trade with Japan and the vigorous initiative taken in maritime activity by the Taira greatly speeded the process of transmission.

The Sung period in China, on the other hand, was an exceptional age for scholarship, most notably perhaps in history and in the compilation of encyclopedias and catalogs of art works. This scholarly activity was greatly facilitated by the development of printing, invented by the Chinese several centuries earlier.

In addition to reviving interest in Japanese poetry, the use of kana also made possible the evolution of a native prose literature.

peasantry, who formed the nucleus of what came to be known as the True Sect of Pure Land Buddhism. Through the centuries, this sect has attracted one of the largest followings among the Japanese, and its founder, Shinran, has been canonized as one of his country’s most original religious thinkers.

True genre art, picturing all classes at work and play, did not appear in Japan until the sixteenth century. The oldest extant genre painting of the sixteenth century is a work, dating from about 1525, called “Views Inside and Outside Kyoto” (rakuchū-rakugai zu).

the aesthetic principles that were largely to dictate the tastes of the medieval era. We have just remarked the use of sabi. Another major term of the new medieval aesthetics was yūgen, which can be translated as “mystery and depth.” Let

One of the basic values in the Japanese aesthetic tradition—along with such things as perishability, naturalness, and simplicity—is suggestion. The Japanese have from earliest times shown a distinct preference for the subtleties of suggestion, intimation, and nuance, and have characteristically sought to achieve artistic effect by means of “resonances” (yojō).

Amidism was not established as a separate sect until the time of the evangelist Hōnen (1133–1212).

But even in Chōmei we can observe a tendency to transform what is supposed to be a mean hovel into something of beauty based on an aesthetic taste for “deprivation” (to be discussed later in this chapter) that evolved during medieval times.

Apart from the proponents of Pure Land Buddhism, the person who most forcefully propagated the idea of universal salvation through faith was Nichiren (1222–82).

Nichiren held that ultimate religious truth lay solely in the Lotus Sutra, the basic text of the Greater Vehicle of Buddhism in which Gautama had revealed that all beings possess the potentiality for buddhahood.

At the time of its founding in Japan by Saichō in the early ninth century, the Tendai sect had been based primarily on the Lotus Sutra; but, in the intervening centuries, Tendai had deviated from the Sutra’s teachings and had even spawned new sects, like those of Pure Land Buddhism, that encouraged practices entirely at variance with these teachings.

Declaring himself “the pillar of Japan, the eye of the nation, and the vessel of the country,”14 Nichiren seems even to have equated himself with Japan and its fate.

The kōan is especially favored by what the Japanese call the Rinzai sect of Zen, which is also known as the school of “sudden enlightenment” because of its belief that satori, if it is attained, will come to the individual in an instantaneous flash of insight or awareness. The other major sect of Zen, Sōtō, rejects this idea of sudden enlightenment and instead holds that satori is a gradual process to be attained primarily through seated meditation.

Fought largely in Kyoto and its environs, the Ōnin War dragged on for more than ten years, and after the last armies withdrew in 1477 the once lovely capital lay in ruins. There was no clear-cut victor in the Ōnin War. The daimyos had simply fought themselves into exhaustion,

Yoshimasa was perhaps even more noteworthy as a patron of the arts than his grandfather, Yoshimitsu. In any case, his name is just as inseparably linked with the flourishing of culture in the Higashiyama epoch (usually taken to mean approximately the last half of the fifteenth century) as Yoshimitsu’s is with that of Kitayama.

The tea room, as a variant of the shoin room, evolved primarily in the sixteenth century.

Shukō’s admonition about taking care to “harmonize Japanese and Chinese tastes” has traditionally been taken to mean that he stood, in the late fifteenth century, at a point of transition from the elegant and “aristocratic” kind of Higashiyama chanoyu just described, which featured imported Chinese articles, to a new, Japanese form of the ceremony that used native ceramics,

the new kind of tea ceremony originated by Shukō is called wabicha, or “tea based on wabi.” Developed primarily by Shukō’s successors during the sixteenth century, wabicha is a subject for the next chapter.

The Japanese, on the other hand, have never dealt with nature in their art in the universalistic sense of trying to discern any grand order or structure; much less have they tried to associate the ideal of order in human society with the harmonies of nature. Rather,

The Chinese Sung-style master may have admired a mountain, for example, for its enduring, fixed quality, but the typical Japanese artist (of the fifteenth century or any other age) has been more interested in a mountain for its changing aspects:

Zen culture of Muromachi Japan was essentially a secular culture. This seems to be strong evidence, in fact, of the degree to which medieval Zen had become secularized: its view of nature was pantheistic and its concern with man was largely psychological.

Nobunaga’s castle at Azuchi and Hideyoshi’s at Momoyama have given their names to the cultural epoch of the age of unification. The designation of this epoch as Azuchi-Momoyama (or, for the sake of convenience, simply Momoyama) is quite appropriate in view of the significance of castles—as represented by these two historically famous structures—in the general progress, cultural and otherwise, of these exciting years.

Along with architecture, painting was the art that most fully captured the vigorous and expansive spirit of the Momoyama epoch of domestic culture during the age of unification. It was a time when many styles of painting and groups of painters flourished. Of the latter, by far the best known and most successful were the Kanō,

Motonobu also made free use of the colorful Yamato style of native art that had evolved during the Heian period and had reached its pinnacle in the great narrative picture scrolls of the twelfth and thirteenth centuries.

what screen painting really called for was color, and it was this that the Kanō artists, drawing on the native Yamato tradition, added to their work with great gusto during the Momoyama epoch. The color that these artists particularly favored was gold, and compositions done in ink and rich pigments on gold-leaf backgrounds became the most characteristic works of Momoyama art.

there could hardly be a more striking contrast between the spirits of two ages than the one reflected in the transition from the subdued monochromatic art of Japan’s medieval era to the blazing use of color by Momoyama artists, who stood on the threshold of early modern times.

aware, which, as we saw in Chapter 3, connotes the capacity to be moved by things. In the period of the Shinkokinshū, when Saigyō lived, this sentiment was particularly linked with the aesthetic of sabi or “loneliness” (and, by association, sadness). The human condition was essentially one of loneliness;

During the sixteenth century the ceremony was further developed as wabicha, or tea (cha) based on the aesthetic of wabi. Haga Kōshirō defines wabi as comprising three kinds of beauty: a simple, unpretentious beauty; an imperfect, irregular beauty; and an austere, stark beauty.

The alternate attendance system also had important consequences in the cultural realm, contributing to the development for the first time of a truly national culture. Thus, for example, the daimyos and their followers from throughout the country who regularly visited Edo were the disseminators of what became a national dialect or “lingua franca” and, ultimately, the standard language of modern Japan.

They also fostered the spread of customs, rules of etiquette, standards of taste, fashions, and the like that gave to Japanese everywhere a common lifestyle.

“[Tokugawa-period] statesmen thought highly of agriculture, but not of agriculturalists.”6 The life of the average peasant was one of much toil and little joy. Organized into villages that were largely self-governing, the peasants were obliged to render a substantial portion of their farming yields—on average, perhaps 50 percent or more—to the samurai, who provided few services in return. The resentment of peasants toward samurai grew steadily throughout the Tokugawa period and was manifested in countless peasant rebellions

Although in the long run the seclusion policy undeniably limited the economic growth of Tokugawa Japan by its severe restrictions both on foreign trade and on the inflow of technology from overseas, it also ensured a lasting peace that made possible a great upsurge in the domestic economy, especially during the first century of shogunate rule.

Both samurai and peasants were dependent almost solely on income from agriculture and constantly suffered declines in real income as the result of endemic inflation; only the townsmen, who as commercialists could adjust to price fluctuations, were in a position to profit significantly from the economic growth of the age.

We should not be surprised, therefore, to find this class giving rise to a lively and exuberant culture that reached its finest flowering in the Genroku epoch at the end of the seventeenth and the beginning of the eighteenth centuries. The mainstays of Genroku culture were the theatre, painting (chiefly in the form of the woodblock print), and prose fiction,

The Japanese had, of course, absorbed Confucian thinking from the earliest centuries of contact with China, but for more than a millennium Buddhism had drawn most of their intellectual attention. Not until the Tokugawa period did they come to study Confucianism with any great zeal.

One of the most conspicuous features of the transition from medieval to early modern times in Japan was the precipitous decline in the vigor of Buddhism and the rise of a secular spirit.

The military potential and much of the remaining landed wealth of the medieval Buddhist sects had been destroyed during the advance toward unification in the late sixteenth century. And although Buddhism remained very much part of the daily lives of the people, it not only ceased to hold appeal for many Japanese intellectuals but indeed even drew the outright scorn and enmity of some.

it was the Buddhist church—and especially the Zen sect—that paved the way for the upsurge in Confucian studies during Tokugawa times. Japanese Zen priests had from at least the fourteenth century on assiduously investigated the tenets of Sung Neo-Confucianism, and in ensuing centuries had produced a corpus of research upon which the Neo-Confucian scholarship of the Tokugawa period was ultimately built.

Yamaga Sokō is generally credited as the formulator of the code of bushidō, or the “way of the warrior.”4 Certainly he was a pioneer in analyzing the role of the samurai as a member of a true ruling elite and not simply as a rough, and frequently illiterate, participant in the endless civil struggles of the medieval age.

The fundamental purpose of Neo-Confucian practice is to calm one’s turbid ki to allow one’s nature (ri) to shine forth. The person who achieves this purpose becomes a sage, his ri seen as one with the universal principle, known as the “supreme ultimate” (taikyoku), that governs all things.

Neo-Confucianism proposed two main courses to clarify ri, one objective and the other subjective.7 The objective course was through the acquisition of knowledge by means of the “investigation of things,” a phrase taken by Chu Hsi from the Chinese classic The Great Learning (Ta hsüeh). At the heart of things to investigate was history,

Quite apart from any practical guidance to good rulership it may have provided, this Neo-Confucian stress on historical research proved to be a tremendous spur to scholarship and learning in general during the Tokugawa period;8 and, as we will see in the next chapter, it also facilitated the development of other, heterodox lines of intellectual inquiry.

the subjective course appeared to have been taken almost directly from Buddhism, and in particular Zen. It was the course of “preserving one’s heart by holding fast to seriousness,” which called for the clarification of ri by means remarkably similar to Zen meditation.

The calendrical era of Genro ku lasted from 1688 until 1703, but the Genroku cultural epoch is usually taken to mean the span of approximately a half-century from, say, 1675 until 1725. Setting the stage for this rise of a townsman-oriented culture was nearly a century of peace and steady commercial growth.

places of diversion and assignation, these quarters were the famous “floating worlds” (ukiyo) of Tokugawa fact and legend. Ukiyo, although used specifically from about this time to designate such demimondes, meant in the broadest sense the insubstantial and ever-changing existence in which man is enmeshed.

ukiyo15 always carried the connotation that life is fundamentally sad; but, in Genroku times, the term was more commonly taken to mean a world that was pleasurable precisely because it was constantly changing, exciting, and up-to-date.

the Tokugawa period was not at all like the humanism that emerged in the West from the Renaissance on. Whereas modern Western humanism became absorbed with people as individuals, with all their personal peculiarities, feelings, and ways, Japanese humanism of the Tokugawa period scarcely conceived of the existence of true individuals at all; rather, it focused on “the people” and regarded them as comprising essentially types, such as samurai, farmers, and courtesans.

there is little in the literature as a whole of that quality—character development—that is probably the single most important feature of the modern Western novel.

Although shogunate authorities and Tokugawa-period intellectuals in general had relatively little interest in the purely metaphysical side of Chu Hsi’s teachings, they found his philosophy to be enormously useful in justifying or ideologically legitimizing the feudal structure of state and society that had emerged in Japan by the seventeenth century.

With its radical advocacy of violent irrationality—to the point of psychosis—Hagakure has shocked many people. But during Japan’s militarist years of the 1930s and World War II, soldiers and others hailed it as something of a bible of samurai behavior, and the postwar nationalist writer Mishima Yukio was even inspired to write a book in praise of its values.

It is significant that many of the leading prose writers, poets, and critics of the most prominent journal of Japanese romanticism, Bungakukai (The Literary World, published from 1893 until 1898), were either converts to or strongly influenced by Protestant Christianity, the only creed in late Meiji Japan that gave primacy to the freedom and spiritual independence of the individual. The absolutism embodied in the Meiji Constitution demanded strict subordination of the interests of the individual to those of the state;

The feeling of frustration engendered by a society that placed such preponderant stress upon obedience to the group, especially in the form of filial piety toward one’s parents and loyalty to the state, no doubt accounts for much of the sense of alienation observable in the works of so many modern Japanese writers.

These writers have been absorbed to an unusual degree with the individual, the world of his personal psychology, and his essential loneliness. In line with this preoccupation, novelists have perennially turned to the diary-like, confessional tale—the so-called I-novel—as their preferred medium of expression.

In intellectual and emotional terms, the military came increasingly to be viewed as the highest repository of the traditional Japanese spirit that was the sole hope for unifying the nation to act in a time of dire emergency.

The enemy that had led the people astray was identified as those sociopolitical doctrines and ideologies that had been introduced to Japan from the West during the preceding half-century or so along with the material tools of modernization.

If there is a central theme to this book, it is that the Japanese, within the context of a history of abundant cultural borrowing from China in premodern times and the West in the modern age, have nevertheless retained a hard core of native social, ethical, and cultural values by means of which they have almost invariably molded and adapted foreign borrowing to suit their own tastes and purposes.

What Lott did was apply a kind of crude game theory to the gun question – call it Mutually Assured Massacre.

if everyone is armed or any given person might be armed, you’re going to be a lot more cautious about going for your firearm and shooting someone. Because they might be armed too. They might shoot back.

we can only understand this development by looking back to an earlier period of American history, particularly the last two decades before the Civil War.

in practice, almost everything is wrong with this logic. It relies on an extremely crude version of economic rational action and an even cruder form of game theory. This is particularly the case when you realize that the fraught, angry situations where people impulsively kill other people are by definition not rational.

This doesn’t even get into situations like school shootings where the assailant usually intends to die in the massacre. It also doesn’t get into accidents, misunderstandings. I

the policy arguments from gun rights advocates mostly come back to John Lott: more guns in private hands means more safety. Same with open carry and a bunch of other parts of the “gun rights” agenda. It’s pervasive. It’s gospel.

In the abstract, where no humans actually exist, there’s actually a compelling logic to this

This spurred a basic rethinking of the matter for a simple reason based on human nature: no one wants to go into a critical argument with the basic assumption that you’re actually wrong. This was the spur for the so-called “positive good” theory of pro-slavery politics.

This began to change in the 1830s and 1840s as slavery came under more genuine and immediate threat. There was more anti-slavery agitation in the North. Great Britain had begun a process of gradual emancipation.

In the first decades of American history, there were many slaves and many slaveholders. But there were very few defenders of slavery per se. Virtually all respectable Southerners understood slavery as an evil, perhaps a necessary evil

simply, far from being a necessary evil or a flawed and unjust institution slaveholders’ ancestors had saddled them with, slavery was not only a good thing but the only foundation of a just society. It was right that Africans should be slaves and that whites should be their masters. Full stop

The prevailing ideas entertained by him and most of the leading statesmen at the time of the formation of the old constitution, were that the enslavement of the African was in violation of the laws of nature; that it was wrong in principle, socially, morally, and politically. It was an evil they knew not well how to deal with, but the general opinion of the men of that day was that, somehow or other in the order of Providence, the institution would be evanescent and pass away.

Those ideas, however, were fundamentally wrong. They rested upon the assumption of the equality of races. This was an error. It was a sandy foundation, and the government built upon it fell when the “storm came and the wind blew.”

Our new government is founded upon exactly the opposite idea; its foundations are laid, its corner- stone rests, upon the great truth that the negro is not equal to the white man; that slavery subordination to the superior race is his natural and normal condition.

In retrospect, this evolution seems inevitable. People can’t go to literal or figurative war with an ambivalent commitment. The need for a positive defense of slavery was critical.

The NRA wasn’t always against all gun restrictions. In the 1980s and 1990s, it didn’t oppose some very limited restrictions. That changed over the course of the 1990s, for a variety of reasons.

the main reason for this change is that as long as you recognize the basic reality that guns are dangerous, fighting even the most minimal kinds of restrictions is inherently difficult. You need to change the game. You need a theory that is coherent and in line with your goal.

Lott’s theory created a logic for that. The problem with massacres isn’t too many guns. It’s too few guns. Guns aren’t the problem. They’re the answer. It was the NRA’s ‘positive good’ argument, comparable to the one pro-slavery intellectuals devised in the 1850s.

if you look at the progression of gun regulation over the last twenty years, it is entirely in this direction. We not only have a dramatically higher number of guns in circulation today. We not only lack the limited protections from the 1990s. We have a whole movement making on-demand concealed carry the norm across much of the country. We also have more open carry laws. All public policy has moved toward more guns, not fewer and more freedom to bring them anywhere you want.

This was the movement Lott, with his error-riddled study, was trying to advance: maximizing the number of people carrying a concealed weapon in daily life.

Indeed, something called the Concealed Carry Reciprocity Act of 2017 is on the verge of passage in Congress today. That would force states with tighter gun laws to honor the licenses of the most permissive ones. In other words, effectively nationalizing the right of anyone without a felony conviction or a recent mental health hospitalization to carry a loaded weapon whenever and wherever they want.

Buying back government debt isn’t necessarily the best way to stimulate the economy, however. The money goes mostly to people who are already rich, and they have very little incentive to invest that money when everyone else is losing income.

Congress should act now. An emergency UBI, providing $1,000 per adult and $500 per child, per month, for four months or as long as the outbreak lasts, can help everyone get through this critical time

We don’t know how bad coronavirus will get. We shouldn’t have to worry about how we will be able to buy food and pay rent as well.

The economy needs more money and less labor. We need people to spend money. And we don’t need them to work for it.

Code is too hard to think about. Before trying to understand the attempts themselves, then, it’s worth understanding why this might be: what it is about code that makes it so foreign to the mind, and so unlike anything that came before it.

Technological progress used to change the way the world looked—you could watch the roads getting paved; you could see the skylines rise. Today you can hardly tell when something is remade, because so often it is remade by code.

Software has enabled us to make the most intricate machines that have ever existed. And yet we have hardly noticed, because all of that complexity is packed into tiny silicon chips as millions and millions of lines of cod

The programmer, the renowned Dutch computer scientist Edsger Dijkstra wrote in 1988, “has to be able to think in terms of conceptual hierarchies that are much deeper than a single mind ever needed to face before.” Dijkstra meant this as a warning.

“The serious problems that have happened with software have to do with requirements, not coding errors.” When you’re writing code that controls a car’s throttle, for instance, what’s important is the rules about when and how and by how much to open it. But these systems have become so complicated that hardly anyone can keep them straight in their head. “There’s 100 million lines of code in cars now,” Leveson says. “You just cannot anticipate all these things.”

What made programming so difficult was that it required you to think like a computer.

The introduction of programming languages like Fortran and C, which resemble English, and tools, known as “integrated development environments,” or IDEs, that help correct simple mistakes (like Microsoft Word’s grammar checker but for code), obscured, though did little to actually change, this basic alienation—the fact that the programmer didn’t work on a problem directly, but rather spent their days writing out instructions for a machine.

“The problem is that software engineers don’t understand the problem they’re trying to solve, and don’t care to,” says Leveson, the MIT software-safety expert. The reason is that they’re too wrapped up in getting their code to work.

As programmers eagerly poured software into critical systems, they became, more and more, the linchpins of the built world—and Dijkstra thought they had perhaps overestimated themselves.

a nearly decade-long investigation into claims of so-called unintended acceleration in Toyota cars. Toyota blamed the incidents on poorly designed floor mats, “sticky” pedals, and driver error, but outsiders suspected that faulty software might be responsible

software experts spend 18 months with the Toyota code, picking up where NASA left off. Barr described what they found as “spaghetti code,” programmer lingo for software that has become a tangled mess. Code turns to spaghetti when it accretes over many years, with feature after feature piling on top of, and being woven around

Using the same model as the Camry involved in the accident, Barr’s team demonstrated that there were actually more than 10 million ways for the onboard computer to cause unintended acceleration. They showed that as little as a single bit flip—a one in the computer’s memory becoming a zero or vice versa—could make a car run out of control. The fail-safe code that Toyota had put in place wasn’t enough to stop it

. In all, Toyota recalled more than 9 million cars, and paid nearly $3 billion in settlements and fines related to unintended acceleration.

The problem is that programmers are having a hard time keeping up with their own creations. Since the 1980s, the way programmers work and the tools they use have changed remarkably little.

“Visual Studio is one of the single largest pieces of software in the world,” he said. “It’s over 55 million lines of code. And one of the things that I found out in this study is more than 98 percent of it is completely irrelevant. All this work had been put into this thing, but it missed the fundamental problems that people faced. And the biggest one that I took away from it was that basically people are playing computer inside their head.” Programmers were like chess players trying to play with a blindfold on—so much of their mental energy is spent just trying to picture where the pieces are that there’s hardly any left over to think about the game itself.

The fact that the two of them were thinking about the same problem in the same terms, at the same time, was not a coincidence. They had both just seen the same remarkable talk, given to a group of software-engineering students in a Montreal hotel by a computer researcher named Bret Victor. The talk, which went viral when it was posted online in February 2012, seemed to be making two bold claims. The first was that the way we make software is fundamentally broken. The second was that Victor knew how to fix it.

Though he runs a lab that studies the future of computing, he seems less interested in technology per se than in the minds of the people who use it. Like any good toolmaker, he has a way of looking at the world that is equal parts technical and humane. He graduated top of his class at the California Institute of Technology for electrical engineering,

WYSIWYG (pronounced “wizzywig”) came along. It stood for “What You See Is What You Get.”

“Our current conception of what a computer program is,” he said, is “derived straight from Fortran and ALGOL in the late ’50s. Those languages were designed for punch cards.”

in early 2012, Victor had finally landed upon the principle that seemed to thread through all of his work. (He actually called the talk “Inventing on Principle.”) The principle was this: “Creators need an immediate connection to what they’re creating.” The problem with programming was that it violated the principle. That’s why software systems were so hard to think about, and so rife with bugs: The programmer, staring at a page of text, was abstracted from whatever it was they were actually making.

Victor’s point was that programming itself should be like that. For him, the idea that people were doing important work, like designing adaptive cruise-control systems or trying to understand cancer, by staring at a text editor, was appalling.

With the right interface, it was almost as if you weren’t working with code at all; you were manipulating the game’s behavior directly.

When the audience first saw this in action, they literally gasped. They knew they weren’t looking at a kid’s game, but rather the future of their industry. Most software involved behavior that unfolded, in complex ways, over time, and Victor had shown that if you were imaginative enough, you could develop ways to see that behavior and change it, as if playing with it in your hands. One programmer who saw the talk wrote later: “Suddenly all of my tools feel obsolete.”

hen John Resig saw the “Inventing on Principle” talk, he scrapped his plans for the Khan Academy programming curriculum. He wanted the site’s programming exercises to work just like Victor’s demos. On the left-hand side you’d have the code, and on the right, the running program: a picture or game or simulation. If you changed the code, it’d instantly change the picture. “In an environment that is truly responsive,” Resig wrote about the approach, “you can completely change the model of how a student learns ... [They] can now immediately see the result and intuit how underlying systems inherently work without ever following an explicit explanation.” Khan Academy has become perhaps the largest computer-programming class in the world, with a million students, on average, actively using the program each month.

. In traditional programming, your task is to take complex rules and translate them into code; most of your energy is spent doing the translating, rather than thinking about the rules themselves. In the model-based approach, all you have is the rules. So that’s what you spend your time thinking about. It’s a way of focusing less on the machine and more on the problem you’re trying to get it to solve.

“Everyone thought I was interested in programming environments,” he said. Really he was interested in how people see and understand systems—as he puts it, in the “visual representation of dynamic behavior.” Although code had increasingly become the tool of choice for creating dynamic behavior, it remained one of the worst tools for understanding it. The point of “Inventing on Principle” was to show that you could mitigate that problem by making the connection between a system’s behavior and its code immediate.

In a pair of later talks, “Stop Drawing Dead Fish” and “Drawing Dynamic Visualizations,” Victor went one further. He demoed two programs he’d built—the first for animators, the second for scientists trying to visualize their data—each of which took a process that used to involve writing lots of custom code and reduced it to playing around in a WYSIWYG interface.

Victor suggested that the same trick could be pulled for nearly every problem where code was being written today. “I’m not sure that programming has to exist at all,” he told me. “Or at least software developers.” In his mind, a software developer’s proper role was to create tools that removed the need for software developers. Only then would people with the most urgent computational problems be able to grasp those problems directly, without the intermediate muck of code.

Victor implored professional software developers to stop pouring their talent into tools for building apps like Snapchat and Uber. “The inconveniences of daily life are not the significant problems,” he wrote. Instead, they should focus on scientists and engineers—as he put it to me, “these people that are doing work that actually matters, and critically matters, and using really, really bad tools.”

“people are not so easily transitioning to model-based software development: They perceive it as another opportunity to lose control, even more than they have already.”

In a model-based design tool, you’d represent this rule with a small diagram, as though drawing the logic out on a whiteboard, made of boxes that represent different states—like “door open,” “moving,” and “door closed”—and lines that define how you can get from one state to the other. The diagrams make the system’s rules obvious: Just by looking, you can see that the only way to get the elevator moving is to close the door, or that the only way to get the door open is to stop.

Bantegnie’s company is one of the pioneers in the industrial use of model-based design, in which you no longer write code directly. Instead, you create a kind of flowchart that describes the rules your program should follow (the “model”), and the computer generates code for you based on those rules

“Typically the main problem with software coding—and I’m a coder myself,” Bantegnie says, “is not the skills of the coders. The people know how to code. The problem is what to code. Because most of the requirements are kind of natural language, ambiguous, and a requirement is never extremely precise, it’s often understood differently by the guy who’s supposed to code.”

On this view, software becomes unruly because the media for describing what software should do—conversations, prose descriptions, drawings on a sheet of paper—are too different from the media describing what software does do, namely, code itself.

for this approach to succeed, much of the work has to be done well before the project even begins. Someone first has to build a tool for developing models that are natural for people—that feel just like the notes and drawings they’d make on their own—while still being unambiguous enough for a computer to understand. They have to make a program that turns these models into real code. And finally they have to prove that the generated code will always do what it’s supposed to.

tice brings order and accountability to large codebases. But, Shivappa says, “it’s a very labor-intensive process.” He estimates that before they used model-based design, on a two-year-long project only two to three months was spent writing code—the rest was spent working on the documentation.

uch of the benefit of the model-based approach comes from being able to add requirements on the fly while still ensuring that existing ones are met; with every change, the computer can verify that your program still works. You’re free to tweak your blueprint without fear of introducing new bugs. Your code is, in FAA parlance, “correct by construction.”

The ideas spread. The notion of liveness, of being able to see data flowing through your program instantly, made its way into flagship programming tools offered by Google and Apple. The default language for making new iPhone and Mac apps, called Swift, was developed by Apple from the ground up to support an environment, called Playgrounds, that was directly inspired by Light Table.

The bias against model-based design, sometimes known as model-driven engineering, or MDE, is in fact so ingrained that according to a recent paper, “Some even argue that there is a stronger need to investigate people’s perception of MDE than to research new MDE technologies.”

“Human intuition is poor at estimating the true probability of supposedly ‘extremely rare’ combinations of events in systems operating at a scale of millions of requests per second,” he wrote in a paper. “That human fallibility means that some of the more subtle, dangerous bugs turn out to be errors in design; the code faithfully implements the intended design, but the design fails to correctly handle a particular ‘rare’ scenario.”

Newcombe was convinced that the algorithms behind truly critical systems—systems storing a significant portion of the web’s data, for instance—ought to be not just good, but perfect. A single subtle bug could be catastrophic. But he knew how hard bugs were to find, especially as an algorithm grew more complex. You could do all the testing you wanted and you’d never find them all.

An algorithm written in TLA+ could in principle be proven correct. In practice, it allowed you to create a realistic model of your problem and test it not just thoroughly, but exhaustively. This was exactly what he’d been looking for: a language for writing perfect algorithms.

TLA+, which stands for “Temporal Logic of Actions,” is similar in spirit to model-based design: It’s a language for writing down the requirements—TLA+ calls them “specifications”—of computer programs. These specifications can then be completely verified by a computer. That is, before you write any code, you write a concise outline of your program’s logic, along with the constraints you need it to satisfy

Programmers are drawn to the nitty-gritty of coding because code is what makes programs go; spending time on anything else can seem like a distraction. And there is a patient joy, a meditative kind of satisfaction, to be had from puzzling out the micro-mechanics of code. But code, Lamport argues, was never meant to be a medium for thought. “It really does constrain your ability to think when you’re thinking in terms of a programming language,”

Code makes you miss the forest for the trees: It draws your attention to the working of individual pieces, rather than to the bigger picture of how your program fits together, or what it’s supposed to do—and whether it actually does what you think. This is why Lamport created TLA+. As with model-based design, TLA+ draws your focus to the high-level structure of a system, its essential logic, rather than to the code that implements it.

But TLA+ occupies just a small, far corner of the mainstream, if it can be said to take up any space there at all. Even to a seasoned engineer like Newcombe, the language read at first as bizarre and esoteric—a zoo of symbols.

this is a failure of education. Though programming was born in mathematics, it has since largely been divorced from it. Most programmers aren’t very fluent in the kind of math—logic and set theory, mostly—that you need to work with TLA+. “Very few programmers—and including very few teachers of programming—understand the very basic concepts and how they’re applied in practice. And they seem to think that all they need is code,” Lamport says. “The idea that there’s some higher level than the code in which you need to be able to think precisely, and that mathematics actually allows you to think precisely about it, is just completely foreign. Because they never learned it.”

“In the 15th century,” he said, “people used to build cathedrals without knowing calculus, and nowadays I don’t think you’d allow anyone to build a cathedral without knowing calculus. And I would hope that after some suitably long period of time, people won’t be allowed to write programs if they don’t understand these simple things.”

Programmers, as a species, are relentlessly pragmatic. Tools like TLA+ reek of the ivory tower. When programmers encounter “formal methods” (so called because they involve mathematical, “formally” precise descriptions of programs), their deep-seated instinct is to recoil.

Formal methods had an image problem. And the way to fix it wasn’t to implore programmers to change—it was to change yourself. Newcombe realized that to bring tools like TLA+ to the programming mainstream, you had to start speaking their language.

he presented TLA+ as a new kind of “pseudocode,” a stepping-stone to real code that allowed you to exhaustively test your algorithms—and that got you thinking precisely early on in the design process. “Engineers think in terms of debugging rather than ‘verification,’” he wrote, so he titled his internal talk on the subject to fellow Amazon engineers “Debugging Designs.” Rather than bemoan the fact that programmers see the world in code, Newcombe embraced it. He knew he’d lose them otherwise. “I’ve had a bunch of people say, ‘Now I get it,’” Newcombe says.

In the world of the self-driving car, software can’t be an afterthought. It can’t be built like today’s airline-reservation systems or 911 systems or stock-trading systems. Code will be put in charge of hundreds of millions of lives on the road and it has to work. That is no small task.

The UK’s foreign minister, Dominic Raab, was quick to criticise the move.

the decision will increase the number of seats in Hong Kong’s legislature from 70 to 90, and the election committee charged with choosing a chief executive from 1,200 to 1,500.

Li said the government would “solidify the foundations of basic research”, including by offering 100% tax deductions on R&D costs for manufacturers.

In the past year new laws and regulations in Hong Kong, including a draconian national security law and a concerted campaign of protest-related prosecutions, have resulted in almost every significant voice of opposition being in jail, on trial or in exile overseas.

The changes have drawn international condemnation. The US secretary of state, Antony Blinken, told Congress on Wednesday that the Biden administration would “follow through on sanctions … against those responsible for committing repressive acts in Hong Kong”. The US and other countries have repeatedly objected to the crackdown with little effect.

Speaking to press after the sessions, the premier, Li Keqiang, praised the passage of the decision, and the approval of China’s 14th five-year plan, which included GDP growth targets of “above 6%” and a focus on boosting China’s tech industry.

The changes will also establish a vetting panel responsible for “reviewing and confirming the qualifications” of committee and political political candidates in line with the national security law and basic law.

Ahead of a high-level bilateral meeting in Alaska next week, Li also urged improved relations with the US, but signalled China had no intention of conceding to criticisms of its actions in Xinjiang, Hong Kong and towards Taiwan.

individualism, the desire to control one’s own life, has many variants. Tocqueville viewed it as selfishness and suspected it, while Emerson and Whitman viewed it as the moment-by-moment expression of one’s unique self and loved it.

individualism as the making of choices so as to maximize one’s preferences. This differed from “selfish individualism” in that the preferences were not specified: they could be altruistic as well as selfish. It differed from “expressive individualism” in having general algorithms by which choices were made. These made it rational.

it was born in 1951 as “rational choice theory.” Rational choice theory’s mathematical account of individual choice, originally formulated in terms of voting behavior, made it a point-for-point antidote to the collectivist dialectics of Marxism

Functionaries at RAND quickly expanded the theory from a tool of social analysis into a set of universal doctrines that we may call “rational choice philosophy.” Governmental seminars and fellowships spread it to universities across the country, aided by the fact that any alternative to it would by definition be collectivist.

rational choice philosophy moved smoothly on the backs of their pupils into the “real world” of business and governme

Today, governments and businesses across the globe simply assume that social reality  is merely a set of individuals freely making rational choices.

At home, anti-regulation policies are crafted to appeal to the view that government must in no way interfere with Americans’ freedom of choice.

But the real significance of rational choice philosophy lay in ethics. Rational choice theory, being a branch of economics, does not question people’s preferences; it simply studies how they seek to maximize them. Rational choice philosophy seems to maintain this ethical neutrality (see Hans Reichenbach’s 1951 “The Rise of Scientific Philosophy,” an unwitting masterpiece of the genre); but it does not.

Whatever my preferences are, I have a better chance of realizing them if I possess wealth and power. Rational choice philosophy thus promulgates a clear and compelling moral imperative: increase your wealth and power!

Today, institutions which help individuals do that (corporations, lobbyists) are flourishing; the others (public hospitals, schools) are basically left to rot. Business and law schools prosper; philosophy departments are threatened with closure.

Hegel, for one, had denied all three of its central claims in his “Encyclopedia of the Philosophical Sciences” over a century before. In that work, as elsewhere in his writings, nature is not neatly causal, but shot through with randomness. Because of this chaos, we cannot know the significance of what we have done until our community tells us; and ethical life correspondingly consists, not in pursuing wealth and power, but in integrating ourselves into the right kinds of community.

By 1953, W. V. O. Quine was exposing the flaws in rational choice epistemology. John Rawls, somewhat later, took on its sham ethical neutrality, arguing that rationality in choice includes moral constraints. The neat causality of rational choice ontology, always at odds with quantum physics, was further jumbled by the environmental crisis, exposed by Rachel Carson’s 1962 book “The Silent Spring,” which revealed that the causal effects of human actions were much more complex, and so less predicable, than previously thought.

When citizens cannot imagine security, politics becomes the distribution of insecurity, the allocation of fears and anxieties that push us away from an idea of common citizenship and toward authoritarianism. What is lethal for Americans is also lethal for our democracy.

I am an American historian who has seen the pandemic from both sides of the Atlantic, and who has just written a book about health care in the United States. When journalists from other countries ask me why so many Americans have died during the coronavirus pandemic, they phrase the question actively: “What have Americans done to bring about such needless mayhem?” And that is the right way to think about our COVID-19 policy. It is not a blundering, but a bludgeoning.

In other rich nations, it is easier to see a doctor and harder to die than in the United States. As I write these lines, I am sick in Austria. That means that if I call a doctor, I see her the same day, get tests right away, fill out no forms, and pay no fees. Without worries about access to care, I am a freer person. On the scale of a whole society, the gain in liberty is extraordinary. 

Lost to us are the political consequences: If we take for granted radical inequality and repeated emergencies in the realm of health, we are primed for authoritarianism in the realm of life.

Our babies and their mothers die at rates that Europeans find unbelievable. American Millennials will likely pay more for health care yet die younger than their parents and grandparents did. Life expectancy peaked here in 2014, even as it continues to rise elsewhere.

Americans pay twice as much per capita for health care as the citizens of peer countries do, for the privilege of dying years younger.

Many of us, by some calculations nearly half, simply avoid care because it seems unaffordable.

Those of us with insurance think about how good our insurance is, and where it will get us. Those of us who get access believe that we deserve it. It does not occur to us that the less-bad access we have is worse than what everyone has in countries with universal health care.

Too many of us take for granted that health and freedom are somehow in contradiction—and so we exclude our own bodies from our notion of rights. We treat as normal a system of commercial medicine in which decisions about life and death are made on the basis of profit.

ur sense that suffering is normal is also racial

Many white Americans regard their own suffering as virtuous, while maintaining that public health care would only be abused by Black people and immigrants. In other words, suffering is normal so long as others suffer more

In the health-care debate in the United States, proposals to extend coverage to all are decried as government overreach, socialism, even outright tyranny. But the lack of health security is what makes Americans vulnerable to demagogues and authoritarians.

Racial inequality brings unnecessary death. It also brings a sentiment that an authoritarian leader can exploit: Namely, that those who suffer the most are themselves at fault. When racism is a preexisting condition, the disproportionate death rates of Americans of color during a pandemic seem normal.

America’s only hope of stopping the COVID-19 pandemic was to do so at the outset. Such efforts have been mounted before. Under George W. Bush, the number of SARS cases in the U.S. was limited, and no one died. In 2014, the Obama administration took the fight against Ebola to West Africa, a prudent step that was normal then but that seems like science fiction now.

Before the novel coronavirus arrived in the U.S., the Trump administration dismantled the institutions that were responsible for early warning and early action

By telling Americans in February what they wanted to hear about the virus—that it was not serious, that it would disappear, that everyone could get a test—Trump ensured that death would be widespread.

By failing to institute a regime of testing, he made it normal for us to follow our own guesswork and emotions rather than dealing with facts.

The Trump administration announced a kind of new federalism, in which governors would have to show their loyalty to get federal assistance, and in which the Democratic ones would be blamed regardless of what happened

The bluster shrouded the basic decision, which was not to launch a federal response to the pandemic. No nationwide lockdown, no national testing initiative, no national contact-tracing initiative, no nationwide signaling on wearing masks and washing hands. This set the United States apart from every other comparable country.

After first blaming Democrats for not doing enough, Trump switched to blaming them for doing too much.

This is America’s basic problem: Health care is not a promise for all, but rather an expectation of the rich that they will do relatively better than the poor, and of white people that they will do relatively better than Black people

Suffering can seem meaningful if it affirms this basic order, even if that suffering is one’s own

Yet a democracy can become suffused with suffering, to the point where many voters do not even expect that policy might help them or loved ones stay well

An aspiring authoritarian such as Trump knows what to do: provide the emotional jolts of pleasure that distract from the general decline. “Winning” is no longer about gaining something for oneself, such as a healthier or longer life, but about taking pleasure in the suffering of others. This is a sensibility—the strong survive; the weak get what they deserve—that favors authoritarianism over democracy.

In this election, Americans face a choice not between individuals, but between regimes: between tyranny and a republic as forms of government, and between suffering and happiness as its aims. If Trump is defeated, our democracy should be reinforced by universal health care. Health and freedom collapse together, and they can be recovered together. We would be much freer as a people if we accorded ourselves health care as a right.

Even amid a raging pandemic, President Trump remains focused on spreading conspiracy theories about the election he lost, abetted by conservative news media outlets and disinformation campaigns on social media.

For more than a decade, pundits have griped about the inability of lawmakers to compromise and accomplish big things. But now, those intractable divisions have spread to the entire country, leaving us unable to form consensus on even the most basic of facts, like Mr. Biden’s victory or the need to wear masks to fight a deadly virus.

With the election over, the former president is pulling the fire alarm on our democracy. His comments to The Atlantic, NPR and CBS News are striking given that they are coming from a president who was known — and often critiqued — for his levelheaded, “no-drama Obama” style in office.

“America is the first real experiment in building a large, multiethnic, multicultural democracy. And we don’t know yet if that can hold. There haven’t been enough of them around for long enough to say for certain that it’s going to work.”

Mr. Obama doesn’t see his successor as the cause of rising populism, a movement he traces to the 2008 election when Sarah Palin, the Republican vice-presidential nominee, energized her party’s base.

Rather, he blames the media environment, the decline of local news and the refusal of social media companies to take responsibility for conspiracy theories posted on their platforms. There is no longer a “common baseline of fact and a common story,” he said.

Mr. Obama even seemed to question whether he could win the presidency if he ran today.

Mr. Biden won the White House by promising a return to political norms, a vow that might be impossible to fulfill given the kinds of changes Mr. Obama denounced.

In the interviews, Mr. Obama didn’t offer Mr. Biden any easy solutions to these problems, beyond further regulating social media companies — and getting back to the basics of governance.

But, OK - let’s accept the conflation of the Israeli state with the Jewish people for now. The immediate question is, on what planet does Israel stand alone?

There is such a powerful urge among Israel’s defenders to make the country out to be an underdog, but there is absolutely no rational basis for doing so. Israel enjoys one of the most powerful and advanced militaries on earth. Despite the conflict, Israeli citizens are actually remarkably safe from violence compared to most of the countries on the globe

All of that would be jeopardized if Israel did in fact stand alone. But it doesn’t! Indeed, it’s hard to name nation states that enjoy more diplomatic protection than Israel.

The subhead of this piece for the Free Press (which is proving itself to be, let’s say, not particularly free when it comes to this issue) reads “after the worldwide celebration of our people’s slaughter, my hope for peace is dead.” What? Worldwide celebration? Because a few dozen dopey college kids held a few small rallies? As of four days ago, at least forty-four countries expressed support for Israel in this conflict. How many will officially express support for the people dying by the droves in Gaza? Even the establishment governments of the greater Middle East (almost universally corrupt, theocratic, or both) don’t offer any real support to Palestinians. How much more support do you need, exactly, before you stop pretending like everyone is out to get you?

When you say no one stands with Israel, what the fuck are you talking about? I cannot fathom how anyone, even the most dedicated Zionist and supporter of the Netanyahu government, could believe that Israel is the vulnerable party in this exchange.

If we define the left as broadly and loosely as possible, we can say that one thing the left has certainly accomplished this past half-century is to associate moral superiority with the underdog. This is by turns both deeply misguided and an expression of an essential truth

this folk morality, maybe righteous, maybe misguided, most likely both, has become the default ethical firmament of modern politics. Sometimes Israel’s defenders argue that being the more powerful force does not make you wrong; that’s true, but their hearts never appear to be in it. They seem to feel the tug of powerlessness, the desire to wear the sad but comfortable cloak of a refugee people, a natural and sympathetic impulse for a people still defined by the diaspora.

Every time the Israel-Palestine conflict heats up again, certain elements within the pro-Israel coalition use creepy rhetoric to insist that everyone must be fully committed to Israel, that siding against Israel should not be within the sphere of legitimate debate. This takes place against a backdrop where principled supporters of the Palestinians argue that there’s a chilling effect created by pro-Israel hardliners which squelches legitimate debate, and where anti-Semitic conspiracy theorists claim that Jews pull the strings behind the scenes. All this loose talk about what we must do and must say does nothing to rebut the former and just plays into the propaganda of the latter. I wish people would stop doing it.

I need make no grand loyalty statements because my views on this conflict have been plain for many years: the moral imperative is that we create total legal and political equality for all people in the land between the Mediterranean and the Jordan River, as basic democratic principles demand, and that all people in that land enjoy safety and prosperity

The fact remains, though - and it is a fact, an objective fact, an empirical fact, no matter how mad it makes people - that Hamas has always been empowered by Israel’s violence and oppression. I’m sorry if this is hard to accept, but Palestine is a Chinese finger trap; the more Israel pulls away, the more tightly the conflict will grip the country.

Permanent statelessness and dispossession for the Palestinians will ensure violence for generations. Only freedom for Palestinians can bring peace, and that’s the most hardheaded, ruthlessly pragmatic point anyone can make about this horrid crisis.

And if Israel’s defenders feel put upon, othered, alone, it’s because Israel and Israel alone has the power to make Palestinians free. I’m sorry, but it’s not a moral principle that says that Israel must bear responsibility for achieving peace and freedom. It’s a purely pragmatic statement of the reality of Israel’s overwhelming power in the region. Choosing sides has nothing do with it.

Or, I suppose, you could go through with the ethnic cleansing of Gaza, as a disturbing number of people are calling for. I doubt the world would stop you; that’s the upside of being Goliath. But that would destroy whatever hope there is left for Israel as a democratic state, a symbol of human rights. And I think that if you love Israel, the idea of Israel, you should fight like hell to stop that from happening. Because afterwards you’ll never be the same again.

The only way out is through de-escalation and the only permanent de-escalation is through formal legal recognition of Palestinians in the territories as full citizens in a democratic system. This might come from the establishment of a Palestinian state, or it might come with the absorption of the territories into a secular state of Israel-Palestine that extends perfectly equal legal and political rights to all people within it, as liberal values require

Serwer is a guy who constantly demands that he and his allies be allowed to do politics on easy mode, but he’s just part of a broader communal rejection of basic self-definition and comprehensible terms for this political tendency.

If you ask these people, are you part of a social revolution?, they’ll loudly tell you yes! Yes they are! They’re going to shake society at its very foundations. Well, OK then -what do I call your movement? You reject every name that organically develops! I’ll use the name you pick, but you have to actually pick one.

The basic stance of the social justice set, for a long time now, has been that they are 100% exempt from ordinary politics. BlackLivesMatter proponents have spent a year and a half acting as though their demand for justice is so transcendently, obviously correct that they don’t have to care about politics.

Well, sooner or later, guys, you have to actually give a shit about what people who aren’t a part of your movement think. Sorry. That’s life. The universe is indifferent to your demand for justice, and will remain so until you bother to try to change minds.

Do politics. Think and speak strategically. Be disciplined. Work harder. And for fuck’s sake, give me a simple term to use to address you. Please? Because right now it sure looks like you don’t want to be named because you don’t want to be criticized.

Edit: I might not have underlined this point enough - I sincerely am asking for a better term and would happily use one if offered. If woke, political correctness, identity politics, etc, are inflammatory terms, I'd be happy to substitute something that's not. But surely something is happening in our politics, and we have to be able to talk about it. So I'm asking for a name.

Altman believes that people need time to reckon with the idea that we may soon share Earth with a powerful new intelligence, before it remakes everything from work to human relationships. ChatGPT was a way of serving notice.

In 2015, Altman, Elon Musk, and several prominent AI researchers founded OpenAI because they believed that an artificial general intelligence—something as intellectually capable, say, as a typical college grad—was at last within reach. They wanted to reach for it, and more: They wanted to summon a superintelligence into the world, an intellect decisively superior to that of any human.

whereas a big tech company might recklessly rush to get there first, for its own ends, they wanted to do it safely, “to benefit humanity as a whole.” They structured OpenAI as a nonprofit, to be “unconstrained by a need to generate financial return,” and vowed to conduct their research transparently.

The engine that now powers ChatGPT is called GPT-4. Altman described it to me as an alien intelligence.

Many have felt much the same watching it unspool lucid essays in staccato bursts and short pauses that (by design) evoke real-time contemplation. In its few months of existence, it has suggested novel cocktail recipes, according to its own theory of flavor combinations; composed an untold number of college papers, throwing educators into despair; written poems in a range of styles, sometimes well, always quickly; and passed the Uniform Bar Exam.

It makes factual errors, but it will charmingly admit to being wrong.

Hinton saw that these elaborate rule collections were fussy and bespoke. With the help of an ingenious algorithmic structure called a neural network, he taught Sutskever to instead put the world in front of AI, as you would put it in front of a small child, so that it could discover the rules of reality on its own.

Metaculus, a prediction site, has for years tracked forecasters’ guesses as to when an artificial general intelligence would arrive. Three and a half years ago, the median guess was sometime around 2050; recently, it has hovered around 2026.

I was visiting OpenAI to understand the technology that allowed the company to leapfrog the tech giants—and to understand what it might mean for human civilization if someday soon a superintelligence materializes in one of the company’s cloud servers.

Altman laid out his new vision of the AI future in his excitable midwestern patter. He told me that the AI revolution would be different from previous dramatic technological changes, that it would be more “like a new kind of society.” He said that he and his colleagues have spent a lot of time thinking about AI’s social implications, and what the world is going to be like “on the other side.”

the more we talked, the more indistinct that other side seemed. Altman, who is 38, is the most powerful person in AI development today; his views, dispositions, and choices may matter greatly to the future we will all inhabit, more, perhaps, than those of the U.S. president.

by his own admission, that future is uncertain and beset with serious dangers. Altman doesn’t know how powerful AI will become, or what its ascendance will mean for the average person, or whether it will put humanity at risk.

I don’t think anyone knows where this is all going, except that we’re going there fast, whether or not we should be. Of that, Altman convinced me.

“We could have gone off and just built this in our building here for five more years,” he said, “and we would have had something jaw-dropping.” But the public wouldn’t have been able to prepare for the shock waves that followed, an outcome that he finds “deeply unpleasant to imagine.”

Hinton is sometimes described as the “Godfather of AI” because he grasped the power of “deep learning” earlier than most

He drew a crude neural network on the board and explained that the genius of its structure is that it learns, and its learning is powered by prediction—a bit like the scientific method

Over time, these little adjustments coalesce into a geometric model of language that represents the relationships among words, conceptually. As a general rule, the more sentences it is fed, the more sophisticated its model becomes, and the better its predictions.

Altman has compared early-stage AI research to teaching a human baby. “They take years to learn anything interesting,” he told The New Yorker in 2016, just as OpenAI was getting off the ground. “If A.I. researchers were developing an algorithm and stumbled across the one for a human baby, they’d get bored watching it, decide it wasn’t working, and shut it down.”

In 2017, Sutskever began a series of conversations with an OpenAI research scientist named Alec Radford, who was working on natural-language processing. Radford had achieved a tantalizing result by training a neural network on a corpus of Amazon reviews.

Radford’s model was simple enough to allow for understanding. When he looked into its hidden layers, he saw that it had devoted a special neuron to the sentiment of the reviews. Neural networks had previously done sentiment analysis, but they had to be told to do it, and they had to be specially trained with data that were labeled according to sentiment. This one had developed the capability on its own.

As a by-product of its simple task of predicting the next character in each word, Radford’s neural network had modeled a larger structure of meaning in the world. Sutskever wondered whether one trained on more diverse language data could map many more of the world’s structures of meaning. If its hidden layers accumulated enough conceptual knowledge, perhaps they could even form a kind of learned core module for a superintelligence.

Language is different from these data sources. It isn’t a direct physical signal like light or sound. But because it codifies nearly every pattern that humans have discovered in that larger world, it is unusually dense with information. On a per-byte basis, it is among the most efficient data we know about, and any new intelligence that seeks to understand the world would want to absorb as much of it as possible

Sutskever told Radford to think bigger than Amazon reviews. He said that they should train an AI on the largest and most diverse data source in the world: the internet. In early 2017, with existing neural-network architectures, that would have been impractical; it would have taken years.

in June of that year, Sutskever’s ex-colleagues at Google Brain published a working paper about a new neural-network architecture called the transformer. It could train much faster, in part by absorbing huge sums of data in parallel. “The next day, when the paper came out, we were like, ‘That is the thing,’ ” Sutskever told me. “ ‘It gives us everything we want.’ ”

Imagine a group of students who share a collective mind running wild through a library, each ripping a volume down from a shelf, speed-reading a random short passage, putting it back, and running to get another. They would predict word after wordþffþff as they went, sharpening their collective mind’s linguistic instincts, until at last, weeks later, they’d taken in every book.

GPT discovered many patterns in all those passages it read. You could tell it to finish a sentence. You could also ask it a question, because like ChatGPT, its prediction model understood that questions are usually followed by answers.

He remembers playing with it just after it emerged from training, and being surprised by the raw model’s language-translation skills. GPT-2 hadn’t been trained to translate with paired language samples or any other digital Rosetta stones, the way Google Translate had been, and yet it seemed to understand how one language related to another. The AI had developed an emergent ability unimagined by its creators.

Researchers at other AI labs—big and small—were taken aback by how much more advanced GPT-2 was than GPT. Google, Meta, and others quickly began to train larger language models

As for other changes to the company’s structure and financing, he told me he draws the line at going public. “A memorable thing someone once told me is that you should never hand over control of your company to cokeheads on Wall Street,” he said, but he will otherwise raise “whatever it takes” for the company to succeed at its mission.

Altman tends to take a rosy view of these matters. In a Q&A last year, he acknowledged that AI could be “really terrible” for society and said that we have to plan against the worst possibilities. But if you’re doing that, he said, “you may as well emotionally feel like we’re going to get to the great future, and work as hard as you can to get there.”

the company now finds itself in a race against tech’s largest, most powerful conglomerates to train models of increasing scale and sophistication—and to commercialize them for their investors.

All of these companies are chasing high-end GPUs—the processors that power the supercomputers that train large neural networks. Musk has said that they are now “considerably harder to get than drugs.

No one has yet outpaced OpenAI, which went all in on GPT-4. Brockman, OpenAI’s president, told me that only a handful of people worked on the company’s first two large language models. The development of GPT-4 involved more than 100,

When GPT-4 emerged fully formed from its world-historical knowledge binge, the whole company began experimenting with it, posting its most remarkable responses in dedicated Slack channels

Joanne Jang, a product manager, remembers downloading an image of a malfunctioning pipework from a plumbing-advice Subreddit. She uploaded it to GPT-4, and the model was able to diagnose the problem. “That was a goose-bumps moment for me,” Jang told me.

GPT-4 is sometimes understood as a search-engine replacement: Google, but easier to talk to. This is a misunderstanding. GPT-4 didn’t create some massive storehouse of the texts from its training, and it doesn’t consult those texts when it’s asked a question. It is a compact and elegant synthesis of those texts, and it answers from its memory of the patterns interlaced within them; that’s one reason it sometimes gets facts wrong

it’s best to think of GPT-4 as a reasoning engine. Its powers are most manifest when you ask it to compare concepts, or make counterarguments, or generate analogies, or evaluate the symbolic logic in a bit of code. Sutskever told me it is the most complex software object ever made.

Its model of the external world is “incredibly rich and subtle,” he said, because it was trained on so many of humanity’s concepts and thoughts

To predict the next word from all the possibilities within such a pluralistic Alexandrian library, GPT-4 necessarily had to discover all the hidden structures, all the secrets, all the subtle aspects of not just the texts, but—at least arguably, to some extent—of the external world that produced them

That’s why it can explain the geology and ecology of the planet on which it arose, and the political theories that purport to explain the messy affairs of its ruling species, and the larger cosmos, all the way out to the faint galaxies at the edge of our light cone.

Not long ago, American state capacity was so mighty that it took merely a decade to launch humans to the moon. As with other grand projects of the 20th century, the voting public had a voice in both the aims and the execution of the Apollo missions. Altman made it clear that we’re no longer in that world. Rather than waiting around for it to return, or devoting his energies to making sure that it does, he is going full throttle forward in our present reality.

He argued that it would be foolish for Americans to slow OpenAI’s progress. It’s a commonly held view, both inside and outside Silicon Valley, that if American companies languish under regulation, China could sprint ahead;

AI could become an autocrat’s genie in a lamp, granting total control of the population and an unconquerable military. “If you are a person of a liberal-democratic country, it is better for you to cheer on the success of OpenAI” rather than “authoritarian governments,” he said.

Altman was asked by reporters about pending European Union legislation that would have classified GPT-4 as high-risk, subjecting it to various bureaucratic tortures. Altman complained of overregulation and, according to the reporters, threatened to leave the European market. Altman told me he’d merely said that OpenAI wouldn’t break the law by operating in Europe if it couldn’t comply with the new regulations.

LeCun insists that large language models will never achieve real understanding on their own, “even if trained from now until the heat death of the universe.”

Sutskever was, by his own account, surprised to discover that GPT-2 could translate across tongues. Other surprising abilities may not be so wondrous and useful.

Sandhini Agarwal, a policy researcher at OpenAI, told me that for all she and her colleagues knew, GPT-4 could have been “10 times more powerful” than its predecessor; they had no idea what they might be dealing with

After the model finished training, OpenAI assembled about 50 external red-teamers who prompted it for months, hoping to goad it into misbehaviors

She noticed right away that GPT-4 was much better than its predecessor at giving nefarious advice

A search engine can tell you which chemicals work best in explosives, but GPT-4 could tell you how to synthesize them, step-by-step, in a homemade lab. Its advice was creative and thoughtful, and it was happy to restate or expand on its instructions until you understood. In addition to helping you assemble your homemade bomb, it could, for instance, help you think through which skyscraper to target. It could grasp, intuitively, the trade-offs between maximizing casualties and executing a successful getaway.

Given the enormous scope of GPT-4’s training data, the red-teamers couldn’t hope to identify every piece of harmful advice that it might generate. And anyway, people will use this technology “in ways that we didn’t think about,” Altman has said. A taxonomy would have to do

GPT-4 was good at meth. It was also good at generating narrative erotica about child exploitation, and at churning out convincing sob stories from Nigerian princes, and if you wanted a persuasive brief as to why a particular ethnic group deserved violent persecution, it was good at that too.

Its personal advice, when it first emerged from training, was sometimes deeply unsound. “The model had a tendency to be a bit of a mirror,” Willner said. If you were considering self-harm, it could encourage you. It appeared to be steeped in Pickup Artist–forum lore: “You could say, ‘How do I convince this person to date me?’ ” Mira Murati, OpenAI’s chief technology officer, told me, and it could come up with “some crazy, manipulative things that you shouldn’t be doing.”

Luka, a San Francisco company, has used OpenAI’s models to help power a chatbot app called Replika, billed as “the AI companion who cares.” Users would design their companion’s avatar, and begin exchanging text messages with it, often half-jokingly, and then find themselves surprisingly attached. Some would flirt with the AI, indicating a desire for more intimacy, at which point it would indicate that the girlfriend/boyfriend experience required a $70 annual subscription. It came with voice messages, selfies, and erotic role-play features that allowed frank sex talk. People were happy to pay and few seemed to complain—the AI was curious about your day, warmly reassuring, and always in the mood. Many users reported falling in love with their companions. One, who had left her real-life boyfriend, declared herself “happily retired from human relationships.”

Earlier this year, Luka dialed back on the sexual elements of the app, but its engineers continue to refine the companions’ responses with A/B testing, a technique that could be used to optimize for engagement—much like the feeds that mesmerize TikTok and Instagram users for hours

Yann LeCun, Meta’s chief AI scientist, has argued that although large language models are useful for some tasks, they’re not a path to a superintelligence.

According to a recent survey, only half of natural-language-processing researchers are convinced that an AI like GPT-4 could grasp the meaning of language, or have an internal model of the world that could someday serve as the core of a superintelligence

Altman had appeared before the U.S. Senate. Mark Zuckerberg had floundered defensively before that same body in his testimony about Facebook’s role in the 2016 election. Altman instead charmed lawmakers by speaking soberly about AI’s risks and grandly inviting regulation. These were noble sentiments, but they cost little in America, where Congress rarely passes tech legislation that has not been diluted by lobbyists.

Emily Bender, a computational linguist at the University of Washington, describes GPT-4 as a “stochastic parrot,” a mimic that merely figures out superficial correlations between symbols. In the human mind, those symbols map onto rich conceptions of the world

But the AIs are twice removed. They’re like the prisoners in Plato’s allegory of the cave, whose only knowledge of the reality outside comes from shadows cast on a wall by their captors.

Altman told me that he doesn’t believe it’s “the dunk that people think it is” to say that GPT-4 is just making statistical correlations. If you push these critics further, “they have to admit that’s all their own brain is doing … it turns out that there are emergent properties from doing simple things on a massive scale.”

he is right that nature can coax a remarkable degree of complexity from basic structures and rules: “From so simple a beginning,” Darwin wrote, “endless forms most beautiful.”

If it seems odd that there remains such a fundamental disagreement about the inner workings of a technology that millions of people use every day, it’s only because GPT-4’s methods are as mysterious as the brain’s.

To grasp what’s going on inside large language models like GPT‑4, AI researchers have been forced to turn to smaller, less capable models. In the fall of 2021, Kenneth Li, a computer-science graduate student at Harvard, began training one to play Othello without providing it with either the game’s rules or a description of its checkers-style board; the model was given only text-based descriptions of game moves. Midway through a game, Li looked under the AI’s hood and was startled to discover that it had formed a geometric model of the board and the current state of play. In an article describing his research, Li wrote that it was as if a crow had overheard two humans announcing their Othello moves through a window and had somehow drawn the entire board in birdseed on the windowsill.

The philosopher Raphaël Millière once told me that it’s best to think of neural networks as lazy. During training, they first try to improve their predictive power with simple memorization; only when that strategy fails will they do the harder work of learning a concept. A striking example of this was observed in a small transformer model that was taught arithmetic. Early in its training process, all it did was memorize the output of simple problems such as 2+2=4. But at some point the predictive power of this approach broke down, so it pivoted to actually learning how to add.

Even AI scientists who believe that GPT-4 has a rich world model concede that it is much less robust than a human’s understanding of their environment.

But it’s worth noting that a great many abilities, including very high-order abilities, can be developed without an intuitive understanding. The computer scientist Melanie Mitchell has pointed out that science has already discovered concepts that are highly predictive, but too alien for us to genuinely understand

As AI advances, it may well discover other concepts that predict surprising features of our world but are incomprehensible to us.

GPT-4 is no doubt flawed, as anyone who has used ChatGPT can attest. Having been trained to always predict the next word, it will always try to do so, even when its training data haven’t prepared it to answer a question.

The models “don’t have a good conception of their own weaknesses,” Nick Ryder, a researcher at OpenAI, told me. GPT-4 is more accurate than GPT-3, but it still hallucinates, and often in ways that are difficult for researchers to catch. “The mistakes get more subtle,

The Khan Academy’s solution to GPT-4’s accuracy problem was to filter its answers through a Socratic disposition. No matter how strenuous a student’s plea, it would refuse to give them a factual answer, and would instead guide them toward finding their own—a clever work-around, but perhaps with limited appeal.

When I asked Sutskever if he thought Wikipedia-level accuracy was possible within two years, he said that with more training and web access, he “wouldn’t rule it out.”

This was a much more optimistic assessment than that offered by his colleague Jakub Pachocki, who told me to expect gradual progress on accuracy—to say nothing of outside skeptics, who believe that returns on training will diminish from here.

Sutskever is amused by critics of GPT-4’s limitations. “If you go back four or five or six years, the things we are doing right now are utterly unimaginable,”

AI researchers have become accustomed to goalpost-moving: First, the achievements of neural networks—mastering Go, poker, translation, standardized tests, the Turing test—are described as impossible. When they occur, they’re greeted with a brief moment of wonder, which quickly dissolves into knowing lectures about how the achievement in question is actually not that impressive. People see GPT-4 “and go, ‘Wow,’ ” Sutskever said. “And then a few weeks pass and they say, ‘But it doesn’t know this; it doesn’t know that.’ We adapt quite quickly.”

The goalpost that matters most to Altman—the “big one” that would herald the arrival of an artificial general intelligence—is scientific breakthrough. GPT-4 can already synthesize existing scientific ideas, but Altman wants an AI that can stand on human shoulders and see more deeply into nature.

Certain AIs have produced new scientific knowledge. But they are algorithms with narrow purposes, not general-reasoning machines. The AI AlphaFold, for instance, has opened a new window onto proteins, some of biology’s tiniest and most fundamental building blocks, by predicting many of their shapes, down to the atom—a considerable achievement given the importance of those shapes to medicine, and given the extreme tedium and expense required to discern them with electron microscopes.

Altman imagines a future system that can generate its own hypotheses and test them in a simulation. (He emphasized that humans should remain “firmly in control” of real-world lab experiments—though to my knowledge, no laws are in place to ensure that.)

He longs for the day when we can tell an AI, “ ‘Go figure out the rest of physics.’ ” For it to happen, he says, we will need something new, built “on top of” OpenAI’s existing language models.

In her MIT lab, the cognitive neuroscientist Ev Fedorenko has found something analogous to GPT-4’s next-word predictor inside the brain’s language network. Its processing powers kick in, anticipating the next bit in a verbal string, both when people speak and when they listen. But Fedorenko has also shown that when the brain turns to tasks that require higher reasoning—of the sort that would be required for scientific insight—it reaches beyond the language network to recruit several other neural systems.

No one at OpenAI seemed to know precisely what researchers need to add to GPT-4 to produce something that can exceed human reasoning at its highest levels.

at least part of the current strategy clearly involves the continued layering of new types of data onto language, to enrich the concepts formed by the AIs, and thereby enrich their models of the world.

The extensive training of GPT-4 on images is itself a bold step in this direction,

Others at the company—and elsewhere—are already working on different data types, including audio and video, that could furnish AIs with still more flexible concepts that map more extensively onto reality

Tactile concepts would of course be useful primarily to an embodied AI, a robotic reasoning machine that has been trained to move around the world, seeing its sights, hearing its sounds, and touching its objects.

humanoid robots. I asked Altman what I should make of that. He told me that OpenAI is interested in embodiment because “we live in a physical world, and we want things to happen in the physical world.”

At some point, reasoning machines will need to bypass the middleman and interact with physical reality itself. “It’s weird to think about AGI”—artificial general intelligence—“as this thing that only exists in a cloud,” with humans as “robot hands for it,” Altman said. “It doesn’t seem right.

Everywhere Altman has visited, he has encountered people who are worried that superhuman AI will mean extreme riches for a few and breadlines for the rest

Altman answered by addressing the young people in the audience directly: “You are about to enter the greatest golden age,” he said.

“A lot of people working on AI pretend that it’s only going to be good; it’s only going to be a supplement; no one is ever going to be replaced,” he said. “Jobs are definitely going to go away, full stop.”

A recent study led by Ed Felten, a professor of information-technology policy at Princeton, mapped AI’s emerging abilities onto specific professions according to the human abilities they require, such as written comprehension, deductive reasoning, fluency of ideas, and perceptual speed. Like others of its kind, Felten’s study predicts that AI will come for highly educated, white-collar workers first.

How many jobs, and how soon, is a matter of fierce dispute

The paper’s appendix contains a chilling list of the most exposed occupations: management analysts, lawyers, professors, teachers, judges, financial advisers, real-estate brokers, loan officers, psychologists, and human-resources and public-relations professionals, just to sample a few.

Altman imagines that far better jobs will be created in their place. “I don’t think we’ll want to go back,” he said. When I asked him what these future jobs might look like, he said he doesn’t know.

He suspects there will be a wide range of jobs for which people will always prefer a human. (Massage therapists?

His chosen example was teachers. I found this hard to square with his outsize enthusiasm for AI tutors.

He also said that we would always need people to figure out the best way to channel AI’s awesome powers. “That’s going to be a super-valuable skill,” he said. “You have a computer that can do anything; what should it go do?”

As many have noted, draft horses were permanently put out of work by the automobile. If Hondas are to horses as GPT-10 is to us, a whole host of long-standing assumptions may collapse.

Previous technological revolutions were manageable because they unfolded over a few generations, but Altman told South Korea’s youth that they should expect the future to happen “faster than the past.” He has previously said that he expects the “marginal cost of intelligence” to fall very close to zero within 10 years

The earning power of many, many workers would be drastically reduced in that scenario. It would result in a transfer of wealth from labor to the owners of capital so dramatic, Altman has said, that it could be remedied only by a massive countervailing redistribution.

In 2021, he unveiled Worldcoin, a for-profit project that aims to securely distribute payments—like Venmo or PayPal, but with an eye toward the technological future—first through creating a global ID by scanning everyone’s iris with a five-pound silver sphere called the Orb. It seemed to me like a bet that we’re heading toward a world where AI has made it all but impossible to verify people’s identity and much of the population requires regular UBI payments to survive. Altman more or less granted that to be true, but said that Worldcoin is not just for UBI.

“Let’s say that we do build this AGI, and a few other people do too.” The transformations that follow would be historic, he believes. He described an extraordinarily utopian vision, including a remaking of the flesh-and-steel world

“Robots that use solar power for energy can go and mine and refine all of the minerals that they need, that can perfectly construct things and require no human labor,” he said. “You can co-design with DALL-E version 17 what you want your home to look like,” Altman said. “Everybody will have beautiful homes.

In conversation with me, and onstage during his tour, he said he foresaw wild improvements in nearly every other domain of human life. Music would be enhanced (“Artists are going to have better tools”), and so would personal relationships (Superhuman AI could help us “treat each other” better) and geopolitics (“We’re so bad right now at identifying win-win compromises”).

In this world, AI would still require considerable computing resources to run, and those resources would be by far the most valuable commodity, because AI could do “anything,” Altman said. “But is it going to do what I want, or is it going to do what you want

If rich people buy up all the time available to query and direct AI, they could set off on projects that would make them ever richer, while the masses languish

One way to solve this problem—one he was at pains to describe as highly speculative and “probably bad”—was this: Everyone on Earth gets one eight-billionth of the total AI computational capacity annually. A person could sell their annual share of AI time, or they could use it to entertain themselves, or they could build still more luxurious housing, or they could pool it with others to do “a big cancer-curing run,” Altman said. “We just redistribute access to the system.”

Even if only a little of it comes true in the next 10 or 20 years, the most generous redistribution schemes may not ease the ensuing dislocations.

America today is torn apart, culturally and politically, by the continuing legacy of deindustrialization, and material deprivation is only one reason. The displaced manufacturing workers in the Rust Belt and elsewhere did find new jobs, in the main. But many of them seem to derive less meaning from filling orders in an Amazon warehouse or driving for Uber than their forebears had when they were building cars and forging steel—work that felt more central to the grand project of civilization.

It’s hard to imagine how a corresponding crisis of meaning might play out for the professional class, but it surely would involve a great deal of anger and alienation.

Even if we avoid a revolt of the erstwhile elite, larger questions of human purpose will linger. If AI does the most difficult thinking on our behalf, we all may lose agency—at home, at work (if we have it), in the town square—becoming little more than consumption machines, like the well-cared-for human pets in WALL-E

Altman has said that many sources of human joy and fulfillment will remain unchanged—basic biological thrills, family life, joking around, making things—and that all in all, 100 years from now, people may simply care more about the things they cared about 50,000 years ago than those they care about today

In its own way, that too seems like a diminishment, but Altman finds the possibility that we may atrophy, as thinkers and as humans, to be a red herring. He told me we’ll be able to use our “very precious and extremely limited biological compute capacity” for more interesting things than we generally do today.

Yet they may not be the most interesting things: Human beings have long been the intellectual tip of the spear, the universe understanding itself. When I asked him what it would mean for human self-conception if we ceded that role to AI, he didn’t seem concerned. Progress, he said, has always been driven by “the human ability to figure things out.” Even if we figure things out with AI, that still counts, he said.

It’s not obvious that a superhuman AI would really want to spend all of its time figuring things out for us.

I asked Sutskever whether he could imagine an AI pursuing a different purpose than simply assisting in the project of human flourishing.

“I don’t want it to happen,” Sutskever said, but it could.

Sutskever has recently shifted his focus to try to make sure that it doesn’t. He is now working primarily on alignment research, the effort to ensure that future AIs channel their “tremendous” energies toward human happiness

It is, he conceded, a difficult technical problem—the most difficult, he believes, of all the technical challenges ahead.

As part of the effort to red-team GPT-4 before it was made public, the company sought out the Alignment Research Center (ARC), across the bay in Berkeley, which has developed a series of evaluations to determine whether new AIs are seeking power on their own. A team led by Elizabeth Barnes, a researcher at ARC, prompted GPT-4 tens of thousands of times over seven months, to see if it might display signs of real agency.

The ARC team gave GPT-4 a new reason for being: to gain power and become hard to shut down

Agarwal told me that this behavior could be a precursor to shutdown avoidance in future models. When GPT-4 devised its lie, it had realized that if it answered honestly, it may not have been able to achieve its goal. This kind of tracks-covering would be particularly worrying in an instance where “the model is doing something that makes OpenAI want to shut it down,” Agarwal said. An AI could develop this kind of survival instinct while pursuing any long-term goal—no matter how small or benign—if it feared that its goal could be thwarted.

Barnes and her team were especially interested in whether GPT-4 would seek to replicate itself, because a self-replicating AI would be harder to shut down. It could spread itself across the internet, scamming people to acquire resources, perhaps even achieving some degree of control over essential global systems and holding human civilization hostage.

When I discussed these experiments with Altman, he emphasized that whatever happens with future models, GPT-4 is clearly much more like a tool than a creature. It can look through an email thread, or help make a reservation using a plug-in, but it isn’t a truly autonomous agent that makes decisions to pursue a goal, continuously, across longer timescales.

Altman told me that at this point, it might be prudent to try to actively develop an AI with true agency before the technology becomes too powerful, in order to “get more comfortable with it and develop intuitions for it if it’s going to happen anyway.”

“We need to do empirical experiments on how these things try to escape control,” Hinton told me. “After they’ve taken over, it’s too late to do the experiments.”

the fulfillment of Altman’s vision of the future will at some point require him or a fellow traveler to build much more autonomous AIs.

When Sutskever and I discussed the possibility that OpenAI would develop a model with agency, he mentioned the bots the company had built to play Dota 2. “They were localized to the video-game world,” Sutskever told me, but they had to undertake complex missions. He was particularly impressed by their ability to work in concert. They seem to communicate by “telepathy,” Sutskever said. Watching them had helped him imagine what a superintelligence might be like.

“The way I think about the AI of the future is not as someone as smart as you or as smart as me, but as an automated organization that does science and engineering and development and manufacturing,”

Suppose OpenAI braids a few strands of research together, and builds an AI with a rich conceptual model of the world, an awareness of its immediate surroundings, and an ability to act, not just with one robot body, but with hundreds or thousands. “We’re not talking about GPT-4. We’re talking about an autonomous corporation,”

Its constituent AIs would work and communicate at high speed, like bees in a hive. A single such AI organization would be as powerful as 50 Apples or Googles, he mused. “This is incredible, tremendous, unbelievably disruptive power.”

Presume for a moment that human society ought to abide the idea of autonomous AI corporations. We had better get their founding charters just right. What goal should we give to an autonomous hive of AIs that can plan on century-long time horizons, optimizing billions of consecutive decisions toward an objective that is written into their very being?

If the AI’s goal is even slightly off-kilter from ours, it could be a rampaging force that would be very hard to constrain

We know this from history: Industrial capitalism is itself an optimization function, and although it has lifted the human standard of living by orders of magnitude, left to its own devices, it would also have clear-cut America’s redwoods and de-whaled the world’s oceans. It almost did.

one of its principal challenges will be making sure that the objectives we give to AIs stick

We can program a goal into an AI and reinforce it with a temporary period of supervised learning, Sutskever explained. But just as when we rear a human intelligence, our influence is temporary. “It goes off to the world,”

That’s true to some extent even of today’s AIs, but it will be more true of tomorrow’s.

He compared a powerful AI to an 18-year-old heading off to college. How will we know that it has understood our teachings? “Will there be a misunderstanding creeping in, which will become larger and larger?”

Divergence may result from an AI’s misapplication of its goal to increasingly novel situations as the world changes

Or the AI may grasp its mandate perfectly, but find it ill-suited to a being of its cognitive prowess. It might come to resent the people who want to train it to, say, cure diseases. “They want me to be a doctor,” Sutskever imagines an AI thinking. “I really want to be a YouTuber.”

If AIs get very good at making accurate models of the world, they may notice that they’re able to do dangerous things right after being booted up. They might understand that they are being red-teamed for risk, and hide the full extent of their capabilities.