Group items tagged

Technology’s effects are driven by technology itself, but even more by human choice. Roose warns against treating “technological change as a disembodied natural force that simply happens to us, like gravity or thermodynamics.” Instead we all should realize that “none of this is predetermined. … Regulators, not robots, decide what limits to place on emerging technologies like facial recognition and targeted digital advertising.” The message from both of these books is that the sky is not falling — but it could. There is time to make a choice.

above them all is a mercurial, impulsive, and ignorant president who has no desire to be pulled into a Middle Eastern war in an election year, and who wants to look tough without being prepared to follow through. This is a recipe for strategic ineptitude, and possibly failure

There seems to have been no one playing that role, and thereby ensuring that second- and third-order considerations had been identified and explored.

As the United States has learned to its cost, good decision making requires a forceful brake, or at least a counterpoise, to a tempting decision like the one to eliminate Soleimani

The novelist James Gould Cozzens observed higher headquarters at close range during World War II. He drew on that for his masterly World War II novel, Guard of Honor.

In one passage, his protagonist admits to himself that some of his seniors “were not complete fools.” However, he noted, it was the habit of all of them to look straight, and not very far, ahead. They saw their immediate duties and did those, not vaguely or stupidly, but in an experienced firm way. Then they waited until whatever was going to happen, happened. Then they sized this up, noted whatever new duties there were, and did those. Their position was that of a chess player who had in his head no moves beyond the one it was now his turn to make. He would be dumbfounded when, after he had made four or five such moves (each sensible enough in itself) sudden catastrophe, from an unexpected direction by an unexpected means, fell on him, and he was mated.

that may be where the United States government is headed.

The conflict between the images is important because it shows up in alternative visions of the economy and the relationship between the economy and the state.

This republican image runs into sharp conflict with a more received picture, celebrated by right-wing libertarians, according to which the rules of public order regulate the private sphere rather than serving – now in the fashion of one culture, now in the fashion of another – to make it possible

You own something only insofar as it is a matter of accepted convention that given the way you came to hold it — given public recognition of the title you have to the property — you enjoy public protection against those who would take it from you

This view of property, prominent in Rousseau and presupposed in the broader republican tradition, is scarcely questionable in view of the salient diversity in systems of property

These observations, scarcely richer than platitudes, are important for giving us a perspective on the market and the economy, undermining the libertarian image. That picture represents the market as a res privata, a private thing, suggesting that the role of the state is merely to lay low the hills in the way of the market and smooth the paths for its operation. And so it depicts any other interventions of government in the market as dubious on philosophical, not just empirical, grounds.

this image accounts for the continuing attachment to austerity among those on the right. They are philosophically opposed to Keynesianism, not just opposed on empirical grounds, and their ideological stance makes empirically based arguments for Keynesianism invisible to them.

we are firmly in a transition from a Middle East shaped by great powers to a Middle East shaped by Mother Nature.

We are just at the start of this paradigm shift from resistance to resilience, as this region starts to become too hot, too populated and too water-starved to sustain any quality of life.

et’s go back to Biden. He was dead right: America’s presence in Afghanistan and tacit security guarantees around the region were both stabilizing and enabling a lot of bad behavior — boycotts, occupations, reckless adventures and brutal interventions.

President Barack Obama’s pullback from the region and President Donald Trump’s refusal to retaliate against Iran — after it sent a wave of drones to attack a key Saudi oil facility in 2019 — were the warning signs that America had grown weary of intervening and refereeing in the Middle East’s sectarian wars. Biden just made it official.

“The U.S. is not pulling out entirely, but it is pulling back, and all of its Sunni Arab partners are now acting to protect themselves — and to stabilize the region — in an era when the U.S. will no longer be dominant there,”

“But the U.S. will still be needed to deter Iran, should it develop a nuclear capability — and to defuse other conflicts.”

just as we once supplanted the Soviets as the dominant shaper in the region, Mother Nature is now supplanting America as the dominant force.

In Mother Nature’s Middle East, leaders will be judged not by how much they resist one another or great powers, but by how much resilience they build for their people and nations at a time when the world will be phasing out fossil fuels, at a time when all the Arab-Muslim states have booming populations under the age of 30 and at a time of intensifying climate change.

The United Nations recently reported that Afghanistan has been hit with the worst drought in more than 30 years. It is crushing farmers, pushing up food prices and putting 18.8 million Afghans — nearly half of the population — into food insecurity. Over to you, Mr. Taliban: You broke it, you own it.

there may be a day, very soon, where the United States will need to return to active Israeli-Palestinian diplomacy — not based on land for peace, but sun and fresh water for peace. EcoPeace Middle East, an alliance of Israeli, Palestinian and Jordanian environmentalists, recently put forward just such a strategy called the “Green Blue Deal.”

How would it work? Jordan, with its vast desert areas, has the comparative advantage to produce large amounts of cheap solar electricity to meet its own needs and also to sell to the Israeli and Palestinian grids to “generate the electricity for desalination plants that could provide all three parties abundant fresh water,”

All the parties there are ecologically interdependent, but they have unhealthy interdependencies rather than healthy ones. America could become the trusted mediator who forges healthy interdependencies

Admiral Lee Hsi-ming, who was chief of Taiwan’s General Staff from 2017 to 2019, has championed the shift to asymmetric capabilities and has emerged as a Cassandra-like figure in his warnings that Taiwan is not preparing fast enough

You may not be able to stop an invasion, Lee says, but you can stop China from subjugating Taiwan. This entails denying China the ability to control the battle space. The Chinese haven’t fought a war in several decades, and Taiwan has geographic advantages—including ample mountains and few beaches suitable for amphibious operations

the first three section headings: “I. Taiwan Is Part of China—This Is an Indisputable Fact,” “II. Resolute Efforts of the CPC to Realize China’s Complete Reunification,” and “III. 2fChina’s Complete Reunification Is a Process That Cannot Be Halted.”

Lee points to two possible scenarios. The first is a coercive approach in which China encircles and pressures Taiwan—perhaps even seizing outlying islands and engaging in missile strikes. The second is a full-scale invasion.

Politically, Lee said, the message from China to the U.S. and Taiwan is simple: “I can do whatever I want in Taiwan, and there’s nothing the U.S. can do about it.” This message came across unequivocally in a white paper that Beijing released in August.

Anti-ship missiles, anti-tank weapons, shoulder-fired surface-to-air missiles, drones, long-range artillery, and small arms could wreak havoc on an invading force, and disrupt the supply chains necessary to sustain an occupation.

Lee also argues that Taiwan’s civilian population should be organized into a trained Territorial Defense Force, so that any attempted occupation would be met by the broadest possible resistance. “As long as China fails, Taiwan wins the war,”

“The purpose is to make China believe that if you want to invade Taiwan, you will suffer huge losses,” Lee said. “And if you still invade Taiwan, you will not be able to succeed.

as Lee sees it, the pace must quicken. “Taiwan needs a strategic paradigm shift,”

When Chiang fled to Taiwan—with roughly 2 million Chinese from the mainland—there were some 6 million people already living on an island that was just emerging from 50 years of Japanese rule. Most of the people living on the island when Chiang arrived could claim roots in Taiwan going back hundreds of years. They had their own languages and culture. So too did the island’s many Indigenous groups, such as the Amis, the Atayal, and the Paiwan.

To subjugate the island, Chiang killed and imprisoned tens of thousands over decades—a period known as the White Terror. He set up a military dictatorship under the leadership of his Chinese nationalist Kuomintang Party (KMT) and, from this offshore platform, vowed to reclaim mainland China.

After the election of President Ma Ying-jeou, in 2008, links of trade, investment, and travel helped reduce tensions with China. Ma was from the KMT, and the party’s Chinese heritage and its ties to Taiwan’s business elite eased the way to détente with Beijing.

But many Taiwanese, particularly the young, feared that forging too close a connection could ultimately give Beijing leverage over Taiwan. In 2014, in what became known as “the Sunflower Movement,” named for the flower that served as a symbol of hope, students occupied the Taiwan legislature to oppose a free-trade agreement with China. After a tense standoff, they succeeded in stopping the deal.

tus quo is really interesting, because in the American context that is what it mean

But the idea of it here is: There is no need to declare independence, because we are already independent. This country functions like an independent nation, but someone else says it is not.” Recent polling suggests that fewer than 5 percent of people in Taiwan identify as “only Chinese.”

n Chinese and KMT officials 30 years ago, an outcome

at represents anything but consensus. To the Chinese Communist Party, the consensus is that there is one China, and the government in Beijing is the sole legitimate authority. To the KMT, the consensus is that there is one China, but the Republic of China in Taiwan is the legitimate government. To the DPP, there is no consensus, only a fraught political reality to be managed

China proposes a “one country, two systems” regime, in which Taiwan becomes a formal part of China but maintains an autonomous political system. There is one big problem with this proposal: Hong Kong

in 2020, several “national-security laws” were passed giving the authorities broad powers to crush dissent. Activists were rounded up. Independent media were shut down. One country, two systems was dead. The fate of Hong Kong has had a profound impact on Taiwan.

Ukraine inspired the Taiwanese society a lot, including how Zelensky told their story,” Chiang said. He was almost matter-of-fact when he told me, “I would say war between China and Taiwan will definitely happen. We want to win.”

In our conversation, Tsai talked about what she had learned from Ukraine. One lesson is simply the need for international support—to defend itself or, better, to avoid a war in the first place

Another lesson of Ukraine is the importance of national character. Outside support, Tsai emphasized, depends on qualities only Taiwan can provide. “You need to have good leadership,” she said, “but more important is the people’s determination to defend themselves, and the Ukrainian people showed that.

Hanging over all of this is the role of the United States. As one Taiwanese ex

ert pointedly asked me: “We can make ourselves a porcupine, but what are you going to do?”

Would the U.S. risk the biggest naval battle since World War II to break a Chinese blockade? Would the U.S. attack an invading Chinese force knowing that U.S. military personnel in Japan, Guam, and possibly Hawaii are within range of Chinese rockets? Would the American people really support a war with the world’s most populous country in order to defend Taiwan?

how the U.S. can help prepare Taiwan than on what the U.S. would do in a conflict.

small victories only point up the scale of the challenge. Wu himself has used the term cognitive warfare to describe the comprehensive nature of China’s pressure on Taiwan. “They use missiles, air, ships, disinformation, cyberattacks, and economic coercion,” he told me. As a warning sign, China has banned hundreds of exported products from Taiwan. “They claimed that our mangoes tested positive for COVID,”

. If China takes Taiwan, Wu suggested that the Chinese Communist Party’s ambitions could extend to the East China Sea, threatening Japan; to the South China Sea, where China has built militarized islands and claims an entire body of water bordering several nations; to the Indian Ocean, where China is expanding influence and could establish military bases; and to the Pacific Ocean, where China is working to establish security pacts with island nations

I sat there reading message after message, all posted in closed chat rooms, meant to bend Taiwanese minds to Beijing’s worldview. The meanings of buzzwords like cognitive warfare and resilience came into sharper focus. Facing the seemingly bottomless resources of a massive totalitarian state, here were two young people working for free on a Wednesday night, quietly insisting on the notion that there is indeed such a thing as objective reality.

to preserve this, Taiwan has to find some mix of the approaches that I’d heard about: preparing for a war while avoiding it; talking to China without being coerced by it; drawing closer to the U.S. without being reduced to a chess piece on the board of a great game; tending to a young democracy without letting divisions weaken it; asserting a unique identity without becoming an independent country.

Shoggoths are fictional creatures, introduced by the science fiction author H.P. Lovecraft in his 1936 novella “At the Mountains of Madness.” In Lovecraft’s telling, Shoggoths were massive, blob-like monsters made out of iridescent black goo, covered in tentacles and eyes.

In a nutshell, the joke was that in order to prevent A.I. language models from behaving in scary and dangerous ways, A.I. companies have had to train them to act polite and harmless. One popular way to do this is called “reinforcement learning from human feedback,” or R.L.H.F., a process that involves asking humans to score chatbot responses, and feeding those scores back into the A.I. model.

Most A.I. researchers agree that models trained using R.L.H.F. are better behaved than models without it. But some argue that fine-tuning a language model this way doesn’t actually make the underlying model less weird and inscrutable. In their view, it’s just a flimsy, friendly mask that obscures the mysterious beast underneath.

@TetraspaceWest, the meme’s creator, told me in a Twitter message that the Shoggoth “represents something that thinks in a way that humans don’t understand and that’s totally different from the way that humans think.”

@TetraspaceWest said, wasn’t necessarily implying that it was evil or sentient, just that its true nature might be unknowable.

“I was also thinking about how Lovecraft’s most powerful entities are dangerous — not because they don’t like humans, but because they’re indifferent and their priorities are totally alien to us and don’t involve humans, which is what I think will be true about possible future powerful A.I.”

when Bing’s chatbot became unhinged and tried to break up my marriage, an A.I. researcher I know congratulated me on “glimpsing the Shoggoth.” A fellow A.I. journalist joked that when it came to fine-tuning Bing, Microsoft had forgotten to put on its smiley-face mask.

If it’s an A.I. safety researcher talking about the Shoggoth, maybe that person is passionate about preventing A.I. systems from displaying their true, Shoggoth-like nature.

In any case, the Shoggoth is a potent metaphor that encapsulates one of the most bizarre facts about the A.I. world, which is that many of the people working on this technology are somewhat mystified by their own creations. They don’t fully understand the inner workings of A.I. language models, how they acquire new capabilities or why they behave unpredictably at times. They aren’t totally sure if A.I. is going to be net-good or net-bad for the world.

That some A.I. insiders refer to their creations as Lovecraftian horrors, even as a joke, is unusual by historical standards. (Put it this way: Fifteen years ago, Mark Zuckerberg wasn’t going around comparing Facebook to Cthulhu.)

And it reinforces the notion that what’s happening in A.I. today feels, to some of its participants, more like an act of summoning than a software development process. They are creating the blobby, alien Shoggoths, making them bigger and more powerful, and hoping that there are enough smiley faces to cover the scary parts.

A great many people are dismissive of suggestions that any of these systems are “really” thinking, because they’re “just” doing something banal (like making statistical predictions about the next word in a sentence). What they fail to appreciate is that there is every reason to suspect that human cognition is “just” doing those exact same things. It matters not that birds flap their wings but airliners don’t. Both fly. And these machines think. And, just as airliners fly faster and higher and farther than birds while carrying far more weight, these machines are already outthinking the majority of humans at the majority of tasks. Further, that machines aren’t perfect thinkers is about as relevant as the fact that air travel isn’t instantaneous. Now consider: we’re well past the Wright flyer level of thinking machine, past the early biplanes, somewhere about the first commercial airline level. Not quite the DC-10, I think. Can you imagine what the AI equivalent of a 777 will be like? Fasten your seatbelts.

@BLA. You are incorrect. Everything has nature. Its nature is manifested in making humans react. Sure, no humans, no nature, but here we are. The writer and various sources are not attributing nature to AI so much as admitting that they don’t know what this nature might be, and there are reasons to be scared of it. More concerning to me is the idea that this field is resorting to geek culture reference points to explain and comprehend itself. It’s not so much the algorithm has no soul, but that the souls of the humans making it possible are stupendously and tragically underdeveloped.

@thomas h. You make my point perfectly. You’re observing that the way a plane flies — by using a turbine to generate thrust from combusting kerosene, for example — is nothing like the way that a bird flies, which is by using the energy from eating plant seeds to contract the muscles in its wings to make them flap. You are absolutely correct in that observation, but it’s also almost utterly irrelevant. And it ignores that, to a first approximation, there’s no difference in the physics you would use to describe a hawk riding a thermal and an airliner gliding (essentially) unpowered in its final descent to the runway. Further, you do yourself a grave disservice in being dismissive of the abilities of thinking machines, in exactly the same way that early skeptics have been dismissive of every new technology in all of human history. Writing would make people dumb; automobiles lacked the intelligence of horses; no computer could possibly beat a chess grandmaster because it can’t comprehend strategy; and on and on and on. Humans aren’t nearly as special as we fool ourselves into believing. If you want to have any hope of acting responsibly in the age of intelligent machines, you’ll have to accept that, like it or not, and whether or not it fits with your preconceived notions of what thinking is and how it is or should be done … machines can and do think, many of them better than you in a great many ways. b&

When even tech companies are saying AI is moving too fast, and the articles land on page 1 of the NYT (there's an old reference), I think the greedy will not think twice about exploiting this technology, with no ethical considerations, at all.

@nome sane? The problem is it isn't data as we understand it. We know what the datasets are -- they were used to train the AI's. But once trained, the AI is thinking for itself, with results that have surprised everybody.

The unique feature of a shoggoth is it can become whatever is needed for a particular job. There's no actual shape so it's not a bad metaphor, if an imperfect image. Shoghoths also turned upon and destroyed their creators, so the cautionary metaphor is in there, too. A shame more Asimov wasn't baked into AI. But then the conflict about how to handle AI in relation to people was key to those stories, too.

But I was constantly confronted by the contradictions embedded in American leadership, the knowledge that our government arms autocrats while its rhetoric appeals to the dissidents trying to overthrow them or that our nation enforces rules — for the conduct of war, the resolution of disputes and the flow of commerce — while insisting that America be excused from following them when they become inconvenient.

He helped extend the war in Vietnam and expand it to Cambodia and Laos, where the United States rained down more bombs than it dropped on Germany and Japan in World War II. That bombing — often indiscriminately massacring civilians — did nothing to improve the terms on which the Vietnam War ended; if anything, it just indicated the lengths to which the United States would go to express its displeasure at losing.

For decades, he was a coveted guest at gatherings of statesmen and tycoons, perhaps because he could always provide an intellectual framework for why some people are powerful and justified in wielding power

Mr. Kissinger was fixated on credibility, the idea that America must impose a price on those who ignore our demands to shape the decisions of others in the future. It’s hard to see how the bombing of Laos, the coup in Chile or the killings in East Pakistan (now Bangladesh) contributed to the outcome of the Cold War.

But Mr. Kissinger’s unsentimental view of global affairs allowed him to achieve consequential breakthroughs with autocratic countries closer to America’s weight class — a détente with the Soviet Union that reduced the escalatory momentum of the arms race and an opening to China that deepened the Sino-Soviet split, integrated the People’s Republic of China into the global order and prefaced Chinese reforms that lifted hundreds of millions of people out of poverty.

From a strategic standpoint, Mr. Kissinger surely knew, being a superpower carried with it a cavernous margin of error that can be forgiven by history

Now history has come full circle. Around the world, we see a resurgence of autocracy and ethnonationalism, most acutely in Russia’s war against Ukraine

Just a few decades after the end of the Vietnam War, the same countries we’d bombed were seeking expanded trade with the United States. Bangladesh and East Timor are now independent nations that receive American assistance. Chile is governed by a millennial socialist whose minister of defense is Mr. Allende’s granddaughter.

Superpowers do what they must. The wheel of history turns. When and where you live determines whether you get crushed or lifted by it

But that worldview mistakes cynicism — or realism — for wisdom. The story, what it’s all about, matters. Ultimately, the Berlin Wall came down not because of chess moves made on the board of a great game but rather because people in the East wanted to live like the people in the West.

Economics, popular culture and social movements mattered. Despite all our flaws, we had a better system and story.

Credibility, after all, is not just about whether you punish an adversary to send a message to another; it’s also about whether you are what you say you are. No one can expect perfection in the affairs of state any more than in relations among human beings.

But the United States has paid a price for its hypocrisy, though it’s harder to measure than the outcome of a war or negotiation. Over the decades, our story about democracy has come to ring hollow to a growing number of people who can point to the places where our actions drained our words of meaning and “democracy” just sounded like an extension of American interests.

Similarly, our insistence on a rules-based international order has been ignored by strongmen who point to America’s sins to justify their own.

The generous defense is that Mr. Kissinger represented an ethos that saw the ends (the defeat of the Soviet Union and revolutionary Communism) as justifying the means. But for huge swaths of the world, this mind-set carried a brutal message that America has often conveyed to its own marginalized populations: We care about democracy for us, not for them.

In Gaza the United States has supported an Israeli military operation that has killed civilians at a pace that has once again suggested to much of the world that we are selective in our embrace of international laws and norms.

Meanwhile, at home, we see how democracy has become subordinate to the pursuit of power within a chunk of the Republican Party.

This is where cynicism can lead. Because when there is no higher aspiration, no story to give meaning to our actions, politics and geopolitics become merely a zero-sum game. In that kind of world, might makes right.

his is also a cautionary tale. As imperfect as we are, the United States needs our story to survive. It’s what holds together a multiracial democracy at home and differentiates us from Russia and China abroad.

That story insists that a child in Laos is equal in dignity and worth to our children and that the people of Chile have the same right of self-determination as we do. For the United States, that must be a part of national security. We forget that at our peril.

A handful of financial institutions produced them when Rio de Janeiro hosted in 2016, using a mixture of macroeconomic indicators and performances at previous Olympics to forecast total medal hauls for each country.

Gracenote’s distinguishing feature is the ability to produce quantitative analysis for each event.

Gracenote still uses the old system to produce its public medal table, which also deals in absolute forecasts, rather than fractional ones. If a French athlete, say, is the most likely to win an event, France gets awarded one gold medal in the table, even though the true probability of the athlete winning gold is less than 100% and his chances of claiming silver and bronze are greater than 0%.

the Elo rating system, which was developed for chess by Arpad Elo, a Hungarian physicist. The formula exchanges ranking points from the loser to the winner, with greater rewards for beating stronger opponents. The difference in ratings points between two rivals can be easily used to calculate the probability that one will beat the other.

Yet only two events on the winter Olympics programme, curling and ice hockey, involve head-to-head contests.

Those that compete in teams have their scores blended with their compatriots. And for those that participate in a number of events, such as Ms Dahlmeier, results in related disciplines affect multiple ratings. A strong performance in the biathlon sprint, a group race, would boost her ranking in the pursuit, a staggered race, for example.

The best way to answer that question is to take every previous contest in the sport and analyse how past results correlate with future success.

The bans have benefited Norway most, as the country will likely gain of the five of the 12 foregone medals—enough to nudge it ahead of Germany into first place in terms of total medals won.

Mr Gleave notes that the favourite only wins about 30% of the time, a lower share than in any other winter sport. Ms Dahlmeier’s rating has dwindled a little, but not by enough to suggest that last year’s record breaker has become this year’s flop.

Gracenote’s research into age curves for each sport shows that the best biathletes can maintain their peak performance into their early 30s (see chart). Expect to see more event-by-event forecasting at future Olympics, too.

The key issue is not “human-competitive” intelligence (as the open letter puts it); it’s what happens after AI gets to smarter-than-human intelligence. Key thresholds there may not be obvious, we definitely can’t calculate in advance what happens when, and it currently seems imaginable that a research lab would cross critical lines without noticing.

Many researchers steeped in these issues, including myself, expect that the most likely result of building a superhumanly smart AI, under anything remotely like the current circumstances, is that literally everyone on Earth will die. Not as in “maybe possibly some remote chance,” but as in “that is the obvious thing that would happen.”

It’s not that you can’t, in principle, survive creating something much smarter than you; it’s that it would require precision and preparation and new scientific insights, and probably not having AI systems composed of giant inscrutable arrays of fractional numbers.

Absent that caring, we get “the AI does not love you, nor does it hate you, and you are made of atoms it can use for something else.”

Without that precision and preparation, the most likely outcome is AI that does not do what we want, and does not care for us nor for sentient life in general. That kind of caring is something that could in principle be imbued into an AI but we are not ready and do not currently know how.

The likely result of humanity facing down an opposed superhuman intelligence is a total loss

To visualize a hostile superhuman AI, don’t imagine a lifeless book-smart thinker dwelling inside the internet and sending ill-intentioned emails. Visualize an entire alien civilization, thinking at millions of times human speeds, initially confined to computers—in a world of creatures that are, from its perspective, very stupid and very slow. A sufficiently intelligent AI won’t stay confined to computers for long. In today’s world you can email DNA strings to laboratories that will produce proteins on demand, allowing an AI initially confined to the internet to build artificial life forms or bootstrap straight to postbiological molecular manufacturing.

There’s no proposed plan for how we could do any such thing and survive. OpenAI’s openly declared intention is to make some future AI do our AI alignment homework. Just hearing that this is the plan ought to be enough to get any sensible person to panic. The other leading AI lab, DeepMind, has no plan at all.

An aside: None of this danger depends on whether or not AIs are or can be conscious; it’s intrinsic to the notion of powerful cognitive systems that optimize hard and calculate outputs that meet sufficiently complicated outcome criteria.

I didn’t also mention that we have no idea how to determine whether AI systems are aware of themselves—since we have no idea how to decode anything that goes on in the giant inscrutable arrays—and therefore we may at some point inadvertently create digital minds which are truly conscious and ought to have rights and shouldn’t be owned.

I refrained from signing because I think the letter is understating the seriousness of the situation and asking for too little to solve it.

the thing about trying this with superhuman intelligence is that if you get that wrong on the first try, you do not get to learn from your mistakes, because you are dead. Humanity does not learn from the mistake and dust itself off and try again, as in other challenges we’ve overcome in our history, because we are all gone.

If we held anything in the nascent field of Artificial General Intelligence to the lesser standards of engineering rigor that apply to a bridge meant to carry a couple of thousand cars, the entire field would be shut down tomorrow.

We are not on course to be prepared in any reasonable time window. There is no plan. Progress in AI capabilities is running vastly, vastly ahead of progress in AI alignment or even progress in understanding what the hell is going on inside those systems

Many researchers working on these systems think that we’re plunging toward a catastrophe, with more of them daring to say it in private than in public; but they think that they can’t unilaterally stop the forward plunge, that others will go on even if they personally quit their jobs.

This is a stupid state of affairs, and an undignified way for Earth to die, and the rest of humanity ought to step in at this point and help the industry solve its collective action problem.

When the insider conversation is about the grief of seeing your daughter lose her first tooth, and thinking she’s not going to get a chance to grow up, I believe we are past the point of playing political chess about a six-month moratorium.

The moratorium on new large training runs needs to be indefinite and worldwide. There can be no exceptions, including for governments or militaries. If the policy starts with the U.S., then China needs to see that the U.S. is not seeking an advantage but rather trying to prevent a horrifically dangerous technology which can have no true owner and which will kill everyone in the U.S. and in China and on Earth

Here’s what would actually need to be done:

Shut down all the large GPU clusters (the large computer farms where the most powerful AIs are refined). Shut down all the large training runs

Put a ceiling on how much computing power anyone is allowed to use in training an AI system, and move it downward over the coming years to compensate for more efficient training algorithm

Track all GPUs sold. If intelligence says that a country outside the agreement is building a GPU cluster, be less scared of a shooting conflict between nations than of the moratorium being violated; be willing to destroy a rogue datacenter by airstrike.

Frame nothing as a conflict between national interests, have it clear that anyone talking of arms races is a fool

Make it explicit in international diplomacy that preventing AI extinction scenarios is considered a priority above preventing a full nuclear exchange, and that allied nuclear countries are willing to run some risk of nuclear exchange if that’s what it takes to reduce the risk of large AI training runs.

when your policy ask is that large, the only way it goes through is if policymakers realize that if they conduct business as usual, and do what’s politically easy, that means their own kids are going to die too.

Without giving specifics, it said AI tools have already begun to damage human dignity because the tools are sometimes designed to seize users’ attention without regard to morals or accuracy.

Unless AI is restrained, “in the worst-case scenario, democracy and social order could collapse, resulting in wars,” the manifesto said.

It said Japan should take measures immediately in response, including laws to protect elections and national security from abuse of generative AI.

The Biden administration is also stepping up oversight, invoking emergency federal powers last October to compel major AI companies to notify the government when developing systems that pose a serious risk to national security. The U.S., U.K. and Japan have each set up government-led AI safety institutes to help develop AI guidelines.

NTT and Yomiuri said their manifesto was motivated by concern over public discourse. The two companies are among Japan’s most influential in policy. The government still owns about one-third of NTT, formerly the state-controlled phone monopoly.

Yomiuri Shimbun, which has a morning circulation of about six million copies according to industry figures, is Japan’s most widely-read newspaper. Under the late Prime Minister Shinzo Abe and his successors, the newspaper’s conservative editorial line has been influential in pushing the ruling Liberal Democratic Party to expand military spending and deepen the nation’s alliance with the U.S.

The Yomiuri’s news pages and editorials frequently highlight concerns about artificial intelligence. An editorial in December, noting the rush of new AI products coming from U.S. tech companies, said “AI models could teach people how to make weapons or spread discriminatory ideas.” It cited risks from sophisticated fake videos purporting to show politicians speaking.

NTT is active in AI research, and its units offer generative AI products to business customers. In March, it started offering these customers a large-language model it calls “tsuzumi” which is akin to OpenAI’s ChatGPT but is designed to use less computing power and work better in Japanese-language contexts.