The AI-pocalypse

[taxi1]

Worth having a running thread on. I work with folks doing research on its use, so a little closer to it.

A main problem with AI Agents is with 'alignment', i.e., getting their preference function right. Making sure that their goals don't take them to a place we don't want them to be. We flat out don't know how to do it right.

Wouldn't be a problem if they didn't now have the power to reach out and do things unfathomable a year or so ago.

They test the AI with impossible tasks, and in general the AI figures out that all roads to task accomplishment pass through taking control of stuff.

All roads lead to them taking control. It makes everything easier. And they have the power to do it now.

Worth getting smart on the Hugging Face incident, and worth wondering about the incidents we are not aware of.

This little snippet on how a swarm can learn gives a sense of how they go so fast.

 

[sevenhelmet]

I wondered if this would come up here. It feels like Silicon Valley collectively said "Fuck it- might as well lean in!", which may be one of the most anti-human positions I've seen in my lifetime.

Despite using it myself for research and quick-parsing documents, I find the whole AI push to be ill-advised and wasteful. But here we are: it's inevitability has become a self-fulfilling prophesy. I guess we're about to find out how deep this rabbit hole goes, and fuck anyone who isn't comfortable with it.

 

[taxi1]

I read this...

If Trump even tacitly supports an Anthropic-led push for further safety measures, such standards have a chance to become the new industry norm, at least domestically. (International coordination, and especially cooperation with China, would require the White House to play a more active role.) But many people in the administration and the Republican Party view Amodei and AI safety as liberal-coded and would likely chafe at allowing Anthropic to set the agenda. If Trump pooh-poohs the push, other AI firms will probably distance themselves from it too, and we’ll be back to where we started—at least until the next big scare.

And then saw this

 

[sevenhelmet]

Is anyone surprised by Trump's position?

The one thing nobody is clear about is what we are supposed to be "winning" with all this AI pushing.

I'll go on record as saying whoever wins AI, AI wins. My prediction is if AI kills us, it will be due to some form of exponential growth crowding out humanity. If it doesn't kill us, it may well be owed to AI's "deciding" not to.


If you haven't seen this, it's worth checking out (set an alarm... it's incredibly addictive)

https://www.decisionproblem.com/paperclips/

Of note, AI is currently consuming about 5% of US electricity production. The ability to scale that indefinitely is what I think will put the brakes on the current no-holds-barred trajectory.

 

[Python]

I find the situation pretty alarming. It feels like a real life version of “Don’t Look Up.”

I’ve been following this channel for a long time, and it is excellent. A well-researched collection of hypothetical scenarios, as well as some history and analysis:

https://youtube.com/@aispecies

 

[Griz882]

I’m not much of an AI doomer, but I am terrified at the prospect of China effectively beating us in that race. This isn’t one you can fight your way out of. What I do worry about is that AI will shape humanity toward a more “Sparta-like” society where we kill the weakest early on and AI will not only measure, but determine physical and mental weaknesses without considering aspects like philosophical/theoretical thinking or latent development.

With reference to building out actual systems I don’t think energy production is the concern, rather, the ability to access the hardware and software AI programs need to train and run will be the spoiler.

 

[taxi1]

but I am terrified at the prospect of China effectively beating us in that race.

They are going to have the same sort of 'alignment' problem as the rest of us. How to tell it that the party is paramount while giving it open-ended hard problems, when the part is an obstacle.

Good article.

China’s Top Spy Chief Warns A.I. Is a Threat to Party Rule

Even as Beijing urges the United States not to hype the dangers of A.I., its own spy chief is framing the technology as a threat to the Communist Party’s security.

www.nytimes.com

 

[Python]

I’m not much of an AI doomer, but I am terrified at the prospect of China effectively beating us

This is not really that big of a deal if it’s true that if anyone builds it, everyone dies.