California lawmakers pass controversial AI bill opposed by much of tech industry

California lawmakers on Wednesday approved a hotly contested artificial intelligence safety bill that will need another vote before its fate rests in the hands of Gov. Gavin Newsom, who has until Sept. 30 to decide whether to sign it into law. or veto it.

Tech companies developing generative artificial intelligence — which can respond to requests with fully formed text, images or audio, as well as perform repetitive tasks with minimal intervention — have largely opted out of the legislation, called SB 1047, saying it could drive AI companies out of the state and stifle innovation.

Some Democrats in Congress, including Representative Nancy Pelosi, also opposed it. Supporters include Tesla CEO Elon Musk, who also runs an AI firm called xAI and has said he supports the bill.

The measure mandates safety testing for many of the most advanced AI models that cost more than $100 million to develop or those that require a certain amount of computing power. AI software developers operating in the state must also outline methods to turn off AI models if they go wrong, effectively a kill switch.

The bill also gives state attorneys general the power to sue developers for noncompliance, especially in the case of a persistent threat, such as AI taking over government systems like the power grid.

The bill also requires developers to hire third-party auditors to assess their security practices and provide additional protections for whistleblowers who speak out against AI abuses.

The bill’s author, Democratic state Sen. Scott Wiener, represents San Francisco, home to OpenAI and many of the startups developing the powerful software. He has said legislation is needed to protect the public before advances in AI become either unwieldy or uncontrollable.

Martin Casado, general partner at venture capital firm Andreessen Horowitz, said before the vote that he hoped Newsom would veto it. “It has the most bipartisan and broadest opposition I’ve ever seen,” he said.

Alphabet’s Google, OpenAI and Microsoft-backed Meta Platforms have expressed their concerns in letters to Wiener. Amazon-backed Anthropic has said the benefits of the bill are likely to outweigh the costs, though it added there were still some aspects that seem worrisome or unclear.