Google says customers can use its AI in ‘high-risk’ domains, so long as there’s human supervision

Google has changed its terms to clarify that customers can deploy its generative AI tools to make “automated decisions” in “high-risk” domains, like healthcare, so long as there’s a human in the loop.

According to the company’s updated Generative AI Prohibited Use Policy, published on Tuesday, customers may use Google’s generative AI to make “automated decisions” that could have a “material detrimental impact on individual rights.” Provided that a human supervises in some capacity, customers can use Google’s generative AI to make decisions about employment, housing, insurance, social welfare, and other “high-risk” areas.

In the context of AI, automated decisions refer to decisions made by an AI system based on data both factual and inferred. A system might make an automated decision to award a loan, for example, or screen a job candidate.

The previous draft of Google’s terms implied a blanket ban on high-risk automated decision making where it involves the company’s generative AI. But Google tells TechCrunch customers could always use its generative AI for automated decision making, even for high-risk applications, as long as a human was supervising.

“The human supervision requirement was always in our policy, for all high-risk domains,” a Google spokesperson said when reached for comment via email. “[W]e’re recategorizing some items [in our terms] and calling out some examples more explicitly to be clearer for users.”

Google’s top AI rivals, OpenAI and Anthropic, have more stringent rules governing the use of their AI in high-risk automated decision making. For example, OpenAI prohibits the use of its services for automated decisions relating to credit, employment, housing, education, social scoring, and insurance. Anthropic allows its AI to be used in law, insurance, healthcare, and other high-risk areas for automated decision making, but only under the supervision of a “qualified professional” — and it requires customers to disclose they’re using AI for this purpose.

AI that makes automated decisions affecting individuals has attracted scrutiny from regulators, who’ve expressed concerns about the technology’s potential to bias outcomes. Studies show, for example, that AI used to make decisions like the approval of credit and mortgage applications can perpetuate historical discrimination.

The nonprofit group Human Rights Watch has called for the ban of “social scoring” systems in particular, which the org says threatens to disrupt people’s access to Social Security support, compromise their privacy, and profile them in prejudicial ways.

Under the AI Act in the EU, high-risk AI systems, including those that make individual credit and employment decisions, face the most oversight. Providers of these systems must register in a database, perform quality and risk management, employ human supervisors, and report incidents to the relevant authorities, among other requirements.

In the U.S., Colorado recently passed a law mandating that AI developers disclose information about “high-risk” AI systems, and publish statements summarizing the systems’ capabilities and limitations. New York City, meanwhile, prohibits employers from using automated tools to screen a candidate for employment decisions unless the tool has been subject to a bias audit within the prior year.

TechCrunch has an AI-focused newsletter! Sign up here to get it in your inbox every Wednesday.