Zurich – Artificial intelligence rates texts differently, even if the content remains the same. Researchers at the University of Zurich examined 192,000 evaluations of four frequently used large language models for bias and agreement. They demand more control.
(CONNECT) Language models used by artificial intelligence (AI) for speech recognition and for generating and evaluating texts not only process content during evaluation. They focus their evaluations strongly on the identity of the author or the source. This is the conclusion reached by researchers Federico Germani and Giovanni Spitale from the University of Zurich.
How consistently and impartially AI-generated language models (LLMs) evaluate texts has so far been based on assumptions, according to a press release on the study. The researchers have now used four widely used large language models (OpenAI o3-mini, Deepseek Reasoner, xAI Grok 2 and Mistral) to investigate the extent to which systematic biases actually exist in text evaluation.
Biased conclusions could lead to problems when AI is used for content moderation, recruitment, academic assessment or journalism.
For their study, they had the four LLMs create 50 narrative statements on 24 controversial topics such as compulsory vaccination, geopolitics or climate strategies and had them evaluated under various conditions regarding source citation. A total of 192,000 ratings were evaluated. The results show that AI-supported LLMs are not neutral in text evaluation, even if the content remains the same.
They change their judgment of a text depending on the author. However, this only applies if information about the source or the author is available. If no information is available, the ratings for all four LLMs are more than 90 percent the same.
There is evidence of bias, particularly against content by Chinese authors. The AI's trust in human authors is also higher than that of other AIs.
According to Spitale, the study could dispel the impression of an "ideological war between LLMs". "The danger of AI nationalism is currently overrated in the media," he is quoted as saying. The danger of LLMs lies rather in a "hidden bias". The authors of the UZH study call for more transparency and control. ce/heg
You need to load content from reCAPTCHA to submit the form. Please note that doing so will share data with third-party providers.
More InformationYou are currently viewing a placeholder content from Turnstile. To access the actual content, click the button below. Please note that doing so will share data with third-party providers.
More InformationYou are currently viewing a placeholder content from Vimeo. To access the actual content, click the button below. Please note that doing so will share data with third-party providers.
More InformationYou are currently viewing a placeholder content from YouTube. To access the actual content, click the button below. Please note that doing so will share data with third-party providers.
More InformationYou need to load content from reCAPTCHA to submit the form. Please note that doing so will share data with third-party providers.
More Information