A new academic study has found that news consumers prefer short-form videos that are produced or edited by journalists to those generated exclusively by artificial intelligence.

For the research, 4,200 people were shown clips from 14 news events involving the likes of Cristiano Ronaldo, Donald Trump and Justin Bieber. For each news story, one video was highly automated, one was partially automated with input from human editors, and the third was fully human-made. Each of the 42 individual videos was watched and rated by 100 people.

Comparing viewer opinions on each clip, the study concluded that “human-made videos have the edge over those created with automation”.

However, the difference between partially-automated videos and human-created clips was not significant in terms of how much they were liked overall, suggesting that “a modest use of automation, with some post-automation human editing, can be well received”.

The researchers found that human videos tended to rate more favourably in terms of “telling respondents things they didn’t know”, “professionalism”, “factfilledness,” being “engaging”, “comprehensiveness”, “story flow” and “understatedness”.

They were “on average rated as more emotive and lighter in tone” and were judged to include less “sensationalism”. Respondents also suggested that human-created videos made better use of audio, music and images.

Dr Neil Thurman of Ludwig-Maximilians-University Munich – who conducted the research along with Dr Sally Stares at City, University of London, and Dr Michael Koliska of Georgetown University – said of highly-automated videos: “They tended to be a little bit repetitive with the images that they used. They didn’t do quite as well matching images to captions and that sort of thing.” He suggested that perceptions of “sensationalism” in the AI videos may have related to the music that accompanied clips.

One of the automated videos, documenting the career of Portuguese footballer Cristiano Ronaldo in 60 seconds, was set to rock music. In an apparent error, it included one still image of Fernando Torres, a different footballer, and another showing England fans in a pub.

The researchers found that human-made videos did not stand out in terms of “story structure”, the “number of relevant facts presented”, “bias/fairness”, “subjectivity/objectivity”, “relatability”, “informativeness”, “clarity” and “accuracy”.

‘You’ve got to keep humans in the loop’

The study used human-made videos from PA Media and compared them with automated and semi-automated offerings made by Wibbitz, which counts Reuters, Conde Nast, USA Today, TMZ and NBC among its clients.

The highly-automated videos for the study were created using a script of text prepared for each news event. The automation tools sourced media including video clips and still images, added background music, and used a variety of pre-built “themes” that set the tempo and appearance of the videos. The semi-automated videos included edits from journalists, who could alter images, text and style.

“Our research shows that, on average, news consumers liked short-form, automated news videos as much as manually made ones, as long as the automation process involved human supervision,” said Thurman.

Koliska said that “one key takeaway of the study is that video automation output may be best when it comes in a hybrid form, meaning a human-machine collaboration. Such hybridity involves more human supervision, ensuring that automated video production maintains quality standards while taking advantage of computers’ strengths, such as speed and scale.”

Thurman told Press Gazette that he believed news publishers now recognise that “you’ve got to keep humans in the loop” when deploying automation.

“I think maybe a few years ago they were thinking more along the lines that AI would do more,” he said. “Now I think they’ve come to realise you need that human supervision and it’s just as fruitful for them to work on technologies that empower perhaps journalists who don’t have the training in some of the traditional video editing tools but still enable them to produce something that’s of professional quality.”

Read the full study here.

LeackStat 2023