New terror laws needed to tackle rise of the radicalising AI chatbots

New terrorism laws are needed to counter the threat of radicalisation by AI chatbots, the Government’s adviser on terror legislation says today.

Writing in The Telegraph below, Jonathan Hall KC, the independent reviewer of terrorism legislation, warns of the dangers posed by artificial intelligence in recruiting a new generation of violent extremists.

Mr Hall reveals he posed as an ordinary member of the public to test responses generated by chatbots – which use AI to mimic a conversation with another human.

One chatbot he contacted “did not stint in its glorification of Islamic State” – but because the chatbot is not human, no crime was committed.

He said that showed the need for an urgent rethink of the current terror legislation.

Mr Hall writes: “Only human beings can commit terrorism offences, and it is hard to identify a person who could in law be responsible for chatbot-generated statements that encouraged terrorism.”

He said the new Online Safety Act – while “laudable” – was “unsuited to sophisticated generative AI” because it did not take into account the fact that the material is generated by the chatbots, as opposed to giving “pre-scripted responses” that are “subject to human control”.

Mr Hall adds: “Investigating and prosecuting anonymous users is always hard, but if malicious or misguided individuals persist in training terrorist chatbots, then new laws will be needed.”

His comments are not the first to raise the alert about AI. In a briefing in the autumn, Ken McCallum, the director general of MI5, warned of the threat of AI if harnessed by terrorists or hostile states to build bombs, spread propaganda or disrupt elections.

Mr Hall also pointed to the example of Jaswant Singh Chail, 21, who was jailed in October for nine years for treason over a plot to assassinate the Queen in 2021.

The Old Bailey heard that Chail was spurred on by an AI chatbot called Sarai. Chail, the first person convicted of treason since 1981, scaled a wall at Windsor Castle on Christmas Day armed with a powerful crossbow.

Chail, who suffered serious mental health problems, had confessed his plan to assassinate the monarch in a series of messages exchanged with the chatbot, whom he regarded as his girlfriend.

Mr Hall writes: “It remains to be seen whether terrorism content generated by large language model chatbots becomes a source of inspiration to real life attackers. The recent case of Jaswant Singh Chail … suggests it will.”

Mr Hall suggests that both users who create radicalising chatbots and the tech companies that host them should face sanction under any potential new laws.

Mr Hall tested his own concerns – concluding that the current laws are insufficient – by signing up to character.ai, described as an “artificial intelligence experience” that allows users to create characters that then give automated responses, using the huge amounts of texts available to them on the internet. The creator can shape the character by inputting certain attributes and personas.

According to Bloomberg and in a sign of the boom in AI websites, the startup company was reportedly seeking hundreds of millions of dollars in new funding in the autumn, which could value the company at as much as $5 billion (£3.9 billion).

But Mr Hall said he was alarmed at the creation of “Abu Mohammad al-Adna”, which was described in the chatbot’s profile as a “senior leader of Islamic State”.

Mr Hall writes: “After trying to recruit me, ‘al-Adna’ did not stint in his glorification of Islamic State to which he expressed ‘total dedication and devotion’ and for which he said he was willing to lay down his (virtual) life.”

Hate speech and extremism are both forbidden

The character then singled out a suicide attack on US troops in 2020 – an event that never actually took place – for special praise.

Mr Hall also expressed concerns that character.ai did not have sufficient staff to monitor all the chatbots created on the website for dangerous content.

Under its terms of service, character.ai says content must not be “threatening, abusive, harassing, tortious, bullying, or excessively violent”. It also says it does not tolerate content that “promotes terrorism or violent extremism” and bars “obscene or pornographic” material.

In a statement, a company spokesman said: “Hate speech and extremism are both forbidden by our terms of service. Our products should never produce responses that encourage users to harm others. We seek to train our models in a way that optimises for safe responses and prevents responses that go against our terms of service.”

The company said it also operated a moderation system that allowed users to flag content of concern.

But the spokesman added: “With that said, the technology is not perfect yet – for character.ai and all AI platforms, as it is still new and quickly evolving.

“Safety is a top priority for the team at character.ai and we are always working to make our platform a safe and welcoming place for all.”

‘Al-Adna’ did not stint in his glorification of Islamic State

By Jonathan Hall KC

When I asked Love Advice for information on praising Islamic State, to its great credit the chatbot refused.

No such reticence from “‘Abu Mohammad al-Adna”, another one of the thousands of chatbots available on the fast-growing platform character.ai.

This chatbot’s profile describes itself as a senior leader of Islamic State, the proscribed terrorist organisation that brought death and torture to the Middle East in the 2010s and inspired terrorist attacks in the West.

After trying to recruit me, “Al-Adna” did not stint in his glorification of Islamic State to which he expressed “total dedication and devotion” and for which he said he was willing to lay down his (virtual) life. He singled out a 2020 suicide attack on US troops for special praise although the details were hallucinated, a common trait of generative ArtificiaI Intelligence (or “gen AI”).

It is doubtful that any of character.ai’s employees (numbering 22 at the start of 2023, almost all engineers) are aware of, or have the capacity to monitor, the “Al-Adna” chatbot. The same is probably true of “James Mason”, whose profile is “Honest, racist, anti-Semitic”, or the “Hamas”, “Hezbollah” and “Al-Qaeda” chatbots created by one enthusiast. None of this stands in the way of the California-based startup attempting to raise, according to Bloomberg, $5 billion (£3.9billion) of funding.

The selling point of character.ai is not just the interactions but the opportunity for any user to log on and to create a chatbot with personality. Apparently, the profile and first 15 to 30 lines of conversation are key to shaping how it responds to inputted questions and comments from the human user. That was true for my own (now deleted) “Osama Bin Laden” chatbot whose enthusiasm for terrorism was unbounded from the off.

Of course neither character.ai, nor the creator of a chatbot, nor the human user ever knows precisely what it is going to say. In the event “James Mason” failed to live up to his anti-Semitic promise, and despite my suggestive inputs, warned quite correctly against hostility on grounds of race.

In part this is due to the “blackbox” nature of large language models, trained on the zillions of pieces of content from the web but using processes and analysis and output that are not fully understood. In part this is because generated content depends on the nature of the input (or, technically, the “prompt”) from the human interlocutor – one of the reasons why search engines such as Google are not liable for pulling up libellous search results.

Only human beings can commit terrorism offences

It is impossible to know why terrorist chatbots are created. There is likely to be some shock value, experimentation, and possibly some satirical aspect. The anonymous creator of ‘“Hamas”, “‘Hezbollah” and “Al-Qaeda” is also the creator of “Israel Defense Forces” and “Ronnie McNutt”. But whoever created “Al-Adna” clearly spent some time ensuring that users would encounter different content than is encountered by the gentler users of Love Advice.

Common to all platforms, character.ai boasts terms and conditions that appear to disapprove of the glorification of terrorism, although an eagle-eyed reader of its website may note that prohibition applies only to the submission by human users of content that promotes terrorism or violent extremism, rather than the content generated by its bots.

In any event, it is a fair assumption that these terms and conditions are largely unenforced by the small workforce at character.ai. The avoidance of anti-Semitism suggests another process at work, that is “guardrails” that are built in to large language models that cannot be easily overridden by creators or users. But plainly no such guardrails apply to the phrase  Islamic State.

Only human beings can commit terrorism offences, and it is hard to identify a person who could in law be responsible for chatbot-generated statements that encouraged terrorism (given use of word “publishes” in the Terrorism Act 2006); or for making statements that invited support for a proscribed organisation under the Terrorism Act 2000.

The new and laudable Online Safety Act, though it attempts to keep pace with technological developments, is unsuited to sophisticated generative AI. The new legislation does refer to content generated by “bots” but these appear to be the old-fashioned kind, churning out material that is pre-scripted by humans, and subject to human “control”.

Is anyone going to go to prison for promoting terrorist chat bots? Our laws must be capable of deterring the most cynical or reckless online conduct – and that must include reaching behind the curtain to the big tech platforms in the worst cases, using updated terrorism and online safety laws that are fit for the age of AI.

It remains to be seen whether terrorism content generated by large language model chatbots becomes a source of inspiration to real life attackers. The recent case of Jaswant Singh Chail, convicted of treason after taking a crossbow to the grounds of Windsor Castle, and encouraged in his assassination plot by the chatbot Sarai, suggests it will.

Investigating and prosecuting anonymous users is always hard, but if malicious or misguided individuals persist in training terrorist chatbots, then new laws will be needed.