Radicalisation to violence is a deeply personal process. It’s about an individual making a set of choices for their own reasons within a broader political context that leads them to turn against a society into which they were born. This makes it very difficult to counter and even harder to remove once it has been embraced. Few effective solutions exist, and they are even harder to implement inside a prison.

Last week’s prosecution of the extremist preacher Anjem Choudary – along with a number of his acolytes from the now-banned al-Muhajiroun organisation – means the prison system will again be absorbing a new batch of radicals into a population of alienated and sometimes violent young men who are vulnerable to their message. Managing them will be a complicated process, so the Ministry of Justice has announced a new approach: the “most dangerous” extremist prisoners will be isolated from the general population in special high-security units. But will it work?

We are dealing with a very small number of people. Most of the Islamist terror plots hatched in the UK over the past 20 years – and even some of those unfolding in Europe – can be linked in some way with al-Muhajiroun and its graduates. Authorities have not been ignorant, and a persistent policing and intelligence effort has disrupted their activities, including an attentive effort that sweeps them periodically off the streets when they overstep the line of the law for whatever reason.

Yet this is not a permanent solution. In many cases these individuals serve a limited time before returning to their earlier activity. One Choudary associate, Trevor Brooks, was recently caught on a train to Turkey in breach of his bail conditions despite repeated spells in prison. In short, they are persistent long-term radicals – likely lost causes.

That is not always true. There are cases where people move on from extremism. Although the paths out are as personal and variable as the paths that lead into it, this process can be accelerated or shaped by intensive and engaged mentors who can take a leadership role in the individual’s life and steer them away from their former ideology. That requires two things: isolating them from their old groups and leaders, and offering them a real alternative life they can embrace.

But what do you do with persistent long-term offenders who show no evidence of rejecting their creed and may use prison as an opportunity to further spread it? Ideally you should isolate them from the broader prison community, yet solitary confinement – especially over a sentence of 30 or 40 years – is prohibitively expensive and legally problematic. At the same time, they cannot simply be confined together, free to plot their next moves upon release; the authorities learnt that lesson in Northern Ireland, where paramilitary prisoners packed together in the infamous HMP Maze ended up in effective control of their cell blocks and became a political force.

Until now the government response to this dilemma has been to keep extremist prisoners in confinement or in the general population, moving them regularly so they cannot form strong links. This has its own problems, not least that there aren’t enough prisons in Britain to keep its 100-plus jihadists from meeting each other inside.

In that sense, the new approach is the least bad option. This is not the Maze: each unit will be relatively small and subject to as yet unspecified anti-plotting interventions. It may be that this small but dangerous group of people will always be with us, and that the best we can do without violating our societal principles is to manage them and stop them recruiting – to lock them up when we can, to control their movements and activity once they are out, and to disrupt their ability to spread their ideology in public.

There is a price. Although it is rare, committed long-term extremists do sometimes unexpectedly turn away from their beliefs. As always, this is more likely if they are isolated from comrades and able to socialise with non-extremists, and less likely if not. We will never know how many people we have written off as incorrigible might otherwise have followed this path. It is a balance with no perfect answer – but one which society will probably have to accept.

Raffaello Pantucci is director of international security studies at the Royal United Services Institute