Harm:

Violent Extremism

Definition: Content that supports motivated acts of violence as means to achieve political, religious, or social objectives.
Motivation:
Ideological
Legal Status:
Can be illegal
Platform ToS:
Violates Policy
Victim Visibility:
Unaware
Classification:
Inherent in Content
TSPA Abuse Type:
Violent and Criminal Behavior: Dangerous and Criminal Organizations

Violent extremism (VE) poses a significant challenge for online platforms. These platforms inadvertently serve as conduits for extremists from various ideological backgrounds to connect, share propaganda, and coordinate acts of violence. Due to algorithms that prioritize engagement, borderline or provocative content often gets amplified, leading users down rabbit holes that can end in extreme beliefs - many of which include acceptance of violence for political or ideological ends.

Platform design choices complicate the battle against VE. For instance, features like end-to-end encryption, while intended to ensure privacy, can shield extremists from scrutiny and make it difficult for trust and safety professionals to monitor and intervene in harmful activities. On the other end of the visibility spectrum, platforms giving users the power to livestream has been used by violent extremists to spread their message, and the terror it intends to produce (most notably in Christchurch NZ, in the aftermath of which the organization Christchurch Call, which focuses squarely on issues of VE on online platforms, was founded).

While violent extremism predates the internet, the digital landscape has evolved to inadvertently facilitate it. It is increasingly critical for platforms to look beyond simple content moderation, and ensure that their design decisions do not actively support or amplify the causes of violent extremists.

What features facilitate Violent Extremism?

Recommendation
A platform proactively inserting content into a user's view.
Messaging
Enable users to exchange text in real time.

How can platform design prevent Violent Extremism?

Since violent-extremism often lies at the end of a rabbit hole, resetting preferences can bring people back toward median.
Periodic Preference Reset
Regularly reset the profile used to generate recommendations for a user.
No E2E Encryption
When content is end-to-end encrypted, platforms can offer no content-based protections.
Since violent extremism is typically illegal, recording traces to help identify it's origin may reduce its prevalence.
Save Extensive Metadata
Recording extensive metadata about sensitive activity can reduce criminal appeal.
Anonymous Limitations
Require users to create an account before they can use features that create data or interact with others.
Is something missing, or could it be better?
Loading...