Harm:

Phishing

Definition: Using impersonation to trick a victim into revealing sensitive information.
Motivation:
Financial
Legal Status:
Almost always illegal
Platform ToS:
Violates Policy
Victim Visibility:
Aware
Classification:
Inherent in Content
TSPA Abuse Type:
Deceptive & Fraudulent Behavior: Cybersecurity

Phishing is a form of impersonation used by cybercriminals to try to trick victims into providing them with credit card numbers or passwords.

Unlike other forms of explicitly malicious online behavior, phishing doesn't rely on bad platform design or technical security flaws to gain access; rather it exploits flaws in the nature of human psychology in order to emotionally manipulate users into acting.

Structure of a Phishing Attack

Typically, phishing attacks are comprised of three components, which might be contained in a single message or spread across multiple messages:

  • Establishing credibility  - this could be through the assertion of a fraudulent or inaccurate identity, or in the case of "spear phishing", the assertion that the sender is a known connection of the victim, often including some publicly accessible details about the relationship between the assumed identity and the victim.
  • Establishing urgency - phishing only works when the victim is not thinking properly, so phishing attacks typically try to induce a state of urgency into the conversation, which has the dual impact of reducing a victim's capacity to think clearly, and reduces their impulse to verify the identity of the sender. Examples include (a) A victim's "boss" needs gift cards for a visiting client, and soon, before the client leaves! (b) A customer support representative is reaching out to see if you want a new laptop for free, but you have to act before they're all gone! (c) your friend needs to make bail, and you were the one friend that they called!
  • Requesting Access or Resources - In the final part of a phishing scam, the attacker asks for resources, like a password, a gift card, a credit card number, a one time verification code, or for you to install software. In all cases, this is the kind of request that you typically wouldn't do if you were asked on the street by a stranger, but by being primed with the assumed identity, and the false sense of urgency, victims frequently aquiesce.

Detection + Mitigation

Because the elements of a phishing scheme (identity, urgency, request for resources) are hard to distinguish (by their content alone) from legitimate communication, user awareness and vigilance play an essential role in stymying phishing attacks. 

It will get worse before it gets better

It's worth noting that the rise of Large Language Models, often colloquially called AI, is going to make phishing attacks dramatically more prevalent and effective. Within the next ten years it will become commonplace to receive a phone call from a computer impersonating a loved one begging for bail money. Better structural interventions are necessary to head off the coming acceleration of this scourge.

What features facilitate Phishing?

Messaging
Enable users to exchange text in real time.

How can platform design prevent Phishing?

Identity Verification
Require users to register for an application with a state issued identity document.
Warn Before Risky Action
Use signals about affinity and content to occasionally warn the user about what they're about to see/download/visit.
No E2E Encryption
When content is end-to-end encrypted, platforms can offer no content-based protections.
Must Request to Message
Only allow friction-less initiation of a conversation between established connections.
Is something missing, or could it be better?
Loading...