Page 79 - March 2019 - v1_june_july_2009.qxd

Human Toll of Protecting the Internet 79

The Human Toll of

Protecting the Internet

from the Worst of

Humanity

By Adrian Chen

Henry Soto worked for Microsoft’s online-
safety team, in Seattle, for eight years. He
reviewed objectionable material on Microsoft’s
products—Bing, the cloud-storage service
OneDrive, and Xbox Live among them—and
decided whether to delete it or report it to the
police. Each day, Soto looked at thousands of
disturbing images and videos, which included
depictions of killings and child abuse.
Particularly traumatic was a video of a girl being
sexually abused and then murdered. The work Regardless of the merits of Soto’s specific case, by members of the alt-right, which found a
took a heavy toll. He developed symptoms of constant exposure to the worst of humanity on a fertile breeding ground on the site.
P.T.S.D., including insomnia, nightmares, daily basis takes an undeniable toll. One former Tech companies like to envision
anxiety, and auditory hallucinations. He began moderator for Facebook described it to me: themselves as neutral platforms, efficiently
to have trouble spending time around his son, “Think like that there is a sewer channel and all governed by code and logic. But users want
because it triggered traumatic memories. In of the mess/dirt/waste/shit of the world flow these companies to be flexible and responsive to
February, 2015, he went on medical leave. towards you and you have to clean it.” A former their needs. They want something more than
This story is laid out in a lawsuit filed moderator for YouTube told me that constant terms of service handed down from policy
against Microsoft, late last year, by Soto and a exposure to brutal combat and animal-abuse teams, or canned responses to a reported abuse,
colleague named Greg Blauert, and first videos sent him into a depression. Studies that which then disappears into a bureaucratic maze.
reported by Courthouse News Service. Soto and examine the impact that exposure to disturbing They want a human relationship with the
Blauert claim that the company did not prepare content has on moderators are rare, but there services that play such an important role in their
them for the stress of the job, nor did it offer have been a number of studies of law- lives. That will never be possible if those
adequate counselling and other measures to enforcement officers who investigate computer services dehumanize the workers that protect
mitigate the psychological harm. Microsoft crimes. A study, conducted by the U.S. Marshals them.
disputes Soto’s story, telling the Guardian in a Service, of six hundred employees of the Justice
statement that it “takes seriously its Department’s Internet Crimes Against Children Conspiracy Quotes
responsibility to remove and report imagery of task force, suggested that a quarter of the
child sexual exploitation and abuse being shared investigators surveyed displayed symptoms of
on its services, as well as the health and secondary traumatic-stress disorder, which is If a man says (for instance) that men
have a conspiracy against him, you cannot
resiliency of the employees who do this akin to P.T.S.D., but is caused by indirect
important work.” exposure to trauma. dispute it except by saying that all the men deny
that they are conspirators; which is exactly what
The lawsuit offers a rare look into a Tech companies don’t like to talk about
conspirators would do. His explanation covers
little-known field of digital work known as the details of content moderation, so it’s difficult
content moderation. Even technology that seems to judge how well they’re caring for the the facts as much as yours.
to exist only as data on a server rests on tedious psychological health of moderators. Silicon - G. K. Chesterton
and potentially dangerous human labor. Valley’s optimistic brand does not fit well with
The truth is, that it is not the Jewish
Although algorithms and artificial intelligence frank discussions of beheading videos and child-
banking conspiracy or the grey aliens or the 12
have helped streamline the process of molestation images. Social-media companies
foot reptiloids from another dimension that are
moderation, most technology companies that are also not eager to highlight the extent to
in control. The truth is more frightening, nobody
host user-generated content employ moderators which they set limits on our expression in the
is in control. The world is rudderless.
like Soto to screen video, text, and images, to digital age—think of the recurring censorship
see if they violate company guidelines. But the controversies involving deleted Facebook pages - Alan Moore
labor of content moderators is pretty much and Twitter accounts. In 2013, an internal
invisible, since it manifests not in flashy new Facebook moderation document was leaked, You simply cannot invent any
conspiracy theory so ridiculous and obviously
features or viral videos but in a lack of filth and revealing that Facebook moderators were
satirical that some people somewhere don't
abuse. Often, the moderators are workers in instructed to flag all attacks on Kemal Atatürk,
already believe it.
developing countries, like the Philippines or the founder of modern Turkey. Kurdish users
- Robert Anton Wilson
India, or low-paid contractors in the United critical of the Turkish government protested the
States. There’s no reliable figure for how many company’s actions. Leaving the moderation
All of those conspiracies are run by
people are employed in this line of work, but it’s process opaque offers a degree of flexibility and
certainly in the tens of thousands. Content plausible deniability when dealing with paranoid fantasists and ham-fisted clowns. If
moderators are recent college graduates and politically sensitive issues. you are on a list targeted by the CIA, you really
stay-at-home mothers, remote workers in Moderation has become an especially have nothing to worry about. If however, you
have a name similar to somebody on a list
Morocco and employees sitting in giant charged subject in the past year, as the tumult
targeted by the CIA, then you are dead.
outsourcing companies in Manila. The number accompanying the Presidential campaign played
- Alan Moore
of moderators will only increase, as more of our out on social media. Both Facebook and Twitter
lives are lived online, requiring more and more were involved in high-profile controversies over
Although Masonry is often denounced as
policing. Whenever I share on Twitter an article their moderation practices, or, rather, their lack
either a political or religious "conspiracy",
about content moderation, I’m always struck by thereof. Facebook caught flak from Barack
the number of people who tell me that they’ve Obama for its role in spreading fake news, while Freemasons are forbidden to discuss either
done the work and how psychologically difficult Twitter was criticized for a surge in hate speech politics or religion within the lodge.
they found it. and targeted harassment, much of it perpetrated - Robert Anton Wilson