Conflict, like most of our social lives, has increasingly moved online and mediators and peacebuilders have reacted to this new world by partnering with platforms to remove harmful content and actors from their ecosystems. This is important work, but it can feel endless as each day brings a new set of harms to address. As Maria Ressa noted in an address to UNESCO in 2023, content moderation can feel like cleaning a glass of water from a dirty river and then dumping the clean water back into the river, rather than dealing with the factory polluting the water upstream. Having worked at a platform on a variety of crisis response efforts, I can tell you that you can learn a lot about how to fix the factory by examining the pollution it creates. But it requires asking that broader question, about what is creating the pollution, rather than focusing solely on cleaning up as much as you can.
Peacebuilders have an important role to play in helping platforms not only address harmful content and actors, but also in making their platforms more robust against those actors in the first place. Emillie de Keulenaar's recent analysis of YouTube videos on the conflict between Armenia and Azerbaijan found that searches about the conflict often led to videos containing inflammatory content. While one could attempt to find policy violations within those videos, conflict actors often can divide people without the need to violate policies.
Rather than attempting to penalise these particular videos and actors, it is more impactful to think of the prominent placement of these videos as a ‘bug’, which YouTube recommendation engineers should fix holistically, rather than dealing with individual pieces of content. In this way, you can prevent the next actors from gaming the algorithm, rather than simply removing today’s examples of harmful content.
Content moderation will never address all the drivers of violence. In one example I worked on in Ethiopia, social media posts suggesting that the arrest of an opposition figure (Jawar Mohammed) was imminent were reported to trigger violence, even as such speech would be considered important to protect in many contexts. Inciting fear of oppression, not hate, is a tactic that is often used on social media across conflicts by divisive actors seeking to consolidate power by perpetuating conflict. Unfortunately, there is no reasonable and legitimate way to disentangle unjust from just expressions of fear.
Such tactics work because human beings pay attention to fear for evolutionary reasons. Algorithms that measure attention, via clicks, shares, and time spent, learn that fear grabs attention and then incentivise this content. The open nature of these platforms means that conflict actors can manipulate algorithms by manufacturing engagement and dominating an information space. The result is that divisive conflict actors find platforms to be useful for perpetuating conflict and even neutral actors feel pulled toward divisive content in order to draw attention to their messaging and compete for attention. Fortunately, the physics of such online phenomena have become known, such that platforms have reformed some of those incentives, by reducing optimisations toward comments and shares for political content and by adding some limits to virality and reach in sensitive contexts. However, these changes certainly do not address all issues and may not be particularly robust, especially in international contexts. Some changes rely on the technical ability to identify what is or is not political content, for example, which is a classification task that will necessarily be limited by language and cultural understanding. Failure is inevitable and platforms need local expertise to help fix emerging issues.
Divisive actors will always be able to spread their messaging to those who seek out that content. That is a reality of online life that is inescapable. However, such content does not have to be pushed by platforms (e.g. among the most popular videos on YouTube) with the resulting warping of incentives for more neutral conflict actors. When such content is being pushed, having concrete cases to anchor on can help engineers treat these cases like ‘bugs’ that can be solved both by removing content and most importantly, by understanding why that content was being incentivised in the first place. Perhaps a political content classifier needs to be retrained with better culturally relevant examples. Perhaps a signal like ‘time spent’ is being used inappropriately within an algorithm, leading to the spread of sensational and dangerous claims. Perhaps a small group of users is willfully spreading content in ways that should be rate limited (i.e. capping how often someone can repeat an action within a certain timeframe). By pointing out specific examples of harmful content and understanding why that content is spreading, one can more proactively make a system more robust against future campaigns of divisiveness.
Mediators and peacebuilders should continue to do crisis response with platforms as trusted partners. But they should also be active partners in addressing the systems that perpetuate harmful content, not just reacting to harmful content and removing individual cases. In doing so, they can help platforms address emerging risk and can help develop new product levers that may help not only their efforts, but the efforts of their allies in peacebuilding across the world.
