Applicability of SIR Models to Disinformation Models

Susceptible-Infected-Recovered (SIR) models such as the one applied to create the synthetic environment (SEN) used in this study have a long history of application across many fields. These models began in epidemiology in the 1920s with work by William Ogilvy Kermack and Anderson Gray McKendrick (Kermack & McKendrick, 1927), but have also long been applied to study the transmission of ideas, narratives, and rumors (Goffman & Newill, 1964; Daley & Kendall, 1964). These models capture key aspects relevant to the spread of disinformation and misinformation in social networks, and provide a parsimonious way to characterize components of the strategic situation faced by those seeking to influence information spread.

SIR models include a population consisting of individuals or agents of at least three types: susceptible, infected, and recovered or resistant¹. Transition probabilities in the SIR model govern the movement of agents from one state to another. Solutions for SIR models have been examined numerically, through simulations, and in recent years for specific parameter values exact analytical solutions have been computed as well (Harko et al., 2014).

The typical results of a SIR model run involve initial infection spread as infected individuals initially encounter mostly susceptible individuals. Then, a peak level of infection intensity as recovery and less availability of susceptible individuals balances new infections. Lastly, there is typically a decline in the number of infected individuals as recovery / resistance combine with diminished numbers of susceptible individuals to end the epidemic, often before all susceptible individuals have become exposed. There are several SIR model variants with alternative assumptions. For example, in the SIS model recovered agents remain susceptible, while in the SIR model, recovered agents are no longer susceptible. In the SIRS model, resistance to infection fades over time. The SEIHFR model has six categories, adding Exposed (but not yet symptomatic), Hospitalized (and thus perhaps less infectious), and Funeral (dead, not buried, and hence potentially still infectious) categories and has been used to model Ebola epidemics (Drake et al., 2015). The model variant used in this project allows for a possibility that infected agents who recover will transition to either the susceptible (S) or resistant (R) categories. Section 3.2 describes how we modified the standard SIR model to fit with the disinformation context.

SIR and related models have long been recognized as an effective framework for studying the spread of misinformation and disinformation. Key early work in the 1960s by Goffman and Newill (Goffman & Newill, 1964>) and Daley and Kendall (Daley & Kendall, 1964) pioneered the application of SIR and related models to the spread of information and rumors. These authors noted that the spread of ideas or information, like the spread of an infection, involved transmission from one individual to another, and that the SIR framework could provide a fruitful approach for modeling this process. At the same time, the models also account for a range of potential modifications such as effects of encountering other infected and/or resistant individuals.

The SIR model has been applied widely to information and idea transmission in fields including politics, economics, marketing, health, and communication. For example, recent work by Nobel prize winning economics professor Robert J. Schiller (Schiller, 2019), applies SIR epidemic models to understand the role of narratives in shaping economic behavior across a wide range of domains from speculation in Bitcoin to economic cycles, stock market bubbles, and many more. Work by Zhao, Weng and co-authors has expanded study of the spread of competing ideas and the dynamics of when and how ideas go 'viral' in social networks (Weng et al., 2012; Weng et al., 2013; Zhao et al., 2013). Bauckhage and colleagues examined attention to social media services (Bauckhage et al., 2014) and viral videos (Bauckhage et al., 2015). Internet memes can also be effectively modeled using an SIR framework, and Beskow and co-authors extended this work to study the evolution of political memes (Beskow et al., 2020). Across domains, epidemic models have provided useful insights into idea, information, and disinformation transmission.

One important distinction between the models involves whether agents assort at random or exist in a network structure. Random assortment is simpler to model for obvious reasons, but network structures often are particularly important for modeling transmission of ideas in realistic settings because they allow for differences in influence between actors. The most relevant models for the analysis of disinformation involve models with network effects and these models are often best analyzed using agent-based models in which the network structure can be directly analyzed (Ji et al., 2017). Infection of widely followed and trusted sources or sites has the potential to super-spread disinformation.

The SIR Synthetic Environment (SEN): Configuration and Settings

Because of the potential for greater realism in a network model, we model disinformation spread in an agent-based network. The networked disinformation spread model used to create the SIR based synthetic environment (SEN) in the wargame was developed by modifying and adapting the "virus on a network" SIR model presented by Stonedahl and Wilensky (2008). The model was programed in NetLogo, an open-source platform for agent-based modeling (Wilensky, 1999). For the purposes of the synthetic environment, the software was used to mimic and visualize the spread of disinformation. As mentioned previously in Section 3.1, related SIR models have a long history of application across many fields and in spite of their highly abstract and reductionist style, the SIR model can effectively capture the way in which disinformation spreads through a network of people.

Agents exist in a spatially clustered networked structure as in Stonedahl and Wilensky (2008). The configuration of our model, illustrated in Table 1, is made possible by initial settings which include the total number-of-nodes (agents in the SEN), the average-node-degree, showing how many other agents each agent in the SEN is connected to, the initial-breakout-size, depicting the scale of the disinformation spread, and by a series of transition or transmission probabilities which we describe below.

Each node can be in one of three states - susceptible (S), infected (I), and resistant (R) (Stonedahl & Wilensky, 2008). Susceptible (S) are vulnerable to disinformation due to low levels of awareness of the issue, lack of rational/critical thinking abilities, and/or other similar limitations. Infected (I) agents have been deceived by disinformation and perceive narratives spread by malicious actors as credible and trustworthy. Infected nodes tend to spread the information they have received and believed, thus becoming unwitting participants in the spread of disinformation or misinformation. Infected nodes are not always aware that they have been 'infected' at least until they 'fact-check'. Even those who do fact-check may still remain 'infected'. Therefore, not all infected nodes 'recover' from the condition of being infected. Resistant (R) agents are no longer vulnerable to disinformation due to fact-checking habits, high levels of awareness and rational/critical thinking abilities, and other cognitive and situational factors. The use of the term resistant which we adopt from Stonedahl and Wilensky (2008) is somewhat at variance with the use of the term recovered in some SIR models, but it is appropriate in our context as we distinguish between recovered agents.

Several parameters govern the transition of agents from one state to another. Infected agents spread disinformation to connected uninfected agents with a specified probability β. Infected agents also engage in fact checking with a specified frequency τ. When fact checking occurs, agents potentially recover (with a specified probability γ) with some failing to develop ongoing resistance to future infection by disinformation (returning to susceptible) and some developing resistance to future infection (with probability ρ.) Unlike most SIR models of disease, in the disinformation model, we also allow for the possibility that resistant agents connected with others infected with disinformation will push back, triggering additional fact checking. With a specified probability (ψ) a resistant agent may trigger fact checking among infected network connections and thereby potentially induce recovery to a susceptible state or the development of resistance.

In every step of the simulation (represented by a tick), each infected agent, marked by a red node, attempts to infect all of its connections with the disinformation. As a consequence, susceptible connections, marked with green nodes, may or may not get infected. The probability of infection is determined by β the disinformation-spread-chance setting. This characteristic represents the real-world equivalent of falling prey to a misleading headline, or to propaganda designed to elicit an emotional response favoring the actor spreading the false information. People that are resistant, marked with gray nodes, do not get infected. This represents the real-world equivalent of highly-aware people who have fact-checked and/or critically analyzed the disinformation and are no longer susceptible to it.

As opposed to this, infected people, marked with red nodes, are not always aware that they have been 'infected' by false information. In this model, every person has the potential to conduct a fact-check with a probability, which is controlled by τ, the fact-check-frequency setting. This represents the real-world event of a learning process in which an individual is being told by a person or an outlet they trust, in verbal or written form, that a particular piece of information is false.

If an agent successfully discovers through a fact check that they have indeed been 'infected', there is a chance that they might 'recover', i.e., get reliable and credible information. The probability of such a recovery is controlled by γ, the recovery-chance setting in the model. At the same time, a person's 'recovery' does not mean they will never get infected again. An appropriate analogy would be that one single human can get scammed or fall victim of phishing attacks many times. Therefore, some nodes may get infected again (modeled by a return to the susceptible group), some may not.

The probability of gaining this 'resistance' or 'immunity' is controlled by ρ, the gain-resistance-chance setting. When a person becomes resistant, the links between them and their connections are darkened, since they are no longer possible vectors for spreading misinformation. Figure 1 shows a screenshot of the simulation in its final stage.

As a result of feedback concerning the match between epidemiological models and the disinformation context in the SEN, we also modified the SIR model to allow for the potential that resistant individuals might actively resist the spread of disinformation triggering fact checks by connected infected agents with probability ψ.

Figure 2 illustrates the impact of differences in the model parameters in the simulation. The key point is that the outcome of a model run is highly contingent upon the parameters. With the same starting values except for the frequency of fact checking (τ), the panel on the left follows a trajectory in which a severe infection develops (fact checking occurs only every 10 ticks). The panel on the right follows a trajectory in which a more rapid development of resistance more rapidly ends the spread of disinformation and prevents it from ever simultaneously attracting a majority of the population (fact checking occurs every tick).

Figure 2. Example model runs with different fact check frequencies.

All simulation parameters could potentially be influenced by the teams playing the DTEX wargame through their strategic choices, as will be discussed in Section 4. This modification of parameters was one of the two ways the wargame-based test of the implementation of the anti-disinformation-spread technologies was evaluated. One half of the choice of the winning team was based upon which team's SEN inputs led to the most rapid elimination of the disinformation in the model (the lowest number of ticks at the end of the simulation). Teams were also judged on their argument concerning the choice of technologies and the strategy for deploying them.

DTEX Process

The DTEX Process used in this simulation was adapted from NATO's Disruptive Technology Assessment Game (DTAG) structure. The latter "is a table-top seminar wargame, used to assess potential future technologies and their impact on military operations and operating environment" (NATO ACT, 2010). Similarly to DTAG, DTEX also adopts the seminar wargame core, but reveals some more nuances in the way the simulation was conducted - in a fully online, synchronous environment.

The DTEX Process, illustrated in Figure 3, incorporated five steps, as follows. First, the participants studied the scenario and the issues described in it. The exact text of the scenarios can be found in Appendix 2. They were also given some supplementary materials and had the opportunity to receive guidance about the scenario and the solutions from a facilitator. Second, the participants reviewed the IoS cards (see Appendix 3) with proposed solutions. Third, each participant individually made a choice of three IoS cards which they found suitable to resolve the issues at hand. Fourth, participants discussed their choices with their teams and debated the rationales behind their choices. Fifth, each of the two teams deliberated on a final selection of IoS cards, based on the merits of the suggested solutions, their combined, synergetic effects, and the impact of the entire set of cards, as tested in SEN. After this process was completed, the participants prepared one-slide presentations with their choices, defended their strategy, and the winner was announced by a subject matter expert, who served as a judge.

Scenarios

The scenarios with which the participants in the simulation were presented focused on social media disinformation. They presupposed that the Supreme Allied Commander Transformation (SACT) formed a small task force that will assist an Allied Command Operations (ACO) team in the ongoing fight against disinformation and the participants were a part of it. Next, they were asked to select three IoS cards (described in Section 4.3) which addressed the various specific issues underscored in both scenarios. The teams qualitatively evaluated the merits of each IoS card (and the combined impact of the chosen cards) and after they made their final choice of IoS cards, the quantitative effects of their choice of IoS cards, based on the expert ratings, was also tested in the SEN provided to them and their facilitator. Teams did not have direct access to the expert ratings of the cards. The faster the SEN eliminates the spread of dis/misinformation (fewer ticks to elimination), the better. The winning team was chosen based on both their rationale for their IoS card choices and on the temporal impact of their choices within the SEN. Equal weight was given to these two criteria to make sure that the solution is supported by qualitative and quantitative factors.

DTEX IoS Cards

As mentioned in Section 4.1 and Section 4.2, scenario play involved a choice of IoS cards by participants. As for the structure of the IoS cards, as shown in Figure 4, each card consists of various sections describing the technology intended to serve as a solution to the problem of disinformation on social media. In the first one, called offerings, the objectives of the technology are outlined, and then the technology itself is introduced through a brief overview. Next, the second section of the cards summarizes the input, the output, the process the technology is using to achieve its goals, and the supported technologies in which it will operate. The third and last section of the cards highlights advantages and limitations of the technology. The purpose of this section is to guide participants in their choices, as they could not obtain information about the proposed technologies directly from the contributors in the NATO Innovation Challenge through which these ideas were gathered. Description of the features of all IoS cards is available in the Appendix 3.

In addition to the content summary of each card, the subject matter experts invited to contribute to this simulation assigned each IoS card a specific impact. The latter was expressed in numerical value calculated as the average of the expert ratings and contributed to visualizing the solutions in SEN. Figure 5 shows the worksheet with all of the IoS cards' SEN inputs that was compiled and used by the facilitators to coordinate the team's activities and to process the inputs in SEN for the participants during the simulation.

Each of the categories of impact on the SEN (A through E) shapes elements of the simulation environment (e.g., fact check frequency τ, probability of disinformation spread β, etc.). Participants did not directly receive information about the ratings on the cards they received, but the ratings informed the way in which the simulated SIR model in the scenarios was modified as a result of group choices. The rated impacts of the cards are discussed in Section 5.2.

The role of the participants

As mentioned in Section 4.2, the participants in the experiment were asked to select three IoS cards and explain why they are the best choices to address the issues highlighted in the scenario. The participants also had to identify the priorities to which they adhered when choosing the cards. These priorities included five different objectives - identification of malicious communication material online, categorization of information (real vs. fake), attribution (finding sources of fake information), additional analyses (processing and analysis of collected information to fulfill other objectives), visualization of analyses, and mitigation of effects (countering disinformation and their effects by shielding the audience being targeted, disseminating counternarratives, etc.) After completing the selection of IoS cards, the participants were invited to test their choices in the SEN, where both the individual effects of their choices and their combined synergetic effects were visualized and assessed. Lastly, during a confrontation session between the different teams, the participants presented their proposed plan to the jury, which consisted of subject matter experts on the topic of disinformation.

Group Dynamics Qualitative Observations

In the first scenario, Group 2 seemed less organized than Group 1. Group 1 used screen share capabilities more effectively to help ground discussion of alternative cards, while Group 2 seemed to struggle a bit more to reach consensus, and as a result did not develop as effective and clear a set of plans for how to address the challenges in the scenario, nor how to present their plans.

In the second scenario, one of the members of Group 2 opened the discussion with a proposal that helped set the tone for a more productive deliberative process which set the stage for the Group 2 win in scenario 2. With her leadership they identified goals and reached consensus about them. Then they developed a combination of technology cards that would allow them to effectively achieve those goals. The structure of the deliberations could have potentially benefitted from more involvement by the moderators and a division of the cards into different categories (e.g., dashboards versus tools for intervention). By the second scenario, Group 2 seemed to have begun to do this kind of sorting of cards into categories on its own, and that process helped the group reach a more effective path to a solution, while Group 1 in the second scenario seemed to have more trouble structuring their deliberations and combining the synergies of the cards. Group 2 reached near-consensus with sufficient time remaining for multiple model runs in the SEN to test which of two alternative strategies would lead to better results. Ultimately, choice of the strategy rejected by Group 2 through this process would have led to less successful model runs than Group 1, and potentially to a loss in scenario 2, so the time the group was able to invest in this aspect of the deliberation seems to have been well spent.

The group dynamics described highlight some of the skills and approaches which determined the winning group. In particular, leadership, level of organization and structure of the decision-making process, along with an effective use of the technical capabilities of the SEN to which the participants had access contributed to Group 1's better performance in the first scenario, and Group 2's in the second scenario. These conclusions about the group dynamics in DTEX provide important insights for the successful selection process of technological solutions with a high level of impact against disinformation. They may be used in future iterations of this simulation to increase the productivity and competitiveness of both teams, thus ensuring a better learning experience for the participants and a more careful re-assessment of the IoS cards, previously ranked by experts, based on their characteristics.

IoS Cards: Strengths and Synergies

As noted at the outset, the purpose of the SEN (SIR model) and wargame virtual simulation in this case was to evaluate proposed anti-disinformation technological tools submitted to NATO through an innovation challenge. This section discusses the results of that evaluation which is based upon the totality of the information collected including the actions and arguments made by wargame participants, expert rankings, and simulation results.

Prior to the DTEX wargame the IoS cards were ranked by experts for their ability to impact five different characteristics of disinformation spread in the SEN, and then evaluated by the competing teams to construct compelling and synergistic combinations of the cards. The characteristics were: A - Reduces Initial Outbreak Size, β - Reduces Disinformation Spread Chance, τ - Increases Fact Check Frequency, γ - Increases Recovery Chance, and ρ - Increases Gain Resistance Chance. The probability that a resistant agent will trigger a fact check by a connected infected agent (Ψ) was added after DTEX based on the simulation experience and so is not included in this section. Based upon the expert rankings and the results of the wargame, including qualitative analysis of participant discussion and arguments we have categorized each card in Table 2 in terms of the best card(s) for addressing each aspect.

Containing initial outbreak size is potentially very important, especially if once the outbreak is identified, effective tools are available to curtail the spread of the outbreak. Card #33 was rated as providing the best impact on initial outbreak size. This technology provides a dashboard for decision-makers that "monitors all aspects of the spread of information (about COVID-19) and predicts what and how other topics will spread." The key aspect of this platform for curtailing initial outbreak size is that ideally this platform will allow rapid identification of outbreaks of disinformation, allowing agile targeting responses to those outbreaks using various other tools before the outbreaks have time to become widespread.

Once an outbreak of disinformation has begun, a critical factor shaping its spread is the extent to which individuals or media infected with disinformation spread it to others. The three best-rated cards for curtailing the disinformation spread chance were implemented in different strategies, suggesting potential for fruitful combination between these cards for larger impact. IoS card #5 SGOOF uses data-mining, classification, and machine learning classification to develop a 'truth score' and classification for information. This could be fed into a dashboard similarly to #33, but it also could potentially be used in public-facing applications. IoS card #20 DeepDetector is a more specialized software application aimed at detecting and identifying deep-fakes in video footage. The current prototype is asserted to have a 95-98% accuracy and could provide an important tool both if fed into a dashboard and as a public-facing application to allow for rapid identification of likely faked video content in order to catalyze actions to limit its spread. Another IoS card - #35 Profiling fake news spreaders on Social Media takes a somewhat different tactic. Potentialize synergizing with #5 and #20, this machine learning application focuses on the profiles of fake news spreaders instead of on the news content itself. This could provide particularly valuable information in order to facilitate rapid response to the spread of fake news that targets accounts being used to spread disinformation.

Once disinformation has begun to spread widely, combatting it involves in part triggering fact checking that potentially leads individuals to believe they should not trust the disinformation. The best rated card for increasing fact check frequency was #29 Intelligence Dashboard. This dashboard proposal utilizes a combination of AI and human fact checking to identify and classify the most prevalent information. As with other dashboard proposals, the primary focus here is on enabling decisionmakers to take effective actions to increase fact check frequency or provide targeted individuals with fact checks of disinformation which they have been exposed to. Individuals who have come to believe disinformation may eventually recover by believing fact checks which disabuse them of belief in the false narratives provided by the disinformation source. The best rated card for increasing recovery chance was #39 PULSE. This proposal emphasizes the important counter-insurgency principle that all combatants are intelligence gatherers. It provides a framework for submissions from "front-line workers" to identify and cluster information on unaddressed issues and challenges. This could be an important component of any dashboard, helping decision-makers operate with better information concerning the current state of play in the spread of disinformation, and potentially facilitating the identification of unaddressed issues.

A key factor in ultimately containing a disinformation outbreak is the development of resistance to it in the form of individuals who are no longer susceptible to the disinformation. Three technology cards received the highest ratings for this element: #7, #9, and #22, and pursue two quite distinct strategies that would need to be synergized for the largest impact. IoS card #7 aims to achieve resistance through counter-spreading measures, a unique and very important aspect of this card compared to most of the other proposed technologies. In essence, the strategy behind using it is to achieve resistance to disinformation by identifying potential spreaders, and swamping the disinformation signal with alternative signals. This more active resistance by jamming disinformation signals moves beyond most other cards which emphasize identification of disinformation rather than active counter-information measures. Card #9 Zetane is a dashboard that aids in visualization of the geographic and regional trends in false information spread. #22 Nunki is another dashboard application which focuses on alerts concerning events and news spread, hopefully facilitating rapid response. Obviously, the dashboard applications would be most fruitfully combined with other measures, such as IoS card #7, since with dashboard strategies the resistance developed would involve societal level rapid-response to renewed spread of disinformation.

Fortunately, as discussed above, multiple technologies can be combined to address the challenges of disinformation. However, if only a single technology was to be used, the best overall technology in terms of impact relative to the others across the five categories is #7 Combat Information Through Social Media. What makes this strategy stand out is its emphasis on active measures. The high ratings given this card suggest that efforts to develop a suite of different active signal-jamming measures to combat disinformation would be well worth while. Combination of such measures with good dashboard and intelligence to identify threats would probably help to magnify the effectiveness of this technology.