Incentives And Information Security

Motivation: incentives break security

IntuitionThe central question

When a system is protected by people who don’t bear the full cost of its failure, and when crucial facts — software quality, a peer’s effort, a firm’s exposure — are hidden, what level of security emerges, and what changes to liability, markets, or architecture would fix it?

The thesis. Security fails at least as often from misaligned incentives as from design mistakes. It is especially likely to fail when the guardian is not the victim. Game theory and microeconomics matter to the security engineer as much as cryptography does.

Theme	Economic concept	Security instance
misaligned incentives	externalities, public goods, moral hazard	infected PCs attacking third parties; banks not liable for fraud
hidden action	moral hazard	routers dropping packets; p2p free-riding
hidden information	adverse selection	unmeasurable software security
correlated risk	insurance market failure	worms, Y2K
architecture	solidarity vs diversity	discretionary vs random p2p
topology	evolutionary arms race	cells vs rings

Misaligned incentives and externalities

ExampleUK vs US bank liability — moral hazard

US: banks are liable for card fraud. On a disputed transaction, the bank must prove the customer is cheating or refund her.
UK: banks could often claim the ATM system was “secure,” so a complaining customer must be mistaken or lying.

“Lucky bankers”? No. UK banks spent more on security and suffered more fraud. Staff knew complaints wouldn’t be taken seriously, so they got careless, and fraud followed.

ExampleAntivirus circa 2000 — a negative externality

Typical virus payloads were DDoS attacks on sites like Amazon or Microsoft. A rational user might pay $20 to protect his own hard disk, but not to prevent an attack on someone else. So protection was underbought.

ExampleMedical privacy

Hospital directors and insurers buy medical information systems. Their interests (account management, cost control, research) are misaligned with patients’ interest in privacy. The result was a decade of HIPAA regulatory wrangling.

ImportantLiability should sit with whoever can best manage the risk

Legal theory has long said this. Online, risk is systematically misallocated, and privacy failures and regulatory battles follow.

RemarkPollution analogy

Connecting an insecure machine is like polluting air or congesting a road: the connector doesn’t bear the full cost, so investment in protection falls below the social optimum. This is the same externality structure as Pigou’s example in routing-games.

Security as a public good: how effort aggregates

Security often depends on many principals’ efforts, and uncoordinated choices underinvest. The outcome depends on how individual efforts combine:

Aggregation (Varian)	System reliability depends on	Who matters	More agents ⇒
total effort (sum)	$\sum_i e_i$	the agent with the highest benefit–cost ratio; everyone else free-rides	more reliable
weakest link (min)	$\min_i e_i$	the agent with the lowest benefit–cost ratio; extra effort by others is wasted	less reliable
best shot (max)	$\max_i e_i$	the single best effort (e.g., a star cryptanalyst)	—

ExampleProgramming vs testing

Code quality is weakest link: the most careless programmer introduces the fatal bug. Testing and validation are total effort: everyone’s bugs found add up.

Implication: hire more testers but fewer, more competent programmers.

NoteModel details not given

The chapter summarizes Varian’s results without the formal model.

DefinitionInterdependent security (Kunreuther–Heal)

A group’s security rests on each member’s. One member’s protection creates a positive externality for others, which in turn discourages their own investment.

ExampleBeyond computers

An apartment owner’s sprinklers protect the neighbours, who may then skip theirs.
Airlines skip screening bags transferred from carriers believed careful.
Parents free-ride on herd immunity instead of vaccinating.

In each case, several Nash equilibria exist, from complete adoption to total refusal, depending on coordination.

Network effects and bootstrapping

DefinitionBootstrapping problem

A security technology’s benefit to each adopter grows with the number of adopters. Until a minimum number adopt, cost exceeds benefit. So everyone waits for others to go first, and the technology is never deployed.

WarningThe economics literature doesn’t quite fit

Following Katz–Shapiro, the literature treats adoption as a coordination problem among players who commit long before a slow rollout. But security software can often be deployed fast, and it may give no special benefit to early adopters. (Compare the critical-mass dynamics in cascading-behavior-in-networks.)

ExampleHow SSH and IPsec escaped

Insecure core protocols (DNS, routing) have secure replacements that aren’t adopted. SSH and IPsec were adopted because they gave intra- organizational benefits (X session support, VPNs), independent of global adoption. Fax machines spread the same way: firms first bought them to link their own offices.

ImportantEscape via clubs

A technology that pays off within a club doesn’t need a global critical mass. Restricting the game to club members recurs below as a fix for hidden action.

Informational asymmetries

Hidden action

DefinitionHidden-action attack

A hidden-action problem arises when one party to a transaction can take unobservable actions that affect it (classically, reckless behaviour by the insured). In security, hidden-action attacks are attractive precisely because observation, and hence punishment, is unlikely.

ExampleNetworks invite hidden action

Routers quietly drop selected packets or falsify routing responses.
Nodes redirect traffic to eavesdrop.
File-sharers hide whether they share, and free-ride (p2p-incentives).

FormulaThe observability window

Let $G$ be the net gain from deviating, and $E_h \le E_o$ the expected penalties when observation is unlikely and likely. Then:

Condition	Outcome
$G < E_h$	no attack, even when hidden
$E_h < G < E_o$	hidden-action attack: it pays only because it’s hard to see
$G > E_o$	attack regardless of observability

IntuitionWhat observability buys

Raising the probability of detection moves $E_h$ toward $E_o$ and closes the window. Everything in the middle band is deterred by visibility alone.

Two remedies.

Contracts (mechanism design). Auto insurers’ deductibles deter recklessness without observing it, at a cost: everyone pays, careful or not.
Change the topology to raise observability.

ExampleP2P flat topologies breed hidden action

Large memberships, a flat topology (anyone can deal with anyone), high turnover and cheap identities all mean few repeated interactions and hard-to-apply penalties. Nodes are predisposed to hidden action.

AlgorithmClub topology

Group nodes into clubs by common interest.
Transact first within your club to establish legitimacy.
Once trust exists, trade across clubs through established inter-group channels.

This creates a credible threat of observation, and long-lived principals (the clubs) against whom sanctions actually hurt.

RemarkSocial-network topologies

Letting honest players choose friends as neighbours reduces informational asymmetry, raises an attacker’s entry cost, and aligns incentives. The downside: less exposure to diverse information, and isolated players get marginalized.

Hidden information: software as a market for lemons

Why don’t vendors ship secure software? Consumers reward features, time-to-market and dominance. Security needs slow testing and simplicity. “Ship it Tuesday and get it right by version 3” is rational. So why doesn’t reputational damage from worms and viruses correct this?

ExampleAkerlof’s market for lemons

A town has 50 good used cars (worth $2,000) and 50 lemons (worth $1,000). Sellers know which is which; buyers don’t.

Naive price: $1,500.
At $1,500, no one sells a good car, so only lemons remain, and the price falls to about $1,000.

Buyers won’t pay a premium for quality they can’t measure, so quality exits the market.

ImportantSoftware is a lemons market

Buyers can’t verify security, and vendors may not know it accurately either. Buyers won’t pay for it, so vendors don’t invest in it. Measurement is the bottleneck.

Three approaches to measuring software security:

Approach	Idea	Status here
statistical	reliability-growth models for vulnerabilities	not discussed
market-based	let prices reveal the cost of finding vulnerabilities	below
insurance-based	premiums encode assessed risk	below

Market-based measurement

DefinitionVulnerability market

An open market, or an auction, for reports of undiscovered vulnerabilities. Its bid, ask and last-sale prices approximate the labour cost of finding a vulnerability, which yields a security metric for comparing products.

Precedent: banking standards for PIN-entry terminals specify a minimum cost of technical compromise.

WarningExisting buyers aren’t socially optimal

Several firms buy vulnerabilities but don’t publish prices. They sell the information to customers and the vendor simultaneously, and they always have an incentive to leak it without safeguards.

DefinitionSoftware security derivatives

Issue contracts in pairs, each paying a fixed value $V$ :

contract 1 pays if no vulnerability is found in program $X$ by date $D$ ;
contract 2 pays if one is found.

The trading price reveals the market’s consensus probability of a vulnerability. Vendors, users, investors and insurers could use such contracts to hedge.

Cyber-insurance and correlated risk

The case for insurance. Underwriters price premiums by a firm’s infrastructure and processes. That produces best-practice guidance and, over time, data putting a money value on practices and products. Yet the market is underdeveloped. Why?

DefinitionTwo kinds of interdependence

Physical: firms’ infrastructure is connected, so another firm’s failure can undermine your defenses.
Logical: attacks exploit a platform many firms share, so one worm hits them all.

CautionGlobally correlated risks are uninsurable

Insurance works by pooling independent risks. Worms, viruses and systemic events (Y2K) are globally correlated: many claims arrive at once. That makes them unattractive to insurers.

If liability shifted to vendors, as many urge, a firm like Microsoft probably couldn’t buy insurance. Currently vendors dump almost all risk, which is also far from socially optimal.

IntuitionA double failure

Firms’ security depends on others, so they underinvest in both security and insurance. Insurers face correlated losses and charge high premiums, which price most firms out. The result is too little volume and liquidity for an efficient market.

The economics of censorship resistance

Early p2p systems aimed at censorship resistance, and put all content in one pot. Different groups ended up protecting one another’s speech: Falun Gong members, Scientology critics, and people distributing imagery legal in California but banned in Tennessee.

IntuitionSolidarity vs diversity

Would each group defend its own content harder in its own system, rather than protecting causes it doesn’t care about or even opposes? The same question arises in social theory (does diversity erode the solidarity behind welfare states?) and in guerrilla warfare (when to aggregate, when to disperse).

Architecture	Content placement	Examples
random / single pot	files scattered across all nodes	Eternity, Freenet, Chord, Pastry, OceanStore (DHTs)
discretionary	nodes serve what they chose to download	Gnutella, Kazaa

These were first compared on technical costs (search, retrieval, communication, storage). Incentives matter too.

The Red–Blue model (Danezis–Anderson)

DefinitionRed–Blue utility

$N$ nodes and two resource types. Node $i$ prefers a red fraction $r_i\in[0,1]$ and a blue fraction $b_i = 1 - r_i$ . Holding $T = R + B$ resources:

U_i(R,B) = -T\left(\frac RT - r_i - 1\right)\left(\frac RT - r_i + 1\right) = T\left[1 - \Big(\frac RT - r_i\Big)^2\right]. \tag{25.1}

Component	Detail
Signature	$U_i : \mathbb{R}_{\ge0}^2 \to \mathbb{R}_{\ge0}$
Domain	how many red and blue files the node holds
Codomain	its satisfaction: grows with volume $T$ , peaks when the mix matches its preference
Maximum	$R = r_iT$ , with value $T$

NoteHomogeneity

$U_i$ is homogeneous of degree 1: $U_i(\rho T, (1-\rho)T) = T\,u_i(\rho)$ with $u_i(\rho) = 1 - (\rho - r_i)^2 \in [0,1]$ . In (25.6)–(25.10) below, " $U_i(r,b)$ " means this per-unit utility $u_i$ . (This reading is mine; it makes the formulas consistent.)

FormulaRandom placement

With $R$ red and $B$ blue files in total, a DHT gives every node on average the systemwide mix $r_s = \frac{R}{R+B}, \qquad b_s = \frac{B}{R+B}. \tag{25.2}$ $U_i(r_iT, b_iT) \ge U_i(r_sT, b_sT), \tag{25.3}$ with equality iff $r_i = r_s$ . This can’t hold for everyone unless preferences are homogeneous.

WarningEveryone wants to tip the pot

Under random placement, each node gains by skewing total supply toward its own preference. With utility slightly more weighted toward serving, the network could be flooded with the dominant colour.

How hard does each node fight the censor?

DefinitionCensorship model

A censor tries to impose a mix $(r_c, b_c)$ . A node under attack diverts effort $t$ from serving (keeping $T - t$ resources) to resisting. It wins with probability $P(t)$ and restores its mix; otherwise the censor imposes $(r_c,b_c)$ . The censor’s cost is assumed equal to the defender’s $t$ .

FormulaExpected utility under censorship

Discretionary:

U = P(t)\,U_i\big(r_i(T-t), b_i(T-t)\big) + (1 - P(t))\,U_i\big(r_c(T-t), b_c(T-t)\big). \tag{25.4}

Random (DHT): the same, with $(r_s, b_s)$ in place of $(r_i, b_i)$ . (25.7)

Take the simplest resistance technology: spending everything guarantees victory, and spending nothing guarantees defeat:

$P(t) = \frac tT. \tag{25.5}$

DerivationOptimal defense budget

Step 1: Use homogeneity. Let $a = u_i(\text{own mix})$ and $c = u_i(r_c)$ . Then $U(t) = (T - t)\Big[c + \frac tT(a - c)\Big].$

Step 2: First-order condition. $U'(t) = (a - 2c) - \frac{2t}{T}(a - c) = 0.$

Step 3: Solve.

t^* = \frac{T\,(a - 2c)}{2(a - c)} = \frac{T\,\big(2U_i(r_c,b_c) - U_i(\cdot)\big)}{2\big(U_i(r_c,b_c) - U_i(\cdot)\big)},

matching (25.6) with $U_i(\cdot) = U_i(r_i,b_i)$ (discretionary, $t_d$ ) and (25.8) with $U_i(r_s,b_s)$ (random, $t_s$ ).

Step 4: Bounds. Since $0 \le c \le a$ , we get $t^* \le T/2$ : a node never spends more than half its capacity fighting. (This bound is my observation.)

FormulaWhen does a node fight at all?

$t^* > 0$ iff $2\,U_i(r_c, b_c) < U_i(\text{own mix}). \tag{25.9–25.10}$ The censor’s mix must be worth less than half the node’s current mix. Otherwise the node sets $t = 0$ .

ExampleRandom placement recruits the censor’s allies

Let $r_s = b_s = 0.5$ and a censor push $r_c = 0$ (all blue). Blue-loving nodes find $U_i(r_c) \ge U_i(r_s)$ : censorship makes them happier, so they spend $t = 0$ . In a single pot, the censor has a built-in fifth column.

TheoremDiscretionary placement is costlier to censor

For each node, $t_d \ge t_s$ , so $\sum_{i\in S} t_{d,i} \ge \sum_{i\in S} t_{s,i} \tag{25.11}$ for any targeted set $S$ . Whatever the censor does, attacking a discretionary network costs at least as much. Equality holds only when preferences are homogeneous ( $r_i = r_s$ for all $i$ ).

ProofDefense is increasing in the value of what you defend

Step 1: Discretionary nodes have more to lose. Their own mix gives $a_d = u_i(r_i) = 1 \ge a_s = u_i(r_s)$ , by (25.3).

Step 2: $t^*$ is increasing in $a$ .

\frac{\partial}{\partial a}\frac{a - 2c}{2(a - c)} = \frac{c}{2(a - c)^2} \ge 0.

Truncation at 0 preserves the order.

Step 3: Sum over nodes. A per-node inequality gives the aggregate one.

(The chapter asserts the per-node inequality; Step 2’s derivative is mine.)

∎

ImportantSolidarity is strongest when you guard your own

Nodes fight hardest for content they chose. Mixing everyone’s content dilutes each node’s stake, and may even align some nodes with the censor. This is a formal case for fan-club constellations over one giant pot.

Topology: attack and defense on complex networks

Real networks (the Internet, p2p overlays, social networks, metabolic pathways) emerge from simple local rules and are heterogeneous at every scale. Network science, at the boundary of sociology and condensed-matter physics, studies them. Here it meets security and evolutionary game theory.

ExampleCovert conflicts on networks

An attacker tries to disconnect a network or raise its diameter by removing nodes or edges, and defenders counter with resilience mechanisms. Examples: the music industry against file sharing, police against terrorist cells, a totalitarian state surveilling activists.

TheoremScale-free robustness (Albert–Jeong–Barabási)

Scale-free networks, like many real networks, get most of their connectivity from a minority of high-degree hubs. They are robust to random failure and fragile to targeted attack: remove the kingpins and connectivity collapses.

NoteStatic vs dynamic

Albert et al. model the static case: find and remove key personnel once. The chapter asks about the dynamic case, where defenders replace and rewire after each round.

The dynamic simulation

AlgorithmAttack–defense rounds

Each round:

Attack: remove a fixed proportion of nodes. Which nodes is the attacker’s strategy.
Replenish lost nodes.
Adapt: defenders rewire using only local knowledge. How is the defenders’ strategy.

Efficiency. An attack is more efficient if it needs a smaller budget to disrupt the network, given the defense. A defense is more efficient if it forces a larger attack budget, given the attack.

Round 1: vertex-order (hub) attack.

Defense	Behaviour (largest component, starting at 400 nodes)
random replenishment	halves to under 200 within 3 rounds
rings (replace hubs with rings, as in Chord)	collapse postponed from about 2 to about 12 rounds, then near-total disconnection
cliques (replace hubs with fully connected clusters)	a few vertices lost per round; stays robustly connected

ImportantWhy revolutionaries use cells

Rings appeal to theoreticians, but real revolutionary movements that left traces in history used cells, which are cliques. The simulation shows why: cliques survive hub-removal indefinitely, while rings only delay collapse.

Round 2: attacking cliques. The best attack found targets centrality: remove the highest-betweenness nodes each round, computed by Brandes’ algorithm.

Rings still collapse after about 12 rounds.
Cliques are hurt significantly: the largest component shrinks a lot.

Round 3: defending against centrality attacks.

DefinitionDelegation defense

A node becoming too well connected picks a neighbour as deputy and transfers some of its links to it. This mirrors peacetime behaviour (busy leaders pass recruits to colleagues), and in wartime it limits exposure. A leader approached by a financier simply introduces him to a militant and then keeps contact with at most one of them.

Defense vs centrality attack	Outcome
delegation alone	like rings: fragmentation postponed (about 14 rounds), not averted
delegation before hostilities + cliques from the start	better than cliques alone: equilibrium reached later, with a larger surviving component

IntuitionWhy the compound defense works

Delegation keeps path lengths short under attack, slowing the growth in path length that hub elimination causes. Cliques keep local connectivity. Both reduce degree skewness, which is exactly what vertex-order attacks exploit. So adopting them doesn’t reopen the earlier attacks.

graph TD
  A1["Attack: remove highest-degree nodes"] --> D1["Defense: rings delay; cliques survive"]
  D1 --> A2["Attack: remove highest-centrality nodes"]
  A2 --> D2["Defense: cliques hurt; delegation alone delays"]
  D2 --> D3["Defense: delegation before attack + cliques works best"]

NoteNumbers from figures

The round counts and component sizes above come from the chapter’s description of Figures 25.1–25.3. Simulation parameters beyond “400 nodes” are not given.

RemarkToward evolutionary game theory

Iterating “find the best attack against the current defense, then the best defense against that” is an arms race over strategies. The chapter presents it as a first bridge between network analysis and evolutionary game theory, and a framework for evaluating strategies where topology matters.

Quick reference

Object	Statement	Note
Core thesis	guardian ≠ victim ⇒ underprotection	externality / public good
UK vs US banks	less liability ⇒ more spending and more fraud	moral hazard
Antivirus 2000	payloads hit third parties ⇒ underbuying	negative externality
Liability principle	assign to the party best able to manage the risk	often violated online
Total effort	best benefit–cost agent provides; others free-ride	more agents ⇒ more reliable
Weakest link	worst benefit–cost agent binds	more agents ⇒ less reliable
Best shot	the star effort decides	cryptanalysis
Testers vs programmers	more testers, fewer and better programmers	total effort vs weakest link
Interdependent security	protection’s positive externality discourages others	multiple equilibria; herd immunity
Bootstrapping	benefit < cost until critical mass	escaped via intra-org benefit (SSH, IPsec, fax)
Hidden-action window	attack iff $E_h < G < E_o$	observability deters the middle band
Remedies	deductibles (contracts); clubs; chosen neighbours	raise observability
Market for lemons	50 good at $2k + 50 lemons at $1k ⇒ price about $1k	unmeasurable quality exits
Software lemons	unmeasurable security ⇒ no premium ⇒ no investment
Vulnerability market	prices ≈ labour cost to find a bug	current buyers hide prices and may leak
Security derivatives	paired “no bug” / “bug by date $D$ ” contracts	price = consensus probability
Cyber-insurance	physical + logical interdependence; global correlation	underinvestment; illiquid market
Red–Blue utility (25.1)	$U_i = T[1 - (R/T - r_i)^2]$	peak at $R = r_iT$
Random mix (25.2)	$r_s = R/(R+B)$	$U_i(r_i) \ge U_i(r_s)$ (25.3)
$P(t)$ (25.5)	$t/T$	linear resistance
Optimal defense (25.6/25.8)	$t^* = T\frac{a - 2c}{2(a-c)}$	$a$ = own-mix utility, $c$ = censor-mix utility
Fight condition (25.10)	$2u_i(r_c) < u_i(\text{own})$	else $t = 0$
(25.11)	$\sum t_d \ge \sum t_s$	equality iff homogeneous preferences
Scale-free	robust to random, fragile to targeted	Albert–Jeong–Barabási
Vertex-order attack	rings delay (about 12 rounds); cliques survive	cells > rings
Centrality attack	hurts cliques; rings still collapse	Brandes betweenness
Delegation + cliques	best defense found vs centrality	shorter paths, less skew

Facts to keep to hand

Every failure here is an externality or an information gap. Bank carelessness, antivirus underbuying, medical privacy and vendor risk-dumping are externalities. Hidden action, lemons and uninsurability are information problems. Fixes either reassign liability or create information (markets, derivatives, insurance data, observability).
The aggregation technology decides free-riding. Sum: one agent carries everyone. Min: one careless agent sinks everyone. Max: one star saves everyone. Policy should follow the technology: add testers (sum), shrink the programmer pool (min).
Clubs recur as the fix. Club-level benefits bootstrapped SSH and IPsec. Clubs make hidden action observable and give sanctions something durable to bite on. Fan-club p2p systems resist censorship better. Cliques/cells survive attack. Local groups align incentives that global pools dilute.
Observability deters a whole band of attacks. Anything whose gain lies between the penalty-if-hidden and the penalty-if-seen disappears once detection is likely. That makes topology a security tool, not just a performance choice.
Insurance needs independence. Globally correlated cyber-risk, from shared platforms and connected infrastructure, breaks pooling. That is why worm risk is uninsurable and why vendor liability collides with insurability.
People defend what they chose. In the Red–Blue model, defense effort rises with the value of your own mix. So discretionary placement always makes censorship at least as costly, and random pots can hand the censor allies.
Static robustness isn’t dynamic robustness. Scale-free networks fall to hub removal. With replenishment and local rewiring, rings only delay collapse. Cliques beat hub attacks, centrality attacks beat cliques, and delegation plus cliques beats centrality. Evaluate strategies as an arms race.

References

Nisan, Roughgarden, Tardos, Vazirani, Algorithmic Game Theory — chapter 25 (Anderson, Moore, Nagaraja, Ozment)

Related