Manipulation-Resistant Reputation Systems
Why reputation matters (ability revelation, moral hazard, adverse selection); the reputation-system pipeline and aggregation across users, time, and transitive trust; three threats (whitewashing, dishonest or missing feedback, phantom/sybil feedback); repeated Prisoner's Dilemma, discounted average payoff, Grim SPNE for delta >= 1/2, reputational vs personalized Grim; whitewashing, entry fees f >= 1/delta, the pay-your-dues (PYD) strategy and its social cost, noisy dynamic model and unavoidable mistrust of newcomers; peer-prediction method (Miller–Resnick– Zeckhauser) with common prior, signals, reference raters and the log scoring rule, truthful strict Nash equilibrium (Proposition 1), worked H/L example, non-truthful equilibria and extensions; transitive trust graphs, PageRank, max-flow (Advogato), Pathrank; monotonicity, symmetry, rank-strategyproofness impossibility; sybil strategies, value- vs rank-sybilproofness, no symmetric rank-sybilproof function (Theorems 27.5, 27.7), max-flow value-sybilproof but not rank-sybilproof (Theorem 27.8), Pathrank value- and rank-sybilproof (Theorem 27.9)
Motivation: trusting strangers online
The Internet enables interactions between parties with no common context. Their value depends on the partner’s ability and reliability. An online shopper may find better deals, or be defrauded with little redress.
Making each entity’s history visible does three jobs:
| Benefit | Economic problem solved |
|---|---|
| reveals ability, so others choose partners and terms | information |
| future visibility deters cheating or shirking now | moral hazard |
| high-ability entities are distinguishable and rewarded, so they join | adverse selection |
IntuitionThe central question
A reputation system is only as good as the histories it records and the identities it attaches them to. When participants can abandon bad names, shade or skip their feedback, and invent fake raters, which guarantees can a reputation system still provide — and which are impossible?
What a reputation system is
DefinitionReputation system
A system that collects, maintains and disseminates reputations: aggregated records of each participant’s past interactions. Records combine objective data (e.g., sale price) with subjective feedback (e.g., buyer and seller rating each other; peers reporting availability). Reputations are usually numeric summaries, for easy comparison.
graph LR I["Interactions between entities"] --> C["Collect objective data and feedback"] C --> U["Aggregate and update reputations"] U --> D["Disseminate reputations"] D --> I
Aggregation happens along three dimensions:
- across users: a statistic of everyone’s reports about ;
- across time: a statistic of past reports;
- transitive trust: reports from highly reputed users weigh more.
NoteScope
Economic models in which actions link to outcomes only probabilistically, or with observation error, are “interesting and complex” but out of scope. The chapter studies threats to the system itself: how histories are collected and attached to identities.
DefinitionThree threats
- Whitewashing: take a new pseudonym and start over with a clean record.
- Incorrect feedback: don’t report it, or don’t report it honestly.
- Phantom feedback: feedback on interactions that never happened, often from sock puppets or sybils, identities created for the purpose.
Threats 1 and 3 both come from cheap pseudonyms.
| Threat | Model | Remedy | Section |
|---|---|---|---|
| whitewashing | repeated PD with random matching | pay-your-dues | Whitewashing |
| dishonest or missing feedback | common-prior signal model | peer prediction | Eliciting effort and honest feedback |
| dishonest and phantom feedback | trust graph | seeded, path-based ranking | Transitive trust and sybils |
Why reputations work: the repeated Prisoner’s Dilemma
DefinitionThe stage game
| Bob: | Bob: | |
|---|---|---|
| Alice: | ||
| Alice: |
One-shot, is dominant, so the outcome is .
FormulaDiscounted average payoff
The factor normalizes: getting every round gives . measures how much future rounds matter (think of interest rates).
DefinitionGrim trigger
Play until anyone has played , then play forever. is a subgame perfect Nash equilibrium (SPNE) when no player wants to deviate at any stage game.
ProofGrim is an SPNE iff
Step 1: One-shot deviations suffice (the single-deviation property from dynamic programming).
Step 2: Compare. Deviate at round 0: . Comply: .
Step 3: Solve. Complying is optimal iff , i.e. . The same holds at every round , since both sides scale by .
∎
ImportantReputations need patience
If players don’t value the future enough (small ), no reputation system can constrain behaviour. Less harsh strategies, which punish for only a few rounds, can also sustain cooperation, for higher .
DefinitionReputational Grim (random matching)
players ( odd) are paired at random each round. Everyone starts good, and stays good by playing against good partners and against bad ones. This is an SPNE for : a deviator faces the same punishment as under full Grim.
WarningPrivate histories don’t scale
Without shared reputations, a player can punish only on her own encounters (personalized Grim, separate games). She meets a given partner about once every rounds, so the SPNE condition becomes which is unreasonably close to 1 for large . Shared histories are what make reputations work at scale. (Compare direct vs indirect reciprocity in p2p-incentives.)
NoteDerivation left as an exercise
The chapter gives without derivation.
RemarkHeterogeneous abilities
Varying ability is modelled by different action sets. A commitment type has one action only (low types only , honest high types only ), or may cost low types more. Strategic types choose actions to mimic high types. With noisy outcomes, strategic types can defect occasionally and hide among honest types who have bad luck.
Whitewashing
WarningCheap identities destroy Grim
A player can play , then re-register under a new name next round. Punishment never reaches the deviator.
Entry fees
DerivationThe fee needed to deter whitewashing:
Charge to register. Whitewash once (defect, pay again, cooperate after):
Comply: Then . So iff is still needed to stop deviation without whitewashing.
Pay your dues
Fees may be infeasible or politically unacceptable. Instead, charge a reputational fee.
DefinitionPay-your-dues (PYD)
- vs a veteran who has never deviated from PYD: play ;
- vs a veteran who has deviated: play ;
- vs a newcomer: play , unless you are a newcomer too, in which case play .
Veterans get to exploit newcomers once; newcomers “pay dues” by cooperating while exploited.
IntuitionWhy this deters whitewashing
A whitewasher returns as a newcomer and gets exploited, which costs as much as the punishment it tried to escape. Newcomers must be penalized at least as much as deviators, because they are indistinguishable.
WarningAn unavoidable social cost
A veteran–newcomer pair plays , with total payoff instead of for . Making newcomers defect against each other too would give , which is even worse. So whitewashing imposes a cost on society that even the best equilibrium can’t avoid. Under PYD, every player still averages a discounted payoff of 1 (exercise).
A more realistic dynamic model
WarningThe naive model is too easy
If the population is fixed, every newcomer is a whitewasher. Always defecting on post-period-1 arrivals (and personalized Grim otherwise) is then fully efficient.
Add real turnover. genuine newcomers arrive each period and as many random veterans leave. There is still a fragile equilibrium: play PYD while exactly newcomers appear, and defect forever if more appear. One deviation collapses everything. Such knife-edge equilibria are artifacts of noiselessness and perfect rationality.
TheoremPYD is the most efficient equilibrium under noise (Friedman–Resnick)
Add noise: in any stage game, a player plays by accident with probability and returns next period as a whitewasher. Then PYD yields the highest fraction of cooperative outcomes of any equilibrium.
NoteProof omitted
Showing PYD is an equilibrium uses dynamic programming. Optimality needs a careful stochastic analysis (Friedman–Resnick 2001), not reproduced.
ImportantDistrust of newcomers is the price of cheap pseudonyms
In practice PYD means mistrusting newcomers until they prove themselves. The only way to treat newcomers better is to make whitewashing harder for existing members than for genuine newcomers. For example: require a true name, give one free pseudonym, and restrict or charge for additional ones.
Eliciting effort and honest feedback
Subjective feedback has two problems:
| Problem | Cause |
|---|---|
| underprovision | forming and reporting an opinion takes effort; the benefit goes to others |
| dishonesty | niceness or fear of retaliation suppresses negatives; conflicts of interest, or wanting to look like a good rater, distort reports |
If objective outcomes are later revealed (weather, sports), reports can be scored against them; see prediction-markets. The hard case is no objective outcome: either none exists (product “quality”), or it exists but is private (breakdown frequency, known only to owners).
WarningRewarding agreement causes herding
Scoring raters by agreement with peers backfires. A rater with a bad experience of a seller with a stellar record still expects the next rater to report good, so she reports good too. This resembles herding and information cascades.
NoteMeta-moderation
Slashdot has users rate the raters. That is inefficient (effort diverted from rating content) and only pushes the incentive problem up one level.
RemarkCurrency of rewards
Scores can become money or discounts, but prestige often suffices: Slashdot “karma,” ePinions status, chess and bridge ratings.
The model
DefinitionSignal model
- A product has a fixed type with a common prior .
- Raters , with , possibly countably infinite. Each privately observes a signal .
- Given the type, signals are i.i.d. with commonly known , which differs across types (so signals are informative).
- Raters report to a center, which has no independent information and pays transfers based only on reports.
- Raters are risk-neutral with utility linear in points.
DefinitionBest response and truthful equilibrium
is a best response if for every signal and every alternative report :
Truthful revelation () is a (strict) Nash equilibrium if truth is a (strict) best response when everyone else is truthful.
ExampleRunning example: two types, two signals
, with and . So and .
Predicting another rater :
The herding trap: even after seeing , rater thinks will probably see (). A pure agreement reward would make her report .
Peer-prediction scoring
DefinitionDefinition 27.1 — strictly proper scoring rule
A rule is strictly proper if a rater uniquely maximizes her expected score by reporting her true beliefs.
ExampleLogarithmic scoring rule
For a binary event with reported probability of : and . With true belief , the expected score has first-order condition .
DefinitionPeer-prediction method (Miller–Resnick–Zeckhauser)
Assign each rater a reference rater . The rater reports only her signal. The center infers her implied forecast of the reference rater’s report and scores it with the log rule:
| Component | Detail |
|---|---|
| Input | ’s report and the reference rater’s report |
| Inferred belief | posterior over given , computed by the center from the common prior |
| Output | log-probability the inferred belief assigned to what actually reported (a penalty ) |
| Not required of the rater | any Bayesian computation |
TheoremProposition 1 — truthful reporting is a strict Nash equilibrium
For any assignment , truthful reporting is a strict Nash equilibrium of the simultaneous reporting game with transfers .
ProofStrict propriety applied to the inferred forecast
Step 1: The reference rater is truthful. Then is stochastically informative about ‘s report.
Step 2: Rater ‘s problem. With true signal , she chooses to maximize
The key step: reporting amounts to announcing the forecast . The true forecast is the one conditioned on . The log rule is strictly proper, and different signals induce different forecasts (signals are informative), so (27.3) is uniquely maximized at .
∎
ExampleThe numbers: honesty wins
Rater saw .
| Report | Expected score |
|---|---|
| honest | |
| lie |
The expectations use ‘s true beliefs (, ). The log terms use the forecast implied by her report.
ImportantWhy peer prediction beats agreement
The score depends on the updated belief about the peer’s signal given your report, not on whether you match. The update already folds in the prior, so there’s no reason to shade your report toward the prior’s likely outcome. The rater just reports what she saw, trusting the center to do the Bayes.
NoteScores are penalties
Since , . Adding a constant that depends only on makes scores positive without changing incentives.
WarningNon-truthful equilibria exist
“Always report ” is also an equilibrium. Against an always- reference rater, reporting scores and reporting scores . Similarly for always-. Mitigations:
- the truthful equilibrium is focal (limited communication; some raters prefer honesty);
- the center can punish everyone on detecting an uninformative pattern.
Extensions.
| Modification | Effect |
|---|---|
| add a constant | raises payoffs; incentives unchanged |
| multiply by a constant | incentives unchanged, but tunes how much costly effort raters spend acquiring signals |
| settle via transfers between raters | budget-balanced, no subsidy from the center |
| alternative proper rules | smaller expected payments (Jurca–Faltings) |
| sequential reporting | score each rater on the belief implied by the prior, previous reports and her own; reporting her own signal stays optimal |
CautionThe weak point: common priors
The center needs common beliefs about types and about signal distributions given types, to infer each forecast. Eliciting the forecast directly fails, because ‘s signal report is also needed to score someone else.
- Partial fix 1: ask each rater for her personal prior over types before she observes, then her signal.
- Partial fix 2: define types empirically by their signal distributions (type 1 gives 10% , type 2 gives 20%, …), which makes the conditional signal beliefs common by construction.
NoteOpen questions
Raters of differing ability, where scores both assess credibility and give incentives; self-selected rather than random interaction partners; robustness to collusion and sybil confirmers.
Transitive trust and sybils
DefinitionTransitive trust
Postulate: a rater’s credibility equals her reputation as an actor. Weight ratings of an entity by the raters’ credibilities, which come from their reputations. This is recursive: bootstrap from a credible set and grow it along positive feedback. Positive feedback from a well-rated eBay member counts more.
DefinitionTrust graph and reputation function
Ignore time order. The trust graph is , where summarizes ‘s trust in . Graphs are sparse, so is listed explicitly. A reputation function induces a ranking of the nodes. is trivial if the ranking is the same for every .
| Reputation function | Definition | Seeded? |
|---|---|---|
| PageRank (simple) | ; web: | no (symmetric) |
| max-flow (Advogato) | = max flow from seed to , with capacities | yes |
| Pathrank | ranked by shortest path from to , edge length (shorter = better) | yes |
In P2P settings each node can use itself as the seed (personalized reputation). For web ranking, pick trusted seeds.
DefinitionMonotonic and symmetric
- Monotonic: adding an incoming edge never lowers ‘s rank relative to any (both strict and weak order preserved). All three functions above are monotonic (PageRank: Chien et al.).
- Symmetric: commutes with relabelling nodes; reputations depend only on graph structure. PageRank is symmetric. Max-flow and Pathrank are not: the seed is privileged.
Honest reporting: rank-strategyproofness
WarningTransitive trust has the worst incentives
Raters are rewarded only for ratings they receive, never for ratings they give. So there is no incentive to rate informatively, and possibly an incentive to rate strategically, to undermine rivals’ credibility.
DefinitionRank-strategyproof
No agent , in any graph, can improve her position in the ranking by changing how she rates others. (Changing scores is allowed; improving rank is not.)
TheoremImpossibility for symmetric systems
No nontrivial, monotonic, symmetric reputation system is rank-strategyproof.
ExamplePageRank manipulation
Node can drop an outgoing edge to a higher-ranked node . That lowers ‘s score, which can move above .
NoteCited, not proved
See Altman–Tennenholtz and related axiomatic work (compare the axiomatic method in cost-sharing). Pathrank escapes: it is monotonic and rank-strategyproof.
Sybil attacks
DefinitionDefinition 27.2 — sybil strategy
Given and user , a sybil strategy is a graph and a set of sybils such that collapsing into one node labelled gives back . The attacker may create any number of sybils with any trust among them, split incoming trust among them in any way that preserves the total , and set outgoing edges arbitrarily.
DefinitionDefinitions 27.3–27.4 — sybilproofness
- Value-sybilproof: no sybil strategy gives any a score . You can’t raise your number.
- Rank-sybilproof: no sybil strategy gives some a score for some with . You can’t overtake anyone.
TheoremTheorem 27.5 — no symmetric rank-sybilproof function
No nontrivial symmetric reputation function is rank-sybilproof. Monotonicity is not assumed.
ProofClone the whole network
Step 1: Pick a target. Nontriviality gives a graph with .
Step 2: Sybil strategy. Let be two disjoint copies of , and let the sybils be together with the whole second copy. Collapsing into returns (up to internal edges).
The key step: by symmetry, swapping the two copies is an automorphism, so the copy of has . A sybil now ties , whom previously trailed. Rank-sybilproofness fails.
∎
graph LR
subgraph Original["Original graph G"]
V1["v: lower rank"] --- W1["w: higher rank"]
end
subgraph Clone["Sybil copy of G, all controlled by v"]
V2["copy of v"] --- W2["copy of w: ties w by symmetry"]
end
DefinitionDefinition 27.6 — -rank-sybilproof
Rank-sybilproof against all sybil strategies with .
TheoremTheorem 27.7 — even one sybil breaks symmetry
No nontrivial symmetric reputation function is -rank-sybilproof for any .
ProofAdd the clone one node at a time
Step 1: Interpolate. With and clone , let contain . So and .
Step 2: A first crossing. At , the best of ‘s identities trails . At , ties (Theorem 27.5). So some has but .
The key step: let be the top identity in . Either or now reaches . So adding the single node is a successful 1-sybil strategy for in .
∎
WarningPageRank
PageRank is symmetric, so it is not rank-sybilproof. About half of all web pages could double their PageRank with a single sybil.
Escape: break symmetry with a trusted seed. Seeding alone isn’t enough; the aggregation rule matters too.
TheoremTheorem 27.8 — max-flow is value-sybilproof
The max-flow ranking from a seed is value-sybilproof.
ProofMin cut bounds every sybil
By max-flow = min-cut, is the capacity of a minimum – cut. All of ‘s sybils lie on ‘s side of that cut, away from . The trust entering them is at most the trust entering , so no sybil’s max flow exceeds .
∎
ExampleFigure 27.2 — max-flow is not rank-sybilproof
Node lies on a max-flow path to a node ranked above it. reroutes its outgoing trust from into a sybil . ’s own value is unchanged, but loses the flow through and falls below . The attacker can’t raise itself, but it can lower others that depend on it.
TheoremTheorem 27.9 — Pathrank is value- and rank-sybilproof
Shortest-path ranking from a seed is both value-sybilproof and rank-sybilproof.
ProofSybils can’t shorten paths, and blockers are already ahead
Value: sybils sit behind , so no path to a sybil is shorter than the shortest path to .
Rank: can affect ‘s value only if lies on the shortest – path. But then is strictly closer to than , so already outranks . There is no one it could overtake by manipulating.
∎
ImportantFlows vs paths
The pattern generalizes. Generalized flow mechanisms (e.g., “leaky pipes”) are value- but not rank-sybilproof. Generalized path mechanisms are both, and in a large class, only path-based mechanisms are rank-sybilproof.
NoteOpen question
PageRank and max-flow are not worst-case sybilproof but are very useful in practice. Are they less manipulable on average? This needs a precise formulation.
Open problems and extensions
NoteDistributed reputation systems
In P2P and ad hoc networks there is no trusted party computing reputations, and users may intercept or alter messages. Techniques: replication (compute at many nodes and detect inconsistencies), and redundant paths so no single user can block or modify messages. Both are vulnerable to collusion. The defense is to require many colluders, which sybils undermine.
NoteDynamic attacks
Users may claim credit for others’ interactions, or copy others’ actions (puzzle solutions, ratings) to build reputation cheaply. Conversely, dynamics can limit attacks: in P2P, sybils need real interactions to gain incoming links, and low-ranked sybils struggle to find partners.
NoteMetrics and benchmarks
Robustness isn’t everything. Giving everyone zero reputation is perfectly secure and perfectly useless. We need metrics (economic welfare, or more direct measures) of how well an aggregation method serves users’ information needs.
NoteOther disciplines, and synthesis
Sociology and social psychology study the broader notion of trust. The grand challenge is a single design combining honest elicitation, whitewashing and sybil robustness, and distributed computation.
Quick reference
| Object | Statement | Note |
|---|---|---|
| Benefits of visible history | ability revelation; deters moral hazard; averts adverse selection | |
| Three threats | whitewashing; incorrect feedback; phantom (sybil) feedback | cheap pseudonyms |
| PD payoffs | , , , | |
| Discounted average | ||
| Grim SPNE | iff | |
| Reputational Grim | SPNE for under random matching | shared history |
| Personalized Grim | needs | private histories don’t scale |
| Entry fee | deters whitewashing iff | |
| PYD | veterans exploit newcomers once; newcomers cooperate | total 1 vs 2 |
| Noisy dynamic model | PYD maximizes cooperation | Friedman–Resnick |
| Fix for newcomer distrust | make new pseudonyms costly for existing members | one free pseudonym |
| Signal model | common prior ; i.i.d. signals ; | center has no outside info |
| Herding | agreement rewards suppress minority signals | Ex.: |
| Strictly proper rule | truth uniquely maximizes expected score | log rule |
| Peer prediction (27.2) | the rater reports only her signal | |
| Prop 1 | truthful is a strict NE for any | |
| Example | honest : ; lie : | |
| Non-truthful equilibria | always-, always- | focal truth; punish uninformative patterns |
| Weak point | common priors on types and signal distributions | empirical types |
| Trust graph | , | |
| PageRank | symmetric | |
| Max-flow / Pathrank | flow / shortest path from seed | asymmetric |
| Rank-strategyproof | can’t improve rank by how you rate | impossible if symmetric, monotonic, nontrivial |
| Sybil strategy (Def 27.2) | collapsing to | split incoming trust arbitrarily |
| Value- / rank-sybilproof | can’t raise own score / can’t overtake anyone | |
| Thm 27.5 | no symmetric rank-sybilproof | clone the network |
| Thm 27.7 | not even 1-rank-sybilproof | add the clone node by node |
| Thm 27.8 | max-flow value-sybilproof | min cut; not rank (Fig. 27.2) |
| Thm 27.9 | Pathrank value- and rank-sybilproof | only path-based is rank-sybilproof |
Facts to keep to hand
- Every remedy trades something away. Pay-your-dues buys whitewashing resistance at the cost of newcomer welfare. Peer prediction buys honesty at the cost of a common-prior assumption and spurious equilibria. Seeded path rankings buy sybilproofness at the cost of symmetry and dependence on a trusted seed.
- Anyone indistinguishable from a deviator must be punished like one. Newcomers look like whitewashers, so they pay dues. This echoes the whitewasher problem in p2p-incentives: generosity toward newcomers subsidizes cheaters.
- Shared history is what makes reputation scale. Private punishment needs as grows. Public reputations keep the threshold at .
- Score beliefs, not agreement. Peer prediction scores the implied forecast of a peer’s signal with a strictly proper rule, which already corrects for the prior, so honest minority reports aren’t penalized. This is the same proper-scoring-rule logic as market scoring rules in prediction-markets.
- Symmetry is fatal for sybil resistance. If ignores labels, cloning the network (or even adding one well-chosen node) lets an attacker tie anyone. Robust systems must privilege a seed.
- Flows can be drained; paths can’t be shortened. Sybils can’t raise their own flow (min cut) but can starve downstream rivals. On shortest paths, anyone able to influence you is already ahead of you, so rank manipulation is impossible.
- Security is not the objective. A system that rates everyone zero is unmanipulable and useless. Robustness results must be paired with measures of informativeness.
References
- Nisan, Roughgarden, Tardos, Vazirani, Algorithmic Game Theory — chapter 27 (Friedman, Resnick, Sami)