Reputation in public goods cooperation under double <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" altimg="si51.svg" display="inline" id="d1e1286"><mml:mi>Q</mml:mi></mml:math>-learning protocol

This paper asks how cooperation in a spatial public goods game (PGG) changes when (i) players’ investment is heterogeneous and depends on both the reputation of group organizers and the population’s cooperation willingness, (ii) individuals’ payoffs are augmented by reputation through a weighted fitness function, and (iii) reputation evolves nonlinearly with potentially abrupt changes rather than monotonic drift. The question matters because cooperation dilemmas are central to evolutionary game theory and to real collective-action settings (public projects, institutions, online communities), where trust and reputation are known to be fragile, state-dependent, and capable of sudden collapse or recovery. Most prior reputation-based PGG models either treat reputation as monotonic or incorporate it in a limited way (e.g., only as a selection/interaction modifier). Here, reputation is integrated simultaneously into investment heterogeneity, into payoff evaluation, and into a nonlinear update rule, creating a richer mechanism for indirect reciprocity.

Methodologically, the authors implement an agent-based evolutionary game on a two-dimensional lattice of size $L\times L$ with periodic boundary conditions, using von Neumann neighborhoods (each player interacts with four nearest neighbors). Each player participates in five groups (centered on itself and on each neighbor). The game is repeated for $E=10^4$ steps per run, with $M=30$ independent runs. Initial strategies are randomly assigned as cooperation (C) or defection (D). The public goods payoff from each group $G_j$ is computed with a synergy factor $r$: cooperators contribute and defectors contribute nothing, and the group’s amplified contributions are shared equally among group members. The total game payoff $p_i(\tau )$ is the sum of payoffs across the five groups. Reputation enters via a weighted “actual income” (fitness) function ${\Pi }_i(\tau )=(1-\eta )p_i(\tau )+\eta R_i(\tau )$, where $\eta$ controls how strongly reputation dominates fitness.

Reputation dynamics follow a nonlinear transfer (NRT) rule. Each player’s reputation $R_i(\tau )$ is initialized uniformly in $[R_{\min },R_{\max }]$ with $R_{\min }=1$ and $R_{\max }=10$. At each step, reputation changes according to the player’s last action: if the player cooperated, reputation increases by $\delta \left(1-\frac{R_i(\tau )}{10}\right)$; if the player defected, reputation decreases by $-\delta \frac{R_i(\tau )}{10}$. This creates diminishing returns for high reputation under cooperation and potentially large drops under defection, with reputation clipped to $[1,10]$. The parameter $\delta$ is the reputation sensitivity.

Crucially, strategy updating is performed using reinforcement learning rather than imitation. The paper replaces traditional Q-learning (TQL) with double Q-learning (DQL) to mitigate overestimation bias. Each agent maintains two Q-tables, $Q_1^i$ and $Q_2^i$, over the state set $\mathbb{S}=\{C,D\}$ and action set $\mathbb{A}=\{C,D\}$. Action selection uses the larger Q-value between the two tables (with $ϵ$-greedy exploration, $ϵ=0.02$ in baseline experiments), while Q-value updates randomly update either $Q_1$ or $Q_2$. When updating one table, the target uses the other table’s evaluation, separating action selection from value estimation—this is the key theoretical mechanism for reducing overestimation bias.

The primary empirical outcome is the stationary cooperation level $f_c$, computed by averaging the cooperation fraction over the final 500 steps of each run (steps $9501$ to $10000$), then averaging across runs. The authors also track time-dependent cooperation $f_c(\tau )$ and analyze Q-values and reputation distributions in the stationary regime.

Key findings are reported through simulation comparisons and parameter sweeps. First, DQL substantially outperforms TQL in promoting cooperation when reputation is meaningfully coupled to fitness (i.e., when $\delta >0$ and $\eta >0$). The paper notes that when $\delta =0$ (reputation effectively fixed/uninformative), TQL can show slightly higher cooperation, but this is not meaningful because reputation is not dynamically connected to fitness. Under realistic reputation sensitivity (e.g., $\delta =2$), increasing $\eta$ and $\delta$ increases cooperation for both algorithms, but the improvement under DQL is “remarkably more pronounced,” with the authors emphasizing that DQL’s advantage is clear for the relevant parameter regimes (the text indicates DQL superiority particularly when $r$ is in the cooperative-supporting range, though the exact p-values are not provided).

Second, the coevolution of strategy and reputation reveals a mechanism: when reputation is weighted heavily in fitness (larger $\eta$), the system can reverse the typical tragedy-of-the-commons outcome. The authors describe spatial snapshots showing that at $\eta =0$ (reputation irrelevant), defection dominates and only low-reputation classes survive. When $\eta$ is increased (e.g., $\eta =0.3$), cooperators can form small clusters and defectors are eliminated even at relatively low synergy $r$, while reputation levels rise and become aligned with cooperation. Importantly, the paper argues that large cooperative clusters do not arise from network reciprocity because the agents’ Q-tables are not directly conditioned on neighbors’ strategies; instead, updates are driven primarily by self-reward (fitness) and the reinforcement learning process. This design makes the emergence of cooperation less dependent on topology and more dependent on the reputation-payoff coupling.

Third, reputation dynamics exhibit a non-monotonic, state-dependent structure. The authors observe that some players maintain moderate reputation and repeatedly flip between cooperation and defection when $\eta >0$, consistent with a trade-off: maintaining high reputation requires sustained cooperation, while defection can yield short-term payoff but erodes reputation. They also report that the middle-reputation class can persist robustly for any $\eta >0$, with its final fraction only mildly dependent on $\eta$. The explanation is that reputation growth slows near saturation due to the NRT rule, so reaching higher reputation requires sustained investment and becomes costly/ambiguous.

Fourth, the combined parameter effects are mapped on the $\delta$-$\eta$ plane. For a representative low synergy case $r=2$, the authors report that high cooperation (red region) appears only when $\delta >0.5$ and $\eta >0.22$. For a higher synergy case $r=3$, the low-cooperation region disappears, and cooperation shifts to medium or high levels (green/yellow). The authors summarize the key message as: $\eta$, $r$, and $\delta$ must be jointly tuned; no single parameter alone reliably produces high cooperation.

Fifth, the paper provides quantitative Q-table evidence. Table 3 reports average Q-values in the stationary state for $r=2$ and $\delta =2$ across $\eta \in \{0,0.1,0.2,0.3\}$. When $\eta =0$, the only nonzero average is ${\bar{Q}}_{(D,D)}=0.33$, while ${\bar{Q}}_{(C,C)}={\bar{Q}}_{(C,D)}={\bar{Q}}_{(D,C)}=0$, indicating defection is optimal. As $\eta$ increases, cooperation-related Q-values rise: at $\eta =0.1$, ${\bar{Q}}_{(C,C)}=13.09$, ${\bar{Q}}_{(C,D)}=17.25$, ${\bar{Q}}_{(D,C)}=17.34$, ${\bar{Q}}_{(D,D)}=17.69$; at $\eta =0.2$, ${\bar{Q}}_{(C,C)}=55.36$, ${\bar{Q}}_{(C,D)}=54.57$, ${\bar{Q}}_{(D,C)}=57.71$, ${\bar{Q}}_{(D,D)}=48.59$; and at $\eta =0.3$, ${\bar{Q}}_{(C,C)}=87.52$, ${\bar{Q}}_{(C,D)}=78.64$, ${\bar{Q}}_{(D,C)}=84.39$, ${\bar{Q}}_{(D,D)}=69.89$. The authors interpret cooperation dominance as occurring when both inequalities hold: ${\bar{Q}}_{(C,C)}>{\bar{Q}}_{(C,D)}$ and ${\bar{Q}}_{(D,C)}>{\bar{Q}}_{(D,D)}$. They further argue that larger gaps between these quantities make strategy switching toward cooperation easier.

Finally, robustness is supported via a mean-field approximation. The authors derive a differential equation for cooperation frequency $f_c(\tau )$ based on transition probabilities ${\mathbb{P}}_{D\to C}$ and ${\mathbb{P}}_{C\to D}$, yielding a stationary expression $f_c=\frac{{\mathbb{P}}_{D\to C}}{{\mathbb{P}}_{C\to D}+{\mathbb{P}}_{D\to C}}$. They report close agreement between mean-field predictions and simulation results, suggesting that the observed cooperation enhancement is not purely a finite-size artifact.

Limitations include the reliance on a specific lattice topology (square lattice with von Neumann neighborhood) and a specific two-action state/action representation for reinforcement learning (state equals current strategy). The paper also fixes baseline learning parameters ($\alpha =0.8$, $\gamma =0.9$, $ϵ=0.02$) and does not provide statistical significance tests (e.g., p-values or confidence intervals) for the cooperation-level differences between DQL and TQL, though it uses $M=30$ runs and reports error bars qualitatively in figures. Additionally, the HIORC mechanism is described conceptually, but the provided excerpt emphasizes the payoff weighting and NRT reputation dynamics more than the full mathematical specification of organizer reputation and bandwagon-driven heterogeneous investment; thus, readers should verify how fully HIORC is operationalized in the simulation.

Practically, the results suggest that cooperation can be stabilized when (1) reputation is not merely an interaction label but enters agents’ effective fitness, (2) reputation dynamics reflect realistic fragility and nonlinear recovery, and (3) decision-making uses reinforcement learning methods that avoid overestimation bias. This is relevant to designing institutions and platforms where reputation affects both resource allocation (who invests) and evaluation (how rewards are computed). Stakeholders in online marketplaces, crowdsourcing, and public project governance—where reputation systems and learning-based adaptation occur—should care because the model predicts that coupling reputation to payoff can shift populations from defection to cooperation, but only when reputation sensitivity and payoff weighting are jointly strong enough relative to the underlying public goods synergy $r$.