SK game of reinforcement learners

The SK game, reinforcement learners whose rewards come from quenched Sherrington-Kirkpatrick couplings.

• Name: sk_game
• Field: physics/criticality
• Formalism: discrete map
• References: Lesmana et al. 2026; Sherrington and Kirkpatrick 1975; Watkins and Dayan 1992

Factory

@register_model(
    field='physics/criticality',
    tags=('spin glass', 'reinforcement learning', 'game', 'disorder', 'discrete map'),
    references=('Lesmana et al. 2026', 'Sherrington and Kirkpatrick 1975', 'Watkins and Dayan 1992'),
    defaults={'until': 200, 'dt': 1},
    observables=_observables,
)
def sk_game(
    n_agents: int = 256,
    *,
    alpha: float = 0.5,
    temperature: float = 0.5,
    random_state: int | np.random.Generator | None = 0,
) -> System:
    """Return the SK game, reinforcement learners whose rewards come from quenched Sherrington-Kirkpatrick couplings.

    Every agent plays `sigma_i = ±1` and earns `sigma_i sum_j J_ij sigma_j`, with symmetric Gaussian couplings of mean
    0, variance `1 / N`, and zero diagonal. It discounts its rewards with the memory `alpha` into the incentive `Q_i`
    and plays from a softmax policy at the temperature `T` (Lesmana et al., Eqs. 1-3). All agents start with zero
    incentives, so their first actions are random, and update in parallel. At low temperature the agents freeze into
    one of many ordered states with a high collective reward, and at high temperature they explore noisily with a low
    one.

    Args:
        n_agents:
            The number of agents `N`.

        alpha:
            The memory parameter.

        temperature:
            The exploration temperature of every agent.

        random_state:
            The seed or generator of the couplings and of the initial actions.

    Returns:
        system:
            The model, with the actions `agents.sigma`, the incentives `agents.Q`, and the coupling matrix in the values
            of the group. Its standard observable `reward` is the mean reward per agent.

    Examples:
        >>> from skcomplex.models.physics.criticality import sk_game
        >>> couplings = sk_game(10).elements[0].values['couplings']
        >>> bool((couplings == couplings.T).all())
        True
    """
    rng = np.random.default_rng(random_state)
    upper = np.triu(rng.normal(0.0, 1.0 / np.sqrt(n_agents), (n_agents, n_agents)), 1)
    agents = GroupElement(
        'agents',
        element=LearningAgentElement,
        size=n_agents,
        values={
            'sigma': rng.choice([-1.0, 1.0], size=n_agents),
            'Q': 0.0,
            'alpha': alpha,
            'temperature': temperature,
            'couplings': upper + upper.T,
        },
    )
    return System(elements=[agents])

Components

class LearningAgentElement(Element):
    """Agent of the SK game that plays an action of ±1 from a softmax policy over its incentive to play +1.

    The incentive follows the reinforcement rule `Q(t + 1) = (1 - alpha) Q(t) + alpha sum_j J_ij sigma_j(t)`, the
    discounted reward of playing +1, and the agent plays `sigma = ±1` with probability `1 / (1 + exp(-+2 Q / T))`.

    Args:
        label:
            The label of the agent or the group.

        visual:
            How the element is drawn, or `None` for `default_visual_`.

        sigma:
            The initial action, -1 or 1.

        Q:
            The initial incentive.

        alpha:
            The memory parameter, the weight of the latest reward.

        temperature:
            The exploration temperature `T` of the softmax policy.

        couplings:
            The coupling matrix `J`, one row and one column per agent.

    Examples:
        >>> from skcomplex.models.physics.criticality import LearningAgentElement
        >>> LearningAgentElement('agent').alpha
        0.5
    """

    def __init__(
        self,
        label: str,
        visual: VisualSpec | None = None,
        *,
        sigma: Annotated[float, StateSpec(domain=ZZ)] = 1.0,
        Q: Annotated[float, StateSpec()] = 0.0,
        alpha: Annotated[float, ParameterSpec(domain=RealSet.open(0, oo))] = 0.5,
        temperature: Annotated[float, ParameterSpec(domain=RealSet.open(0, oo))] = 0.5,
        couplings: npt.ArrayLike | None = None,
    ) -> None:
        super().__init__(label, visual=visual)
        self.sigma = sigma
        self.Q = Q
        self.alpha = alpha
        self.temperature = temperature
        self.couplings = couplings

    def _get_laws(self, s: SimpleNamespace) -> list[BaseLaw]:
        """Return the reinforcement of the incentive and the softmax draw of the action, both in parallel.

        Args:
            s:
                The namespace of the symbols the laws can use.

        Returns:
            laws:
                The laws of the component.
        """
        reward = total(s.sigma, weights=self.couplings)
        up = 1 / (1 + exp(-2 * s.Q / s.temperature))
        return [
            UpdateLaw(s.Q, (1 - s.alpha) * s.Q + s.alpha * reward),
            UpdateLaw(s.sigma, where(uniform('u') < up, 1, -1)),
        ]

Equations

agents.sigma[i]: sigma_agents(t + 1) == where(uniform_u_agents < (1/(e^(-2*Q_agents(t)/temperature_agents) + 1)), 1, -1)
agents.Q[i]: Q_agents(t + 1) == _reduced_0*alpha_agents - (alpha_agents - 1)*Q_agents(t)