Skip to content

Neural networks

Neural networks: Hopfield's memory, its diluted version, the echo state network, and Elman's recurrent network.

BinaryWeightElement(label, on=0.0)

Bases: Element

Weight of a binary perceptron, where the state 1 stands for +1 and the state 0 for -1.

Parameters:

Name Type Description Default
label str

The label of the weight or the group.

 required
on Annotated[float, StateSpec(domain=FiniteEnumeratedSet([0, 1]))]

Whether the weight is +1.

 0.0

Examples:

>>> from skcomplex.models.computer_science.neural_networks import BinaryWeightElement
>>> BinaryWeightElement('weight', on=1.0).on
1.0
Source code in src/skcomplex/models/computer_science/neural_networks/_binary_weight.py 
30
31
32
def __init__(self, label: str, on: Annotated[float, StateSpec(domain=FiniteEnumeratedSet([0, 1]))] = 0.0) -> None:
    super().__init__(label)
    self.on = on

DilutedNeuronElement(label, visual=None, *, spin=1.0, pattern=1.0, beta=0.06)

Bases: Element

Binary neuron of an asymmetric diluted Hopfield network, updated in parallel with the others at every step.

The logit choice of diluted_hopfield draws the value s = ±1 of the neuron with probability 1 / (1 + exp(-2 s β h)), where h = Σ_j J_ij s_j is its local field, so its mean is tanh(β h).

Parameters:

Name Type Description Default
label str

The label of the neuron or the group.

 required
visual VisualSpec | None

How the element is drawn, or None for default_visual_.

 None
spin Annotated[float, StateSpec(domain=ZZ)]

The initial value, -1 or 1.

 1.0
pattern Annotated[float, ParameterSpec()]

The value of the first stored pattern at the neuron, whose overlap with the network is recorded.

 1.0
beta Annotated[float, ParameterSpec(domain=closed_open(0, oo))]

The inverse temperature β.

 0.06

Examples:

>>> from skcomplex.models.computer_science.neural_networks import DilutedNeuronElement
>>> DilutedNeuronElement('neuron').beta
0.06
Source code in src/skcomplex/models/computer_science/neural_networks/_diluted.py 
71
72
73
74
75
76
77
78
79
80
81
82
83
def __init__(
    self,
    label: str,
    visual: VisualSpec | None = None,
    *,
    spin: Annotated[float, StateSpec(domain=ZZ)] = 1.0,
    pattern: Annotated[float, ParameterSpec()] = 1.0,
    beta: Annotated[float, ParameterSpec(domain=RealSet.closed_open(0, oo))] = 0.06,
) -> None:
    super().__init__(label, visual=visual)
    self.spin = spin
    self.pattern = pattern
    self.beta = beta

HiddenUnitElement(label, visual=None, *, s=0.5, u=0.0, b=0.0)

Bases: Element

Hidden unit of a simple recurrent network, whose activity is a sigmoid of its recurrent input and its drive.

The drive u holds the weighted input of the previous step, and the bias b is adaptive, so training changes it.

Parameters:

Name Type Description Default
label str

The label of the element or group.

 required
visual VisualSpec | None

How the element is drawn, or None for default_visual_.

 None
s Annotated[float, StateSpec()]

The activity in (0, 1).

 0.5
u Annotated[float, StateSpec()]

The weighted input of the previous step.

 0.0
b Annotated[float, ParameterSpec(adaptive=True)]

The bias, an adaptive parameter.

 0.0

Examples:

>>> from skcomplex.models.computer_science.neural_networks import HiddenUnitElement
>>> HiddenUnitElement('unit').s
0.5
Source code in src/skcomplex/models/computer_science/neural_networks/_hidden_unit.py 
39
40
41
42
43
44
45
46
47
48
49
50
51
def __init__(
    self,
    label: str,
    visual: VisualSpec | None = None,
    *,
    s: Annotated[float, StateSpec()] = 0.5,
    u: Annotated[float, StateSpec()] = 0.0,
    b: Annotated[float, ParameterSpec(adaptive=True)] = 0.0,
) -> None:
    super().__init__(label, visual=visual)
    self.s = s
    self.u = u
    self.b = b

HopfieldUpdateInteraction(label, *, target)

Bases: MapInteraction

Asynchronous update of a Hopfield network, where every neuron in turn takes the sign of its weighted input.

The neurons update one after the other in a new random order every step, each reading the states the neurons before it set, so the energy -sum w_ij s_i s_j / 2 never increases. A neuron whose input is zero keeps its state.

Parameters:

Name Type Description Default
label str

The label of the interaction.

 required
target Annotated[str, RoleSpec()]

The path of the group of neurons.

 required

Examples:

>>> from skcomplex.models.computer_science.neural_networks import HopfieldUpdateInteraction
>>> HopfieldUpdateInteraction('update', target='neurons').check().schedule
'random-sequential'
Source code in src/skcomplex/models/computer_science/neural_networks/_hopfield_update.py 
35
36
37
38
39
40
41
42
43
def __init__(self, label: str, *, target: Annotated[str, RoleSpec()]) -> None:
    super().__init__(
        label,
        target=target,
        state='spin',
        function=self._get_spin,
        schedule='random-sequential',
        form='agent',
    )

InputCouplingInteraction(label, *, target, source, D=0.0)

Bases: CouplingInteraction

Weighted input of a simple recurrent network, which sets the drive u = D x of every hidden unit each step.

The weights D are an adaptive matrix with one row per hidden unit and one column per input unit, so training changes them. The coupling runs in the phase drive of the step, which the system declares before the phases hidden of the recurrence and readout of the readout, so the hidden units read the drive of the same step.

Parameters:

Name Type Description Default
label str

The label of the interaction, unique among the parts of its system.

 required
target Annotated[str, RoleSpec()]

The path of the group of hidden units.

 required
source Annotated[str, RoleSpec()]

The path of the group of input units.

 required
D Annotated[object, ParameterSpec(shape=(target, source), adaptive=True)]

The input weights, a matrix of shape (hidden units, input units) or a number for every entry.

 0.0

Examples:

>>> from skcomplex.models.computer_science.neural_networks import InputCouplingInteraction
>>> InputCouplingInteraction('input', target='hidden', source='inputs').write
'u'
Source code in src/skcomplex/models/computer_science/neural_networks/_input_coupling.py 
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
def __init__(
    self,
    label: str,
    *,
    target: Annotated[str, RoleSpec()],
    source: Annotated[str, RoleSpec()],
    D: Annotated[object, ParameterSpec(shape=('target', 'source'), adaptive=True)] = 0.0,
) -> None:
    super().__init__(
        label,
        target=target,
        source=source,
        read='x',
        write='u',
        function=self._input,
        mode='group',
        weights='D',
        form='map',
        phase='drive',
    )
    self.D = D

InputUnitElement(label, visual=None, *, x=0.0)

Bases: Element

Input unit of a simple recurrent network, whose value a dataset or a schedule sets at every step.

Parameters:

Name Type Description Default
label str

The label of the element or group.

 required
visual VisualSpec | None

How the element is drawn, or None for default_visual_.

 None
x Annotated[float, ParameterSpec()]

The input.

 0.0

Examples:

>>> from skcomplex.models.computer_science.neural_networks import InputUnitElement
>>> InputUnitElement('bit').x
0.0
Source code in src/skcomplex/models/computer_science/neural_networks/_input_unit.py 
31
32
33
34
35
36
37
38
39
def __init__(
    self,
    label: str,
    visual: VisualSpec | None = None,
    *,
    x: Annotated[float, ParameterSpec()] = 0.0,
) -> None:
    super().__init__(label, visual=visual)
    self.x = x

NeuronElement(label, visual=VisualSpec('agents', {'color_by': 'spin', 'color_range': (-1.0, 1.0)}), *, spin=1.0, weights=0.0)

Bases: Element

Binary neuron of a Hopfield network, which the update aligns with the sign of its weighted input.

The neuron sits on its node of a complete graph and holds the weights of its incoming links, one per node.

Parameters:

Name Type Description Default
label str

The label of the agent or group.

 required
visual VisualSpec | None

How the element is drawn, or None for default_visual_.

 VisualSpec('agents', {'color_by': 'spin', 'color_range': (-1.0, 1.0)})
spin Annotated[float, StateSpec(domain=ZZ)]

The initial state, -1 or 1.

 1.0
weights Annotated[object, StateSpec()]

The weights of the links from every node, an array with one value per node.

 0.0

Examples:

>>> from skcomplex.models.computer_science.neural_networks import NeuronElement
>>> group = GroupElement('neurons', element=NeuronElement, size=4, values={'spin': 1.0})
>>> group.label, group.element.__name__, group.size
('neurons', 'NeuronElement', 4)
Source code in src/skcomplex/models/computer_science/neural_networks/_hopfield.py 
55
56
57
58
59
60
61
62
63
64
65
def __init__(
    self,
    label: str,
    visual: VisualSpec | None = VisualSpec('agents', {'color_by': 'spin', 'color_range': (-1.0, 1.0)}),
    *,
    spin: Annotated[float, StateSpec(domain=ZZ)] = 1.0,
    weights: Annotated[object, StateSpec()] = 0.0,
) -> None:
    super().__init__(label, visual=visual)
    self.spin = spin
    self.weights = weights

OutputUnitElement(label, visual=None, *, y=0.5, c=0.0, target=0.0)

Bases: Element

Output unit of a simple recurrent network, whose activity predicts a target.

The bias c is adaptive, so training changes it, and the target is a parameter a dataset or a schedule sets.

Parameters:

Name Type Description Default
label str

The label of the element or group.

 required
visual VisualSpec | None

How the element is drawn, or None for default_visual_.

 None
y Annotated[float, StateSpec()]

The prediction in (0, 1).

 0.5
c Annotated[float, ParameterSpec(adaptive=True)]

The bias, an adaptive parameter.

 0.0
target Annotated[float, ParameterSpec()]

The value the prediction is compared with.

 0.0

Examples:

>>> from skcomplex.models.computer_science.neural_networks import OutputUnitElement
>>> OutputUnitElement('prediction').target
0.0
Source code in src/skcomplex/models/computer_science/neural_networks/_output_unit.py 
39
40
41
42
43
44
45
46
47
48
49
50
51
def __init__(
    self,
    label: str,
    visual: VisualSpec | None = None,
    *,
    y: Annotated[float, StateSpec()] = 0.5,
    c: Annotated[float, ParameterSpec(adaptive=True)] = 0.0,
    target: Annotated[float, ParameterSpec()] = 0.0,
) -> None:
    super().__init__(label, visual=visual)
    self.y = y
    self.c = c
    self.target = target

ReadoutCouplingInteraction(label, *, target, source, W=0.0)

Bases: CouplingInteraction

Readout of a simple recurrent network, y_i <- sigmoid(sum_j W_ij s_j + c_i) with the logistic sigmoid.

Every output unit reads the activities of the hidden units through the adaptive matrix W, adds its bias c, and takes the sigmoid of the sum as its next prediction.

Parameters:

Name Type Description Default
label str

The label of the interaction, unique among the parts of its system.

 required
target Annotated[str, RoleSpec()]

The path of the group of output units.

 required
source Annotated[str, RoleSpec()]

The path of the group of hidden units.

 required
W Annotated[object, ParameterSpec(shape=(target, source), adaptive=True)]

The readout weights, a matrix of shape (output units, hidden units) or a number for every entry.

 0.0

Examples:

>>> from skcomplex.models.computer_science.neural_networks import ReadoutCouplingInteraction
>>> ReadoutCouplingInteraction('readout', target='outputs', source='hidden').write
'y'
Source code in src/skcomplex/models/computer_science/neural_networks/_readout_coupling.py 
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
def __init__(
    self,
    label: str,
    *,
    target: Annotated[str, RoleSpec()],
    source: Annotated[str, RoleSpec()],
    W: Annotated[object, ParameterSpec(shape=('target', 'source'), adaptive=True)] = 0.0,
) -> None:
    super().__init__(
        label,
        target=target,
        source=source,
        read='s',
        write='y',
        function=self._activity,
        mode='group',
        weights='W',
        form='map',
        phase='readout',
        combine=self._activate,
    )
    self.W = W

RecurrenceCouplingInteraction(label, *, target, A=0.0)

Bases: CouplingInteraction

Recurrence of a simple recurrent network, s_i <- sigmoid(sum_j A_ij s_j + u_i + b_i) with the logistic sigmoid.

Every hidden unit reads the activities of every hidden unit, itself included, through the adaptive matrix A, adds its drive u and its bias b, and takes the sigmoid of the sum as its next activity (Gros 2015 Eq. 10.18).

Parameters:

Name Type Description Default
label str

The label of the interaction, unique among the parts of its system.

 required
target Annotated[str, RoleSpec()]

The path of the group of hidden units.

 required
A Annotated[object, ParameterSpec(shape=(target, target), adaptive=True)]

The recurrent weights, a square matrix with one row and one column per hidden unit, or a number for every entry.

 0.0

Examples:

>>> from skcomplex.models.computer_science.neural_networks import RecurrenceCouplingInteraction
>>> RecurrenceCouplingInteraction('recurrence', target='hidden').include_self
True
Source code in src/skcomplex/models/computer_science/neural_networks/_recurrence_coupling.py 
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
def __init__(
    self,
    label: str,
    *,
    target: Annotated[str, RoleSpec()],
    A: Annotated[object, ParameterSpec(shape=('target', 'target'), adaptive=True)] = 0.0,
) -> None:
    super().__init__(
        label,
        target=target,
        read='s',
        write='s',
        function=self._activity,
        mode='group',
        weights='A',
        include_self=True,
        form='map',
        phase='hidden',
        combine=self._activate,
    )
    self.A = A

ReservoirUnitElement(label, visual=None, *, r=0.0, w_in=1.0, leak=0.3, amplitude=1.0, frequency=0.2)

Bases: Element

Leaky tanh unit of an echo state network, with its activity and the weight, leak, and sine of its input.

The map of echo_state_network sets r(t + 1) = (1 - a) r + a tanh(sum_j W_ij r_j + w u(t)) with the input u(t) = A sin(f t).

Parameters:

Name Type Description Default
label str

The label of the unit or the group.

 required
visual VisualSpec | None

How the element is drawn, or None for default_visual_.

 None
r Annotated[float, StateSpec()]

The initial activity.

 0.0
w_in Annotated[float, ParameterSpec()]

The weight w of the input.

 1.0
leak Annotated[float, ParameterSpec(domain=open(0, oo))]

The leak rate a, 1 for units without memory of their own.

 0.3
amplitude Annotated[float, ParameterSpec()]

The amplitude A of the input.

 1.0
frequency Annotated[float, ParameterSpec()]

The angular frequency f of the input, per step.

 0.2

Examples:

>>> from skcomplex.models.computer_science.neural_networks import ReservoirUnitElement
>>> ReservoirUnitElement('unit').leak
0.3
Source code in src/skcomplex/models/computer_science/neural_networks/_echo_state.py 
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
def __init__(
    self,
    label: str,
    visual: VisualSpec | None = None,
    *,
    r: Annotated[float, StateSpec()] = 0.0,
    w_in: Annotated[float, ParameterSpec()] = 1.0,
    leak: Annotated[float, ParameterSpec(domain=RealSet.open(0, oo))] = 0.3,
    amplitude: Annotated[float, ParameterSpec()] = 1.0,
    frequency: Annotated[float, ParameterSpec()] = 0.2,
) -> None:
    super().__init__(label, visual=visual)
    self.r = r
    self.w_in = w_in
    self.leak = leak
    self.amplitude = amplitude
    self.frequency = frequency

diluted_hopfield(n_neurons=4000, *, connectivity=50.0, load=0.2, temperature_scale=3.0, overlap=0.3, random_state=0)

Return the asymmetric diluted Hopfield network, which recalls a stored pattern below its memory capacity.

The network stores p = load K random patterns in the couplings J_ij = C_ij Σ_μ ξ_i ξ_j, where every C_ij is 1 with probability K / N independently of C_ji, so the couplings are sparse and asymmetric. All neurons update together at the inverse temperature β = temperature_scale / K. Started with an overlap m with the first pattern, the overlap follows m(t + 1) = f(m(t)) for large K, which keeps a nonzero fixed point below the critical load 2 / π.

Parameters:

Name Type Description Default
n_neurons int

The number of neurons N.

 4000
connectivity float

The mean number K of inputs of a neuron.

 50.0
load float

The ratio p / K of the number of patterns to the connectivity.

 0.2
temperature_scale float

The product β K of the inverse temperature and the connectivity.

 3.0
overlap float

The initial overlap with the first pattern.

 0.3
random_state int | Generator | None

The seed or generator of the patterns, the dilution, and the initial values.

 0

Returns:

Name Type Description
system System

The model, with the values neurons.spin and the observable overlap.

Examples:

>>> from skcomplex.models.computer_science.neural_networks import diluted_hopfield
>>> diluted_hopfield(200, connectivity=10).elements[0].check().prepare().size_
200
Source code in src/skcomplex/models/computer_science/neural_networks/_diluted.py 
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
@register_model(
    field='computer_science/neural_networks',
    tags=('neural network', 'associative memory', 'spin glass', 'discrete map'),
    references=('Derrida, Gardner, and Zippelius 1987', 'Bertin 2021'),
    defaults={'until': 20, 'dt': 1},
    observables=lambda system: [ObservableSpec('overlap', _overlap, group='neurons')],
)
def diluted_hopfield(
    n_neurons: int = 4000,
    *,
    connectivity: float = 50.0,
    load: float = 0.2,
    temperature_scale: float = 3.0,
    overlap: float = 0.3,
    random_state: int | np.random.Generator | None = 0,
) -> System:
    """Return the asymmetric diluted Hopfield network, which recalls a stored pattern below its memory capacity.

    The network stores `p = load K` random patterns in the couplings `J_ij = C_ij Σ_μ ξ_i ξ_j`, where every `C_ij` is
    1 with probability `K / N` independently of `C_ji`, so the couplings are sparse and asymmetric. All neurons update
    together at the inverse temperature `β = temperature_scale / K`. Started with an overlap `m` with the first pattern,
    the overlap follows `m(t + 1) = f(m(t))` for large `K`, which keeps a nonzero fixed point below the critical load
    `2 / π`.

    Args:
        n_neurons:
            The number of neurons `N`.

        connectivity:
            The mean number `K` of inputs of a neuron.

        load:
            The ratio `p / K` of the number of patterns to the connectivity.

        temperature_scale:
            The product `β K` of the inverse temperature and the connectivity.

        overlap:
            The initial overlap with the first pattern.

        random_state:
            The seed or generator of the patterns, the dilution, and the initial values.

    Returns:
        system:
            The model, with the values `neurons.spin` and the observable `overlap`.

    Examples:
        >>> from skcomplex.models.computer_science.neural_networks import diluted_hopfield
        >>> diluted_hopfield(200, connectivity=10).elements[0].check().prepare().size_
        200
    """
    rng = np.random.default_rng(random_state)
    n_patterns = max(1, round(load * connectivity))
    patterns = rng.choice([-1.0, 1.0], size=(n_patterns, n_neurons))
    spins = np.where(rng.random(n_neurons) < (1 + overlap) / 2, patterns[0], -patterns[0])
    neurons = GroupElement(
        'neurons',
        element=DilutedNeuronElement,
        size=n_neurons,
        values={
            'spin': spins,
            'pattern': patterns[0],
            'beta': temperature_scale / connectivity,
        },
    )
    glauber = ChoiceInteraction(
        'glauber',
        target='neurons',
        state='spin',
        options=(-1.0, 1.0),
        utility=functools.partial(_alignment, couplings=_couplings(patterns, connectivity, rng)),
    )
    return System(elements=[neurons], interactions=[glauber])

echo_state_network(n_units=200, *, spectral_radius=0.9, leak=0.3, connectivity=0.1, input_scaling=1.0, amplitude=1.0, frequency=0.2, random_state=0)

Return an echo state network, a reservoir of leaky tanh units with random recurrent weights driven by a sine.

The recurrent weights are sparse and Gaussian, scaled to a spectral radius below 1, so the reservoir has the echo state property: its activities forget their initial values and become a function of the input history alone. A readout trained on these activities, as skcomplex.analysis.ReservoirAnalysis does, turns the reservoir into a forecaster.

Parameters:

Name Type Description Default
n_units int

The number of units.

 200
spectral_radius float

The largest modulus of the eigenvalues of the recurrent weights.

 0.9
leak float

The leak rate of every unit, 1 for units without memory of their own.

 0.3
connectivity float

The share of the recurrent weights that are not zero.

 0.1
input_scaling float

The scale of the input weights, drawn uniformly from [-input_scaling, input_scaling].

 1.0
amplitude float

The amplitude of the sine input.

 1.0
frequency float

The angular frequency of the sine input, per step.

 0.2
random_state int | Generator | None

The seed or generator of the weights and of the initial activities.

 0

Returns:

Name Type Description
system System

The model, with the group units of activities r, one per unit.

Examples:

>>> from skcomplex.models.computer_science.neural_networks import echo_state_network
>>> echo_state_network(50).elements[0].check().prepare().size_
50
Source code in src/skcomplex/models/computer_science/neural_networks/_echo_state.py 
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
@register_model(
    field='computer_science/neural_networks',
    tags=('neural network', 'reservoir computing', 'echo state', 'discrete map'),
    references=('Jaeger 2001', 'Cestnik and Martens 2025'),
    defaults={'until': 200, 'dt': 1},
)
def echo_state_network(
    n_units: int = 200,
    *,
    spectral_radius: float = 0.9,
    leak: float = 0.3,
    connectivity: float = 0.1,
    input_scaling: float = 1.0,
    amplitude: float = 1.0,
    frequency: float = 0.2,
    random_state: int | np.random.Generator | None = 0,
) -> System:
    """Return an echo state network, a reservoir of leaky tanh units with random recurrent weights driven by a sine.

    The recurrent weights are sparse and Gaussian, scaled to a spectral radius below 1, so the reservoir has the echo
    state property: its activities forget their initial values and become a function of the input history alone. A
    readout trained on these activities, as `skcomplex.analysis.ReservoirAnalysis` does, turns the reservoir
    into a forecaster.

    Args:
        n_units:
            The number of units.

        spectral_radius:
            The largest modulus of the eigenvalues of the recurrent weights.

        leak:
            The leak rate of every unit, 1 for units without memory of their own.

        connectivity:
            The share of the recurrent weights that are not zero.

        input_scaling:
            The scale of the input weights, drawn uniformly from `[-input_scaling, input_scaling]`.

        amplitude:
            The amplitude of the sine input.

        frequency:
            The angular frequency of the sine input, per step.

        random_state:
            The seed or generator of the weights and of the initial activities.

    Returns:
        system:
            The model, with the group `units` of activities `r`, one per unit.

    Examples:
        >>> from skcomplex.models.computer_science.neural_networks import echo_state_network
        >>> echo_state_network(50).elements[0].check().prepare().size_
        50
    """
    rng = np.random.default_rng(random_state)
    weights = _recurrent_weights(n_units, connectivity, spectral_radius, rng)
    units = GroupElement(
        'units',
        element=ReservoirUnitElement,
        size=n_units,
        values={
            'r': rng.uniform(-1.0, 1.0, n_units),
            'w_in': rng.uniform(-input_scaling, input_scaling, n_units),
            'leak': leak,
            'amplitude': amplitude,
            'frequency': frequency,
        },
    )
    reservoir = MapInteraction(
        'reservoir',
        target='units',
        state='r',
        function=functools.partial(_leaky_tanh, weights=weights),
    )
    return System(elements=[units], interactions=[reservoir])

elman_network(n_hidden=4, *, n_bits=3000, window=12, batch_size=16, learning_rate=0.05, truncation=None, random_state=0)

Return Elman's simple recurrent network, which learns to predict the next bit of a temporal XOR stream.

The nested discrete map net holds one input unit, n_hidden hidden units, and one output unit. The input sets the drive u = D x of the hidden units, the hidden units take s <- sigmoid(A s + u + b) (Gros 2015 Eq. 10.18), and the output predicts y <- sigmoid(W s + c), in three phases of every step, so the prediction of a step reads its input. The training interaction training trains D, A, W, b, and c with Adam on the squared error of predicting the next input from the inputs so far, over windows of window bits that start at every third bit of a stream in which every third bit is the exclusive or of the two before it. Only those bits can be predicted, so the metric xor_error falls toward 0 while random_error, the error on the other bits, stays near 0.25. One step of the system is one epoch.

Parameters:

Name Type Description Default
n_hidden int

The number of hidden units.

 4
n_bits int

The length of the stream.

 3000
window int

The number of bits of a sequence.

 12
batch_size int

The number of sequences of a batch.

 16
learning_rate float

The learning rate of Adam.

 0.05
truncation int | None

The number of steps the gradient flows back through the hidden activities, 1 for Elman's original rule that holds the previous activities as constant context, or None for full backpropagation through time.

 None
random_state int | Generator | None

The seed or generator of the stream and of the initial weights.

 0

Returns:

Name Type Description
system System

The model, with the nested system net and the training interaction training.

Examples:

>>> from skcomplex.models.computer_science.neural_networks import elman_network
>>> [item.label for item in elman_network().systems]
['net']
Source code in src/skcomplex/models/computer_science/neural_networks/_elman.py 
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
@register_model(
    field='computer_science/neural_networks',
    tags=('neural network', 'recurrent network', 'learning', 'backpropagation through time', 'discrete map'),
    references=('Gros 2015 §10.5.1', 'Elman 1990'),
    defaults={'until': 200, 'dt': 1},
)
def elman_network(
    n_hidden: int = 4,
    *,
    n_bits: int = 3000,
    window: int = 12,
    batch_size: int = 16,
    learning_rate: float = 0.05,
    truncation: int | None = None,
    random_state: int | np.random.Generator | None = 0,
) -> System:
    """Return Elman's simple recurrent network, which learns to predict the next bit of a temporal XOR stream.

    The nested discrete map `net` holds one input unit, `n_hidden` hidden units, and one output unit. The input sets
    the drive `u = D x` of the hidden units, the hidden units take `s <- sigmoid(A s + u + b)` (Gros 2015 Eq. 10.18),
    and the output predicts `y <- sigmoid(W s + c)`, in three phases of every step, so the prediction of a step reads
    its input. The training interaction `training` trains `D`, `A`, `W`, `b`, and `c` with Adam on the squared error
    of predicting the next input from the inputs so far, over windows of `window` bits that start at every third bit of
    a stream in which every third bit is the exclusive or of the two before it. Only those bits can be predicted, so
    the metric `xor_error` falls toward 0 while `random_error`, the error on the other bits, stays near 0.25. One step
    of the system is one epoch.

    Args:
        n_hidden:
            The number of hidden units.

        n_bits:
            The length of the stream.

        window:
            The number of bits of a sequence.

        batch_size:
            The number of sequences of a batch.

        learning_rate:
            The learning rate of Adam.

        truncation:
            The number of steps the gradient flows back through the hidden activities, 1 for Elman's original rule
            that holds the previous activities as constant context, or `None` for full backpropagation through time.

        random_state:
            The seed or generator of the stream and of the initial weights.

    Returns:
        system:
            The model, with the nested system `net` and the training interaction `training`.

    Examples:
        >>> from skcomplex.models.computer_science.neural_networks import elman_network
        >>> [item.label for item in elman_network().systems]
        ['net']
    """
    rng = np.random.default_rng(random_state)
    data = _dataset(_temporal_xor(n_bits, rng), window)
    net = System(
        'net',
        elements=[
            GroupElement('inputs', element=InputUnitElement, size=1),
            GroupElement('hidden', element=HiddenUnitElement, size=n_hidden),
            GroupElement('outputs', element=OutputUnitElement, size=1),
        ],
        interactions=[
            InputCouplingInteraction(
                'input',
                target='hidden',
                source='inputs',
                D=rng.normal(0.0, WEIGHT_SCALE, (n_hidden, 1)),
            ),
            RecurrenceCouplingInteraction(
                'recurrence',
                target='hidden',
                A=rng.normal(0.0, WEIGHT_SCALE, (n_hidden, n_hidden)),
            ),
            ReadoutCouplingInteraction(
                'readout',
                target='outputs',
                source='hidden',
                W=rng.normal(0.0, WEIGHT_SCALE, (1, n_hidden)),
            ),
        ],
        time=DiscreteTime(schedule=PhasesScheduleSpec(PHASES)),
    )
    training = TrainingInteraction(
        'training',
        model='net',
        trained=TRAINED,
        loss=ObservableSpec('error', group='outputs', reduce='sum', expression=_squared_error),
        data=data,
        learning_rate=learning_rate,
        batch_size=batch_size,
        truncation=truncation,
        metrics=(
            ObservableSpec('xor_error', group='outputs', reduce='sum', expression=_squared_error),
            ObservableSpec('random_error', group='outputs', reduce='sum', expression=_squared_error),
        ),
    )
    return System('elman', systems=[net], interactions=[training], time=DiscreteTime())

hopfield(patterns=None, *, n_neurons=64, n_patterns=3, noise=0.2, random_state=0)

Return a Hopfield network that stores patterns with Hebb's rule and recalls the first from a corrupted copy.

Neurons update one at a time in random order, so the energy -Σ w_ij s_i s_j / 2 never increases and the network settles in a stored pattern when few patterns are stored, about fewer than 0.14 n for n neurons.

Parameters:

Name Type Description Default
patterns ArrayLike | None

The stored patterns of -1 and 1, one row per pattern, or None for random patterns.

 None
n_neurons int

The number of neurons of random patterns.

 64
n_patterns int

The number of random patterns.

 3
noise float

The probability that a neuron starts flipped from the first pattern.

 0.2
random_state int | Generator | None

The seed or generator of the random patterns and of the corruption.

 0

Returns:

Name Type Description
system System

The model, with the group neurons of states spin and weights on a complete graph.

Examples:

>>> from skcomplex.models.computer_science.neural_networks import hopfield
>>> hopfield(n_neurons=16, n_patterns=2).elements[0].check().prepare().size_
16
Source code in src/skcomplex/models/computer_science/neural_networks/_hopfield.py 
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
@register_model(
    field='computer_science/neural_networks',
    tags=('neural network', 'associative memory', 'agents', 'attractor'),
    references=('Hopfield 1982', 'Trzaska 2023'),
    defaults={'until': 10},
)
def hopfield(
    patterns: npt.ArrayLike | None = None,
    *,
    n_neurons: int = 64,
    n_patterns: int = 3,
    noise: float = 0.2,
    random_state: int | np.random.Generator | None = 0,
) -> System:
    """Return a Hopfield network that stores patterns with Hebb's rule and recalls the first from a corrupted copy.

    Neurons update one at a time in random order, so the energy `-Σ w_ij s_i s_j / 2` never increases and the network
    settles in a stored pattern when few patterns are stored, about fewer than `0.14 n` for `n` neurons.

    Args:
        patterns:
            The stored patterns of -1 and 1, one row per pattern, or `None` for random patterns.

        n_neurons:
            The number of neurons of random patterns.

        n_patterns:
            The number of random patterns.

        noise:
            The probability that a neuron starts flipped from the first pattern.

        random_state:
            The seed or generator of the random patterns and of the corruption.

    Returns:
        system:
            The model, with the group `neurons` of states `spin` and `weights` on a complete graph.

    Examples:
        >>> from skcomplex.models.computer_science.neural_networks import hopfield
        >>> hopfield(n_neurons=16, n_patterns=2).elements[0].check().prepare().size_
        16
    """
    rng = np.random.default_rng(random_state)
    stored: Array = np.asarray(
        rng.choice([-1.0, 1.0], size=(n_patterns, n_neurons)) if patterns is None else patterns,
        dtype=np.float64,
    )
    size = stored.shape[1]
    spins = np.where(rng.random(size) < noise, -stored[0], stored[0])
    neurons = GroupElement(
        'neurons',
        element=NeuronElement,
        size=size,
        values={'spin': spins, 'weights': _hebbian_weights(stored)},
    )
    return System(
        elements=[neurons],
        interactions=[HopfieldUpdateInteraction('update', target='neurons')],
        space=NetworkSpace(nx.complete_graph(size)),
    )

perceptron_satisfiability(n_weights=21, *, load=1.5, random_state=0)

Return a binary perceptron that must store random patterns, revealed one per step, as cardinality constraints.

A pattern xi of output sigma holds when sigma sum_i w_i xi_i > 0, so with an odd number N of weights w_i = +-1, when at least (N + 1) / 2 of the literals w_i = sigma xi_i hold. The patterns are satisfiable up to the critical load p / N of about 0.83 in the limit of many weights (Bertin 2021 §6.3.3, Ex. 6.4). The patterns are drawn from random_state, so a disorder average sweeps it through GridDesign.

Parameters:

Name Type Description Default
n_weights int

The number of weights, which is odd.

 21
load float

The number of patterns over the number of weights.

 1.5
random_state int | Generator | None

The seed or generator of the patterns.

 0

Returns:

Name Type Description
system System

The model, with the group weights and the reports decide.satisfiable and decide.revealed.

Raises:

Type Description
ValidationError

The number of weights is even, so a pattern can tie.

Examples:

>>> from skcomplex.models.computer_science.neural_networks import perceptron_satisfiability
>>> perceptron_satisfiability(3, load=1.0).interactions[0].constraints[0].bound
2
Source code in src/skcomplex/models/computer_science/neural_networks/_perceptron.py 
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
@register_model(
    field='computer_science/neural_networks',
    tags=('perceptron', 'satisfiability', 'disorder', 'critical load'),
    references=('Bertin 2021', 'Krauth and Mezard 1989'),
    defaults={'until': 31},
)
def perceptron_satisfiability(
    n_weights: int = 21,
    *,
    load: float = 1.5,
    random_state: int | np.random.Generator | None = 0,
) -> System:
    """Return a binary perceptron that must store random patterns, revealed one per step, as cardinality constraints.

    A pattern `xi` of output `sigma` holds when `sigma sum_i w_i xi_i > 0`, so with an odd number `N` of weights
    `w_i = +-1`, when at least `(N + 1) / 2` of the literals `w_i = sigma xi_i` hold. The patterns are satisfiable up to
    the critical load `p / N` of about 0.83 in the limit of many weights (Bertin 2021 §6.3.3, Ex. 6.4). The patterns are
    drawn from `random_state`, so a disorder average sweeps it through `GridDesign`.

    Args:
        n_weights:
            The number of weights, which is odd.

        load:
            The number of patterns over the number of weights.

        random_state:
            The seed or generator of the patterns.

    Returns:
        system:
            The model, with the group `weights` and the reports `decide.satisfiable` and `decide.revealed`.

    Raises:
        ValidationError:
            The number of weights is even, so a pattern can tie.

    Examples:
        >>> from skcomplex.models.computer_science.neural_networks import perceptron_satisfiability
        >>> perceptron_satisfiability(3, load=1.0).interactions[0].constraints[0].bound
        2
    """
    if n_weights % 2 == 0:
        error_msg = f'perceptron_satisfiability has n_weights={n_weights!r}. Use an odd number, so no pattern ties.'
        raise ValidationError(error_msg)
    rng = np.random.default_rng(random_state)
    n_patterns = round(load * n_weights)
    patterns = rng.choice([-1, 1], size=(n_patterns, n_weights)) * rng.choice([-1, 1], size=(n_patterns, 1))
    indices = np.arange(1, n_weights + 1)
    constraints = [
        CardinalitySpec(tuple((indices * pattern).tolist()), (n_weights + 1) // 2, 'at-least') for pattern in patterns
    ]
    weights = GroupElement('weights', element=BinaryWeightElement, size=n_weights)
    decide = SatisfiabilityInteraction('decide', target='weights', state='on', constraints=constraints)
    return System(elements=[weights], interactions=[decide], time=DiscreteTime())