2024-11-27
If you use neural networks, you may not realize that these were among the earliest forms of machine structures studied as interest in computing grew in the 1940s and 1950s. In the 1940s, there was growing interest in cybernetic systems. This early work attempted to frame such systems as deterministic machines that are guided by feedback derived over the course of operation. The most obvious of such systems is your thermostat. When you set the temperature of your thermostat governing your heating unit, you will find that so long as the measured temperature is below the temperature you have set, the heat will remain on. Once the temperature reaches the target, the heater turns off. Then, when the detected temperature falls below the target, the heater turns back on, and so on.
One of the great innovations of early research modeling the nervous system was to envision the nervous system as a deterministic machine. In their path-breaking work, McCulloch and Pitts modeled the nervous system so that knowledge of the system at time $t$ translated to knowledge of the future path of the system absent external interventions. Although researchers then and since are well aware that the interpretation of McCulloch and Pitts was a dimensionally reduced view of the system, their contribution allowed for the articulation of the nature of information transmission across and of learning by neural nets. And, conveniently for cyberneticists out of which grew artificial intelligence research, the McCulloch-Pitts neural nets was a Turing machine. Marvin Minksy reflects on the significance of the work of McCulloch and Pitts over two decades later:
It should be understood clearly that neither McCulloch, Pitts, nor the present writer considers these devices and machines to serve as accurate physiological models of nerve cells and tissues. They were not designed with that purpose in mind. They are designed for the representation and analysis of the logic of situations that arise in any discrete process, be it in brain, computer, or anywhere else. In theories which are more seriously intended to be brain models, the 'neurons' have to be much more complicated. The real biological neuron is much more complex than our simple logical units -- for the evolution of nerve cells has led to very intricate and specialized organs. At this point, the McCulloch-Pitts 'cells' or 'neurons' are quite sufficient for our purposes. Our present goal is only to indicate how, starting with a set of very simple elements, one can construct machines of all sorts. (Minsky 1967, 32)
Later in the same chapter, Minsky provides examples of programs that can be straight-forwardly accomplished using neural nets. For example, binary counting can be accomplished when nodes in the chain represent the digit (that is, $2^0$, $2^1$, . . . $2^k$). For each count of some node ($2^j$) that moves that node from $1$ to $0$, a signal is sent to the next node ($2^{j+1}$) to change its state from $0$ to $1$. Values from each neuron are transmitted in the last step to a final node that aggregates the inputs from the nodes to a single value. While this may seem like a pedantic exercise, simple models like this were critical in elaborating programs (machines) that could be implemented by a general Turing machine.
Building McCulloch and Pitts Neurons
In application, what we refer to as the McCulloch and Pitts neuron (which is simpler than the cases presented by the authors) receives an input vector $\boldsymbol{s}$ of length $k$. Each element in the vector may be a $1$ or a $0$. Each receptor $i$ that receives each $s_i$ is subject to a weight, $w_i$. Thus, adjacent the vector $\boldsymbol{s}$ is the vector $\boldsymbol{w}$. If the sum of these weighted values are greater than the threshold value, $T$, then the gate is opened and the neuron is activated.
For the purpose of learning about neural networks that are part of the modern repertoire, it is helpful to build a very simple neuron, which we can do with python script (I must express my appreciation for Introduction to Neural Network Models of Cognitions, a hands-on text that elaborates theory using examples in python).
The neuron that I build below is conveyed by a plot of a networkx graph. The state of the neuron, $T$, is indicated in the title and using color. If the neuron fires, it is green. If it rests, it is red. If we do not know the state of $T$, its color is grey. Let's start by creating an explanatory visualization. After that, we will create data to simulate a neuron. Instead of elaborating in paragraph form, I have sufficiently noted the script and variables to make this script easy to interpret.
import networkx as nx
import matplotlib.pyplot as plt
def build_nn_graph(k, S, W):
G = nx.DiGraph()
G.add_node('T')
for i in range(k):
G.add_node(S[i], )
G.add_edge(S[i], 'T', label=W[i])
return G
def graph_nn(G, pass_treshold = None, label="McCulloch-Pitts Neuron"):
# set layout after creating nodes and edges
pos = nx.spring_layout(G)
# draw nodes and edges with labels
if pass_treshold == False:
# set T green if threshold passed
color_map = ['C3' if node == 'T' else 'C0' for node in G]
elif pass_treshold == True:
# set T red if threshold failed
color_map = ['C2' if node == 'T' else 'C0' for node in G]
else:
# color grey if we do not know if there is access or not
color_map = ['grey' if node == 'T' else 'C0' for node in G]
nx.draw(G, pos, node_color=color_map, with_labels=True, node_size = 2000, arrowsize=25)
edge_labels = nx.get_edge_attributes(G, 'label')
nx.draw_networkx_edge_labels(G, pos, edge_labels=edge_labels)
plt.title(label)
k = 3
template_G = build_nn_graph(k, S = [f"$s_{i}$" for i in range(k)],
W = [f"$w_{i}$" for i in range(k)])
graph_nn(template_G, None)
We see that each input, $s_i$, is intermediated by a weight, $w_i$. The aggregate value of the weighted values from $S$ determine whether or not the gate opens. If $S\cdot W$ meets or passes the threshold, $T$, the gate opens.
Let's create some values of our own to see how this works in application. Our first neuron fails to fire because the dot product fails to meet or exceed the threshold, $T$.
import numpy as np
import pandas as pd
def build_mp_neuron(k=3, random_seed=None):
if random_seed:
np.random.seed(random_seed)
# create a vector of length k composed of 0s and/or 1s
S = np.random.randint(0, 2, k)
# create a weights vector of length k comosed of floats between 0 and 1
W = np.random.rand(k).round(3)
return S, W
S, W = build_mp_neuron(k, random_seed = 4)
# let's view these in a dictionary
signal_and_weights_df = pd.DataFrame({'S': S, 'W': W})
# calculate values that will be added
signal_and_weights_df['S dot W'] = signal_and_weights_df['S'].mul(signal_and_weights_df['W'])
signal_and_weights_df
| S | W | S dot W | |
|---|---|---|---|
| 0 | 0 | 0.173 | 0.000 |
| 1 | 0 | 0.856 | 0.000 |
| 2 | 1 | 0.609 | 0.609 |
# threshold, T, could be any value less than k for the sake of this example
# set either at 1 or 2
# T = np.random.randint(1, k)
# or just choose 1 ...
T=1
# The dot product of S and W is the sum of the products.
# Let's just use the numpy dot function
D = np.dot(S, W)
flip_state = D >= T
graph_simulated_mp_neuron = build_nn_graph(k, S = [f"$S_{i} = {s}$" for i, s in enumerate(S)],
W = W)
firing = {True: "Firing", False: "Resting"}
graph_nn(graph_simulated_mp_neuron, flip_state, label=f"McCulloch-Pitts Neuron {firing[flip_state]}\nThreshold: {T}")
Now let's observe values that exceed the threshold. Since we used functions to construct, this is straight forward. Keep the same $T$ value and regenerate $S$ and $T$:
S, W = build_mp_neuron(k, random_seed=5)
signal_and_weights_df = pd.DataFrame({'S': S, 'W': W})
signal_and_weights_df['S dot W'] = signal_and_weights_df['S'].mul(signal_and_weights_df['W'])
signal_and_weights_df
| S | W | S dot W | |
|---|---|---|---|
| 0 | 1 | 0.831 | 0.831 |
| 1 | 0 | 0.364 | 0.000 |
| 2 | 1 | 0.979 | 0.979 |
D = np.dot(S, W)
flip_state = D >= T
graph_simulated_mp_neuron = build_nn_graph(k, S = [f"$S_{i} = {s}$" for i, s in enumerate(S)],
W = W)
graph_nn(graph_simulated_mp_neuron, flip_state, label=f"McCulloch-Pitts Neuron {firing[flip_state]}\nThreshold: {T}")
So far we the results are not terribly interesting. Either the threshold is passed or it is not. The next step is calibration. If the value of the weights adjust automatically, we can use feedback to improve the weights of the neuron for the purpose of predicting the identity of inputs. In abstract, this may seem obscure, but in the near future I will discuss Rosenblatt's development of the Perceptron which includes this feedback and adaptation and will discuss its significance for laying the foundations for modern neural network applications. Rosenblatt published his work on the Perceptron 15 years after the McCulloch-Pitts neuron. The next post in the series should be published sooner.