CS479 Course Notes

Professor: Jeff Orchard | Lecture Videos

Hodgkin-Huxley Neuron Model

Goal: To see the basics of how a neuron works, in the form of the Hodgkin-Huxley neuron model.

A neuron is a special cell that can send and receive signals from other neurons. A neuron can be quite long, sending its signal over a long distance; up to 50m long. Most are shorter.

Middle of a neuron is the soma (body). The long stem is the axon, and the signal travels up the axon. Dendrites collect electrical signals from other neurons and send toward the soma.

Ions are molecules or atoms in which the number of electrons does not match the number of protons, resulting in a net charge. Several ions float around in cells. The cell’s membrane, a lipid bi-layer, stops most ions from crossing. Ion channels are embedded in the cell membrane can allow ions to pass.

Sodium-Potassium Pump exchanges $3\mathrm{Na}{}^{+}$ ions inside the cell for $2\mathrm{K}{}^{+}$ ions outside the cell. This causes a higher concentration of $\mathrm{Na}{}^{+}$ outside the cell, and higher concentration of $\mathrm{K}{}^{+}$ inside the cell. Also creates a net positive charge outside, and negative charge inside.

This difference in charge across the membrane induces a voltage difference called the membrane potential.

Neurons have a peculiar behaviour, they produce a spike of electrical activity called an action potential. The electrical burst travels along the neuron’s axon to its synapses, where it passes signals to other neurons.

Hodgkin-Huxley Model

Model of an action potential spike based on nonlinear interaction between membrane potential and the opening and closing of $\mathrm{Na}{}^{+}$ and $\mathrm{K}{}^{+}$ ion channels.

Let $v$ be the membrane potential. A neuron usually keeps a membrane potential of around $-70\mathrm{mV}$.

The fraction of $\mathrm{K}{}^{+}$ channels that are open is $n(t{)}^4$, where

$$\begin{array}{r}\frac{dn}{dt}=\frac{1}{{\tau }_n(v)}(n_{\infty }(v)-n)\end{array}$$

Dynamics of opening and closing depends on the voltage.

The fraction of $\mathrm{Na}{}^{+}$ ion channels open is $(m(t){)}^{3}h(t)$, where

$$\begin{array}{rl}\frac{dm}{dt}& =\frac{1}{{\tau }_m(v)}(m_{\infty }(v)-m)\\ \frac{dh}{dt}& =\frac{1}{{\tau }_h(v)}(h_{\infty }(v)-h)\end{array}$$

The two channels allow ions to flow into and out of the cell, inducing a current, which affects the membrane potential, $V$.

$$\begin{array}{r}\underset{\text{capacitance}}{\underbrace{c}}\frac{dV}{dt}=J_{in}-\stackrel{\text{leak current}}{\overbrace{\underset{\text{max conductance}}{\underbrace{g_L}}(V-V_L)}}-\stackrel{\mathrm{Na}{}^{+}\text{ current}}{\overbrace{g_{\mathrm{Na}}{m}^{3}h(V-V_{\mathrm{Na}})}}-\stackrel{\mathrm{K}{}^{+}\text{ current}}{\overbrace{g_{\mathrm{K}}{n}^4(V-V_{\mathrm{K}})}}\end{array}$$

This system of four differential equations govern the dynamics of the membrane potential.

Simpler Neuron Models

Goal: To look at other, less complicated neuron models.

The HH model is already simplified. A neuron is treated as a point in space, conductances are approximated with formulas. Only considers $\mathrm{K}{}^{+}$, $\mathrm{Na}{}^{+}$ and generic leak currents.

But to model a single action potential takes many time steps of this system. Spikes are fairly generic, and the presence of a spike is more important than its specific shape.

Leaky Integrate-and-Fire (LIF) Model

The LIF model only considers the sub-threshold membrane potential, but does not model the spike itself. It records when a spike occurs.

$$\begin{array}{rl}c\frac{dV}{dt}& =J_{in}-g_L(V-V_L)\text{Remember }{g}_L=\frac{1}{R}\\ \underset{{\tau }_m}{\underbrace{RC}}\frac{dV}{dt}& =\underset{\text{Ohm’s Law}}{\underbrace{R{J}_{in}}}-(V-V_L)\end{array}$$

So voltage can be modelled as

$$\begin{array}{r}{\tau }_m\frac{dV}{dt}=V_{in}-(V-V_L)\text{for }V<V_{th}\end{array}$$

We can change variables

$$\begin{array}{r}v=\frac{V-V_L}{V_{th}-V_L}\end{array}$$

Then, $v\to 0$ if $v_{in}=0$. And $v=1$ is the threshold.

We end up with

$$\begin{array}{r}{\tau }_m\frac{dv}{dt}=v_{in}-v\end{array}$$

We integrate the differential equation for a given input current until $v$ reaches the threshold value of 1. Then we record a spike at time $t_1$.

After it spikes, it remains dormant during its refractory period. Then it can start integrating again.

LIF Firing Rate

If we hold the input $v_{in}$ constant, we can solve the DE analytically between spikes.

Claim:

$$\begin{array}{r}v(t)=v_{in}\left(1-e^{-\frac{t}{\tau }}\right)\text{ is a solution of }\tau \frac{dv}{dt}=v_{in}-v,v(0)=0\end{array}$$

Proof: Plug in solution to the differential equation and show that LHS = RHS.

This solution approaches $v_{in}$ asymptotically. The interspike interval is the time between two spikes. It is split up into two parts, the refactor period, and the time it takes to go from $v=0\to v=1$.

It can be shown that the steady-state firing rate for a constant input $v_{in}$ is

$$G(v_{in})=\{\begin{array}{ll}\frac{1}{{\tau }_{\mathrm{ref}}-t_m\ln (1-\frac{1}{v_{in}})}& v_{in}>1\\ 0& v_{in}\le 1\end{array}$$

A tuning curve tells us the spikes per second for $v_{in}$.

Sigmoid Neuron

The activity of a neuron is low when the input is low, and activity goes up and approaches the maximum as the input increases.

This general behaviour can be represented by a number of different activation functions.

Logistic Curve

$$\begin{array}{r}\sigma (z)=\frac{1}{1+e^{-z}}\end{array}$$

Arctan

$$\begin{array}{r}\sigma (z)=\arctan (z)\end{array}$$

Hyperbolic Tangent

$$\begin{array}{r}\sigma (z)=\tanh (z)\end{array}$$

Threshold

$$\begin{array}{r}\sigma (z)=\{\begin{array}{ll}0& z<0\\ 1& z\ge 0\end{array}\end{array}$$

Rectified Linear Unit (ReLU)

$\mathrm{ReLU}(\mathrm{z})=\max (0,z)$

Multi-Neuron Activation Functions: Some activation functions depend on multiple neurons.

SoftMax

$\mathrm{SoftMax}$ is as probability distribution, so its elements add to 1. If $\vec{z}$ is the input to a set of neurons, then

$$\begin{array}{r}\mathrm{SoftMax}(\vec{z}{)}_i=\frac{e^{z_i}}{{\sum }_j{e}^{z_j}}\end{array}$$

So by definition, $\sum \mathrm{SoftMax}(\vec{z}{)}_i=1$.

One-Hot

One-Hot is the extreme of SoftMax, where only the largest element remains nonzero, while the others are set to zero.

Synapses

Goal: To get an overview of how neurons pass information between them, and how we can model those communication channels.

So far, we’ve just looked at individual neurons, and how they react to their input. But that input usually comes from other neurons. When a neuron fires an action potential, this travels along its axon.

The junction where one neuron communicates with the next neuron is called a synapse.

A pre-synaptic action potential $⟹$ causes release of neurotransmitter $⟹$ which binds to receptors on the past-synaptic neuron $⟹$ opening ion channels and changing the membrane potential.

Even though an action potential is very fast, the synaptic process by which it affects the next neuron takes time. Some synapses are fast, some are slow. If we represent that time constant using ${\tau }_s$, then the current entering the post-synaptic neuron can be written as

$$\begin{array}{r}h(t)=\{\begin{array}{ll}k{t}^n{e}^{-\frac{-t}{{\tau }_s}}& \text{if }t\ge 0\text{ for some }n\in {\mathbb{Z}}_{\ge 0}\\ 0& \text{if }t<0\end{array}\end{array}$$

where $k$ is chosen so that

$$\begin{array}{r}{\int }_0^{\infty }h(t)\text{d}t=1\to k=\frac{1}{n!{\tau }_s^{n+1}}\end{array}$$

This function $h(t)$ is called Post-Synaptic Current (PSC) filter, or Post-Synaptic Potential (PSP) filter.

Multiple spikes form what we call a “spike train”, and can be modelled as a sum of Dirac delta functions.

Example

$$\begin{array}{r}a(t)=\sum _{p=1}^3\delta (t-t_p)\end{array}$$

Dirac Delta function

$$\begin{array}{rl}& \delta (t)=\{\begin{array}{ll}\infty & \text{if }t=0\\ 0& \text{otherwise}\end{array}\\ & {\int }_{-\infty }^{\infty }\delta (t)=\text{d}t=1{\int }_{-\infty }^{\infty }f(t)\delta (T-t)\text{d}t=f(T)\end{array}$$

Question

How does a spike train influence the post-synaptic neuron?

You add together the PSC filters, one for each spike. This is convolving the spike train with the PSC filter.

Connection Weight

The total current induced by an action potential onto a particular post-synaptic neuron can vary widely, depending on the number and sizes of the synapses, the amount and type of neurotransmitter, the number and types of receptors etc.

We can combine all those functions into a single number, the connection weight. Thus, the total input to a neuron is a weighted sum of filtered spike-trains.

When we have many pre-synaptic neurons, it is more convenient to use matrix-vector notation to represent the weights and activities.

Suppose we have 2 populations, $X$ and $Y$. $X$ has $N$ nodes, $Y$ has $M$ nodes.

If every node sends its output to every node in $Y$, then we will have a total of $N\times M$ connections, each with its own weight.

flowchart LR

subgraph Y
direction LR
y1
y2
y3
end
x1 -- w11 --> y1
x1 -- w21 --> y2
x1 -- w31 --> y3
x2 -- w21 --> y1
x2 -- w22 --> y2
x2 -- w23 --> y3
subgraph X
direction LR
x1
x2
end

$$\begin{array}{r}\begin{array}{cc}{w}_{11}& w_{12}\\ w_{21}& w_{22}\\ w_{31}& w_{32}\end{array}=W\in {\mathbb{R}}^{M\times N}\end{array}$$