CS479 Course Notes

Professor: Jeff Orchard | Lecture Videos

Hodgkin-Huxley Neuron Model

Goal: To see the basics of how a neuron works, in the form of the Hodgkin-Huxley neuron model.

A neuron is a special cell that can send and receive signals from other neurons. A neuron can be quite long, sending its signal over a long distance; up to 50m long. Most are shorter.

Middle of a neuron is the soma (body). The long stem is the axon, and the signal travels up the axon. Dendrites collect electrical signals from other neurons and send toward the soma.

Ions are molecules or atoms in which the number of electrons does not match the number of protons, resulting in a net charge. Several ions float around in cells. The cell’s membrane, a lipid bi-layer, stops most ions from crossing. Ion channels are embedded in the cell membrane can allow ions to pass.

Sodium-Potassium Pump exchanges $3\mathrm{Na}{}^{+}$ ions inside the cell for $2\mathrm{K}{}^{+}$ ions outside the cell. This causes a higher concentration of $\mathrm{Na}{}^{+}$ outside the cell, and higher concentration of $\mathrm{K}{}^{+}$ inside the cell. Also creates a net positive charge outside, and negative charge inside.

This difference in charge across the membrane induces a voltage difference called the membrane potential.

Neurons have a peculiar behaviour, they produce a spike of electrical activity called an action potential. The electrical burst travels along the neuron’s axon to its synapses, where it passes signals to other neurons.

Hodgkin-Huxley Model

Model of an action potential spike based on nonlinear interaction between membrane potential and the opening and closing of $\mathrm{Na}{}^{+}$ and $\mathrm{K}{}^{+}$ ion channels.

Let $v$ be the membrane potential. A neuron usually keeps a membrane potential of around $-70\mathrm{mV}$.

The fraction of $\mathrm{K}{}^{+}$ channels that are open is $n(t{)}^4$, where

$$\begin{array}{r}\frac{dn}{dt}=\frac{1}{{\tau }_n(v)}(n_{\infty }(v)-n)\end{array}$$

Dynamics of opening and closing depends on the voltage.

The fraction of $\mathrm{Na}{}^{+}$ ion channels open is $(m(t){)}^{3}h(t)$, where

$$\begin{array}{rl}\frac{dm}{dt}& =\frac{1}{{\tau }_m(v)}(m_{\infty }(v)-m)\\ \frac{dh}{dt}& =\frac{1}{{\tau }_h(v)}(h_{\infty }(v)-h)\end{array}$$

The two channels allow ions to flow into and out of the cell, inducing a current, which affects the membrane potential, $V$.

$$\begin{array}{r}\underset{\text{capacitance}}{\underbrace{c}}\frac{dV}{dt}=J_{in}-\stackrel{\text{leak current}}{\overbrace{\underset{\text{max conductance}}{\underbrace{g_L}}(V-V_L)}}-\stackrel{\mathrm{Na}{}^{+}\text{ current}}{\overbrace{g_{\mathrm{Na}}{m}^{3}h(V-V_{\mathrm{Na}})}}-\stackrel{\mathrm{K}{}^{+}\text{ current}}{\overbrace{g_{\mathrm{K}}{n}^4(V-V_{\mathrm{K}})}}\end{array}$$

This system of four differential equations govern the dynamics of the membrane potential.

Simpler Neuron Models

Goal: To look at other, less complicated neuron models.

The HH model is already simplified. A neuron is treated as a point in space, conductances are approximated with formulas. Only considers $\mathrm{K}{}^{+}$, $\mathrm{Na}{}^{+}$ and generic leak currents.

But to model a single action potential takes many time steps of this system. Spikes are fairly generic, and the presence of a spike is more important than its specific shape.

Leaky Integrate-and-Fire (LIF) Model

The LIF model only considers the sub-threshold membrane potential, but does not model the spike itself. It records when a spike occurs.

$$\begin{array}{rl}c\frac{dV}{dt}& =J_{in}-g_L(V-V_L)\text{Remember }{g}_L=\frac{1}{R}\\ \underset{{\tau }_m}{\underbrace{RC}}\frac{dV}{dt}& =\underset{\text{Ohm’s Law}}{\underbrace{R{J}_{in}}}-(V-V_L)\end{array}$$

So voltage can be modelled as

$$\begin{array}{r}{\tau }_m\frac{dV}{dt}=V_{in}-(V-V_L)\text{for }V<V_{th}\end{array}$$

We can change variables

$$\begin{array}{r}v=\frac{V-V_L}{V_{th}-V_L}\end{array}$$

Then, $v\to 0$ if $v_{in}=0$. And $v=1$ is the threshold.

We end up with

$$\begin{array}{r}{\tau }_m\frac{dv}{dt}=v_{in}-v\end{array}$$

We integrate the differential equation for a given input current until $v$ reaches the threshold value of 1. Then we record a spike at time $t_1$.

After it spikes, it remains dormant during its refractory period. Then it can start integrating again.

LIF Firing Rate

If we hold the input $v_{in}$ constant, we can solve the DE analytically between spikes.

Claim:

$$\begin{array}{r}v(t)=v_{in}\left(1-e^{-\frac{t}{\tau }}\right)\text{ is a solution of }\tau \frac{dv}{dt}=v_{in}-v,v(0)=0\end{array}$$

Proof: Plug in solution to the differential equation and show that LHS = RHS.

This solution approaches $v_{in}$ asymptotically. The interspike interval is the time between two spikes. It is split up into two parts, the refactor period, and the time it takes to go from $v=0\to v=1$.

It can be shown that the steady-state firing rate for a constant input $v_{in}$ is

$$G(v_{in})=\{\begin{array}{ll}\frac{1}{{\tau }_{\mathrm{ref}}-t_m\ln (1-\frac{1}{v_{in}})}& v_{in}>1\\ 0& v_{in}\le 1\end{array}$$

A tuning curve tells us the spikes per second for $v_{in}$.

Sigmoid Neuron

The activity of a neuron is low when the input is low, and activity goes up and approaches the maximum as the input increases.

This general behaviour can be represented by a number of different activation functions.

Logistic Curve

$$\begin{array}{r}\sigma (z)=\frac{1}{1+e^{-z}}\end{array}$$

Arctan

$$\begin{array}{r}\sigma (z)=\arctan (z)\end{array}$$

Hyperbolic Tangent

$$\begin{array}{r}\sigma (z)=\tanh (z)\end{array}$$

Threshold

$$\begin{array}{r}\sigma (z)=\{\begin{array}{ll}0& z<0\\ 1& z\ge 0\end{array}\end{array}$$

Rectified Linear Unit (ReLU)

$\mathrm{ReLU}(\mathrm{z})=\max (0,z)$

Multi-Neuron Activation Functions: Some activation functions depend on multiple neurons.

SoftMax

SoftMax is as probability distribution, so its elements add to 1. If $\vec{z}$ is the input to a set of neurons, then

$$\begin{array}{r}\mathrm{SoftMax}(\vec{z}{)}_i=\frac{e^{z_i}}{{\sum }_j{e}^{z_j}}\end{array}$$

So by definition, $\sum \mathrm{SoftMax}(\vec{z}{)}_i=1$.

One-Hot

One-Hot is the extreme of SoftMax, where only the largest element remains nonzero, while the others are set to zero.

Synapses