Neural Networks

From Ufldl

Revision as of 05:41, 26 February 2011 by Ang (Talk | contribs)
Jump to: navigation, search

Consider a supervised learning problem where we have access to labeled training examples (x(i),y(i)). Neural networks give a way of defining a complex, non-linear form of hypotheses hW,b(x), with parameters W,b that we can fit to our data.

To describe neural networks, we will begin by describing the simplest possible neural network, one which comprises a single "neuron." We will use the following diagram to denote a single neuron:

SingleNeuron.png

This "neuron" is a computational unit that takes as input x1,x2,x3 (and a +1 intercept term), and outputs h_{W,b}(x) = f(W^Tx) = f(\sum_{i=1}^3 W_{i}x_i +b), where f : \Re \mapsto \Re is called the activation function. In these notes, we will choose f(\cdot) to be the sigmoid function:


f(z) = \frac{1}{1+\exp(-z)}.

Thus, our single neuron corresponds exactly to the input-output mapping defined by logistic regression.

Although these notes will use the sigmoid function, it is worth noting that another common choice for f is the hyperbolic tangent, or tanh, function:


f(z) = \tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}},

Here are plots of the sigmoid and tanh functions:

{{#if:Sigmoid_Function.pngTanh_Function.png|

{{#if:|

{{{header}}}

}}{{#if:Sigmoid_Function.png|

Sigmoid activation function

{{#if:Sigmoid activation function|

Sigmoid activation function

}}

}}{{#if:Tanh_Function.png|

Tanh activation function
{{#if:Tanh activation function|
Tanh activation function

}}

}}{{#if:|

[[file:{{{image3}}}|{{#if:400 | 400 | {{{width3}}} }}px{{#ifeq::|:|||link={{{link3}}}}}|alt=|]]
{{#if:|
{{{caption3}}}

}}

}}{{#if:|

[[file:{{{image4}}}|{{#if:400 | 400 | {{{width4}}} }}px{{#ifeq::|:|||link={{{link4}}}}}|alt=|]]
{{#if:|
{{{caption4}}}

}}

}}{{#if:|

[[file:{{{image5}}}|{{#if:400 | 400 | {{{width5}}} }}px{{#ifeq::|:|||link={{{link5}}}}}|alt=|]]
{{#if:|
{{{caption5}}}

}}

}}{{#if:|

[[file:{{{image6}}}|{{#if:400 | 400 | {{{width6}}} }}px{{#ifeq::|:|||link={{{link6}}}}}|alt=|]]
{{#if:|
{{{caption6}}}

}}

}}{{#if:|

[[file:{{{image7}}}|{{#if:400 | 400 | {{{width7}}} }}px{{#ifeq::|:|||link={{{link7}}}}}|alt=|]]
{{#if:|
{{{caption7}}}

}}

}}{{#if:|

[[file:{{{image8}}}|{{#if:400 | 400 | {{{width8}}} }}px{{#ifeq::|:|||link={{{link8}}}}}|alt=|]]
{{#if:|
{{{caption8}}}

}}

}}{{#if:|

[[file:{{{image9}}}|{{#if:400 | 400 | {{{width9}}} }}px{{#ifeq::|:|||link={{{link9}}}}}|alt=|]]
{{#if:|
{{{caption9}}}

}}

}}{{#if:|

[[file:{{{image10}}}|{{#if:400 | 400 | {{{width10}}} }}px{{#ifeq::|:|||link={{{link10}}}}}|alt=|]]
{{#if:|
{{{caption10}}}

}}

}}{{#if:Two cards used by football referees|

Two cards used by football referees

}}

}}

Personal tools