スキップしてメイン コンテンツに移動

Theorem of Logistic regression.

Introduction


Today, I will write about Logistic regression. Logistic regression is the basis of Machine Learning. Logistic regression is the model to classify two value

Overview

This post is written by using PRML for reference.
Optimization is used for Iterative reweighted least squares method.

  • First, I will introduce the sigmoid function
  • Second, I will define probability to classify
  • Third, I will write cross-entropy error function
  • Fourth, I will explain IRLS
Firstly, We define Sigmoid function.
sigmoid function is following.

\[\sigma(a) = \frac{1}{1+\exp(a)}\]

I will compute differential of this function.

\begin{eqnarray*} \frac{d}{d a} \frac{1}{1+\exp(a)} &=& \frac{\exp(-a)}{(1+\exp(-a))^2} \\ &=& \frac{1}{1+\exp(-a)} \frac{\exp(-a)}{1+\exp(-a)}\\ &=& \frac{1}{1+\exp(-a)} \{ \frac{1+\exp(-a)}{1+\exp(-a)} - \frac{1}{1+\exp(-a)} \} \\ &=& \sigma(a)(1-\sigma(a)) \end{eqnarray*}

This function is very important in terms in terms of Machine Learning.
Because Sigmoid function has the following form.
enter image description here
The sigmoid function has the following characteristic.
  • Sigmoid function is defined on \((-\infty,\infty)\)
  • and, range of y is (0,1).
  • The sigmoid function is monotonic increase function.
  • Sigmoid function is point-symmetry in (0,0.5)
We regard the value of the sigmoid function as the probability.

Probability of classifying

First, We think of probability that data is properly classified.
enter image description here

This line is a hyperplane.
a hyperplane is expressed as follows.

\[w^T x= 0\]

w is a normal vector of the hyperplane.
  • Data point exist red domain when \(w^T x > 0\)
  • Data point exist blue domain when \(w^T x < 0\)
If data is exists in red domain, the data is belong \(C_1\).
If data exists in the blue domain, the data is belonging \(C_2\)
\(C_1\) have 1 as a label. \(C_2\) have 0 as a label.

We can have confidence that I know the class of data exist far from the hyperplane, but we do not know the class of data when data point exists near hyperplane.

We want not to believe the class of data predicted near hyper plane.

Therefore!!, I use disposition of Sigmoid function. I change the distance from
the hyperplane to data point into probability.
  • I handle that data point exist red domain\((C_1)\) when the probability is higher than 0.5.
  • When probability has just 0.5, I do not know the class of data. (The data exist on the hyperplane)
  • I handle that data point exist blue domain \((C_2)\) when the probability is lower than 0.5.
I summarize this paragraph. I want to handle as follows.
  • The farther the distance between a data point and the hyperplane is the farther keep probability away from 0.5.
  • The nearer the distance between a data point and the hyperplane is, the nearer probability is 0.5.
  • When the probability is near 1, I have confidence that class of the data is \(C_1\)
  • When the probability is near 0, I have confidence that class of the data is \(C_2\)
I implement this way of thinking by using the sigmoid function.
Sigmoid function fulfill these way of thinking.

Thus, I handle probability value which output of sigmoid function by input distance between datapoint and hyperplane.

Cross-entropy error function

when Data set is \(\{\phi_n,t_n\}\), I define probability by using Sigmoid function.
here, \(t_n \in {0,1}\)
I define likelihood function as follows.
\[p(t|w) = \Pi_{n=1}^{N} y_{n}^{t_n} \{1-y_n\}^{1-t_n}\]
here,\[y_n = \sigma(w^T \phi_n) \]
\[t = (t_1,t_2,,,t_n)^T\]
\(w^T x\)is distance between \(\phi\) and hyper plane.
y_n is probability that \(\phi_n\) is \(C_1\)
I get negative logarithm about likelihood function.
\[E(w) = -\log p(t|w) = - \sum_{n=1}^{N} \{t_n \log(y_n) + (1-t_n ) \log(1-y_n)\}\]

\(E(w)\) is called Cross-entropy error function

IRLS

IRLS is iterative reweighted least squares method. This method estimate w by using the Newton-Raphson method.

I get gradient of E(w).
\begin{eqnarray*} \nabla E(w) &=& \nabla - \sum_{n=1}^{N} \{t_n \log(y_n) + (1-t_n ) \log(1-y_n)\}\\ &=& \nabla - \sum_{n=1}^{N} \{t_n \log(\sigma(w^T \phi_n) + (1-t_n ) \log(1-\sigma(w^T \phi_n)\}\\ &=&-\sum_{n=1}^{N} \{ \frac{t_n \sigma(w^T \phi_n)(1-\sigma(w^T \phi_n))}{\sigma(w^T \phi_n)}\phi_n - \frac{(1-t_n) \sigma(w^T \phi_n)(1-\sigma(w^T \phi_n))}{1-\sigma(w^T \phi_n)}\phi_n \}\\ &=& -\sum_{n=1}^{N} \frac{t_n (1-\sigma(w^T \phi_n))-(1-t_n)\sigma(w^T \phi_n)}{\sigma(w^T \phi_n) (1-\sigma(w^T \phi_n))} \{\sigma(w^T \phi_n) (1-\sigma(w^T \phi_n))\} \phi_n \\ &=& -\sum_{n=1}^{N} \{t_n (1-\sigma(w^T \phi_n))-(1-t_n)\sigma(w^T \phi_n)\} \phi_n \\ &=& -\sum_{n=1}^{N} \{t_n - t_n \sigma(w^T \phi_n) - \sigma(w^T \phi_n) - \sigma(w^T \phi_n) + t_n \sigma(w^T \phi)\} \phi_n\\ &=&\sum_{n=1}^{N} \{\sigma(w^T \phi_n) - t_n \} \phi_n\\ &=& \sum_{n=1}^{N} (y_n - t_n) \phi_n\\ &=& \phi^T (y - t)\\ \end{eqnarray*}
I get Hessian matrix of E(w)
\begin{eqnarray*} H &=& \nabla \nabla E(w) \\ &=& \nabla \phi^T(y-t) \\ &=& \nabla \sum_{n=1}^{N} \phi_{n}^T (y_n-t_n) \\ &=& \nabla\sum_{n=1}^{N} (y_n-t_n) \phi_{n}^T \\ &=&\sum_{n=1}^{N} y_n(1-y_n) \phi_n \phi^T = \phi^T R \phi \end{eqnarray*}
here, R is daiagonal matrix and have \(y_n(1-y_n)\) element of (n,n)
I use These form to estimate by Newton-Raphson method.
\begin{eqnarray*} w_{new} &=& w_{old} - \{ \phi^ T R \phi \}^ {-1} \phi^T (y-t) \\ &=& \{ \phi^ T R \phi \}^ T \{ \phi^ T R \phi w_{old} - \phi^T(y-t)\} \\ &=& \{ \phi^ T R \phi \} ^ {-1} \phi^ T Rz \end{eqnarray*}

Thus, w is updated by this form.
This post is Theory ed of Logistic Regression.
Next, I implement Logistic Regression.
I am glad to see my next post.

*I wrote the implementation of Logistic Regression.
Implementation of Logistic Regression

コメント

このブログの人気の投稿

Thorem of SVM part 2

Introduction 日本語 ver This post is written about theorem of SVM. This post is part 2. Part 1 is written about deriving the objective function. I will write about dual problem. The objective function which deriving in Part 1 is called the main problem. Typically, We optimize not the main problem, but the dual problem. I will write about deriving the dual problem from the main problem. If you until look up part 1, please look up Theorem of SVM part 1 . I implement SVM. It post is Implement linear SVM Implement kernel SVM Overview Main problem Dual problem  Lagurange function maximize L about dual variable, minimize L about primal variable minimize L about primal variable, maximize L about dual variable. Main problem I will review the main problem in Part 1. $$\min_{w,b} \frac{1}{2}||W||^2 + C\sum_{i \in N} \epsilon_i$$ $$~~s.t~~ \forall i \in N, y_i (w^T \phi(x_i) + b) \geq 1 - \epsilon_i ,~~~, \forall i \in N~\epsilon \geq 0$$ This problem is called s...

K-means 理論編

Introduction English ver 今日はK-meansアルゴリズムの理論について書きます。 K-meansアルゴリズムはクラスタリングのためのアルゴリズムです。 K-meansの実装の記事は カーネルK-meansの実装 を御覧ください。 この記事はカーネルK-menasの実装についての記事ですが、通常のK-meansの実装も行っています。カーネルK-meansについてはまた、今度別の記事で紹介したいと思います。 概要 1 of K 符号化法 プロトタイプ 歪み尺度 最適化 1 of K 符号化法 K-meansはK個のクラスについて分類することを考えます。 K-meansでは $x_n$がkのクラスに属していることを次のように表します。 ベクトル$r_n:1 \times K$ を $$r_n := (0,0,..,1,..,0)$$ このベクトルはk番目にのみ1を持ち、それ以外は0を要素に持つようなベクトルです。 こののような表現の仕方を1 of K符号化法と呼びます。 プロトタイプ K-meansではプロトタイプと呼ばれるベクトルを選びます。このベクトルは各クラスに一つあり、そのクラスの代表のようなベクトルです。 K-means ではそのようなベクトルは各クラスの平均ベクトルとなります。これは目的関数から自然と導かれます。 歪み尺度 プロトタイプベクトルを $\mu_i ~\forall k \in K$とします。 この時、k-meansの目的関数は次のようになります。 $$J = \sum_{n=1}^{N} \sum_{k=1}^{K} r_{nk} ||x_n-\mu_k||^2$$ ここで、 $r_{nk}$ は$r_n$のk番目の要素です。 この目的関数について少し説明をします。$r_{n}$は$x_n$が属しているクラスのラベルの場所だけ1で他は0であるので、 $$J = \sum_{n=1}^{N} ||x_n - \mu_{x_n}||$$ ここで、$\mu_{k_n}$は$x_n$が属しているクラスのプロトタイプです。 よって、 $$J = ||x_1 - \mu_{x_1}|| + ||x_2 -\mu_{x_2}|| + ...

Definition of Topology space

Introduction 日本語 ver Today, I will write about General Topology. General topology is very impossible for studying mathematics. General topology defines phase to define continuity of function. Today, I will explain phase. Also, This post is my review of General Topology. Overview  Distance space Axiom of phase Topology space Open set Distance space Firstly, Distance space is defined. Let X is set, $d:X\times X ->\mathbb{R}$, (X,d) is distance space $\iff$ $\forall x,y \in X,~~~~~d(x,y) \geq 0$ $\forall x,y \in X,~~~~~x = y \implies d(x,y) = 0$ $\forall x,y \in X,~~~~~d(x,y) = d(y,x)$ $\forall x,y,z \in X, ~~~~d(x,y) + d(y,z) \geq d(x,z)$ this condition is called axiom of distance. d is called distance function. Topology  space Let (X,d) is distance space. $\mathbb{O} \in 2^X$ is phase in (X,d) $\iff$ $\phi,X \in \mathbb{O}$ $\forall O_1,O_2 \in \mathbb{O} \implies O_1 \cap O_2 \in \mathbb{O}$ $\forall \Lambda ,~~\forall \{O_\l...