Category: Tutoring

Information Gain in Decision Trees

One of the ways to determine HOW to break apart a decision tree is by using the Information Gain on that column. The idea is that it tells you how much you gain by splitting on that column. For example, if you split and ALL of the values in the left child have the same value in the Target column you don’t gain anything and your value is 0. If they are split in the middle you get a gain of 1.

The equation is:

IG(A) = Entrophy(Target-or-Prior) - Remainder(A)
This breaks down to:
Entropy(Target-or-Prior) = I(P_1,P_2,..P_n) = \sum_{n=1}^{n} -p_i log_2 p_i
Remainder(A) = \sum_{i=1}^{v}\frac{a_i + b_i}{a+b} I(\frac{a_i}{a_i + b_i};\frac{b_i}{a_i + b_i})

Now that we have the ugly formula out of the way lets put in some actual numbers to determine how we want to split on this tree

SampleCloudsSunRain
1YesYesNo
2YesYesNo
3NoYesNo
4NoNoYes
5YesYesYes
6NoNoNo
7NoNoYes
8YesYesYes
9NoYesNo
10YesNoNo

In this case ‘Rain’ is our Target column. We have 6 no and 4 yes values. Our first step is to find the Entropy of Rain. We then need to find the Remainder of Clouds and Sun. Finally, we find the Information Gain and use that to break out tree.

Entropy of Rain

If we remember I(P_1,P_2) = \sum_{i=1}^{n} -p_i log_2 p_i

We can now put in the actual numbers:

I(\frac{6}{10},\frac{4}{10}) = - \frac{6}{10} log_2 \frac{6}{10} + - \frac{4}{10} log_2 \frac{4}{10}

I(\frac{6}{10},\frac{4}{10}) = -0.6 * -0.7 + -0.4 * -1.3

I(\frac{6}{10},\frac{4}{10}) = 0.42 + 0.52

I(\frac{6}{10},\frac{4}{10}) = 0.94

We know that the Entropy of Rain is 0.94.

On a side note, I(0,1) = 0 AND I(\frac{1}{2},\frac{1}{2}) = 1.

Remainder of Cloud and Sun

If we remember the formula is Remainder(A) = \sum_{i=1}^{v}\frac{a_i + b_i}{a+b} I(\frac{a_i}{a_i + b_i};\frac{b_i}{a_i + b_i})

Let get some actual numbers and break down the formula.

CloudNoYes
No33
Yes22
SunNoYes
No24
Yes22

For the remainder we need to calculate the number of Yes/No for each of the Yes/No from Rain. These are the values in the grids above.

Lets start with Cloud:

Remainder(A) = \sum_{i=1}^{v}\frac{a_i + b_i}{a+b} I(\frac{a_i}{a_i + b_i},\frac{b_i}{a_i + b_i})

Remainder(Cloud) = \frac{6}{10}I(\frac{3}{6},\frac{3}{6})+\frac{4}{10}I(\frac{2}{4},\frac{2}{4})

WHERE \frac{6}{10} comes from the number of No’s total. \frac{3}{6} comes from the number of No’s that are ALSO No’s in the Rain column. The second \frac{3}{6} from the number of Yes’s that are ALSO No’s in the Rain column.

\frac{4}{10} comes from the number of Yes’s total. \frac{2}{4} comes from the number of No’s that are ALSO Yes’s in the Rain column. The second \frac{2}{4} from the number of Yes’s that are ALSO Yes’s in the Rain column.

Remainder(Cloud) = \frac{6}{10}(-\frac{3}{6} log_2 \frac{3}{6} + -\frac{3}{6} log_2 \frac{3}{6} )+\frac{4}{10}I(-\frac{2}{4} log_2 \frac{2}{4} + -\frac{2}{4} log_2 \frac{2}{4} )

Remainder(Cloud) = \frac{6}{10}(-.5 * -1 + -.5 * -1)+\frac{4}{10}I( -.5 * -1 + -.5 * -1 )

Remainder(Cloud) = \frac{6}{10}+\frac{4}{10}

Remainder(Cloud) = 1

Now, lets do Sun

Remainder(Sun) = \frac{6}{10}I(\frac{2}{6},\frac{4}{6})+\frac{4}{10}I(\frac{2}{4},\frac{2}{4})

WHERE \frac{6}{10} comes from the number of No’s total. \frac{2}{6} comes from the number of No’s that are ALSO No’s in the Rain column. \frac{4}{6} from the number of Yes’s that are ALSO No’s in the Rain column.

\frac{4}{10} comes from the number of Yes’s total. \frac{2}{4} comes from the number of No’s that are ALSO Yes’s in the Rain column. The second \frac{2}{4} from the number of Yes’s that are ALSO Yes’s in the Rain column.

Remainder(Sun) = \frac{6}{10}(-\frac{2}{6} log_2 \frac{2}{6} + -\frac{4}{6} log_2 \frac{4}{6} )+\frac{4}{10}I(-\frac{2}{4} log_2 \frac{2}{4} + -\frac{2}{4} log_2 \frac{2}{4} )

Remainder(Sun) = \frac{6}{10}(-.33 * -1.58 + -.66 * -.58)+\frac{4}{10}I( -.5 * -1 + -.5 * -1 )

Remainder(Sun) = \frac{6}{10}(.52 +.38)+\frac{4}{10}(1)

Remainder(Sun) = \frac{6}{10}(.9)+\frac{4}{10}(1)

Remainder(Cloud) = .54 + .4

Remainder(Cloud) = .94

Information Gain

Remember IG(A) = Entropy(Rain) - Remainder(A)

IG(Cloud) = 0.94 - 1 = -0.06

IG(Sun) = 0.94 - 0.94 = 0.0

So, Sun has the highest Information gain and that is how you should start your decision tree.

Now, I made this simple since I only had 2 columns but if you had more columns you would need to do this again using Sun as the Target and build off of that.

Master Theorem

Since I have started tutoring college level computer science I have had to relearn a lot of things that I haven’t used since college (both undergrad and masters). One of them is the Master’s Theorem that is used to analyze the running time for divide-and-conquer algorithms.

Every time I look at these I have to take a minute to remind myself how to determine the run times. So, this post is to handle just that.

First, the general form of the equations:

T(n) = aT(\frac{n}{b}) + f(n)

This has two parts. The first aT(\frac{n}{b}) is the sub problem where the algorithm does the divide-and-conquer. The second part f(n) shows the time it takes to recreate the problem.

There are 3 cases to determine the running time of this algorithm. Each of the is determined by the time it takes to run each part.

Determine the “cost” of each part of the equation:
To determine the sub problem speed you need to solve log_b a.
To determine the recreation you need to solve based on ‘c’ which is different for each case.

Case 1: f(n) = O(n^c)
Case 2: f(n) = O(n^c log^k n)
Case 3: f(n) = \Omega(n^c)

Case 1: When the work to combine the problem is dwarfed by the sub problem.

aT(\frac{n}{b}) > f(n)

Case 2: When the work to combine the problem is comparable to the sub problem.

aT(\frac{n}{b}) \approx f(n)

Case 3: When the work to combine the problem dominates the sub problem

aT(\frac{n}{b}) < f(n)

To start I am going to work through a single instance of each (from wikipedia) and then give multiple examples of each.

Case 1 Example:
T(n) = 8T(\frac{n}{2}) + 1000n^2
First, we need to determine the variables a, b, and c from f(n).
Here, a = 8 and b = 2.
For c, f(n) = 1000n^2. Since we know f(n) = O(n^c). So we get c=2.

For case 1 we need to have log_b a > c
Doing the math log_2 8= 3 which is greater than 2 from c. This confirms we are in case 1.
Using the formula T(n) = O(n^c) we get O(n^3)

Case 2 Example:
T(n) = 2T(\frac{n}{2}) + 10n
First, we need to determine the variables a, b, and c from f(n).
Here, a = 2 and b = 2.
For c, f(n) = 10n. Since we know f(n) = O(n^c log^k n). So we get c=1.

For case 2 we need to have log_b a \approx c
Doing the math log_2 2= 1 which is the same as 1 from c. This confirms we are in case 2.
Using the formula T(n) = O( n^c log^k n ) we get O(n log n)

Case 3 Example:
T(n) = 2T(\frac{n}{2}) + n^2
First, we need to determine the variables a, b, and c from f(n).
Here, a = 2 and b = 2.
For c, f(n) = n^2. Since we know f(n) = O(f(n)). So we get c=2

For case 3 we need to have log_b a < c
Doing the math log_2 2= 1 which is the same as 2 from c. This confirms we are in case 3.
Using the formula T(n) = O( f(n) ) we get O(n^2)

Samples:

Now, I am going to bang through a few examples of each case. Case 1 and 3 are pretty straight forward but we start getting some interesting cases in 2.

Each of these will be broken into 4 sections. The first column will be the formula. The second column will be the results of log_b a. The third column will be the value of c. The last column will be the notation.

These were pulled from Abdul Bari’s YouTube channel.

Case 1 Samples:
T(n) = 2T(\frac{n}{2}) + 1 -> log_b a = 1 -> c=0 -> O(n^1)
T(n) = 4T(\frac{n}{2}) + 1 -> log_b a = 1 -> c=0 -> O(n^2)
T(n) = 4T(\frac{n}{2}) + n -> log_b a = 2 -> c=1 -> O(n^2)
T(n) = 8T(\frac{n}{2}) + n^2 -> log_b a = 3 -> c=2 -> O(n^3)
T(n) = 16T(\frac{n}{2}) + n^2 -> log_b a = 4 -> c=2 -> O(n^4)

Case 3 Samples:
T(n) = T(\frac{n}{2}) + n -> log_b a = 0 -> c=1 -> O(n)
T(n) = 2T(\frac{n}{2}) + n^2 -> log_b a = 1 -> c=2 -> O(n^2)
T(n) = 2T(\frac{n}{2}) + n^2 log n -> log_b a = 1 -> c=2 -> O(n^2 log n)
T(n) = 4T(\frac{n}{2}) + n^3 log n -> log_b a = 2 -> c=3 -> O(n^3 log n)
T(n) = 2T(\frac{n}{2}) + \frac{n^2}{log n} -> log_b a = 1 -> c=2 -> O(n^2)

Case 2 Samples:
Remember, you are multiplying f(n) times log n.
T(n) = T(\frac{n}{2}) + 1 -> log_b a = 0 -> c=0 -> O(log n)
T(n) = 2T(\frac{n}{2}) + n -> log_b a = 1 -> c=1 -> O(n log n)
T(n) = 2T(\frac{n}{2}) + n log n -> log_b a = 1 -> c=1 -> O(n log^2 n)
T(n) = 4T(\frac{n}{2}) + n^2 -> log_b a = 2 -> c=2 -> O(n^2 log n)
T(n) = 4T(\frac{n}{2}) + (n log n)^2 -> log_b a = 2 -> c=2 -> O(n^2 log^3 n)
T(n) = 2T(\frac{n}{2}) + \frac{n}{log n} -> log_b a = 1 -> c=1 -> O(n log log n)
T(n) = 2T(\frac{n}{2}) + \frac{n}{log^2 n} -> log_b a = 1 -> c=1 -> O(n)

Conclusion:

I hope this helps anyone that is struggling through this stuff. After writing all this down with pen and paper it cleared it up for me.