One of the ways to determine HOW to break apart a decision tree is by using the Information Gain on that column. The idea is that it tells you how much you gain by splitting on that column. For example, if you split and ALL of the values in the left child have the same value in the Target column you don’t gain anything and your value is 0. If they are split in the middle you get a gain of 1.
The equation is:
This breaks down to:
Now that we have the ugly formula out of the way lets put in some actual numbers to determine how we want to split on this tree
Sample | Clouds | Sun | Rain |
1 | Yes | Yes | No |
2 | Yes | Yes | No |
3 | No | Yes | No |
4 | No | No | Yes |
5 | Yes | Yes | Yes |
6 | No | No | No |
7 | No | No | Yes |
8 | Yes | Yes | Yes |
9 | No | Yes | No |
10 | Yes | No | No |
In this case ‘Rain’ is our Target column. We have 6 no and 4 yes values. Our first step is to find the Entropy of Rain. We then need to find the Remainder of Clouds and Sun. Finally, we find the Information Gain and use that to break out tree.
Entropy of Rain
If we remember
We can now put in the actual numbers:
We know that the Entropy of Rain is 0.94.
On a side note, AND
.
Remainder of Cloud and Sun
If we remember the formula is
Let get some actual numbers and break down the formula.
Cloud | No | Yes |
No | 3 | 3 |
Yes | 2 | 2 |
Sun | No | Yes |
No | 2 | 4 |
Yes | 2 | 2 |
For the remainder we need to calculate the number of Yes/No for each of the Yes/No from Rain. These are the values in the grids above.
Lets start with Cloud:
WHERE comes from the number of No’s total.
comes from the number of No’s that are ALSO No’s in the Rain column. The second
from the number of Yes’s that are ALSO No’s in the Rain column.
comes from the number of Yes’s total.
comes from the number of No’s that are ALSO Yes’s in the Rain column. The second
from the number of Yes’s that are ALSO Yes’s in the Rain column.
Now, lets do Sun
WHERE comes from the number of No’s total.
comes from the number of No’s that are ALSO No’s in the Rain column.
from the number of Yes’s that are ALSO No’s in the Rain column.
comes from the number of Yes’s total.
comes from the number of No’s that are ALSO Yes’s in the Rain column. The second
from the number of Yes’s that are ALSO Yes’s in the Rain column.
Information Gain
Remember
So, Sun has the highest Information gain and that is how you should start your decision tree.
Now, I made this simple since I only had 2 columns but if you had more columns you would need to do this again using Sun as the Target and build off of that.