Decision Tree is a classical Machine Learning model which as the name suggests has a hierarchical structure where every decision(or condition) either branches into two other decisions or is the final decision(output class in classification and value in regression). It is a non-linear model widely used when the number of data points are low and the number of features(variables) are not too high. Other complex ML models like random forest, gradient-boosted trees, xgboost, etc. are all derived from decision trees.

In this article, we will implement Decision Tree from scratch in Python and also understand the mathematical foundations behind it.