Editing Random forest (section)

==== Permutation importance ====

To measure a feature's importance in a data set <math>\mathcal{D}_n =\{(X_i, Y_i)\}_{i=1}^n</math>, first a random forest is trained on the data.  During training, the [[out-of-bag error]] for each data point is recorded and averaged over the forest. (If bagging is not used during training, we can instead compute errors on an independent test set.)

After training, the values of the feature are permuted in the out-of-bag samples and the out-of-bag error is again computed on this perturbed data set. The importance for the feature is computed by averaging the difference in out-of-bag error before and after the permutation over all trees.  The score is normalized by the standard deviation of these differences.

Features which produce large values for this score are ranked as more important than features which produce small values. The statistical definition of the variable importance measure was given and analyzed by Zhu ''et al.''<ref>{{cite journal | vauthors = Zhu R, Zeng D, Kosorok MR | title = Reinforcement Learning Trees | journal = Journal of the American Statistical Association | volume = 110 | issue = 512 | pages = 1770–1784 | date = 2015 | pmid = 26903687 | pmc = 4760114 | doi = 10.1080/01621459.2015.1036994 }}</ref>

This method of determining variable importance has some drawbacks:
* When features have different numbers of values, random forests favor features with more values. Solutions to this problem include [[partial permutation]]s<ref>{{cite conference | author=Deng, H.| author2=Runger, G. | author3=Tuv, E. | title=Bias of importance measures for multi-valued attributes and solutions | conference=Proceedings of the 21st International Conference on Artificial Neural Networks (ICANN) | year = 2011 | pages=293–300 | url = https://www.researchgate.net/publication/221079908 }}</ref><ref>{{cite journal | vauthors = Altmann A, Toloşi L, Sander O, Lengauer T | title = Permutation importance: a corrected feature importance measure | journal = Bioinformatics | volume = 26 | issue = 10 | pages = 1340–7 | date = May 2010 | pmid = 20385727 | doi = 10.1093/bioinformatics/btq134 | doi-access = free }}</ref><ref name=":02">{{Cite journal |last1=Piryonesi S. Madeh |last2=El-Diraby Tamer E. |date=2020-06-01 |title=Role of Data Analytics in Infrastructure Asset Management: Overcoming Data Size and Quality Problems |journal=Journal of Transportation Engineering, Part B: Pavements |volume=146 |issue=2 |page=04020022 |doi=10.1061/JPEODX.0000175 |s2cid=216485629}}</ref> and growing unbiased trees.<ref>{{cite journal | last1 = Strobl | first1 = Carolin | last2 = Boulesteix | first2 = Anne-Laure | last3 = Augustin | first3 = Thomas | name-list-style = vanc | title = Unbiased split selection for classification trees based on the Gini index | journal = Computational Statistics & Data Analysis | volume = 52 | year = 2007 | pages = 483–501 | url = https://epub.ub.uni-muenchen.de/1833/1/paper_464.pdf | doi = 10.1016/j.csda.2006.12.030 | citeseerx = 10.1.1.525.3178 }}</ref><ref>{{cite journal|last1=Painsky|first1=Amichai|last2=Rosset|first2=Saharon| name-list-style = vanc |title=Cross-Validated Variable Selection in Tree-Based Methods Improves Predictive Performance|journal=IEEE Transactions on Pattern Analysis and Machine Intelligence|date=2017|volume=39|issue=11|pages=2142–2153|doi=10.1109/tpami.2016.2636831|pmid=28114007|arxiv=1512.03444|s2cid=5381516}}</ref>
* If the data contain groups of correlated features of similar relevance, then smaller groups are favored over large groups.<ref>{{cite journal | vauthors = Tolosi L, Lengauer T | title = Classification with correlated features: unreliability of feature ranking and solutions | journal = Bioinformatics | volume = 27 | issue = 14 | pages = 1986–94 | date = July 2011 | pmid = 21576180 | doi = 10.1093/bioinformatics/btr300 | doi-access = free }}</ref>
* If there are collinear features, the procedure may fail to identify important features. A solution is to permute groups of correlated features together.<ref name=":2">{{Cite web |title=Beware Default Random Forest Importances |url=http://explained.ai/decision-tree-viz/index.html |access-date=2023-10-25 |website=explained.ai}}</ref>