1
Assessing transferability of ecological models: an underappreciated aspect of statistical validation.
Methods Ecol Evol. 2012 Apr; 3(2):260-7
All selected articles haveThis article has been added to your saved articles
You can follow/unfollow articles via MyF1000 or via the article or browse pages.
15 Jun 2012
Technical Advance
DOI: 10.3410/f.716597813.792002812
Wenger and Olden point out that the usual practices for finding 'best' species distribution models, based on various measures of within-sample predictive accuracy, fail to incorporate spatial variation -- and thus do a poor job assessing out-of-sample accuracy. The authors show that allowing for spatial...Continue reading
Wenger and Olden point out that the usual practices for finding 'best' species distribution models, based on various measures of within-sample predictive accuracy, fail to incorporate spatial variation -- and thus do a poor job assessing out-of-sample accuracy. The authors show that allowing for spatial variability can provide a better assessment of transferability (a form of out-of-sample accuracy).
This lesson, delivered with a good deal of common sense, reminds us more generally that model selection and assessment tools can never be magic bullets, but must be tempered with an awareness of the scope and limitations of the data.
Researchers frequently hope to identify models with good out-of-sample predictive power. However, model evaluation typically uses only (penalized or corrected forms of) in-sample prediction error, which only correctly assesses out-of-sample accuracy under ideal circumstances when the data are sampled from homogeneous conditions. While cross-validation and information theoretic metrics provide powerful, automated methods for detecting overfitted models, they make strong implicit assumptions about the homogeneity of the data. The narrow message of this paper is that incorporating spatial heterogeneity in cross-validation techniques can improve transferability; the broader message is that overestimating the quality of models is human nature, and can be minimized but never eliminated by using better tools.
This lesson, delivered with a good deal of common sense, reminds us more generally that model selection and assessment tools can never be magic bullets, but must be tempered with an awareness of the scope and limitations of the data.
Researchers frequently hope to identify models with good out-of-sample predictive power. However, model evaluation typically uses only (penalized or corrected forms of) in-sample prediction error, which only correctly assesses out-of-sample accuracy under ideal circumstances when the data are sampled from homogeneous conditions. While cross-validation and information theoretic metrics provide powerful, automated methods for detecting overfitted models, they make strong implicit assumptions about the homogeneity of the data. The narrow message of this paper is that incorporating spatial heterogeneity in cross-validation techniques can improve transferability; the broader message is that overestimating the quality of models is human nature, and can be minimized but never eliminated by using better tools.
Disclosures
None declared
Add Comment
Bolker B and McCoy M: F1000Prime Recommendation of [Wenger SJ and Olden JD, Methods Ecol Evol 2012, 3(2):260-7]. In F1000Prime, 15 Jun 2012; DOI: 10.3410/f.716597813.792002812. F1000Prime.com/716597813#eval792002812
F1000Prime Recommendations, Dissents and Comments for [Wenger SJ and Olden JD, Methods Ecol Evol 2012, 3(2):260-7]. In F1000Prime, 24 May 2013; F1000Prime.com/716597813
