Image of Spatially autocorrelated training and validation samples inflate performance assessment of convolutional neural networks

Text

Spatially autocorrelated training and validation samples inflate performance assessment of convolutional neural networks

Deep learning and particularly Convolutional Neural Networks (CNN) in concert with remote sensing are becoming standard analytical tools in the geosciences. A series of studies has presented the seemingly outstanding performance of CNN for predictive modelling. However, the predictive performance of such models is commonly estimated using random cross-validation, which does not account for spatial autocorrelation between training and validation data. Independent of the analytical method, such spatial dependence will inevitably inflate the estimated model performance. This problem is ignored in most CNN-related studies and suggests a flaw in their validation procedure. Here, we demonstrate how neglecting spatial autocorrelation during cross-validation leads to an optimistic model performance assessment, using the example of a tree species segmentation problem in multiple, spatially distributed drone image acquisitions. We evaluated CNN-based predictions with test data sampled from 1) randomly sampled hold-outs and 2) spatially blocked hold-outs. Assuming that a block cross-validation provides a realistic model performance, a validation with randomly sampled holdouts overestimated the model performance by up to 28%. Smaller training sample size increased this optimism. Spatial autocorrelation among observations was significantly higher within than between different remote sensing acquisitions. Thus, model performance should be tested with spatial cross-validation strategies and multiple independent remote sensing acquisitions. Otherwise, the estimated performance of any geospatial deep learning method is likely to be overestimated.

Ketersediaan
18621.3678Perpustakaan BIG (Eksternal Harddisk)Tersedia
Informasi Detail
Judul Seri
ISPRS Open Journal of Photogrammetry and Remote Sensing
No. Panggil
621.3678
Penerbit
Amsterdam : Elsevier.,
Deskripsi Fisik
10 hlm PDF, 5.589 KB
Bahasa
Inggris
ISBN/ISSN
1872-8235
Klasifikasi
621.3678
Tipe Isi
text
Tipe Media
-
Tipe Pembawa
-
Edisi
Vol.5, August 2022
Subjek
Machine Learning
Deep learning
Convolutional neural networks
Spatial autocorrelation
Mapping
Reference data
Info Detail Spesifik
-
Pernyataan Tanggungjawab
Versi lain/terkait

Tidak tersedia versi lain

Lampiran Berkas
  • Spatially autocorrelated training and validation samples inflate performance assessment of convolutional neural networks
    Deep learning and particularly Convolutional Neural Networks (CNN) in concert with remote sensing are becoming standard analytical tools in the geosciences. A series of studies has presented the seemingly outstanding performance of CNN for predictive modelling. However, the predictive performance of such models is commonly estimated using random cross-validation, which does not account for spatial autocorrelation between training and validation data. Independent of the analytical method, such spatial dependence will inevitably inflate the estimated model performance. This problem is ignored in most CNN-related studies and suggests a flaw in their validation procedure. Here, we demonstrate how neglecting spatial autocorrelation during cross-validation leads to an optimistic model performance assessment, using the example of a tree species segmentation problem in multiple, spatially distributed drone image acquisitions. We evaluated CNN-based predictions with test data sampled from 1) randomly sampled hold-outs and 2) spatially blocked hold-outs. Assuming that a block cross-validation provides a realistic model performance, a validation with randomly sampled holdouts overestimated the model performance by up to 28%. Smaller training sample size increased this optimism. Spatial autocorrelation among observations was significantly higher within than between different remote sensing acquisitions. Thus, model performance should be tested with spatial cross-validation strategies and multiple independent remote sensing acquisitions. Otherwise, the estimated performance of any geospatial deep learning method is likely to be overestimated.
Komentar
Anda harus masuk sebelum memberikan komentar