Predicting Food Security Outcomes Using CNNs for Satellite Tasking

Swetava Ganguli, Jared Dunnmon, Darren HauDepartment of Computer Science, 353 Serra Mall, Stanford, CA 94305

{swetava, jdunnmon, dhau}@cs.stanford.edu

Abstract

Obtaining reliable data describing local Food SecurityMetrics (FSM) at a granularity that is informative topolicy-makers requires expensive and logistically dif-ficult surveys, particularly in the developing world. Wetrain a CNN on publicly available satellite data describ-ing land cover classification and use both transfer learn-ing and direct training to build a model for FSM predic-tion purely from satellite imagery data. We then proposeefficient tasking algorithms for high resolution satel-lite assets via transfer learning, Markovian search algo-rithms, and Bayesian networks.

Introduction and BackgroundIn a recent paper (Neal Jean 2016), Convolutional Neu-ral Networks (CNNs) were trained to identify image fea-tures that explain up to 75% of the variation in local leveleconomic outcomes. This ability potentially enables policy-makers to measure need and impact without requiring theuse of expensive and logistically difficult local surveys((United Nations 2014), (World Bank 2015), (Devarajan2013) and (Jerven 2013)). This ability is promising for someAfrican countries, where data gaps may be exacerbated bypoor infrastructure and inadequate government resourcing.According to World Bank data, during the years 2000 to2010, 39 of 59 African countries conducted fewer than twosurveys from which nationally representative poverty mea-sures could be constructed. Of these countries, 14 conductedno such surveys during this period (World Bank 2015). Clos-ing these data gaps with more frequent household surveys islikely to be both prohibitively costly, perhaps costing hun-dreds of billions of U.S. dollars to measure every target ofthe United Nations Sustainable Development Goals in ev-ery country over a 15-year period ((Jerven 2014)) and in-stitutionally difficult, as some governments see little bene-fit in having their lackluster performance documented ((De-varajan 2013), (Sandefur 2015)). A popular recent approachleverages passively collected data like satellite images of lu-minosity at night (nightlights) to estimate economic activity((Henderson, Storeygard, and Weil 2011), (Chen and Nord-haus 2011), (Pinkovskiy 2017)).

Copyright c© 2019, Association for the Advancement of ArtificialIntelligence (www.aaai.org). All rights reserved.

Over the last several years, an increasing supply of satel-lite data has enabled a variety of important studies in ar-eas of environmental health, economics, and others (Safyan2015). The unique insight attainable from such datasets hascontinued to be integrated into the intelligence operationsof both public and private sector entities to the point that anumber of modern processes would not be possible in theirabsence (Meyer 2014). Additionally, the increasingly com-mon integration of hyperspectral sensors into commercialsatellite payloads has extended the types of conclusions thatcan be drawn from satellite imagery alone (Romero, Gatta,and Camps-Valls 2014). Recent developments deep CNNshave greatly advanced the performance of visual recogni-tion systems. In addition, using convolutional neural net-works and GANs on geospatial data in unsupervised orsemi-supervised settings has also been of interest recently;especially in domains such as food security, cybersecurity,satellite tasking, etc. ((Ganguli, Garzon, and Glaser 2019),(Dunnmon et al. 2019), (Perez et al. 2019)). In this pa-per, we demonstrate the use of CNNs for predicting foodsecurity metrics in poor countries in Africa using a com-bination of publicly available satellite imagery data fromthe work of (Saikat Basu 2015) (“DeepSat” dataset) andUN data made available by the Stanford Sustainability Lab(“SustLab” dataset).1 We also demonstrate the use of theseprediction models for tasking of relatively expensive, high-resolution satellites in a manner that maximizes the amountof imagery collected over food insecure areas. The task con-sidered in this paper is that of reaching the region of lowestfood security while traversing a path that is globally optimalin the sense that the most food-insecure regions possible aretraversed.

Our project thus consists of two main challenges: (i) de-signing a CNN that ingests satellite images as an input andoutputs successful predictions of key food security met-rics such as z-scores of stunting or wasting, and (ii) us-ing the food security predictions as rewards in a reinforce-ment learning process to train a satellite to automatically tra-verse areas of interest. The first task is a machine-learning-intensive task which requires using a large dataset of imagesto train a CNN model. We first utilize a publicly available

1Note that the DeepSat dataset is available online while theSustLab dataset is currently proprietary.

arX

iv:1

902.

0543

3v2

[cs

.CV

] 2

5 A

pr 2

019

Figure 1: A schematic describing the methodology for train-ing CNN model which acts as an input for the path searchalgorithms

labeled dataset of 28 x 28 image tiles at 1m resolution la-beled with one of six ground cover classifications to traina CNN focused on making predictions from overhead im-agery (Saikat Basu 2015). This trained network can be usedto extract features from satellite imagery, which can in turnbe used as the input to a ridge regression procedure aimedat accurately predicting food security metrics. Alternatively,we can also use both direct learning and transfer learningto enable direct prediction of food security metrics from apure CNN model, which will represent an interesting com-parison to the regression procedure. We choose to use con-volutional neural networks for this task due to their superiorperformance on image classification problems. Their perfor-mance generally results from effective integration of spatiallocality in convolutional features that aids accurate classi-fication predictions (Alex Krizhevsky 2012). We use a rela-tively simple VGG-style architecture in this work (Karen Si-monyan 2014).

The second task is using the output FSMs to define thestate space for a set of path search algorithms. The generaltask of training a model to change the orientation of asatellite can be modeled as the problem of an agent findingits way to a high terminal reward on a grid via a Markovianoptimal path. For brevity, we will chose demonstrativenumbers for easier explanation. Consider a very large image(e.g. 400 x 400 pixel image showing a 1000 X 1000 kmarea). Let us call our trained CNN H. We train the CNNon a dataset of 28 x 28 images (corresponding to 28 m x28 m areas) and regress on a dataset of values of a givenpoverty/food security metric so that when an image isprovided to the trained network, the output is a numberquantifying this metric. More precisely:

28 x 28 Resolution Image =⇒ H[Image] =⇒ Anumber corresponding to food security metric.

We can then tile the large 400 x 400 image into 332 12x 12 tiles, which are then upsampled to 28 x 28 to matchthe resolution of the original training set. For each tile, weuse our model to generate an associated food security metricwhich is treated as that tile’s reward in a search problem.Given this setup, we will sequentially tackle three linkedsub-problems that each add a layer of complexity and un-certainty to solve the complete problem. These sequentially

linked problems are described in the section titled “Auto-mated Satellite Tasking for Food Security Measurements”.A full diagram describing the setup of this problem can befound in Fig. 1.

Predicting Food Security Metrics with CNNsApproachOur first task is to effectively predict an FSM of interest di-rectly from satellite data using a CNN. There exist multiplepotential mechanisms by which to approach this problem,and we perform this task with the aim both of generating areward grid for satellite tasking as well as obtaining usefulclassification performance. The first approach, which repre-sents our baseline, entails using the pre-trained network of(Jean et al. 2016) to extract features directly from the Sust-Lab images and predict the relevant food security metric.This network used a model originally trained on ImageNetto predict nighttime light levels from satellite imagery af-ter transfer learning to fine-tune coefficients. Features wereextracted using this network and food security data was pre-dicted from these features via ridge regression. The outputwas transformed to a ”classification” output by discretelybinning the stunting percentage (percentage height-for-ageZ-scores below two standard deviations of the referencepopulation) in bins of 0, 0−0.1, 0.1−0.2, 0.2−0.3, 0.3−0.4,and≥ 0.4, which yielded a reasonably Gaussian distributionof the training data to the different bins. A ”classification ac-curacy” could then be easily determined by comparing thepredicted bin for each example to the bin in which its groundtruth value fell. Using this metric, the baseline attains a clas-sification accuracy of 39 %.

We attempt several different approaches to improving thisbaseline prediction. First, we hypothesize that using a net-work pretrained on aerial imagery would yield features moreuseful in predicting food security metrics from satellite im-agery than a network originally trained on ImageNet, whichdescribes optical images of a variety of different objects withfeatures very different than those observed in satellite im-ages. We therefore utilize the publicly available DeepSat-6dataset of (Saikat Basu 2015) as an initial dataset, whichconsists of 405,000 28 x 28 x 4 images (at 1m resolution)labeled with one of six ground cover classifications: barren,grassland, trees, road, building, and water. In order to enabletransfer learning, it is optimal that the resolution of the twodatasets match – thus, we downsample the 1m DeepSat im-agery to an effective resolution of 2.4m to match that of theSustLab data.

Once good classification performance on this DeepSatdataset was observed, we performed a tiling operation verysimilar to that of (Saikat Basu 2015) on the SustLab data,where we resized each 400 x 400 image to 420 x 420 andthen extracted 1089 12 x 12 tiles covering the full field ofview, which were then upsampled to 28 x 28 x 3 to align withthe dimensions of the DeepSat data. Note that because theSustLab data did not include a fourth near-IR channel (onlyRGB), the near-IR channel was dropped from this analysis.This tiling procedure results in a dataset in excess of 500,000image tiles, where each tile is labeled with the discrete bin

commensurate with the stunting percentage value of the fullimage from which it was taken. This tiled dataset can be usedin three different ways: 1) direct learning, where we attemptto learn the image label directly from the tiles, 2) transferinitialization learning, where we perform a similar task, butinitialize the network weights with those learned from train-ing on the DeepSat classification task, or 3) transfer learningby directly using the DeepSat-trained model to extract fea-tures from SustLab tiles, then using ridge regression to com-pute a prediction of stunting percentage from these features.

While it is possible to treat this as a regression problem,we have chosen to treat it as a classification problem dueto the following considerations. First, the data from whichFSMs are drawn is survey data taken at a number of house-holds within a given geographic area, the latitude and lon-gitude of which are then correlated with those of the corre-sponding satellite image. This data is fundamentally noisy innature, and it is less important that an exact value be recov-ered than that the general state of food security in these ar-eas be accurately reflected. Thus, a classification frameworkseems appropriate, since binning the continuous results intodiscrete classes eliminates some of this noise. Second, the r2metric characteristic of a regression heavily penalizes largesingle errors, whereas in this task, it is far more important toaccurately classify an areas as ”food secure” or ”food inse-cure” than to be exactly correct about its stunting percentagevalue, and the differential practical cost of large errors andsmall errors in stunting percentage for single observations isnegligible.

InfrastructureTo enable the studies described above, we created acustom codebase from scratch, which handles all as-pects of the above approach including data extract-transform-load (ETL), CNN training/testing, and ridge re-gression/classification computation. Note that all code canbe found in the repository

ETL Data for the food security task is drawn fromsatellite images labeled with FSMs drawn from UnitedNations Demographic and Health Surveys (DHS). Datais loaded from a set of .npy files describing the foodsecurity metrics of a target set of household surveys,and the location (in terms of latitude and longitude) isextracted from each location. These metrics are applied toa square of 0.09 in lat-lon coordinate length centered oneach reported survey location. Next, for each image in thedataset for Nigeria, which we focus on here, we assign itto the cluster in which its location is contained. All of thisis accomplished in the create clust dict2.pyfile in ./test in the code directory. Next, in./test/create test train set.py a numberof different options can be specified for creating test/trainsets. In our case, we tile one image from each survey clusterinto 12 x 12 squares (afterwards upsampled to 28 x 28) andpopulate the train/test sets with an 80/20 split of tiles fromeach image.

Training and Monitoring Utilities to create, train, andmonitor the network can be found in the ”./utils” direc-

Figure 2: Computational graph for VGG-style network.

tory, which contains several thousand lines of code. For ex-ample, ”layer utils.py” contains various layer types(i.e. general conv2d, batch normalization, maxpool), andseveral initialization functions (Xavier, He) incorporatedfrom recent literature. Model testing and training scripts canbe found in ”./transfer learning,” which utilize theutilities in a modular fashion. Further, Google TensorBoardfunctionality is integrated into the underlying training rou-tines in a way that enables direct visualization of trainingcurves, gradient flows, network activations, and the com-putational graph. A computational graph for the VGG-stylenetwork used here for DeepSat classification can be found inFig. 2. Further, completed notebooks for training, monitor-ing, and performance of ridge regression can be found in the”./notebooks” folder in the provided code repository.

Results and DiscussionDeepSat Land Cover Classification Task Before per-forming the FSM prediction task, we first want to ensurethat we have a viable network for land cover classificationfrom which to perform transfer learning. To accomplish thistask, we have implemented a neural net architecture in thestyle of the classic VGGNet in TensorFlow, where the ma-jority of layers are either 3 x 3 convolutional layers or 2x 2 max pooling layers (Karen Simonyan 2014). Using aGPU-powered environment allows us to substantially de-crease training time (by over an order of magnitude) usingan NVIDIA Titan X GPU as opposed to the CPU environ-ment to which we were previously confined. Our first passat training a viable network was performed on the Deepsatimage set, with an architecture as described in Table 1:

Note that in Table 1, ”Lrn” refers to local response nor-

Table 1: Layout of VGG-Style Net Implementation.Layer Type Parameters

0 Input 4 x 28 x 281 Conv-Relu-Lrn 96 x 3 x 32 Conv-Relu-Lrn 96 x 3 x 33 Conv-Relu-Lrn 96 x 3 x 34 Maxpool (2 x 2) -5 Conv-Relu-Lrn 32 x 3 x 36 Conv-Relu-Lrn 32 x 3 x 37 Conv-Relu-Lrn 32 x 3 x 38 Maxpool (2 x 2) -9 FC 3210 Dropout 0.911 Output 6

Figure 3: Training curve for DeepSat classification task.

malization, a procedure that normalizes pixels over a localspatial domain – each layer was also batch normalized tooptimize gradient propagation.

As shown in Fig. 3, training this network for one run overthe full dataset yields a validation accuracy of 95% and test-ing accuracy of 87 % – note that these are comparable toperformance observed in (Saikat Basu 2015), and additionaltraining facilitated by the GPU environment should yield im-proved model performance. We work with 80/20 test-trainsplit with a validation set of 10,000 images. We performeda hyperparameter search to ensure effective training, andwe use local response normalization, dropout, and L2 reg-ularization to avoid overfitting. Our best parameters were0.00071 for learning rate, 0.001 for the L2 regularization co-efficient, and 0.1 for the dropout probability. The confusionmatrix for this network can be found below in Fig. 4.

The high level of performance observed above may indi-cate that the network analyzes these images in a very similarway to how a human reader would – for instance, the con-fusion matrix indicates that the most commonly confusedclasses are grassland and barren, which many humans wouldalso have trouble differentiating. These accuracy results inand of themselves are worth noting in comparison to thework of (Saikat Basu 2015), who use labor-intensive hand-designed features and do not outperform this deep architec-ture by a significant amount.

Figure 4: Confusion matrix for the prediction of land coverclasses

Food Security Prediction: Stunting Percentage Nowthat we have attained useful results for the land cover classi-fication problem, we can use this trained network to extractfeatures from SustLab imagery that would be relevant forFSM prediction. We use the exact same network architecturespecified above in order to enable particularly straightfor-ward transfer learning. Table 2 contains the results of severalmethods for FSM prediction, with stunting percentage as thetarget FSM. While only test accuracies are reported here, weobserved validation accuracies very similar to training accu-racies, indicating that the network is not overfitting on ourdata.

We can now elaborate upon several of the prediction pro-cedures described in Table 2. One method, referred to as”Image-Mean DeepSat Features + RR,” utilized the aver-age of the features extracted from the SustLab tiles for eachfull image and used ridge regression to predict the full im-age stunting percentage. This gives accuracy nearly 10 %worse than the baseline classification. We suspect that av-eraging may cause some features captured by activations tobe lost, though there may be alternative strategies. The ”Di-rect CNN Learning” models directly train the network onSustLab image tiles labeled with the stunting percentage oftheir parent image. As shown here, the number of featuresdoes not play a particularly important role in the test accu-racy of the classification output, which is generally around45 %. These results imply that if given a single 28m x 28mtile from a given image is supplied, we can predict the stunt-ing percentage of that image correctly to within 10 % of theactual value with 50 % probability, which is surprising. Fur-thermore, we are able to outperform the nightlight baselineby over 5 % just based on these tile scale predictions, whichimplies that some features relevant to stunting are on the or-

Table 2: CNN FSM Prediction Results.Model Output Layer Feature Number Tile or Full Image Test Accuracy

Nightlight Features 4092 Full Image 39%Image-Mean DeepSat Features + RR 32 Full Image 31.1 %Image-Mean DeepSat Features + RR 256 Full Image 30.8 %

Direct CNN Learning 32 Tile 43.8 %Direct CNN Learning 256 Tile 45.0 %Direct CNN Learning 1024 Tile 46.5 %

Direct Learning with Transfer Initialization 256 Tile 46.4 %

der of less than 28 meters. In the future, it would be helpfulto look at layer activations to determine the exact nature ofthe features that are providing this predictive capacity. Wealso attempted transfer learning in a different manner, byinitializing the network with weights learned for DeepSat,then continuing to train directly on the SustLab images andassociated stunting percentage. This procedure slightly out-performed the pure Direct CNN Learning approach with asimilar number of output layer features (256), but did notprovide substantial improvement.

Automated Satellite Tasking for Food SecurityMeasurements

Images to Reward Grids: Methodology andAlgorithms

The prediction pipeline obtained from the analysis aboveproduces a food security metric (FSM) given an image,which becomes an input for an automated satellite task-ing algorithm. The FSM we use is the stunting percentage,which is normalized between 0 and 1 such that the numberscloser to 0 denote food insecure regions whereas numberscloser to 1 denote regions with higher food security.

Figure 1 shows the project pipeline. In the prior sec-tions, we have discussed the first stage in this methodologywhere we train a model on the labeled images from DeepSat(Saikat Basu 2015) and SustLab (Jean et al. 2016).

This excercise helps us train a model that can predictthe FSMs over various sub-tiles of a larger image. To givean analogy, even though the Bay Area is food secure,sub-regions within the Bay Area are not homogenous (i.e.Palo Alto versus East Palo Alto). In other words, transferlearning helps us go from a low resolution prediction toa high resolution prediction. The process of extractinga reward grid from the tiled image is the quantitativedescription of this idea. The reward grid is produced bytiling the large image into small regions (making sure thatthe resolution of the training dataset for the model matchesthe resolution of the tiles) and then using the trained modelto predict the FSM at each tile. Once this matrix has beencreated, we can look at the satellite tasking problem as a setof three sequentially linked sub-problems that each add alayer of complexity and uncertainty to solve the completeproblem. After pre-processing the SustLab images, we endup with a 33 x 33 grid of 28 x 28 pixel images.

Sub-Problem 1: Finite Horizon Deterministic Ap-proach: Given our trained CNN H, we can use the constanttime prediction property of neural networks to traversethrough each tile of the large image and create a 33 x 33grid of predicted food security metrics in O(332) time. Onthis grid, we can start at a randomly chosen location andset up a Markovian search problem to find the tile with thelowest food metric (corresponding to the poorest region inour map) in the the least number of tile-to-tile moves (wechoose the number of moves as the metric of cost and definean optimal path as the least cost path, acknowledging thatthis path is not guaranteed to be unique). We use UniformCost Search for this sub-problem.

Sub-Problem 2: Finite Horizon Stochastic Approach:Aleatoric Uncertainty: In the previous model of theproblem, we assumed that there are bounded and finitehorizons to our search space. We also spent time traversingeach tile to calculate the food security metric to set up theMarkovian search problem. Inherent in the prediction taskis an error incurred from the model having a finite accuracy.As discussed above, a ridge regression is performed on thefeatures produced by the CNN when the satellite imageis the input. Therefore, the prediction of the FSM that isobtained is the expected value of the FSM with variancecharacterized by the coefficient of determination of theregression fit. Recall that the coefficient of determinationis the deviation from unity of the ratio of the residual sumof squares and the total sum of squares. In a general form,R2 can be seen to be related to the fraction of unexplainedvariance (FUV), since the ratio compares the unexplainedvariance (variance of the model’s errors) with the totalvariance (of the data). To get an estimate of this predictionvariance, we use the following procedure:

Input: Images tagged with FSMsfor Image i = 1, 2, 3, . . . do

for On Image i, Tile t = 1, 2, 3, . . . doGet FSM Prediction on tile t. Call it Pit

end forV ari = Variance of Pit over all t

end forGlobal Variance = σ2 = Average of V ari over i

This variance characterizes up to the second moment of thebackground probabilistic distribution from which the rewardgrid is predicted. Sampling from this distribution and using

Figure 5: A schematic for the path from an image to generating an FSM

Figure 6: Visualization of the high resolution FSM predictedby our model

Figure 7: The Bayesian Network driven Markov DecisionProcess. The Bayesian Network completely specifies theweather distribution which in turn completely specifies theMDP

the Monte Carlo approach to calculate the relevant statisticsallows us to account for this uncertainty and calculate aprediction error margin. This margin is calculated for thepredicted optimal path. The schematic in figure 5 and thefollowing procedure outlines our approach.

Input: Expected Reward Grid (µ), FSM Model PredictionVariance (σ2)Each tile on reward grid has added parameter onPathCreate a pathProbabilityMatrix of size equal to the size ofthe reward grid. Initialize this matrix to zerosfor Monte Carlo Realization r = 1, 2, 3, . . ., MCIters do

for On Reward Grid, Tile t = 1, 2, 3, . . . doReward Value at tile t sampled from N (µt, σ2)Perform UCS on this realization of reward gridAssign onPath = 1 to tiles that are on optimal pathAssign onPath = 0 to tiles that are not on path

end forpathProbabilityMatrix += onPathMatrixend forpathProbabilityMatrix /= MCIters

The path probability matrix assigns a probability of theoptimal path passing through a given tile. This probabilityserves as a metric for a two-pronged validation of theproposed model. One, a sharper path with less varianceimplies that the trained model works well and is sufficientfor the given geographic area and builds confidence in ourprediction of FSMs. Two, a large variance indicates a mis-match between the dataset on which training is conductedversus the dataset on which prediction is attempted andsignals that either increasing the training database or addingmore geographically relevant data into the database wouldbe advisable.

Sub-Problem 3: Finite Horizon Stochastic Approach:Epistemic Uncertainty: When satellite images are used, the

natural question that comes up is “What happens when thesatellite view is occluded due to clouds?” To address this,we propose a Markovian cloud cover model with Bernoulliprobabilities for cloudy tiles. This model is completely spec-ified by the Bayesian network in Fig. 7. Thus, the futurestep in the path search depends on the cloud cover (p(C))that exists in the current state and the probability of tran-sitioning (p(Ct|Ct−1)) to a tile with/without cloud covergiven the current state of the cloud cover. An additionalbinary variable is added to our minimal description of thestate to account for this. Note that both these probabilities(which has a degree of freedom 3 for the 6 required probabil-ities) can be estimated from historical data using a frequen-tist estimation strategy like maximum likelihood. A morephysical weather model would incorporate spatial correla-tions and temporal statistics which can be modeled usinga deeper Bayesian network and a higher dimensional state-space. Note that the runtimes of all the path search algo-rithms used scale exponentially in the size of the state-space.For the purposes of this project and a proof-of-concept,our limited validity model suffices. Once both p(C) andp(Ct|Ct−1) are prescribed, the epistemic uncertainty is in-corporated into the problem specification and transforms thedeterministic search problem into a Markov Decision Pro-cess. The state subgraph of the full MDP search graph isschematically shown in Fig. 7. On this MDP, value iter-ation is performed so as to obtain the optimal movementpolicy given a state. It is instructive to note that once thispolicy is computed, we do not need to re-compute the op-timal policy since prescribing the probability space of theMDP completely determines the optimal policy. This is thedesired and required behavior for an algorithm that needsto be deployed on light hardwares - typical of satellites.We perform this value iteration process on our MDP withp(C = 1) = 0.2, p(Ct = 1|Ct−1 = 0) = 0.5 andp(Ct = 1|Ct−1 = 1) = 0.5. The figure on the right inthe middle row of Fig. 8 shows one particular realization ofthe cloud cover. The figure on the left in the bottom of Fig.8 shows the occluded view seen from a satellite due to thecloud cover in contrast to the clean view (top left of Fig. 8)that would otherwise be expected to be seen on a clear day.Value Iteration, in contrast to UCS tries to maximize the re-ward. The reward matrix described above is scaled affinelyand linearly so that all values lie between 0 and 1 and the val-ues close to 1 signify lower food security. The goal is onceagain to reach the most food insecure point while traversingfood insecure regions such the path is globally optimal inthe sense that the locations traversed pass through the mostfood insecure regions within the horizon. In order to makesure that the path search avoids clouds, a zero value (highfood security) is assigned to the locations where clouds ex-ist. Shown on the bottom right in Fig. 8 is the optimal pathfound when the realized cloud cover occludes the view. Itis seen that the optimal policy learns to meander around theclouds. The optimal path in this case is also significantly dif-ferent than the optimal path obtained from UCS.

Conclusions and Future WorkIn the end, we have accomplished several goals with thiswork. First, we have shown that CNNs can accurately pre-dict land cover classification on the DeepSat dataset withup to 87 % accuracy, which is on par with state-of-the-artresults from other methods. Second, we have shown thatspecification of a tile-scale CNN model either with directtraining on tiled SustLab images or via transfer initializa-tion from a network trained on DeepSat data yields accu-racy of over 45 % on the test set, which outperforms thebaseline of 39 % generated by the current state-of-the-art.The fact that this tile-scale CNN sourced from training onsatellite-relevant datasets performs better than features ex-tracted on the whole images from nightlight data impliesthat direct training on the metric of interest is indeed a use-ful contributor to improved accuracy. In the future, there area variety of directions that the machine learning side of thiswork could be taken, including iterating over network archi-tectures (GoogLeNet, ResNet, etc.) and obtaining additionaldata to allow for direct training on the larger-scale image.Use of either GoogLeNet or ResNet seems particularly al-luring given their ability to incorporate (and simultaneouslyoutput) features at multiple different scales while incorpo-rating skip connections, which may be particularly helpfulin ensuring that lower-level features are emphasized in theclassification layer.

On the satellite tasking portion of our project, we haveshown that tasking under uncertainty can be handled bothby UCS with a CNN-based probabilistic cost function andMDPs driven by Bayesian networks. Future work on thisfront should include incorporating physical, non-Markovianweather models to more accurately simulate actual satelliteconditions and identification of high-resolution satellite as-sets that could benefit from these types of algorithms in prac-tice.

AcknowledgmentsProf. Stefano Ermon, Prof. Marshall Burke, and StanfordSustainability and Artificial Intelligence Lab

Figure 8: The results derived after an image has gone through the entire pipeline

References[Alex Krizhevsky 2012] Alex Krizhevsky, Ilya Sutskever,e. a. 2012. Imagenet classification with deep convolutionalneural networks. Advances in neural information processingsystems.

[Chen and Nordhaus 2011] Chen, X., and Nordhaus, W. D.2011. Using luminosity data as a proxy for economic statis-tics. PNAS.

[Devarajan 2013] Devarajan. 2013. Rev. Income Wealth 59,S9S15.

[Dunnmon et al. 2019] Dunnmon, J.; Ganguli, S.; Hau, D.;and Husic, B. 2019. Predicting us state-level agriculturalsentiment as a measure of food security with tweets fromfarming communities. arXiv preprint arXiv:1902.07087.

[Ganguli, Garzon, and Glaser 2019] Ganguli, S.; Garzon, P.;and Glaser, N. 2019. Geogan: A conditional gan with recon-struction and style loss to generate standard layer of mapsfrom satellite images. arXiv preprint arXiv:1902.05611.

[Henderson, Storeygard, and Weil 2011] Henderson, V.;Storeygard, A.; and Weil, D. N. 2011. A bright idea formeasuring economic growth. Am. Econ. Rev. 101:194–199.

[Jean et al. 2016] Jean, N.; Burke, M.; Xie, M.; Davis,W. M.; Lobell, D. B.; and Ermon, S. 2016. Combiningsatellite imagery and machine learning to predict poverty.Science Magazine 353(6301):790–794.

[Jerven 2013] Jerven. 2013. Poor numbers: How we are mis-led by african development statistics and what to do about it.Cornell Univ. Press.

[Jerven 2014] Jerven. 2014. Benefits and costs of the data fordevelopment targets for the post-2015 development agenda.Data for Development Assessment Paper Working Paper,September (Copenhagen Consensus Center, Copenhagen,2014).

[Karen Simonyan 2014] Karen Simonyan, e. a. 2014. Verydeep convolutional networks for large-scale image recogni-tion. arXiv.

[Meyer 2014] Meyer, e. a. 2014. Silicon Valley’s new spysatellites. The Atlantic.

[Perez et al. 2019] Perez, A.; Ganguli, S.; Ermon, S.; Azzari,G.; Burke, M.; and Lobell, D. 2019. Semi-supervised multi-task learning on multispectral satellite images using wasser-stein generative adversarial networks (gans) for predictingpoverty. arXiv preprint arXiv:1902.11110.

[Pinkovskiy 2017] Pinkovskiy, M. L. 2017. Growth discon-tinuities at borders. J. Econ. Growth.

[Romero, Gatta, and Camps-Valls 2014] Romero, A.; Gatta,C.; and Camps-Valls, G. 2014. Unsupervised deep featureextraction of hyperspectral images. In IEEE GRSS Work-shop on Hyperspectral Image and Signal Processing (WHIS-PERS). IEEE, volume 6.

[Safyan 2015] Safyan, e. a. 2015. Overview of the planetlabs constellation of earth imaging satellites. ITU Sympo-sium/Workshop, Prague.

[Saikat Basu 2015] Saikat Basu, Sangram Ganguly, e. a.2015. Deepsat a learning framework for satellite imagery.arXiv.

[Sandefur 2015] Sandefur, e. a. 2015. Benefits and costs ofthe data for development targets for the post-2015 develop-ment agenda. J. Dev. Stud. 51, 116132.

[United Nations 2014] United Nations. 2014. A world thatcounts: Mobilising the data revolution for sustainable devel-opment.

[World Bank 2015] World Bank. 2015. Pov-calnet online poverty analysis tool, http:// ire-search.worldbank.org/povcalnet/.