Process Model

Premathmod D3.2. Database Implementation IST-2000-28177

D3 2_1PG 1

PREMATHMOD

IST–2000-28177

STATISTICAL AND MATHEMATICAL MODELLING, DATA ANALYSIS, SIMULATION AND OPTIMISATION METHODOLOGIES FOR PRECISION FARMING

Database implementation

D3.2.

Report Version: Final

Report Preparation Date: 01/06/2002

Status: public

Contract Start Date: 01/12/2001 Duration: 18 months

Project Co-ordinator: Lesprojekt sluzby

Martinov 197, 277 13 Zaryby, CZ

Project Officer: Mr. Johan HAGMAN

Project funded by the European Community under the “Information Society Technology” Programme (1998-2002)


D3 2_1PG 2

Contractual Date of Delivery to the CEC: 01/06/2002

Actual Date of Delivery to the CEC: 01/06/2002

Author(s): Kafka S., Holy S., Charvat K. Gnip P, Gaspardo L, Dohmen B.,

Participant(s): Help Service Remote Sensing, Lesprojekt sluzby, MJM, Technofarming, and AgroSat

Workpackage: WP5

Est. person : 8 months

Nature: R

Version: 1

Total number of pages: 23

Abstract:

The report describes the model implemented for data storage methods within the Premathmod system. The Premathmod system is an Internet solution based on

a combination of two Open Source tools: Mapserver and GRASS. Mapserver is an interface tool. It is used for communication with users. Grass is a powerful analytical tool. The combination of both instruments is a new way of spatial data processing in precision farming. The system works with raster and vector data formats, but the data used for an analysis are in raster format supported by Grass. The system also allows to access data on other servers via WMS services. There are two existing basic variants of Premathmod system:

• A farmers’ version for data display

• An analytical version for data analysis

Keyword list: Precision farming, data storage, GIS, Grass, Mapserver, Open Source, WMS services


D3 2_1PG 3

1. EXECUTIVE SUMMARY

The database for storing, access and analysis of field measurement data was implemented. The measured data are converted into unique raster format. The grid-based model is used for this purpose. All measured data are converted into this grid model. Original data are stored in the system and can be displayed. The workpackage task was:

• To define a standard structure of data

• To create the software tools for conversion of data from various sources and its transformation to the standard format

The definition of the grid structure is important for the mathematical analysis. This structure is suitable for Grass analysis. It enables a user to combine and analyse data from different sources. It is also possible to display the data using Mapserver functionality. The data security is very important as the private and sensitive data are used within the system. User ID and a password are used to protect data of the system. Data in the system are processed by a service company and distributed to the farmer. It is possible to combine private and public data using Web Mapping services. The Premathmod system also offers distribution of data to the variable rate application machines (belonging either to farmers or service companies).


D3 2_1PG 4

2. CONTENTS

1. Executive summary .................................................................................................................... 3

2. Contents ..................................................................................................................................... 4

3. Introduction ................................................................................................................................ 5

4. Data sources ............................................................................................................................... 6

5. Data processing in the Central database of Precision farming ................................................... 8

6. Data storage................................................................................................................................ 9

7. Data interpolation ..................................................................................................................... 13

8. WMS DATA ACCESS ............................................................................................................ 14

9. Data security ............................................................................................................................ 15

10. Premathmod analysis tool ........................................................................................................ 16

Data import. ............................................................................................................................... 16

Map ............................................................................................................................................. 16

Analysis ...................................................................................................................................... 17

Statistics ..................................................................................................................................... 18

Metadata .................................................................................................................................... 20

Settings....................................................................................................................................... 21

11. Future plans: ............................................................................................................................. 22

12. References ................................................................................................................................ 23


D3 2_1PG 5

3. INTRODUCTION

Premathmod project works with new technologies in the field of precision farming. These methods have to be presented to farmers and managers operating in agriculture. Premathmod system is becoming a very powerful GIS tool in farm management. A farmer can get by Internet technologies much more easily essential information for crop growing, weather, farm management, grain market and so on. One of the most important knowledge a farmer acquire when adopting this different way of farm management is an elementary experience of using IT technologies, Internet environment and Precision farming system in rural daily farm work. Premathmod system brings new information and communication solution for precision farming. This solution is based on connection of following advanced technologies:

• Full internet data access based on Open source solution

• Open source geocomputing

• WEB mapping services

• Mobile data access

• Access to different data sources on different places It is expected, that the data will be stored, processed and analysed by service companies, but farmers will have a full access to functionality of the system.


D3 2_1PG 6

4. DATA SOURCES

Maps - The very important data sources for precision agriculture are field maps. These maps are typically accompanied by soil maps, parcel maps etc. Such data are digitised from existing paper maps but the majority of analysed parameters is not described in such maps (e.g. density), therefore soil samples positioned by GPS device have to be collected in the field. In central database of Precision farming can such data be used as a background information layer for field boundary maps and various control points. GPS measurement, soil sampling and field measurement - Soil samples have to be analysed for the main nutrients to show their availability for plants in the surface and in subsurface layers of soil. There are also other parameters affecting the mobility of nutrients through soil profile and soil intensity and capacity factors. Fluctuation of individual analysed parameters varies in time and field relief. Field boundaries are measured in WGS 84. Signal GPS and DGPS are received by TRIMBLE AgGPS 132 DGPS. Receiving data are collected by H-GIS software in Arc view or MapInfo data form (.shx, .shp, .mid, .mif, .dbf). GPS and DGPS signals are receiver with sub meter accuracy. Data of each field is transferred to the central database separately under central geographic coordination (wgs_Lat, wgs_Lon) with six decimal places. Area and distance of a field can be presented in metric units (hectares, square meters, square kilometres, meters, kilometres). Any other information, which is coming to database, is sorted and assigned to field boundary. Future data processing is limited by field boundary. Field boundary is on the top of database tree. All future maps will be created on this field boundary. In addition, this field boundary can be changed according to end user needs up to specific crop rotation. Field area can be separated into several parts and reassembled back after particular use. Basic Lab analysis is done to determinate a level of phosphorus (P), potash (K), magnesium (Mg), calcium (Ca) in mg in 1kg of soil and pH of soil. The Lab determination of content of each nutrient is done by method MEHLICH III. The Lab results of soil analysis are converted into .xls or .dbf formats. The data are then transferred to geographic coordinates in central database, according to ID number of soil sample. Yield maps – measuring of crop - Yield maps are important data source for controlling the fluctuation of soil parameters. The yield maps construction is connected with spatial technological equipment on the harvesting machines. The goal of this project is not to directly develop the equipment for measuring yield on harvesting machines, but the methodology of yield maps constructions. Another important part is measuring of nutrients level, other chemical elements, and moisture during harvest and etc. Data are collected in row format according to producer technology. (.yld –Ag Leader, .fsy – Massay Ferguson and etc.) In fact, data are collect as follows:

• Mass flow in kg.m-2

• Moisture in %

• Yield volume kg.m-3

• Harvest speed km.hour-1

• Id field

• Crop

• GPS time and etc. Those raw data are processed and the final result is presented as grain in kg*ha-1 in calibration moisture. For example: Winter wheat…6,000 kg.ha-1, moisture 15% Satellite imagery - Remote sensing images were not usually used in Europe. Small image resolution and cloudy weather are the most important reason for that. High-resolution data are also affected by cloudy weather, so that radar data seems to be the most interesting remote sensing data.


D3 2_1PG 7

To set a management zone on the field, to check a lack of Nitrogen during spring vegetation or to predict a yield before the harvest, data from Landsat 5 and Landsat 7 TM were used with accuracy 30x30 m one pixel in multispectral. Landsat 5 – 7 spectral bands, Landsat 7 – 8 spectral bands Data format: CEOS, Level: SYSCOR, Cal: Pre-Flight, Rsmlp: NN Air pictures - Aerial photographs can be used until now more effectively than RS data. Colour infrared materials are mostly used in agriculture, because plant production can be easily identified on these images. Raw data was made up as a spectrozonal photo in geodetic format –3 bands (infrared, near infra red and false colours). Altitude 2,500m, accuracy one pixel = 60cm x 60cm, 1 scene 5km x 5km. Fertilisation maps – amount of fertilisation- the fertilisation maps from previous season are also important information source for evaluation of soil parameters. Application data are stored in central database as an information layer. Data could be stored as an application map - .sti, .tif (simple or multi level variable application maps) or “as applied” data - .rpt (monitoring of application what was done by application machines). All variable application maps are in .tif format (demo maps) or .sti format (controller map).


D3 2_1PG 8

5. DATA PROCESSING IN THE CENTRAL DATABASE OF PRECISION FARMING

The collected data have to be processed individually according to the character of the data.

• Field boundary mapping – error data are corrected in the file, changes are updated

• Electro-magnetic conductivity mapping- error data eliminated out of the file,

• Data processing and data classification of the selected field. Data processing and data classification of the monitored area. Establishing the soil zones. Setting up a grid for manual soil sampling or soil sampling by soil sample machines

• Soil sampling data – Controlling of the process of soil sampling in the net.

• Pre-processing of the soil samples before starting the lab analysis –air drying, sorting and aliquotation, lab analysing after “Mehlich III”

• Soil test maps – data of the lab analysis are integrated in the central database and they are processed there.

• Soil test maps are an important information layer for further recommendations.

• Yield data- the raw yield data are processed to a yield map. Such a yield map is

an additional important information layer for recommendations

• Agronomic data – data are coming from an end user (crop rotation, organic matter, plan of yield and so on.)


D3 2_1PG 9

6. DATA STORAGE

The following types of data are stored in Premathmod system: Field measurement

• Field boundaries – original measured vector data in ShapeFile polygon form (the data are stored in separate file for every farm and this layer is accessible only after a user login. Information of the field and field boundary is divided as follows:

o Year o Crop rotation o Organic matter spreading

• Sampling data –original measured vector data in ShapeFile polygon form (the data are stored in separate file for every farm and this layer is accessible only after a user login. There are following attributes

o Year o Control points GPS o NDVI o Ca o K o Mg o P o PH o CEC o K/Mg


D3 2_1PG 10

• Yield measuring - original measured vector data in ShapeFile polygon form (the data are stored in separate file for every farm and this layer is accessible only after a user login

• Conductivity measuring - - original measured vector data in ShapeFile polygon form (the data are stored in separate file for every farm and this layer is accessible only alter a user login

• N sensor measuring - - original measured vector data in ShapeFile polygon form (the data are stored in separate file for every farm and this layer is accessible only after a user login

• Digital aerial photos – colour or infrared digital photos in raster form (data are stored in form of colour composition for displaying, but also in the form of R, G, B components for data analysis)

• Digital satellite images (Thematic Mapper, Spot), data in raster dorm (data are stored in form of colour composition for displaying, but also in the form of spectral bands for data analysis

Data for an analysis The data for an analysis are stored in Premathmod system in raster form. The raster form allows very easy and effective data analysis using Grass system. These entire layers are accessible only after a user login. These layers are:

• Maps of nutrients content – raster maps derived from soil sampling. These data are sorted according to nutrients and years Differences in nutrients content in different years – data in raster form. Soil test data are monitored also in time of sampling. This part can give to a user a quick overview of soil nutrients level changes in topsoil layer. Color maps are prepared for each nutrient separately, for example:

- Ca 1999– 2002 - CEC 1999– 2002 - K 1999– 2002 - K/Mg 1999– 2002 - P 1999– 2002 - PH 1999– 2002


D3 2_1PG 11

Maps show only very general results in three levels of categories as are: - Decrease - No change - Increase

• Yield maps – raster maps derived from yield monitoring. These data are sorted by years

• Conductivity maps - - raster maps derived from conductivity measurements. This data are sorted by years

• N sensor maps - - raster maps derived from N sensor measurement. This data are sorted by years

• Maps derived from satellite and aerial picture data as NDVI (normalised vegetation index

Outputs data The data in raster form, which are generated by Premathmod system.

• VRA recommendation - data Variable Rate Application or Multi Variable Rate Application by table or variable color map for following nutrients:

- A phosphorus (in fertilizer) - A potassium (in fertilizer) - A calcium (in fertilizer)


D3 2_1PG 12

• Price map helps to make a short overview of prices per hectare (including application) for following nutrients:

- A potassium - A phosphorus - A calcium - Both potassium and phosphorus

Supporting data Here could be included also, other data layers supporting easier orientation, it could be:

• Cadastral maps

• Basic maps Both types of data could be used in vector or raster form. Data could be accessed from distant servers using WMS services.


D3 2_1PG 13

7. DATA INTERPOLATION

The working raster maps are generated from original input maps using interpolation methods. There are three interpolation methods used in GRASS:

• Inverse distance weighted. It fills a raster matrix with interpolated values generated from a set of irregularly spaced data points using numerical approximation (weighted averaging) techniques. The interpolated value of a cell is determined by values of nearby data points and the distance of the cell from those input points. In comparison with other methods, numerical approximation allows representation of more complex surfaces (particularly those with anomalous features), restricts the spatial influence of any errors, and generates the interpolated surface from the data points. This method is rather usable for data of different types than DEM, like chemical component spreading etc. This method is available in Premathmod analysis tool for nutrient content computations. Detailed program description is in [1].

• Regularized spline with tension and smoothing. This method has been originally developed for DEM computation. It is derived from known methods of bicubic splines interpolation, but involves smoothing factor, which permits resulting DEM not to go exactly through original points. This is very usable for DEM derived from contours for removing artificial waves. Program enables computing surface from point as well as contour (vector) input data. Resulting DEM may be affected by many user-set parameters. Detailed description of algorithms used, and of program parameters are in [2], [3], [4].

• Recently new Kriging interpolation algorithm was introduced into GRASS. It was originally developed for element ore content evaluation, so we considered it useful for nutrient components interpolation. More detailed description of algorithms used and of program parameters is in [5].


D3 2_1PG 14

8. WMS DATA ACCESS

The technological background WMS is based on OGC standards. This approach ensures base conditions for interoperability of used geodata services. Until this date OGC published several standards for sharing geodata over WWW. To the most developed and also adopted by wide community belongs WMS. The WMS is a Web Map Service (specifically an OGC Web Map Service). A WMS is capable of producing maps drawn into a standard image format (PNG, JPEG, etc) based on a standard set of input parameters. The resulting map can contain "transparent" pixels where there is no information and thus several independently drawn maps can be laid on top of each other to produce an overall map. This is possible even when the maps come from different Web Map Servers. The exact specification of the WMS can be found at the following web address: http://www.opengis.org/techno/specs/01-047r2.pdf. The last specification has version number 1.1.0, but majority of contemporary solutions support previous version 1.0.0. WMS is based on a capability to answer several formalized requests. To the requests belong: GetCapabilities first allows a client (or client proxy) to instruct a server to expose its

mapping content and processing capabilities. GetMap enables a client to instruct multiple servers to independently craft "map layers"

that have identical spatial reference system, size, scale, and pixel geometry. The client then can display these overlays in specified order and transparency such that the information from several sources is rendered for immediate human understanding and use.

GetFeatureInfo enables a user to click on a pixel to inquire about the schema and metadata values of the feature(s) represented there.

WMS is based on a transfer of georeferenced maps i.e. images or drawings generated on request. In some cases a transfer of geofeatures is demanded instead of maps. This possibility belongs to communication among geodata sources. Such type of the service is covered by another OGC specification WFS – Web Feature Service. This specification defines formal request over WWW, which returns OGC simple features in GML format. GML is an XML dialect for geographic features. Specification consist of two requests:

GetFeatue DescribeFeatureType

Self-WFS isn’t still in version 1.0, but GML is well defined. Specification GML ver. 2.1.1 can be found at http://www.opengis.net/gml/02-009/GML2-11.pdf. Because of very few implementations of WFS, for needs of Premathmod we try to develop simple web feature

broker, which cover the necessary subset of WFS capabilities (more in [8]).


D3 2_1PG 15

9. DATA SECURITY

Every user can have different data access to the map application layers according to his data access rights. With this mechanism it is possible to publish some of the maps or data of the focused application, which was impossible in the previous solution. Now the registered user can edit only layers of every application reserved for him. Thus layers with protected information are available only for the registered user. � Every login and user passwords are saved now into proper file in the encrypted form. � All graphic data are saved off the www server tree. � All configuration files of map applications are saved off the www server tree.

The settings enable to open certain applications from the central computer to Internet environment. This modification brings to a network administrator and to users these advantages: � In case the data are placed at the more powerful internet server, connected to the

spinal internet network, it enables several times faster data access comparing to situation, when the data are stored at the company’s central computer and the data access is available via mobile users (using GSM, GPRS or HSCSD technology) to the database.

� This new solution significantly decreases call fees to all the system users (shorter connect time of a user via provider to GSM+GPRS (HSCSD))


D3 2_1PG 16

10. PREMATHMOD ANALYSIS TOOL

The application is developed for precision farming experts as a tool for management zones creation

and fertilizer application recommendation. Thus most of the functions are not targeted for farmers,

but there is a set of functions to be accessible by both groups. A farmer will be able to introduce his

parameters to application, e.g. crop rotation plan and will be able to see resulting application maps

as well as economic calculations as seen in of-line Premathmod application. Expert can use this tool

for an interactive check, to analyze and correct input data, to combine and analyze different types of

data to set potential management zone estimation, to discover potential problems and to compute

application maps. We presume, that this tool will enable a farmer to find the optimal algorithms of

application map formulas and then in the second step these formulas will be implemented as a

“black box”. Currently the application enables a farmer or a manager to upload, analyze and

visualize data in 2D and 3D view.

Currently supported functions: Data import.

Currently only soil samples and field boundary shapefiles are allowed to be uploaded. Other (mainly raster) data may be imported with ftp access. The structure of data is described on the application Help page. For faster upload the data are compressed in the zip archive. User has to set data cartographic system. After upload these actions are performed on the server side:

• Unzipping an archive

• Filenames checking

• Required data structure checking

• Transformation from uploaded data cartographic to target system (it is set by administrator in the process of creating new farm)

• Importing data into GRASS internal format (sites for soil samples, raster for field boundaries)

Map

Shows the map of farm. It enables a user to display the results of an analysis as well as original data. The topographic map is accessible via WMS (web mapping services) from different server. Map application enables a user to edit points, lines and polygons including multipart ones and many other functions (length/area measuring, GPS support, searching/location, data filtering etc.). Training areas for supervised classification may be added/edited.


D3 2_1PG 17

Analysis

Three forms are on the analysis page: 1) Nutrient content computation. It computes the nutrient content raster maps from soil

samples point data. A user has to select an appropriate year of soil sampling and

an appropriate year of field boundaries data (the farm property may vary in time). Two interpolation methods are currently supported:

• Inverse distance weighted. It is simply common method for interpolation. The GRASS function s.surf.idw is used.

• Regularized splines with tension and smoothing. This special method was developed originally for elevation models created by Lubos Mitas and

Mitasova [3]. The GRASS command r.surf.rst is used for interpolation. This computation is better but slower.

2) Management zones estimation. This allows a user to use unsupervised or supervised

classification on an arbitrarily chosen set of raster maps. User may check normalizing values to preprocess data to have the same weight (or scale) in consequent computations. For unsupervised classification the desired number of clusters has to be set. Classification methods used:

• Unsupervised. The GRASS i.cluster is used to compute signatures. Then maximum- likelihood algorithm is performed (i.maxlik).

• Classification report is stored as the part of metadata.

• Supervised. The GRASS i.gensig is used to determine signatures for supervised classification. Then maximum-likelihood algorithm is performed (i.maxlik).

• Supervised+unsupervised. Initial signatures are determined by training areas with GRASS i.gensig. A clustering algorithm (i.cluster) is used to set the result signatures. Maximum-likelihood algorithm is then performed (i.maxlik).

• Supervised-texture. This algorithm has not been implemented yet.


D3 2_1PG 18

Results are stored with names zones.*, where * is user set extension name.

3) Map calculator. A user is allowed to perform various raster operations with this tool:

to set a name of the result map, description, mask raster if desired and the right-side formula. Underneath GRASS r.mapcalc command is then performed. It is a very powerful tool for various analyses. Some examples:

• K content difference k2002-k1999

• vegetation index float(tm4–tm3)/ float(tm4+tm3)

• making mask if(k2002>400) Statistics

Some univariate statistics, histograms, correlation matrices and 2D histograms (correlation graphs) are shown for user chosen raster maps.


D3 2_1PG 19


D3 2_1PG 20

3D view Perspective view of raster data is allowed. A user can select elevation raster - raster holding heights in 3D View. Default is ELEVATION but you may use whatever raster, e.g. nutrient content etc. colour raster - cover raster to be shown on elevation. The colour depth is limited for the present however.

After confirmation, 3D View is shown in the new window. You may use these parameters to change the 3D View appearance: Exaggeration Multiply factor of elevation raster to highlight a terrain dynamics. Lines density Covering net lines density. If 0, no lines are shown. View azimuth Azimuth (degrees) of "user eye". If 0, The view from south to north is set. View distance

Distance (meters) of "user eye" to the centre of the map.

Vertical angle

The angle between user view and horizontal plane.

Target height Height of the target of view point (the point on/below the centre of map)

Field of View Horizontal angle (degrees) of the user's view (changing "camera focal length").

Metadata

The metadata system based on native GRASS metadata is extended according to

the Premathmod specification. Here a user may delete raster maps he created before as well as to edit some metadata of maps.


D3 2_1PG 21

Settings

It enables a user to set raster resolution of raster maps. To take the effect, all data must be reloaded to system. Client system requirements - MS IE > 4.0 - Java 1.2 or higher virtual machine browser support


D3 2_1PG 22

11. FUTURE PLANS:

- Uploading raster data through forms - Kriging interpolation for soil samples - Importing and processing yield and N-sensor data - Implementing ISO TC/211 compatible metadata subset - Interactive editing management zones via internet browser - Application recommendation maps computation - Converting results (e.g. management zones and application recommendation) to vector

format - Download results back to client


D3 2_1PG 23

12. REFERENCES

1. GRASS r.surf.idw manual page (http://www.geog.uni-hannover.de/grass/gdp/html_grass5/html/s.surf.idw.html). Hannover, 2002.

2. GRASS r.surf.rst manual page (http://www.geog.uni-hannover.de/grass/gdp/html_grass5/html/s.surf.rst.html). Hannover, 2002.

3. Mitasova H. and Mitas L. 1993: Interpolation by Regularized Spline with Tension: I. Theory and Implementation, Mathematical Geology 25, 641-655.

4. Talmi, A. and Gilat, G., 1977: Method for Smooth Approximation of Data, Journal of Computational Physics, 23, p.93-123.

5. GRASS r.surf.krig manual page (http://grass.itc.it/gdp/html_grass5/html/s.surf.krig.html) Hannover, 2002.

6. GRASS d.3d manual page (http://www.geog.uni-hannover.de/grass/gdp/html_grass5/html/d.3d.html). Hannover, 2002.

7. Mapserver homepage. (http://mapserver.gis.umn.edu/). Minnesota, 2002. 8. M.Konecny, K.Stanek, M.Vesely, K.Charvat WirelessInfo IST IST 1999-21056 D12.1

User interface

Process Model

Science

Transcript of Process Model