DIS Working Group Report LBA Science Steering Committee Meeting - 24 rd Manaus - Amazonas - Brazil...

DIS Working Group ReportDIS Working Group Report

LBA Science Steering Committee Meeting - 24rd

Manaus - Amazonas - Brazil

December 07-08, 2009

Laurindo Campos (INPA)

Luiz Horta (CPTEC/INPE)

Outline• Data Registration and Archive

Status

• Data Registration status for EU and BR teams

• Worries – Infra-Structure and System Updates

• LBA 2 – DIS Future (?)

Data Registration and Archive Status

LBA Overall Status:

Metadata Registered in Beija-florLBA Overall Status:

Metadata Registered in Beija-flor

260

62

8

118

63

133

83

152

80

231

92

325

100

379

207

413

211

418

211

469

215

538

217

633

217

573

215

575

214

573

207

581

200

601

200

544

193

475

187

0

100

200

300

400

500

600

700

06/99 06/00 10/00 02/01 05/01 11/01 05/02 11/02 05/03 11/03 05/04 11/04 06/05 11/05 05/06 04/07 11/07 05/08 03/09 12/09

# of data sets registered

# of posters registered

N=662

LBA Overall Status:LBA Overall Status:

Data Volume Archived at LBA DISData Volume Archived at LBA DISLBA Overall Status:LBA Overall Status:

Data Volume Archived at LBA DISData Volume Archived at LBA DIS

0

100

200

300

400

500

600G

igab

ytes

Unrestricted DataRestricted DataPostersDocumented

Unrestricted Data 4.0 14.8 69.0 89.1 120.5 210.6 322.1 364.4 312.4 428.3 497 499.2 573.2 575.4

Restricted Data 0.5 8.7 8.9 18.7 20.1 22.1 22.3 22.0 76.80 77.87 78.81 80.43 80.43 114.98

Posters 0.6 1.1 1.2 1.5 1.5 1.7 1.5 1.7 1.68 1.68 1.68 1.68 1.68 1.68

Documented 0.000 0.25 0.79 0.79 0.83 0.83

11/01 11/02 05/03 11/03 05/04 11/04 06/05 11/05 05/06 04/07 11/07 05/0803/0

912/09

NOTE: Increase of 32 GB of restricted data due to LC23 Airborne SIVAM military flights.

Metadata Registration Status by Metadata Registration Status by ComponentComponent

97 14 366

42 5 27

23 4 21

25 1 37

0 100 200 300 400 500

BR-US

BR-EU

BR

Pre-LBA / Other

Data Sets with data available 366 27 21 37

Data Sets w/o data 14 5 4 1

Posters 97 42 23 25

BR-US BR-EU BR Pre-LBA / Other

N=48

N=74

N=477

N=63

There are still a few data sets with unavailable or restricted data and several BR and BR-EU teams have only posters registered -- no data.

Data Progress toward Long-Term Archive November 2007 – December 2009

Data Progress toward Long-Term Archive November 2007 – December 2009

0

100

200

300

400

# o

f D

ata

Sets

Preliminary 2 13 65 33 32 17 1 2

Final 23 19 345 321 195 133 7 36

Preparation for Archive 21 60 90 90 9

Submitted for Archive 26 43 60 70

Archived at ORNL DAAC (LBA-ECO only) 10 10 31 49

Archived at LBA DIS 10 10 31 51

BR BR-EU

LBA-

ECO

(11/ 07

LBA-

ECO

(5/ 08)

LBA-

ECO

(3/ 09)

LBA-

ECO

(12/ 09

LBA-

Hydro

met

Pre-

LBA &

Other

Data Maturity

Investigator responsibility (including documentation)

Preliminary

DocumentedFinal, QA’d

Archive-ready

Project Office responsibility

Data Set DocumentationData Set Documentation• Investigator uses the LBA Metadata Editor (LME) to add the

documentation to original metadata file• Documentation fields:

• Data Set Overview• Data Characteristics• Data Application and Derivation• Quality Assessment• Data Acquisition Materials and Methods• References

• LME metadata (including the documentation fields above) is exported as a Data Set User’s Guide and placed on the ftp site along with the data. – There are currently 200 User’s Guides online in the LBA-ECO

archive prep area at ORNL DAAC.

• This documentation is desirable for all LBA data sets, but required for data sets to be archived at ORNL DAAC

Data Documentation Status- November 2007 -

Data Documentation Status- November 2007 -

0

100

200

300

400

# o

f D

ata

Sets

0

3

6

0 25 30 339 16 33

1 2 7 1 0

2 10 0 0

3 11 0 0

4 11 1 1

5 11 3

6 21 2

In archive process 57

Archived/Documented 10

BR BR-EULBA-ECO

HydroMet

Pre-LBA & Other

Number of documentation fields

completed in metadata

Undocumented

Documentation is complete, and data has been formatted to ORNL DAAC archive specifications

Data Documentation Status- May 2008 -

Data Documentation Status- May 2008 -

0

100

200

300

400

# o

f D

ata

Sets

0

3

6

0 25 30 319 16 33

1 2 6 1 0

2 7 0 0

3 10 0 0

4 11 1 1

5 19 3

6 115 2

In archive process 102


BR BR-EULBA-ECO

HydroMet

Pre-LBA & Other



LBA-ECO data sets are moving from here, i.e.

undocumented

…to here, i.e. documented and formatted for ORNL DAAC archive

Data Documentation Status- March 2009 -

Data Documentation Status- March 2009 -

0

100

200

300

400

# o

f D

ata

Sets

0

3

6

0 25 30 238 16 33

1 2 7 1 0

2 6 0 0

3 3 0 0

4 5 0 1

5 15 0 3

6 158 1 2

In archive process 151 9


BR BR-EULBA-ECO

HydroMet

Pre-LBA & Other



Should documentation be an LBA priority as well ?


undocumented


Data Documentation Status- December 2009 -

Data Documentation Status- December 2009 -

0

100

200

300

400

# o

f D

ata

Sets

0

3

6

0 25 24 149 14 28

1 8 8 1 4

2 11 0 0

3 2 0 0

4 6 1 1

5 14 0 3

6 172 1 2

In archive process 160 9


BR BR-EULBA-ECO

HydroMet

Pre-LBA & Other



Should documentation be an LBA priority as well ?


undocumented


Why the increase in documented LBA-Why the increase in documented LBA-ECO data sets?ECO data sets?

• All LBA-ECO data sets will be archived at ORNL DAAC and must be documented to satisfy DAAC requirements

• Diane: “Don’t make me come up there!” (Sept. ’07)• Hired another “data chaser” (Aug. ’07)

– Now a staff of two (Gentry & McGroddy)

• Additional data chaser has enabled us to provide even more one-on-one assistance to data providers in archive preparation– Documentation support– Data reformatting / reorganization

Data Registration & ArchiveData Registration & ArchiveOngoing tasks ….Ongoing tasks ….

• Continue the effort to get data into the archive and corresponding metadata registered

• Some early LBA projects (BR and BR/EU teams) have still not delivered data to LBA DIS.

• Email announcement was sent out to LBA community requesting each to review their contributions. Shall we send another reminder?

• There may not be any way to identify all data that should be delivered to the archive: LBA-ECO has used publications as the guide and this method has proved somewhat successful.

• Continue to identify and repair broken data links in the metadata

Data Registration & ArchiveData Registration & ArchiveOngoing tasks ….Ongoing tasks ….

• Continue the effort to prepare data for long-term archive

• LBA-ECO data sets are becoming final; fully quality-assured data is replacing preliminary data; and the data are being reformatted and documented for archive at the ORNL DAAC. • These final data sets & documentation will be provided to LBA DIS

as well.• Preliminary data replaced by final data are identified; users are

directed to the final, documented version.

• This process requires close coordination between LBA-DIS and LBA-ECO DIS staff to ensure that the two archives are in sync, and involves many “data housekeeping” chores to ensure that the most recent versions of data are made available.

• And of course, all of the data are being backed up regularly, per archive protocols.

Data Registration & ArchiveData Registration & ArchiveOngoing tasks , cont.Ongoing tasks , cont.

• Increase in the number of LBA-ECO data sets in the archive process is mainly due to increased LBA-ECO investment in data ‘chasing’ and requires a dedicated level of support for this task.

Number of files in LBA DIS Number of files in LBA DIS Archive, by disk areaArchive, by disk area

504695

8152 1104 11690

100000

200000

300000

400000

500000

600000

Disk Areas

Number of files

TOTAL Nov 2005: 349.664TOTAL Nov 2005: 349.664

TOTAL May 2006: 361.766

TOTAL Apr 2007: 384.415

TOTAL Nov 2007: 498.283

TOTAL May 2008: 499,356

TOTAL Mar 2009: 504,536

TOTAL Dec 2009: 515,120

Updated December 03, 2009

Increase in number of data sets archived -- >

Over 1.5 Million Files(Data + System files)

Many requests for LBA Data

Thank you note from researcher for LBA data sent on Nov 2009.

High demand for LBA Data(ex.: ftp log from Dec 03,2009

University of Illinois UIUC-NCSA IP:130.126.146.5)

Worries – Infra-Structure andWorries – Infra-Structure andSystem UpdatesSystem Updates

• Central Office – Manaus-AM A T I V I D A D E S: Adm. de Infra-estrutura 1 - Administração da infra-estrutura de rede do Escritório Central. 2 - Administração do cluster Supermicro. 3 - Administração do Sistema NEC Sx8i. 4 - Suporte aos usuários do Escritório. 5 - Suporte às ações de telemática na ZF2. 6 - Suporte ao curso de Mestrado e Doutorado em Clima e Ambiente

(CLIAMB). 7 - Suporte a biblioteca do INPA, no sistema bibliopac. 8 - Suporte aos sistemas remotos, que incluem, VPN, estação GPS e

máquina do WWLLN.

Technical Team: 1 Support Analyst – 8h 1 Network Administrator – 4hour


• Central Office – Manaus-AM A T I V I D A D E S: Software Development

1 - Databases: Projects, Research Groups, Logistics, Meteorological

2 – Sistemas Web – LBA Portal (Maintenance). 3 – Mo Porã – Repository Manager vs 3.0

Development Team: 1 System Analyst – 8h 2 Programmers – 4hours each

Need 2 Programmers 8hours each


• Central Office – Manaus-AM

Hardware: Upgrade of Desktop for Development and Network Administrator.Investments required - (R$ 25mil).

Software: New licenses – Anti-virus and Operational Systems for specific purposes.Investments required - (R$ 35mil).

TI Team: 1 Network Administrator – 8hour1 System analysts – 4 hours. Need 2 System Analyst 8hours.

Investments required - (R$ 55-75mil/ano).

Version 3.0 is now available.

User Community: LBA, PPBio, Geoma, CTPetro, PSA, UFAL, UFES, e Rede GeoLab da Amazônia.


• Regional Office - Cachoeira Paulista-SP

2009, the Central Office just PURCHASED A NEW SERVER !!!

• New server Specs:– 2 quad core processors (8 CPUs)

• 8 Giga bytes RAM

– 500 G bytes drive for the operating system– 6 TERA bytes of storage for data– Brand new technology!– Deliver date set to December 18, 2009– 21th SSC recommendation fulfilled now!

• PS: Current server was bought in 2001.


• Regional Office - Cachoeira Paulista-SP (cont.)

Hardware: need to acquire new equipments.Investments required - (R$ 20mil).


TI Team: 1 Data Management – 8hour - Need 1 System Analyst - 8hours.

Investments required - (R$ 20mil/ano).


Regional Office - Santarém-PA- TI Support for LBA and partner institutions and Projects:

SFB,UFOPA, Geoma and SIGES.

Hardware: All hardware dates back 2003 -2005. It is required URGENT upgrade, if not, DIS will be no longer able to provide support (this can happen anytime).Investments required - (R$ 75mil).


TI Team: 1 Network Administrator – 8hour 2 System analysts – 4 hours each.

Need 8hours each.Investments required - (R$ 35-45mil/ano).


Additional Investments Required:

Central Office: R$ 135milRegional Office – Cachoeira Paulista: R$ 65milRegional Office – Santárem: R$ 150mil

Investments for Technical Training – R$ 25mil

TOTAL NEEDED: R$ 375MIL Reais.

LBA 2 – DIS Future (?)

• What does SSC expect from DIS?

• Planning for 2010-2011

• Revitalization of Offices’ infra-structure

• Plans for IT Team Training

• Budget for IT activities

Thank You !

DIS Working Group Report LBA Science Steering Committee Meeting - 24 rd Manaus - Amazonas - Brazil...

Documents

Transcript of DIS Working Group Report LBA Science Steering Committee Meeting - 24 rd Manaus - Amazonas - Brazil...