IceCube Project Monthly Report February 2008

Accomplishments

• All on-ice drilling and installation activities were closed after completing the drilling and string deployment of a total of 18 strings.

• Developed a complete drilling equipment inventory with maps and photos and retrograded

damaged components of the EHWD for repair. • Verified the timing performance and calibrated the geometry of all new strings that are now

ready to be used for IC40 running. • Installed the monitoring software at the South Pole and upgraded the verification web page for

posting and viewing updated plots. • The first successful runs with the 40 string detector have been taken following improvements

to the DAQ software. The current planned start of IC40 running is set for April 1st. • Overall 92.3% detector uptime for the month. Resumed normal IC22/IceTop/Amanda data

taking after a very successful and productive commissioning season. Cost and Schedule Performance – The project is 81.7% complete versus the plan of 82.2% complete, as measured using earned value techniques.

15.0

39.5

85.7

140.1

199.4

235.3

261.0

274.5 276.2 276.6

263.9

231.6

0

50

100

150

200

250

300

02 03 04 05 06 07 08 09 10 11

FISCAL YEAR

DO

LL

AR

S IN

MIL

LIO

NS

Total Funds (US & NonUS)

Current Budget (BCWS)

Original Budget (BCWS0)

(excluding contingency)

Earned Value (BCWP)

Actual Cost (ACWP)

CONTINGENCY %ETC

INITIAL %

CURRENT % 12.76 26.3

40.14 23

Scheduled, %

Performed, %

Actual Cost, %

%

82.2

81.7

81.2

DEPLOY INITIAL STRINGS & TANKS

INITIAL OPERATIONS & DATA

ANALYSIS CAPABILITY

FULL SCALE DOM PRODUCTION

FULL OPERATIONS

& DATA ANALYSIS

CAPABILITY

DETERMINE FULL DEPLOYMENT RATE

COMPLETE EHWD INTEGRATION & TEST

Initial In-Ice Strings & IceTop Tanks Installed Jan-2005

Initial Operational Capability Mar-2007

Project Completion & Closeout Sep-2011

Total Project Cost $276.6

Value of Foreign Contributions $34.5

NSF Funding $242.1

Contingency as % of Remaining Costs 26.3

# Strings/Tanks 70/160

31-Jan-08Performance and Cost through:

IceCube Project Baseline (M$)

2

Note 1

Actual Cost Latest

Work Work of Work Budgeted Revised

OBS Structure L2 Scheduled Performed Performed Schedule Cost Assigned % ETC AY $s Estimate Sched Perf

Project Support 22,506.2 22,506.2 22,199.9 0.0 306.3 205.6 5.0% 27,293.4 26,344.7 948.7 82.5% 82.5% 81.3%

Implementation 32,978.5 33,059.2 33,236.7 80.7 -177.4 2,519.2 19.8% 44,477.5 45,953.8 -1,476.3 74.1% 74.3% 74.7%

Instrumentation 65,162.9 63,733.5 63,096.1 -1,429.3 637.5 586.5 6.3% 72,374.2 72,374.2 0.0 90.0% 88.1% 87.2%

Data Acquisition 33,073.4 33,042.2 32,980.4 -31.2 61.8 114.5 13.2% 33,846.2 33,846.2 0.0 97.7% 97.6% 97.4%

Data Systems 22,520.7 22,378.0 22,480.9 -142.7 -102.9 110.0 2.6% 26,781.5 26,781.5 0.0 84.1% 83.6% 83.9%

Detector Comm. &

Verification17,695.6 17,684.1 17,955.6 -11.5 -271.6 369.6 9.2% 21,962.7 21,962.7 0.0 80.6% 80.5% 81.8%

Pre Operations 412.0 412.0 373.5 0.0 38.5 0.0 0.0% 533.8 533.8 0.0 77.2% 77.2% 70.0%

Subtotal 194,349.3 192,815.3 192,323.1 -1,534.0 492.3 3,905.4 11.0% 227,269.4 227,796.9 -527.5 85.5% 84.8% 84.6%

RPSC SUPPORT 21,644.9 21,835.7 21,121.8 190.8 713.9 1,530.9 10.9% 35,339.9 35,184.2 155.6 61.2% 61.8% 59.8%

NSF 803.5 803.5 803.5 0.0 0.0 27.6 6.0% 1,263.0 1,263.0 0.0 63.6% 63.6% 63.6%

Total 216,797.8 215,454.5 214,248.4 -1,343.2 1,206.2 5,463.8 10.9% 263,872.3 264,244.2 -371.9 82.2% 81.7% 81.2%

CONTINGENCY Notes 3,4 12,755.2 12,383.3 371.9

IceCube Total Note 2 216,797.8 215,454.5 214,248.4 -1,343.2 1,206.2 5,463.8 10.9% 276,627.5 276,627.5 0.0 82.2% 81.7% 81.2%

Notes: 1 Incorporates approved baseline changes.

2 Total Budget at Completion includes non-US contributions 4,857 K over the amount in the post Hartill III baseline of:

3 Budgeted contingency is: 26.3% of the Budgeted cost of work remaining.

26.1%

5 All latest revised estimates detailed planning for PY6-10

Complete (%)

$29,698 K

Budgeted Cost 2

Variance Contingency

of the Estimated Cost to Complete (ETC)

Actl

Cost

4 Budgeted contingency is:

Variance

6 Contingency is assigned to each L-2 element based on the ETC, a bottom-up risk assessment model, management judgement, and cost constraints.

Cost Schedule Status Report

IceCube Neutrino Observatory

Cumulative To Date (AY K$) At Completion (AY K$) Note 5

Reporting Period Ending: 1/31/08

01/31/08 Change Log – IceCube Total Project Budget Baseline ($K)

No. Description Date Approved

Total Baseline

Allocated Budget

Allocated Budget Change

Contingency Budget

Estimate To

Complete (ETC)

Budgeted Cost of Work

Remaining

Contingency % of

Remaining Work

NA Status as of December 2007

276,628 263,737 0 12,891 50,871 50,472 25.5%

CR118 CR 0118 PY5 WISDM PMCS Reconciliation

03/05/08 276,628 263,872 135 12,755

NA Status as of January 2008

276,628 263,872 0 12,755 48,790 48,418 26.3%

3

Risk Assessment & Potential Contingency Allocations

Item Description Potential Contingency

Allocation

Assign contingency to mitigate technical, cost and schedule risks associated with the defined scope of work. Risk model applied at WBS-Level 4 to assess appropriate contingency as a percent of the cost of work remaining.

$5,464K

The cost to retro IceCube equipment/materials from the South Pole at the end of the project was not included in the baseline budget. This number represents a worst-case scenario if all drill equipment is returned after the 2010/11 drilling season. A more likely scenario would cost less than one half of this amount.

$1,500K

Additional budget for drill cables, reel motors, strain reliefs and spare hose segments based on season 07-08 results. The increased hose service life gained by increasing the storage temperature is not yet quantified.

$500K

Pre-Operations activities may be extended beyond the one year currently budgeted. During the first three years of initial operations it may be necessary to conduct engineering runs concurrent with operations to debug software and ensure reliability of the installed equipment.

$667K

Restore 80-string configuration. This is the current estimate of the cost to restore the array to the 80-string configuration described in the original IceCube proposal.

$3,294K

Estimate of the RPSC fixed cost for the 2010/11 drilling season. This will be addressed in the following month. $1,330K

26.3%

0%

10%

20%

30%

40%

Jan-

04

Apr-

04

Jul-

04

Oct-

04

Jan-

05

Apr-

05

Jul-

05

Oct-

05

Jan-

06

Apr-

06

Jul-

06

Oct-

06

Jan-

07

Apr-

07

Jul-

07

Oct-

07

Jan-

08

IceCube Contingency: % of Remaining Work

Annual Baseline Re-Plan

Additional Installation Season

10 Ice-Top Stations

Annual Baseline Re-Plan Annual Baseline Re-Plan

Instrumentation

Re-Plan for

75+ Str.

Implementation 58%

Instrumentation 29%

DAQ 11%

RPSC 8%

Project support -10%

Others 4%

Contingency Usage

4

Drill Operation and Installation – All on-ice drilling and installation activities were closed and all but a few personnel left the South Pole on February 3rd. A complete equipment inventory with maps and photos was developed. A preliminary activity list was developed in preparation for off-ice schedule development. An equipment and spares list was generated for procurement. Damaged components of the EHWD were retrograded for repair in CONUS. Digital Optical Module and Cable Production Status and Plans - DOM production is in progress at all 3 sites and is progressing smoothly. Surface cables for the 2009/2010 season have been ordered and will start production at Ericsson soon. Surface to DOM cable production is normal.

IceCube DOM Integration PY6 (April, 2007 to March, 2008) - Plan vs. Actual

0

200

400

600

800

1000

1200

1400

1600

1800

2000

4/7/

2007

4/21

/200

7

5/5/

2007

5/19

/200

7

6/2/

2007

6/16

/200

7

6/30

/200

7

7/14

/200

7

7/28

/200

7

8/11

/200

7

8/25

/200

7

9/8/

2007

9/22

/200

7

10/6

/200

7

10/2

0/20

07

11/3

/200

7

11/1

7/20

07

12/1

/200

7

12/1

5/20

07

12/2

9/20

07

1/12

/200

8

1/26

/200

8

2/9/

2008

2/23

/200

8

3/8/

2008

3/22

/200

8

Week ending

Qu

an

tity

of

DO

Ms r

ead

y f

or

DF

L

DOMs ready for DFL @ PSL - cumulative plan

DOMs ready for DFL @ DESY - cumulative plan

DOMs ready for DFL @ Swed - cumulative plan

DOMs ready for DFL - cumulative plan all sites

DOMs ready for DFL - cumulative actual all sites

The difference will be made up in PY7 and

is not a problem

Detector Commissioning and Verification - Effort in late January and February focused on verifying the timing and geometry of the newly-deployed strings. All new strings have had their timing performance verified and their relative z-positions calibrated and are ready to be used for IC40 running. This critical path deliverable was scheduled for March 23, 2008, and was actually delivered about one month early on February 28, 2008. The data acquisition at Pole took only 20 minutes per string for geometry and eight minutes per string for timing. Both analyses use in situ flashers. The subsequent analysis of the data takes several hours of CPU time, and was done on a dedicated cluster at the South Pole. Timing resolution was found to be considerably worse than in 2007. When the measurement was done in 2007, the TestDAQ was used since the official DAQ system, pDAQ, did not yet have support for running the in situ flashers. The pDAQ system used a different algorithm for correcting DOM local time to UTC and at the high data rates at which this measurement was made this resulted in stale calibration values being used. The problem was quickly fixed by pDAQ personnel and new data taken after the fix showed the expected resolution. Although this

5

problem is not expected to affect 2007 physics data, all of which was taken with pDAQ, studies are underway to try to quantify the effect. IceCube string 48 flasher data was taken to verify the relative AMANDA/IceCube timing. Since then a new and better method for finding this timing offset was devised, and this data will be used primarily as confirmation that the new technique has worked. Additional data with string 48 was taken for the purpose of further understanding the hole ice properties. The monitoring group installed the software needed to provide personnel at Pole with access to the standard monitoring pages. An upgraded web page for viewing plots produced by verification was created and installed at the South Pole and will also be used in the North. Three modifications to AMANDA software and hardware were tested together at the end of the season to insure that they would work in concert when we change over to IC40 running. These modifications were: 1) A change in firmware to reduce the sampling rate on a subset of channels by a factor of two,

to reduce the bandwidth requirement of AMANDA TWR data. 2) A replacement of the Trigger MB to remove the time jitter that is currently in the TWR data. 3) An upgrade to the trigger, which may or may not be implemented for the 2008 data taking

season pending TFT discussion. Data Acquisition Software - The first successful runs with the 40 string detector have been taken following improvements to the DAQ software. Initial tests at the South Pole with 40 strings were not successful due to a bottleneck, which was eventually found and resolved. Software effort was focused on the following tasks:

Increasing the throughput of the StringHub to allow passage of all hits through the system - not just those that have passed a neighboring-channel-coincidence in the ice.

Preparing additional DAQ triggers for the IC40 run. Bug-fix release 'Warners' was prepared and deployed to Pole on February 28, 2008. Work on low-level DOM calibration code including a new release of the DOM-resident

calibration application to calibrate out a newly discovered system variable: the delay time between pulses digitized by different digitizer channels in the 'ping-pong' mode.

Release of new DOM main board software to incorporate last bullet and also to provide additional functionality for IceTop DOMs - PMT pulse charge histograms are now computed inside the DOM and passed into the monitoring stream.

Improvements to the northern hemisphere DAQ testing framework including a new mode, which allows capture of raw data at Pole, which can then be 'played back' on SPTS system.

The current planned start of IC40 running is set for April 1st. We expect approximately one to two weeks following that point where the data quality will need careful scrutiny and quick turnaround for bug fixes.

6

IceCube Detector Operations

Detector Up Time 92.3% IceCube Clean Run Up-Time not including AMANDA Array 83.3% IceCube Clean Run Up-Time including AMANDA Array 68.6% Unscheduled Downtime 0.1% Events from DAQ 1.59 billion

IceCube Detector Operation for February 2008

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29

Day

Pe

rce

nta

ge

of

Da

y

Unscheduled Downtime

Maintainance

C&V

Special Tests / commiss. of new strigs

DOM Calibration

Weekly Benchmark/Calibrations

Partial IceCube + AMANDA

Partial IceCube

AMANDA Only

IceCube Only

IceCube+AMANDA

Data taking during February 2008 achieved an overall uptime of 92.3%. On February 11th, the last system experts left the South Pole. With their departure we resumed normal IC22+IT26+Amanda data taking after a very successful and productive commissioning season. Before they departed, as was the case in January, tight coordination kept the physics data-taking (with IC22) going while interleaving low-level commissioning, system maintenance and high-level commissioning activities. This effort was very successful as demonstrated by the 83.3% clean runs Up-Time for IceCube (in-ice). A problem with the DAQ implementation of the IceTop configurations in use since November 5, 2007, which exchanged the usage two discriminators of the DOM main board, was discovered in February and configurations were reverted to the previous scheme. The IceTop data from November 5, 2007 until February, 28 2008 is of questionable quality. The following commissioning and calibrations activities took place:

Low level commissioning tasks on new strings, including initial calibration High level commissioning activities including timing and geometry determination Changes to the AMANDA triggering system (hardware and software) that allow changes

of configurations through software

7

Tests of DAQ runs IC22+12 with all of IceTop and various runs with the entire IC40 (excluding late-freeze or trouble DOMs). Some of these tests included down-stream systems (merging, monitoring, verification, etc).

A new version of the pDAQ software (Warners) was deployed during February. Last, due to a combination of problems/factors the satellite transmissions of the physics data accumulated a large backlog of ~400 Gigabytes, compared to the nominal daily transfer of 30G, or the summer values in the 50-60G regime. Amongst the causes were North-operator delays on queuing GRB stream data, satellite outages and software problems handling a spike in queued files. Solutions to the current backlog are being considered and a plan is expected soon. Data Systems - South Pole systems in the IceCube Laboratory continued normal IC-22 physics operation in February. The figure below shows the daily satellite data transfer rates. There is periodic testing and preparation of the South Pole computing system for the upcoming IC-40 data run. The systems are ready with no delays in IT systems at pole. Continued operation, maintenance and upgrades to the UW data center and data warehouse to keep pace with incoming engineering data from IC-22. Continued upgrades of disk to the data warehouse to accommodate higher level processing of IC-22 data, store IC-22 simulation data and prepare for IC-40 data starting April 1, 2008.

8

The Joint-Event Builder and Processing and Filter server (JEB/PnF) merging of IceCube-22 and Amanda TWR data streams continues at South Pole sending filtered data sets of 30-60 GB/day over the satellite. This higher than average data rate is to catch up on a backlog of filtered data at the South Pole that resulted from the TDRSS system issues earlier this year. Planning for online filters for this year’s run of IC-40 continues, with all filter proposals to the Trigger-Filter-Transmission Board (TFT). The new IC-40 run is scheduled to start with filters in place on April 1, 2008. There are no significant issues in the filtering system that would cause a delay in the start of IC-40 physics running. In offline software, the official release of IceTray version 2 is now in standard use for online and offline reconstruction and has continued to go smoothly with no major issues. IceTray Version 2 supplied several key improvements to the IceTray framework, including improved I/O performance and flexibility to allow higher data rates from DAQ as the detector grows and provide more efficient and speedier processing of data in the north. Improvements of code testing, documentation and code reviews continue. Offline processing for IC-22 data has made significant progress recently. The data processing through level 2 (the high level reconstructed data set ready for the physics working groups) is now caught up to the data coming over the satellite. This is the first report where the offline processing through level 2 is up to date and represents a milestone in the offline processing system. We expect to keep up with the data through the end of the run with the resources in place at the data center. Mass simulation production continues with IceSim Version 2 to produce large background and signal Monte Carlo datasets for the physics working groups to prepare for IC-22 physics analysis. The distributed production system along with better tracking and monitoring of CPU time supplied at each institution continues to monitor the production sites. Continued improvement on the simulation towards a new release in 2008 for production Monte Carlo runs to be used for the IC-40 run next year. The experiment control (EC) system is running at Pole and coordinating all subsystem configuration start/stop and status information for all IceCube subsystems (i.e. p-daq, JEB/PnF/Spade/TWR-daq). The system now coordinates and runs the flasher calibration runs from the run control console using the pDAQ system instead of the old testDAQ system. Formal work at LBNL for experiment control in the MREFC is scheduled to be completed at the end of this project year, with the system transitioning to M&O on April 1, 2008. A closeout review occurred on December 3rd at UW to make final preparations for the hand-off and documentation of the experiment control system into the M&O phase with personnel at UW. This is major milestone and represents the upcoming closeout of this L3 MREFC category at the end of this project year.

9

Quality Assurance and Safety – No significant issues to report. Education and Outreach – IceCube team members from University of Wisconsin - Madison and the University of Maryland attended the NSF's Budget Rollout and Exhibitions event on February 4, 2008 at the NSF Headquarters in Arlington, Virginia. In addition to a display of a DOM and ECW gear, we had a slideshow of images from this season. The University of Maryland IceCube team showed events captured just days earlier (January 29, 2008) using their event display software. The software allows the user to rotate the event in three dimensions and to scale the event. IceCube was represented at the National Societies of Black and Hispanic Physicists annual meeting in Washington February 20 to 23, 2008. Students were recruited for the REU program and encouraged to consider the IceCube project for graduate and post graduate opportunities. The monthly reports are posted at IceCube Monthly Reports. The On-Ice Weekly Construction reports are posted at On-Ice Weekly Construction Report. Meetings and Events Acoustic Neutrino Detection Working Group @ UW-Madison April 27 – 28, 2008 GRB Workshop @ UW-Madison April 27 – 28, 2008 Collaboration Meeting @ UW-Madison April 29 – May 3, 2008 Science Advisory Committee Meeting @ UW-Madison May 5 – 6, 2008 Neutrino 2008 @ Christchurch, NZ May 26-31, 2008 International Oversight and Finance Group @ UW (proposed) June 10, 2008 NSF Annual Review of IceCube @ UW-Madison (proposed) June 11-12, 2008