TransPAC4 Award #1450904 Year 4 Annual and Quarter 4 1 … Y4...1 TransPAC4 Award #1450904 Year 4...

1

TransPAC4 Award #1450904

Year 4 Annual and Quarter 4 1 Dec 2017 through 30 Nov 2018

Jennifer M. Schopf, Andrew Lee – Principal Investigators

Summary The TransPAC project supports circuits and services for the use of 100G networks between the US and Asia, with a focus on measurement and end user support. Year 4 highlights include the deployment of two additional 10G circuits between Guam and Hong Kong, the subsequent unification of several open exchanges in Hong Kong, signing MOUs with three partners, support for network experiments at the Sapporo Snow Festival and SC18, use of NetSage tools to detect routing and performance anomalies, and significant science engagement.

1. TransPAC Overview The TransPAC project supports circuits and network services between the US West Coast and Asia. The TransPAC-‐PacWave 100G Circuit is a 100 Gbps link between Seattle, Washington, and Tokyo, Japan. This circuit has been in production since February 2016 and is the primary project circuit for production traffic for TransPAC4. This link is fully supported by NSF and is managed in cooperation with Pacific Wave and Pacific Northwest GigaPop (PNWGP). We also support two 10G circuits between Guam and Hong Kong, starting in September 2018. These circuits are used to support a wide variety of science applications and demonstrations of advanced networking technologies. In addition, the TransPAC award supports science engagement, experimental network research, measurement deployments, and security activities.

2. Staffing Changes during Year 4 to staff associated with TransPAC were minimal. Midyear, it was decided that Andrew Lee would shift his focus towards a stronger concentration on engineering issues, including investigating routing issues that have been discovered, understanding the causes of traffic fluctuations over the NSF-‐supported links, and helping to develop additional Grafana dashboards to better understand flow data for TransPAC. We will not be hiring an additional analyst at this time as had been mentioned in previous reports.

2

On October 1, Doug Southworth joined the team to assist with perfSONAR training and science engagement. Southworth has worked most of his career in tech related fields, and spent the last 10 years as a network administrator in K-‐12 education and the United States Courts. At the end of Year 4, project staff included: ● Jennifer Schopf, IN@IU Director ● Andrew Lee, Network Analysis ● Hans Addleman, Primary TransPAC Network Engineer ● Edward Moynihan, Science Engagement Specialist ● Scott Chevalier, perfSONAR and Training ● Doug Southworth, perfSONAR and Training ● Heather Hubbard, Project Support

We do not expect additional staff changes to take place in Year 5 until a project ramp-‐down is needed.

3. Conference and Workshop Travel TransPAC staff participated in various meetings to support their role in collaborations in Asia. Some of these trips were funded by sources other than TransPAC. The travel for the first 3 quarters of the year, detailed in those project reports, included: ● Addleman attended OIN 23 in New Orleans, LA, on December 5-‐6,

http://www.oinworkshop.com/. ● Addleman attended the Data Transfer Node/Fiona Workshop planning

meeting in San Diego, CA, on December 17-‐20. ● Moynihan attended the Winter Federation of Earth Science Information

Partners (ESIP) meeting in Bethesda, MD, on January 9-‐11, http://www.esipfed.org/meetings/upcoming-‐meetings/esip-‐winter-‐meeting-‐2018.

● Schopf and Lee attended the Trans Pacific Research and Education Network (TPREN) workshop in Honolulu, HI, on January 20.

● Schopf and Lee attended PTC’18 in Honolulu, HI, on January 21-‐24, https://www.ptc.org/ptc18/.

● Schopf attended the CENIC Spring Member Meeting, March 6-‐8, in Monterey, CA, https://cenic.org/conference.

● Moynihan attended the Large Hadron Collider Open Network Environment (LHCONE) meeting in Abingdon, United Kingdom, on March 6-‐7, https://indico.cern.ch/event/681168/.

● Lee and Brian Tierney attended APAN45 in Singapore, on March 25-‐29, https://apan45.singaren.net/.

● Schopf, Lee, and Moynihan attended Internet2 Global Summit in San Diego, CA, on May 6-‐9, https://meetings.internet2.edu/2018-‐global-‐summit/.

3

● Schopf, Lee, and Moynihan attended TNC18, Trondheim, Norway, June 10-‐14, https://tnc18.geant.org/.

● Addleman and Lee attended APAN46 in Auckland, New Zealand, August 6-‐10, http://apan46.nz/apan46.

● Jared Schlemmer and Jeff Terzino, members of the GlobalNOC team who provide engineering services for TransPAC, installed equipment in Hong Kong to support the Guam-‐Hong Kong circuits on August 5-‐8.

● Addleman attended the NSF Cyber Security Summit in Alexandria, VA, August 21-‐23, https://trustedci.org/2018-‐nsf-‐cybersecurity-‐summit/.

In Quarter 4, travel included: ● Moynihan attended the 2018 Mekong Research Symposium in Ho Chi Minh

City, Vietnam, September 6-‐7, https://www.lowermekong.org/events/lower-‐mekong-‐research-‐symposium, where he moderated and presented in a session on new technical solutions for data collaborations.

● Lee and Moynihan attended the NORDUnet 18 meeting https://events.nordu.net/display/NDN2018 and the GLIF Americas and GLIF Annual meetings https://www.glif.is/meetings/2018, co-‐located in Helsingor, Denmark, September 18-‐22. During NORDUnet 18, they met with our TransPAC partner Pacific Wave to discuss peering arrangements, migration of the PacWave switch from WIDE to KDDI Otemachi, and upcoming maintenances of the TransPAC-‐PacWave 100G circuit. Moynihan gave a lightning talk on Science Engagement at NORDUnet 18. Lee presented on routing issues during the GLIF Americas meeting.

● Schopf attended the Internet2 Technical Exchange, Orlando, FL, October 14-‐19, https://meetings.internet2.edu/2018-‐technology-‐exchange. Schopf met with representatives from NSF and several IRNC partners to discuss project futures. This conference is one of the key coordination points for all of the IN@IU projects.

● Schopf visited NOAA in Boulder, Colorado, on October 22-‐25, 2018, to meet with the NOAA and EUMetSat weather satellite groups, who are currently evaluating additional network support for work between both groups and Japan and India. Additional meetings took place to evaluate an extended perfSONAR mesh to specifically support the weather satellite teams.

● Moynihan attended the LHCONE meeting at Fermilab in Chicago, IL, October 30-‐31, https://indico.cern.ch/event/725706/. Moynihan met with partners to get updates on LHCONE infrastructure and to provide updates on IN@IU LHCONE and science engagement activities.

● Schopf, Lee, Addleman, Chevalier, and Southworth attended the SC18 Conference in Dallas, TX, November 11-‐18, 2018 https://sc18.supercomputing.org/. Schopf and Lee met up with researchers from organizations including NASA, the National Radio Astronomy Observatory (NRAO), the KEK High Energy Accelerator Research Program (Japan), the Mozambique Research and Education Network (MoRENet), NOAA, the European Bioinformatics Institute (EBI), NIH, and others, about supporting their project use of TransPAC resources. Addleman was SCinet

4

WAN Team Deputy Chair and worked during staging, setup, and show. Chevalier worked on the SCinet DevOps team for setup and show with focus on pSConfig communication in nodes and display using MaDDash.

On August 17, International Networks at Indiana University (IN@IU) celebrated its Twentieth Anniversary. Several members of the community joined the current team in Bloomington for an afternoon reception, including Inder Monge (ESnet), Cathrin Stover (GEANT), Kazunori Konishi (APAN-‐JP), and Kevin Thompson (NSF). Indiana University President McRobbie, who started the IN@IU project in 1999, gave a keynote address. Konishi-‐san was awarded the Thomas Hart Benton Mural Medallion for his long-‐term collaboration with the team. A press release for the event is available online at https://news.iu.edu/stories/2018/08/iu/releases/17-‐international-‐networks-‐celebrates-‐20-‐years.html.

4. Collaborative Activities 4.A Collaborations with IRNC Partners Collaboration with the IRNC AMIS awardee, NetSage, is moving forward successfully. NetSage is currently capturing ongoing NetFlow, and SNMP data for TransPAC. Live network statistics from TransPAC can be viewed on the NetSage portal at https://portal.netsage.global/grafana/dashboard/db/bandwidth-‐dashboard?refresh=1d&orgId=2. We provided feedback for the flow data dashboards that were released in October. Moynihan also worked with the NetSage project to populate the Science Registry with information to tag by science domain, project, location, and educational institution endpoints, focusing on the TransPAC Top Ten Talkers list. We also worked with the NetSage project to develop dashboards to aid in the creation of the graphs and charts in this report, and to provide time-‐constrained views of flow data to enable our analysis work, as discussed in Section 5.C. In Year 5, we plan on continuing our collaboration with NetSage on the Science Registry and further developing these dashboards to provide additional visualizations of traffic patterns and to combine different data sources into a single dashboard for easier identification of performance anomalies. At the IRNC PI meeting in May at the Internet2 Global Summit, Lee presented on TransPAC and Moynihan presented his experience in adding TransPAC data to the Science Registry. Schopf and Lee also attended the IRNC PI meeting held at SC18 and provided updates on TransPAC as well as our other projects.

5

We maintained a close collaboration with Pacific Wave, not only through our joint support of the TransPAC-‐PacWave 100G circuit but through bi-‐weekly calls to coordinate activities support ensure that our services and resources are complimentary. This collaboration in part has also led to additional engagement between our group and the Pacific Research Platform (PRP) and ESNet. In addition, we are collaborating on their pilot deployment of the Automated GLIF Open Light Path Exchange (AutoGOLE) using Network Services Interface (NSI), discussed in Section 7. In January, we participated in the annual Trans-‐Pacific Research & Education Networking (TPREN) workshop hosted at the University of Hawaii, and we continue to collaborate with PIREN on work with the Guam Open Exchange (GOREX), wider trans-‐Pacific network connectivity, and engagement in the Pacific Islands. We rely on this relationship to identify relevant Pacific Island-‐Asia research in need of our science engagement support and training.

We worked closely throughout the year with the Universities of Guam and Hawaii in planning and executing our Guam to Hong Kong expansion. GOREX staff at the Universities of Guam and Hawaii were very helpful during the procurement process, working with the local Guam telco during the installation of our circuits, and the installation of the IXIA tester we used to verify clean operation of the circuits. In Year 5, we will continue work with the GOREX engineers and our peer networks in Hong Kong to adjust routing so that data between Asia and GOREX participants flows directly across the Guam-‐Hong Kong links rather than through the mainland US.

The IRNC NOC provides Tier 1 support services including monitoring the state of the trans-‐Pacific circuits and the equipment installed in Seattle and Hong Kong. TransPAC contracts with the IU GlobalNOC to supply Tier 2 and Tier 3 services. 4.B Collaborations with Asian Partners We continue to build valuable collaborations with international partner organizations around the world. In Year 4, this included signing MOUs, participating in networking experiments, and several additional collaborations.

4.B.1 MOUs In Year 4, we signed three MOUs, each emphasizing the alignment of user engagement efforts with our partner organizations. Our goal is to better understand the research need for networking resources and to identify practical areas where we can support those research projects. The MOU with the Joint University Computer Centre (JUCC) (Hong Kong) was signed in March at APAN45. It enables closer collaboration based on mining flow data to

6

identify universities in Hong Kong that are sharing data with US researchers, increased collaboration as part of our Guam-‐Hong Kong circuit deployment, as well as cooperation on routing and peering issue. Also at APAN45, we signed an MOU with the Singapore Advanced Research and Education Network (SingAREN) (Singapore). The SingAREN MOU likewise enables closer collaboration for science engagement and acknowledges the role that the TransPAC-‐PacWave 100G circuit serves backing up the Internet2/SingAREN 100G that runs between Singapore and Los Angeles. At TNC18, we signed an MOU with the China Education and Research Network (CERNET) aimed at improving communications between our networks, increasing peering, additional science engagement activities, and promoting further collaboration among the exchange points in Hong Kong. Following the signing of the MOUs, we worked with JUCC and SingAREN to set up teams for ongoing dialogue, shared existing knowledge of research collaborations, and set goals for our initial outreach to international research collaborations. We also began looking more deeply into the available flow data to help identify additional researchers for potential outreach and support activities, detailed in Section 4.C.4. In Year 5, we will continue our collaborative engagement work as outlined in the MOUs. We plan to expand on our previous work by focusing on potential engagement opportunities that arise out of specific flow analysis cases or are tied to specific science disciplines. We also are planning to re-‐engage CERNET to discuss opportunities for supporting Bioinformatics and other US-‐China collaborations, see Section 4.C.4. Additionally, we will have further discussions with Academica Sinica Grid Computing Centre (ASGC) (Taiwan) to potentially sign an MOU to codify our ongoing collaborative work with their team.

4.B.2 Network Experiments During the annual Sapporo Snow Festival in early February 2018, we supported a network demonstration conducted by Japan’s National Institute of Information and Communication Technology (NICT) in partnership with Pacific Wave and SingAREN involving 8k uncompressed videos. The Sapporo Snow festival is one of Japan’s largest winter events, and attracts over 2 million people from Japan and abroad every year. An 8k camera and monitor were set up at the festival in Sapporo, Japan, with a matching set up in Singapore. The video was transmitted between the sites using IP multicast on multipath, where one path went directly between Singapore and Japan directly and the other path went via the US. This allowed the experiment to load balance the traffic between the paths, even though one was significantly longer than the other. In addition to testing the video and network technologies, the demonstration made it possible to share the festival and reach a broader international audience.

7

In November, the TransPAC-‐PacWave 100G circuit was used by two separate demonstrations for SC18 that increased the traffic on the link for the week to over 80 Gbps, as shown in Figures 1 and 2. To make space for the experimental traffic, production traffic was rerouted to the backup connection.

Figure 1: Average utilization on the TransPAC-PacWave 100G circuit during the week of SC18.

Figure 2: Maximum utilization on the TransPAC-PacWave 100G circuit during the week of SC18. The values above the 100G maximum for the circuit are due to statistical anomalies that can be introduced in the polling data due to the delay in the response to a query, causing a polling interval to be slightly longer than expected followed by a slightly shorter interval. For example, if a circuit running at 100Gbps is polled 0.07 seconds later than expected, the first interval will report 107Gbps and the second interval will report 93Gbps.

The first experiment was a network visualization demonstration where production SC18 traffic, as well as test traffic generated across the TransPAC-PacWave 100G circuit using an IXIA test set, was fed into an ALAXALA router. Using proprietary technology, the data was visualized in real time. Traffic was transmitted from an IXIA tester at the SC18 venue out to Seattle, across the TransPAC-‐PacWave 100G to a loop in Tokyo that sent it back to the venue. They also received a 10G feed from the SC18 commodity internet connection that provided real user data through a tap provided by SCinet. Using processors built into the router, they classified both the 100G and 10G traffic based on the unsampled IP header information at line rate for source and destination country. The country to country statistics were then queried by a server using Simple Network Monitoring Protocol (SNMP) counters and the NIRVANA-‐Kai program developed by the NICT. The country-‐to-‐country transfer data was visualized in real time.

8

A second experiment also used the ALAXALA router to dynamically reprovision network and virtual machine resources around the globe to control an autonomous vehicle. The resources were allocated dynamically to reduce the latency between the car and its network-‐based controller. This allowed the car and controller to react quickly to real time events. The third experiment at SC18 was by Professor Hiraki of the University of Tokyo, where he was testing a new file transfer protocol that was designed to have very high performance over long distances. His goal is to be able to support sensitive data applications, such as medical research, with an encrypted high performance transfer tool. The experiment showed that it was possible to transfer data that was fully encrypted at high speeds and without high-‐end server hardware. The demo was able to achieve 75 Gbps encrypted disk-‐to-‐disk data transfers. In Year 5 they plan to try to extend this experiment to involve four 100G connections. We fully expect to continue to support demos similar to these in Year 5 at venues such as TNC19 and SC19. When we get requirements from our partners we will engage with them, gather requirements, and allocate resources as is reasonable.

4.B.3 Additional Collaborative Activities International Networks at Indiana University became a sponsor of the Global Lambda Integrated Facility (GLIF) (https://www.glif.is/). We have been participants in GLIF meetings and activities for many years, and this sponsorship shows our commitment to global R&E network research and collaboration. Similarly, we participated in the Global Network Architecture Technical working group (GNA-‐Tech) to develop documents regarding high level network architecture, exchange point operations, and measurement issues. At the GLIF meeting in Denmark in September, a community discussion began to evaluate the pros and cons of combining the technical activities of the GNA and GLIF. Our team will continue to participate in these discussions as they continue into Year 5. Lee serves as the co-‐chair of the APAN Backbone Committee (BBC), which provides a forum for NRENs in the region to discuss upcoming deployments, exchange points, and routing issues. We are also working with the APAN board to revamp the BBC to broaden its community appeal and better synchronize its activities with the Network Engineering Working group and Asi@Connect (formerly TEIN) community. In May, a proposal to Asi@Connect project was funded to conduct a pair of perfSONAR training workshops. These workshops will expand our training efforts to include participants from Lower Middle Income Countries that have not previously had access to dedicated perfSONAR training. These workshops are important to US researchers collaborating in these regions, as we will have better information about the state of the networks and will be able to troubleshoot both network and file transfer issues more effectively. The funding from this program is supporting in-‐

9

region participants to travel to the workshop sites, local workshop expenses, and a small amount of equipment. This work is detailed in Section 6.D.2. As part of the deployment of the Guam-‐Hong Kong circuits, we worked with several Asian collaborators to not only set up proper peering relationships but to establish better linkages between the various open exchanges present at iAdvantage’s MEGA-‐i data facility in Hong Kong. This is detailed in Section 5.A. In Year 4, we also evaluated the status of our outstanding collaborative items as outlined in the proposal. This included re-‐evaluating our efforts to support US branch campuses in Asia. At the Internet2 Global Summit in May, we met with staff from several US universities with campuses in Asia, Internet2 staff, and representatives from Asian NRENs who were involved with US branch campuses in Asia to continue discussions on the need for support and areas of collaboration. We discussed the potential for new perfSONAR deployments, for liaising with NRENs on behalf of US branch campuses, and for hosting workshops. However, similar to our previous attempts, these discussions did not yield useful results or identify mutually beneficial ways forward. Because these efforts were once again unsuccessful, we have decided that dedicated support for Asian branch campuses is not needed at this time. We will continue to monitor the environment and to support transnational education where appropriate, but continuing to pursue these efforts is not an efficient use of project resources. In Quarter 4, we learned about three 100G circuit projects being developed to support direct connectivity between Asia and Europe. The first is being overseen by a collaboration made up of GEANT (EU), SURFnet (Netherlands), NORDUnet (Nordic), AARNet (Australia), SingAREN (Singapore), and TEIN*CC (Asia). It includes a 15 year IRU for capacity that will connect Singapore to London. The second circuit will be supported by NII (Japan), and will run between Tokyo and Amsterdam. A potential third circuit will be supported by the Academia Sinica (Taiwan) and will connect Taipei, Singapore, and Amsterdam at 100G. All of these circuits are in the process of being procured and will likely be operational in Spring 2019. As the circuits become operational in Year 5, we will continue to work with our partners to determine potential areas of collaboration. 4.C Science Engagement 4.C.1 High Energy Physics In March, we followed up on the plan for better connectivity for LHCONE with Academia Sinica Grid Computing Centre (ASGC) (Taiwan), NICT (Japan), and ESNet (US). An agreement was reached to extend the LHCONE overlay network across the TransPAC-‐PacWave 100G circuit and down to Hong Kong. The connectivity was implemented in Quarter 2. With the new TransPAC capacity between Guam and Hong Kong, discussions have started to extend the LHCONE to directly connect in Hong Kong. We expect actions to take place during Year 5.

10

In Year 4, the WLCG Management Board reacted positively to a proposal that allows the Belle-‐II experiment in Japan to use the LHCOPN and LHCONE networks to transfer data. This will allow US researchers more efficient access to data generated by Belle-‐II.

4.C.2 Geoscience/Climate We sustained our efforts throughout Year 4 to target the US geoscience community, specifically through attendance at the Federation of Earth Science Information Partners (ESIP) winter meeting, where we worked to identify ways TransPAC can support geoscience researchers with large data transfer needs. At that meeting, we also met with NASA representatives to discuss current network use and to better understand future network requirements for major international atmospheric science applications that utilize TransPAC resources, including the Moderate Resolution Imaging Spectroradiometer (MODIS) , the Suomi National Polar Orbiting satellite (SNPP,) and the Visible Infrared Imaging Radiometer Suite (VIIRS). In October, we also met with representatives from NOAA and EUMetSat to discuss connectivity related to weather data. We are planning to attend AGU in December to further extend out collaborations with this community. We coordinated with Dr. Tho Nguyen from the University of Virginia (UVA) to plan the 2018 Lower Mekong Basin (LMB) Research Symposium that took place September 6-‐7 in Ho Chi Minh City, Vietnam. This workshop explored obstacles to international collaboration and data sharing and provided a venue for discussing future high-‐impact collaborations among LMB researchers, institutions, and global partners. Prior to the workshop, we shared results from IN@IU’s 2014 Lower Mekong workshop and worked with UVA to leverage the lessons learned and best practices. We also identified US scientists with collaborations in the Lower Mekong region. We partnered with UVA to help fund participation for Dr. Soe Myint, Professor of Geological Science and Urban Planning at Arizona State University, and Dr. Andrew Grimshaw, Professor of Computer Science at UVA. This funding allowed each participant to attend the workshop and present on their research and potential data sharing solutions. Moynihan attended the workshop, chaired a session on data transfer, and presented on how R&E networking infrastructure helps support international science collaborations. At the workshop, he met with members of PRAGMA, USGS, and the US State Department to discuss sharing community resources and coordinating future projects in support of US research in the region. Moynihan also visited Can Tho University and met with members of the Delta Research and Global Observation Network (DRAGON), an institute working to build a collaborative ecosystem for scientists from Vietnam and the US to share data on water and water-‐related topics. On the request of UVA, following the workshop, we connected with engineers at Thoylui University in Hanoi to better understand the connectivity between UVA and the Vietnamese Research and Education Network (VINAREN) to see if we could help improve performance. These discussions will continue into Year 5.

11

4.C.3 Astronomy We concluded an engagement with the Hawaii Pan-‐STARRS group and the Space Telescope Science Institute at Johns Hopkins to support large scale data movement, which resulted in a nearly 3 times speed-‐up of their transfers. This work was supported jointly between IN@IU and the IRNC NOC Performance Engagement Team (PET). Based on our findings, the Pacific Islands Research and Education Network (PIREN) was able to secure additional funding from the NSF IRNC program to upgrade parts of their network and deploy additional measurement tools. We are staying engaged during the build out and continue to provide advice and assistance. As part of our performance anomaly detection work, we also observed several large transfers over the TransPAC-‐PacWave 100G circuit related to astronomy traffic between Europe and Asia. There is a VLBI correlator located at NICT’s Kashima Space Technology Center (http://ksrc.nict.go.jp/about_e.html) in Kashima, Japan. VLBI correlators are servers responsible for getting data feeds from radio telescopes that are geographically distant and are aimed at the same celestial object. The correlator combines the incoming data to produce a result. From March to June, the source of the data was primarily the Onsala Space Observatory, a radio telescope facility in Sweden. From June through the end of the year, large spikes were observed for data coming from the Istituto di Radioastronomia in Italy, with the distinctive pattern of usage as shown in Figure 3. These data transfers were discussed with our network partners at GEANT and NICT to ensure the performance was meeting expectations.

Figure 3: Examples of the recurring data transfers that support VLBI astronomy research between Europe and Japan October 5 through November 7.

4.C.4 Other Science Engagement Jointly with ESnet engagement staff, we began planning a Science Engagement Deep Dive workshop at the University of Guam. This workshop will offer hands-‐on training with current research technologies (Science DMZ, DTNs, perfSONAR) as well as interactive, small group science engagement designed to understand and improve researcher data movement and workflow. Targeted attendees include water and energy researchers from the University of Guam and their collaborators,

12

astronomy researchers from the University of Hawaii, and researchers from nearby Pacific Islands with data transfer needs. We had originally targeted this for late August, however after surveying Guam EPSCOR researchers and other potential attendees in the Pacific region, we determined that more participation would be likely if we moved the workshop to a later date. We are currently waiting for the University of Guam to determine the best date for this to continue. As a result of the MOU with JUCC, we updated and shared research project databases to begin identifying researchers working jointly between the US and Hong Kong. This included work by Yi-‐Kuen Lee, Hong Kong University of Science and Technology, who has data-‐intensive IOT collaborations with MIT, and Dr. Tommy Tsan-‐Yuk Lam, The University of Hong Kong, who frequently downloads data from Argonne National Laboratory and the University of Chicago. We held initial discussions with each of these collaborations and continue working with JUCC to try to understand performance between these sites. At SC18, we reached out to many of the computing centers that transfer data using TransPAC circuits, including NASA, the National Radio Astronomy Observatory (NRAO), the KEK High Energy Accelerator Research Program (Japan), the Mozambique Research and Education Network (MoRENet), NOAA, the European Bioinformatics Institute (EBI), NIH, and others. We agreed to follow-‐up on these interactions with the hope that continued dialogue will lead to a better understanding of current and future data transfer requirements. Our researcher engagement efforts in Year 4 included sending outreach emails to 15 US-‐funded researchers to provide more information about the TransPAC project, assess if these projects have international data transfer needs, and offer assistance where necessary. These efforts focused on reaching out to researchers in regions supported by our new MOU partnerships (Hong Kong, Singapore, and China) and to NRENs of least developed countries in the APAN region to help identify potential projects in need of support. We continued our participation in several international science engagement and coordination projects. Moynihan is on the Steering Committee of GÉANT’s Task Force for Researcher Engagement Development (TF-‐RED) and participates in the Special Interest Group for Transnational Education. We also collaborated and shared best practices with the Pacific Research Platform, PRAGMA, the perfSONAR consortium, and the Joint Engineering Team (JET). In addition to continuing our active Year 4 engagements, in Year 5 we are also planning to develop a more targeted approach to our science engagement efforts. This approach will provide greater focus to our outreach efforts and help us become more integrated into the individual science communities we support. This approach will also help us answer specific questions about how different science communities use TransPAC resources and will provide greater insight into how effective our outreach and engagement efforts are in enhancing science outcomes

13

5. Circuit Deployments and Technical Updates

5.A Guam-Hong Kong Circuit Procurement The Guam Open Exchange (GOREX) announced they were officially operational on January 18, 2018, including a 100G circuit connecting GOREX to the University of Hawaii and onward to Pacific Wave in Los Angeles. At the January TPRE meeting, the most urgent missing path was identified as the link between Guam and Hong Kong, so it was decided this would be the best path for a second TransPAC circuit. We selected iAdvantage’s MEGA-‐i data facility in Hong Kong as the location on the Asian end in which to locate our equipment. This is the same facility that houses the Hong Kong Internet eXchange Research and Education node (HKIX-‐RE), the Hong Kong Open eXchange (HKOX), and NICT router that terminates the 100G Singapore-‐Hong Kong-‐Tokyo circuit that is part of the Asia Pacific Ring, which enables us to connect easily to all of them. In February, we published a Request For Proposals (RFP) for circuits with one end point in either Guam or Hawaii and the other in Hong Kong. The process concluded in May and, based on the pricing that we received, we opted to award two separate contracts each for a 10G circuit. Telstra was contracted for a 10G circuit using the Australia-‐Japan Cable (AJC) north to Japan and then south to Hong Kong on the Flag telecom North Asia Loop (FNAL) system, which became operational on August 16. AT&T was contracted for a 10G circuit using the Asia-‐America Gateway (AAG) system, which became operational on September 12. The AT&T circuit has a slightly shorter latency, but the Telstra circuit historically has been more resilient. On August 9, our engineers completed the installation of our equipment in Hong Kong. They deployed an Arista 7280R switch, perfSONAR server, file transfer server, and other supporting equipment. The Arista switch was chosen because it had the most effective power profile and the most capable buffering/forwarding for a switch in this class. Physical connections to the HKIX-‐RE, HKOX, and the NICT router were completed. In this way, we will be able to assist the Asian community in making a clustered open exchange point, combining all of these efforts, to enable better peering, routing, and higher speed traffic through the region. Once the circuits were put into production, we established peering relationships with networks in Asia and North America. The University of Hawaii/PIREN extended connectivity for VLANs put in place for these circuits over their 100G circuit from Guam to Pacific Wave in Los Angeles via Hawaii. This enabled direct peerings between US R&E networks and our router in Hong Kong, to better control the traffic levels. Our North American peerings established during Quarter 4 included Internet2, the Universities of Guam and Hawaii, and ESNet. Asian peerings included TEIN-‐HK and the Hong Kong Academic Research and Education Network (HARNET). Additional Asian peerings that we are working toward establishing in the next quarter include Preginet/ASTI (Philippines), Academia Sinica (Taiwan),

14

HKIX, KISTI (Korea), and CERNET (China) via a new physical connection to the CERNET GXP in Hong Kong.

5.B Technical Updates for the TransPAC-PacWave 100G Circuit A backup peering for the TransPAC-‐PacWave 100G circuit was put in place for APAN across the SINET 100G link during February. SINET did not request a reciprocal backup. In April, we established a JGN-‐ESnet peering over to better support LHCONE and the high energy physics community. We also established a new peering with REANNZ, the New Zealand NREN, across Pacific Wave. In Year 4 we also worked with Pacific Wave as part of their AutoGOLE deployment for self provisioning between the US and Asia. This is detailed in Section 7. The TransPAC-‐PacWave 100G circuit was very stable during Project Year 4. Please see Section 6.E for uptime statistics. One of the only times the circuit entered a degraded condition was in August. The performance was impacted but not to the point of an outage, so no alarms or alerts were triggered. On August 5, the Medical Working Group at APAN46 was setting up a demo, noticed poor performance over the circuit, and contacted the APAN-‐JP NOC. The APAN-‐JP NOC worked directly with Addleman and Lee to resolve the issue. Peering with APAN-‐JP (AS 7660) was turned off to force traffic onto our preconfigured backup path, and a ticket was opened with Pacific Northwest Gigapop (PNWG) who contacted TATA. TATA reset a card in their optical system and restored service. This was tested and confirmed and the BGP peering was restored. In October, a server was ordered that would be deployed to serve as a file transfer testing server and temporary archival storage to be located in our point of presence in Seattle. The hardware arrived in November, but due to the proximity to the Supercomputing conference, deployment will be delayed until next quarter. This equipment will enable temporary data storage on a well-‐connected and tuned host to increase the efficiency of long distance file transfers, similar to the server we deployed in Hong Kong. In Year 5, we plan to use these servers as part of the “Data Super Highway” supported by the Pacific Research Platform (PRP), and as local caches and archive nodes for large-‐scale collaborative science project to speed up their data transfers. In Quarter 4, we began tracking three new projects that together will create a 100G ring between the Asia-‐Pacific and Europe (See Section 4.A.3). In Year 5, we are planning to work with these collaborations to make sure that traffic routed between Europe and Asia is handled appropriately. This may result in a decrease in the load on the TransPAC-‐PacWave 100G circuit, as traffic that is currently sent between Europe and Asia via the US may be routed instead on one of these more direct circuits.

15

5.C Network Analysis At the end of 2017, we developed a map based on analysis of flow data for the TransPAC-‐PacWave 100G circuit to visualize the countries around the world that were sources and/or destinations of traffic. Our initial research showed that in 2017, 163 countries (84% of the world) sent or received data over the circuit, as shown in Figure 4.

Figure 4: World map showing the connectivity for TransPAC in 2017 and the 163 countries that are end points for that traffic.

5.C.1 Routing Anomalies As part of our analysis work for TransPAC, we have found a number of potential routing anomalies related to the traffic being carried. Many of these reflect historical routing decisions that have not been updated, and in part came to light when we visualized the map shown in Figure 4. Lee and Moynihan presented initial work describing some of the TransPAC and NEAAR routing anomalies at TNC18, along with a call to the community to address some of these problems. One example included large data transfers from a M-‐Root DNS root server in Japan being accessed by groups in South Africa. It was unclear why this server was being used instead of the available instances of that root server that are available in Europe. Following discussions at TNC18 with South African National Research Network (SANReN), we are no longer seeing this issue. We believe more of these issues can be corrected if network operators become aware of them and if there is an appropriate venue for discussion. Going forward we will continue to develop our tools, research, and advocate for a broad and open forum for discussion on this topic. As we have deployed our equipment in Hong Kong and turned up new peerings, we worked with our network engineers in the GlobalNOC to fully document our routing policy as it relates to the TransPAC routers in Seattle and Hong Kong and the

16

bilateral peerings that are being established. The defined policy now includes the following general guidelines: ● General Routing safety policies should be applied.

○ Do not advertise or accept private (RFC 1918) IP addresses. ○ Do not accept or advertise longer than a /24 of IP space. ○ Do not advertise or accept IP addresses that are reserved or

unassigned. ● Label routes with identifiers to track the source of routes (US, Europe, Africa,

Asia), and document those identifiers used on the TransPAC web site. ● IPv6 policy should match IPv4 policy. ● Multicast is enabled as per current community practice. ● Jumbo frames should be enabled wherever possible.

TransPAC specific policies: ● Routes from Asia should be advertised to all North American peers. ● Routes from the North America should be advertised to all Asian peers. ● North American routes should not be advertised to North American peers. ● By default, routes from Asian should not be advertised to Asian peers. There

maybe be exceptions to improve connectivity in Southeast Asia, for example if we discover traffic that is being routed to the US and then back to Asia.

● TransPAC also peers with AARNet and REANNZ across the Pacific Wave fabric. These routes should be advertised both to North American and Asian networks.

5.C.2 Performance Anomalies Separate from the work on routing anomalies, we also began to identify instances of unusual, increased utilization on the TransPAC-‐PacWave circuit. Such traffic anomalies often indicate a large data transfer on behalf of a particular project. We are working with the NetSage project to create a dashboard to allow us to pick out interesting traffic patterns as well as automatically identifying changes in performance behavior on the circuit. One examples of this is the VLBI data transfer, discussed in Section 4.C.3. When interesting anomalies are identified, we will work with NetSage to make sure they are properly documented in the Science Registry and will engage with our NREN partners to reach out to the scientists to follow up on any network performance issues.

6. Circuit Status and Performance For the TransPAC-‐PacWave 100G circuit, we currently collect sampled flow data, SNMP data, and perfSONAR data for this circuit. Starting in September, we started to collect sampled flow data, SNMP data, and perfSONAR data for the two 10G circuits between Guam and Hong Kong.

6.A. Traffic Graphs Figures 5 and 6 show the traffic on the TransPAC-‐PacWave 100G Circuit during the period of December 1, 2017 to November 30, 2018. The largest, continued variation

17

over the year was caused by traffic being rerouted to this circuit from the Internet2-‐SingAREN 100G circuit between Los Angeles to Singapore when it experienced outages. In mid-‐November, that circuit was re-‐engineered and stabilized, so the outages are not expected to continue into the future. The gap in the graph around April 29 was caused by a configuration error on the Pacific Wave switch in Seattle that made it unreachable by our monitoring systems for a short time. Actual traffic levels were not impacted. Our analysis project for Year 4 identified several performance anomalies over the year. For example, the large spike in traffic in February is related to HDTV streaming of the annual Sapporo snow festival from Japan to the US and Singapore, discussed in Section 4.B. The dip in the graphs on August 3 reflects network outage during the APAN45 meeting that is discussed in Section 5.B. The large spike in November reflects usage during SC18, discussed in Section 4.B. Additional spikes in March to June and June to November were due to the VLBI radio telescope traffic discussed in Section 4.C.3.

Figure 5: TransPAC-PacWave 100G Circuit traffic using smoothed daily averages.

Figure 6: TransPAC-PacWave 100G Circuit traffic using maximum daily averages.

Figures 7 and 8 show the traffic on the two 10G connections between Guam and Hong Kong. These graphs start in September when the circuits began to carry

18

production traffic. A stair step pattern is evident as more peers were enabled throughout Quarter 4, drawing more traffic across the circuit. Peering with the Universities of Guam and Hawaii was enabled September 19th. Peering with HARNet was enabled via HKOX on September 24th. The peering with ESNet was enabled October 5th. The Internet2 peering was enabled October 23rd. The peering with TEIN-‐HK was enabled November 14th.

Figure 7: Traffic using smoothed daily averaged for the two TransPAC Guam-Hong Kong 10G circuits.

Figure 8: Traffic using maximum daily averages for the two TransPAC Guam-Hong Kong 10G circuits.

Table 1 shows that more than 3 petabytes of data were transferred over the TransPAC-‐PacWave and Guam-‐Hong Kong links during Quarter 4. Table 2 shows full volume of traffic transferred over the TransPAC-‐PacWave and Guam-‐Hong Kong links during Year 4. The elevated traffic levels in Quarters 2 and 4 are partially due to the VLBI data transfers discussed in Section 4.B.3. In addition, there is increased traffic in November due to the SC18 experiments discussed in Section 4.A.2. Over 7 Petabytes of data has been transferred over the links during the year.

19

Table 1: Traffic in terabytes transferred over TransPAC links, September 1, 2018 - November 30, 2018.

TB Sept Oct Nov Total

Seattle-‐Tokyo 426.69 691.21 707.22 1,825.13

Tokyo-‐Seattle 226.90 189.09 707.22 1,123.22

Guam-‐Hong Kong 0.01 18.27 71.02 89.30

Hong Kong-‐Guam 11.46 18.88 52.45 82.79

Total 665.07 917.45 1,537.92 3,120.44 Table 2: Traffic in terabytes transferred over TransPAC links, December 1, 2017 - November 30, 2018.

Dec-‐Feb Mar-‐May Jun-‐Aug Sep-‐Nov Total

Seattle-‐Tokyo 1,561.68 1,791.40 1,502.55 1,825.13 5,119.08

Tokyo-‐Seattle 567.11 530.65 568.48 1,123.22 2,222.35

Guam-‐Hong Kong 0 0 0 89.30 89.30

Hong Kong-‐Guam 0 0 0 82.79 82.79

Total 2,128.79 2,322.05 2,071.03 3,120.44 7,513.52

6.B Flow Data for Year 4 Quarter 4 At the beginning of the year, the TransPAC-‐PacWave 100G circuit was collecting sampled flow data and unsampled Tstat data. De-‐identified versions of this data were also shared with the IRNC NetSage project. In Quarter 4, it was discovered that Tstat cannot gather data accurately for asymmetric paths. Due to how many science organizations have their routes set up, a significant portion of the traffic on the TransPAC circuits are asymmetric. In addition, it was discovered that the Tstat server had challenges when there were large volumes of flows, for example during SC18. The NetSage project is investigating these issues, but in the meantime we have halted the collection of Tstat data. Figures 9 and 10 display the Top 10 Talkers for outbound traffic to Asia by autonomous system sources and destinations for Quarter 4. Figures 11 and 12 display the Top 10 Talkers for inbound traffic to United States by autonomous systems sources and destinations for this quarter. VBLI data transfers to Japan dominated traffic outbound from the United States to Asia. These transfers alone accounted for over 30% of the total outbound traffic during the quarter. We continue to see traffic from European sources transiting the United States to get to Asian destinations. When the direct 100G links between Europe and Asia are upgraded, it is likely this traffic will shift to a different route.

20

Otherwise the sources of traffic from the United States remained fairly consistent with earlier quarters. Outbound destinations remained fairly consistent as well, featuring sites in Japan, Pakistan, and Singapore as in previous quarters. Traffic inbound into the United States did not coalesce into strong patterns in Quarter 4, with both sources and destinations spread across a large number of institutions. Inbound data tended to show significant high energy physics usage, as has been the case in the past with sources such as the Chinese University of Hong Kong (CUHK) and the Pakistan Education and Research Network (PERN) sending physics related data to the cluster at the University of Chicago that is used by Atlas and the Atlas Tier 2 cluster at Simon Frasier University in Canada. Other science areas include VLBI transfers between facilities hosted by National Institute of Informatics / Science Information Network (SINET) in Japan and NASA as well as WIDE, National Astronomical Observatory of Japan (NAOJ), and Atacama Large Millimeter Array (ALMA) in Chile. The USGS was the destination for geoscience data from locations such as the Earth Remote Sensing Analysis Center (ERSAC) in Japan and the National University of Singapore.

Figure 9: Top 10 Talkers by autonomous system source, outbound from the US for Sept 1, 2018 - Nov 30, 2018.

21

Figure 10: Top 10 Talkers by autonomous system destination, outbound from the US for Sept 1, 2018 - Nov 30, 2018.

Figure 11: Top 10 Talkers by autonomous system source, inbound to the US for Sept 1, 2018 - Nov 30, 2018.

22

Figure 12: Top 10 Talkers by autonomous system destination, inbound to the US for Sept 1, 2018 - Nov 30, 2018.

6.C Flow Data for Year 4 Inclusive Figures 13 and 14 display the Top 10 Talkers for outbound traffic to Asia by autonomous system sources and destinations for Year 4. Figures 15 and 16 display the Top 10 Talkers for inbound traffic to the United States by autonomous systems sources and destinations for Year 4. Looking at the year in full, some trends emerge for the outbound data. For example, astronomy data has been steadily growing over time, and certainly stands out in the annual charts, especially for VBLI sources in Italy and Sweden discussed in Section 4.C.3 sending large amounts of data to their corresponding destinations at APAN-‐JP/NICT and WIDE in Japan. The University of Chicago appears at the top of the outbound sources mainly due to a Cancer Genomic Data Commons run by the National Cancer Institute (https://gdc.cancer.gov/), which is hosted at the University of Chicago. That traffic did not reappear in Quarter 4. Other sources of data involved geoscience from the USGS, genomics from JISC (including the European Bioinformatics Institute -‐ EBI) and LHC data from Stanford and JISC (GridPP). On our inbound traffic, Geoscience is a contributor with large amounts of data being sent from the Japan Aerospace Exploration Agency (JAXA) Earth Observation Research Center (EORC) to NASA and other data being sent to the US Geological Survey (USGS). High energy physics is also a large data mover inbound as their

23

community continues to transfer data between Tier 2 centers in Hong Kong and Taiwan to the United States. Some of this data also goes directly to or from CERN or LHC related computing facilities in the UK and Germany. Astronomy is represented with VLBI transfers between facilities hosted by NII/SINET to NASA and encrypted data transfers from WIDE/NAOJ to ALMA. With the continuing instability of the Singapore-‐US 100G circuit throughout Year 4, we also see traffic on TransPAC for Singapore on both inbound and outbound, though this traffic may drop off in the Year 5 as that link was re-‐engineered to be more reliable in November 2018.

Figure 13: Top 10 Talkers by autonomous system source, outbound from the US for December 1, 2017 - November 30, 2018.

24

Figure 14: Top 10 Talkers by autonomous system destination, outbound from the US for December 1, 2017 - November 30, 2018.

Figure 15: Top 10 Talkers by autonomous system source, inbound to the US for December 1, 2017 - November 30, 2018.

25

Figure 16: Top 10 Talkers by autonomous system destination, inbound to the US for December 1, 2017 - November 30, 2018.

6.D PerfSONAR

6.D.1 Deployments The TransPAC project supports perfSONAR servers in Hong Kong and Seattle that provide periodic testing between several US and Asian sites. TransPAC participates in the IRNC mesh available at http://data.ctc.transpac.org/maddash-‐webui/index.cgi?dashboard=IRNC%20Mesh. We also participate in the APAN testing matrix, http://ps2.jp.apan.net/maddash-‐webui/. In April, Howard Peng at the State Department briefed his senior leadership on the status of available throughput data for the State Department Southeast Asia network, using data from the perfSONAR deployment the TransPAC team helped him to set up. This was the basis of his proposal for additional capacity and an extended perfSONAR deployment to support enterprise network. His presentation generously acknowledged the support of the TransPAC team. Bangladesh Research and Education Network (BdREN) deployed a perfSONAR node that was included in the APAN MaDDash in October of 2018. During this installation process, the configuration files for this MaDDash instance were reviewed and an error that was precluding joining members from showing was found by APAN personnel and corrected. 6.D.2 Training TransPAC funding supports training activities in the region. At APAN46 in New Zealand in August, Addleman and Lee executed a 1-‐day perfSONAR tutorial and a

26

look at the different types of file transfer tests. This training included live troubleshooting using two perfSONAR small nodes to show the effect of latency on large sustained file transfers. Despite having offered several similar workshops at past APAN meetings, this session was well attended with 24 delegates from 7 different countries. A subsequent survey of participants indicated that these continue to be valued by the APAN community and that most attendees would like them to continue. Planning continues for the two 3-‐day perfSONAR workshops to be held in 2019, partially funded by Asi@Connect/TEIN*CC. The Southeast Asia workshop, March 6-‐8, in Laos, will host attendees from Laos, Vietnam, Cambodia, Thailand, and Myanmar. A second South Asia workshop will be held July 8-‐10 at the IU Global Gateway office in New Delhi, India, with attendees from India, Afghanistan, Nepal, Bhutan, Bangladesh, and Sri Lanka. The funding secured from Asi@Connect will be used to pay for workshop logistics, some equipment, and travel expenses for in-‐region attendees. TransPAC will fund our instructors to develop the training materials needed and travel expenses for the instructors. The team has also been approached by several members of the Pakistan Education and Research Network (PERN) about the possibility of having a perfSONAR workshop specifically for that community. Due to the challenges of IN@IU staff traveling to Pakistan, we have encouraged them to request funding from Asi@Connect to enable their engineers to travel to an APAN meeting, where we could conduct a multi-‐day workshop for several participating regions. In Year 5 we will continue to assess training requests as they are received. We will work with the community to provide training where it is needed and where it most meets the overall goals of the project.

6.D.3 General Support Addleman and Lee passed their support assistance roles for the APAN MaDDash to Chevalier this reporting period. Chevalier now helps to monitor and provide guidance to the dashboard administrator on problems seen within the APAN MaDDash. At least one issue, with a perfSONAR node in Japan having fallen out of working order, has already been caught and corrected through this partnership. Currently APAN’s MaDDash is up-‐to-‐date on software versions and ready for new participants receiving small nodes at the Laos workshop in March and the India workshop in July.

27

6.E Trouble Tickets During Year 4, there was only five scheduled maintenances, as shown in Table 3, and eight unscheduled outages, as shown in Table 4. Table 3: Scheduled Maintenances for TransPAC equipment and circuits, December 1, 2017 - November 30, 2018.

Ticket Number

Customer

Impact

Network Impact

Title Maint Type

Source Impact

Current State

Start Time (UTC)

End Time (UTC)

1912:62 3 -‐ Moderate

2 -‐ High Maintenance Completed -‐ TransPAC Backbones SEAT-‐TOKY and SEAT-‐SEAT

Circuit Vendor Closed 2017-‐12-‐14 00:00:00

2017-‐12-‐14 00:00:00

1913:62 3 -‐ Moderate

2 -‐ High Maintenance Completed -‐ TransPAC Backbones SEAT-‐TOKY and SEAT-‐SEAT


2018-‐01-‐10 00:00:00

CHG0032635 3 -‐ Moderate

3 -‐ Normal

Maintenance 1 of 2 Completed -‐ TransPAC Backbone SEAT-‐TOKY


2018-‐08-‐27 15:57:55


3 -‐ Normal

Maintenance 2 of 2 Completed -‐ TransPAC Backbone SEAT-‐TOKY


2018-‐08-‐30 16:27:44


3 -‐ Normal

Maintenance Complete -‐ TransPAC Backbones SEAT-‐TOKY and SEAT-‐SEAT


2018-‐11-‐26 01:17:30

Table 4 : Unscheduled Outages for TransPAC equipment and circuits, December 1, 2017 - November 30, 2018.

Incident Number

Cust Impact

NW Impact

Title Outage Type

Source of

Impact

Current State

Start Time (UTC)

End Time (UTC)

1908:62 2 -‐ High 2 -‐ High Stability -‐ TransPAC Backbone SEAT-‐TOKY

Unann. Maint

Vendor Closed 2017-‐12-‐06 15:03:00

2017-‐12-‐06 23:27:00

1914:62 2 -‐ High 2 -‐ High Outage Resolved -‐ TransPAC Backbone SEAT-‐TOKY

Circuit Dam Fiber

Vendor Closed 2017-‐12-‐20 22:32:00

2017-‐12-‐22 02:27:00

1931:62 2 -‐ High 2 -‐ High Outage Resolved -‐ TransPAC Backbone SEAT-‐TOKY

Power -‐ UPS Power

Vendor Closed 2018-‐04-‐23 03:36:00

2018-‐04-‐23 08:18:00

INC0019773

2 -‐ High 2 -‐ High Outage Resolved -‐ TransPAC Backbone HONG-‐HONG

Circuit -‐ Other

Vendor Closed 2018-‐09-‐27 06:23:07

2018-‐09-‐27 06:42:18

INC0021161

4 -‐ Low 2 -‐ High Availability -‐ TransPAC Backbone HONG-‐GUAM

Undetermined

Vendor Closed 2018-‐10-‐14 12:08:00

2018-‐10-‐14 12:09:00

INC0021670

2 -‐ High 2 -‐ High Availability -‐ TransPAC Backbone HONG-‐GUAM

Undetermined

Vendor Closed 2018-‐10-‐21 03:46:00 2018-‐10-‐21

2018-‐10-‐21 03:47:00 2018-‐10-‐21

28

03:48:00 2018-‐10-‐21 04:00:00

03:53:00 2018-‐10-‐21 04:35:00

INC0022144

2 -‐ High 2 -‐ High Availability -‐ TransPAC Circuit HONG-‐GUAM

Undetermined

Vendor Closed 2018-‐10-‐28 01:07:02

2018-‐10-‐28 09:50:30

INC0022948

4 -‐ Low 2 -‐ High Availability -‐ TransPAC Backbone HONG-‐GUAM

Undetermined

Vendor Closed 2018-‐11-‐08 08:35:01

2018-‐11-‐08 08:36:02

6.F Downtime and Availability

Table 5 shows that the core nodes for the project did not experience any down time during Project Year 4. Table 6 lists the overall downtime for the project circuits, which was approximately 2 days. Table 5 : Downtime and availability for TransPAC core nodes.

TransPAC Nodes Down Time Quarter 4 Availability

Project Year 4 Availability

dcp.seat.transpac.org 00 hr 00 min 100% 100% netsage-‐probe.seat.transpac.org 00 hr 00 min 100% 100% oob.seat.transpac.org 00 hr 00 min 100% 100% rtr.seat.transpac.org 00 hr 00 min 100% 100% test.seat.transpac.org 00 hr 00 min 100% 100% dtn.hong.transpac.org 00 hr 00 min 100% 100% netsage-‐probe.hong.transpac.org 00 hr 00 min 100% 100% oob.hong.transpac.org 00 hr 00 min 100% 100% perf.hong.transpac.org 00 hr 00 min 100% 100% rtr.hong.transpac.org 00 hr 00 min 100% 100%

Table 6: Downtime and availability for TransPAC circuits.

TransPAC Backbone Circuits Down Time Quarter 4 Availability

Project Year 4 Availability

TP2-‐SEAT-‐TP-‐TOKY-‐100GE-‐01522 (100G TransPAC-‐PacWave circuit)

1 days 18 hr 13 min 99.52% 99.53%

TP2-‐SEAT-‐TP-‐SEAT-‐TP-‐100GE-‐01523 (Cross Connect between TP router and Pacific Wave switch)

00 hr 00 min 100% 100%

TP2-‐HONG-‐GUAM-‐10GE-‐01527 (Telstra Hong Kong-‐Guam 10G)

8 hr 44 min 99.90% 99.90%

TP2-‐HONG-‐GUAM-‐10GE-‐01528 (AT&T Guam-‐Hong Kong 10G)

0 hr 42 min 99.99% 99.99%

TP2-‐HONG-‐HONG-‐10GE-‐01525 (10G Connection to HKOX)

00 hr 00 min 100% 100%

TP2-‐HONG-‐HONG-‐10GE-‐01526 (10G Connection to HKIX-‐RE)

0 hr 19 min 99.99% 99.99%

29

7. Software and Systems Work In Year 4 we adjusted our software research goals after community feedback. The BGP Path Hinting project, originally proposed by Brent Sweeny, has received little to no interest from the community. Path Hinting relied on broad participation to be successful. Due to the lack of interest we will no longer pursue this deliverable. Addleman's DDOS detection work also saw a setback when SciPass lost ongoing support. There is a new OpenFlow controller, Faucet, and a related plug-‐in, Nozzle, that may be able to take the place of SciPass, although both are still in active development and not yet stable for third party deployments. Addleman has been in contact with the developers and anticipates release in Year 5 allowing research to continue. This research, if successful, will allow TransPAC to track DDOS traffic on it’s international links. During Year 4, Pacific Wave included the Tokyo and Seattle switches in their pilot deployment of Automated GLIF OPen Light Path Exchange (AutoGOLE) using the Network Services Interface(NSI), bookending the TransPAC-‐PacWave circuit so thereby including it as well. The Tokyo switch has a direct NSI peering with SINET and the Seattle switch has direct NSI peerings with Starlight, JGN-‐Seattle, ESNet, and the Pacific Wave switch in Sunnyvale, as shown in Figure 17. The AutoGOLE fabric allows researchers and network engineers to self provision network connectivity from Japan, across the TransPAC-‐PacWave 100G link, and around the world. We had preliminary discussions after the Guam-‐Hong Kong infrastructure was put into production with GOREX, HKIX, and HKOX to begin to evaluate whether or not it would be useful to enable NSI in Hong Kong. If our partners feel it is advantageous to do so, this will occur in Year 5.

Figure 17: The map published by the AutoGOLE group showing all the networks involved in the AutoGOLE using NSI pilot.

30

8. Security Events and Activities During Year 4, the security documents were upgraded to include all resources that are part of IN@IU. The hardware inventory was maintained by the team and upgraded to better display pertinent information. We have also added all IN@IU servers to the GlobalNOC database in an effort to work more closely with the GlobalNOC systems team. Indiana University is starting a review of the Cyber Risk Mitigation Responsibilities policy (IT-‐28). More information on this can be found at https://kb.iu.edu/d/bdls. We are working closely with the GlobalNOC systems team to be in full compliance with this policy. Security measures were maintained throughout the year and there were no incidents. IN@IU Security documents can be reviewed at: https://internationalnetworks.iu.edu/about/policies.html.

9. Year 5 Planning The current TransPAC4 project officially runs through February 29, 2020, so we are entering the last planned year for the project. Due to program savings, detailed in Section 11, we believe we will be able to continue the project an additional 7 months, and are planning to request a No Cost Extension. This will see us through the minimum current contracts, September 2020. This includes extending other circuit contracts to meet that date.

9.A Circuits WBS 1.2.1, 1.3.1, 3.2.3, 3.3.1, 3.5.4,3.5.1, 3.5.2, 3.5.3 Sections 6, 4.B The current TransPAC-‐PacWave 100G circuit is scheduled to run until mid December 2019. We have already begun discussions with Pacific Wave to extend this for 4 months through the end of the TransPAC4 project (Feb 28, 2020). Starting in November 2019, we will begin the procedure for shutting down the link -‐ turning off the contracts for co-‐lo and ports, as well as informing our partners that we will need to evaluate and re-‐map routes. The current contracts for to two 10G circuits between Guam and Hong Kong, due to delays in the RFP process, actually run past the end of the project lifespan until September 2020. Financially, what this means is that all TransPAC activities except those vital to the support of these circuits will end on or about February 29, 2020, but these support activities (Tier 2/3 support, PI support, minimal engineering and systems support) will continue until September 2020. Three months prior to this we will begin the shut down procedures for these services as well. During Year 5, we will also track the progress of the three 100G connections being put in place between Asia and Europe. It is likely this will require adaptations to existing routes and peerings.

31

9.B Year 5 Travel and Training WBS 2.6.1 Section 3 We expect Year 5 travel and Training to be similar to what was done in prior years, with attendance at APAN, TNC, SC, and Internet2 meetings continuing. In Year 5, we will teach two multi-‐day perfSONAR workshops, co-‐funded with Asi@Connect, one in Laos and one in India. We will likely also do two one-‐day perfSONAR workshops at the APAN meetings. We will seek out options for additional multi-‐day perfSONAR workshops in Year 5.

9.C Year 5 Collaborations WBS 1.2.2, 1.3.2, 1.3.3, 2.6.2 Sections 4.A, 4.B In Year 5, we will continue our collaborations with the IRNC partners. It is likely we will expand the work with the NetSage project, as we will have an additional focus on analysis in support of science engagement (which has grown out of our augmented MOUs), so we plan to work closely with NetSage as they update and modify their tools. With out Asian partners, we plan to sign at least one more MOU with Academica Sinica (Taiwan). We will also continue to support network experimentation for SC and TNC as requested by our partners.

9.D Year 5 Science Engagement WBS 2.2.1, 2.6.3, 2.6.4, 2.6.5, 2.6.6, 3.5.3, Section 4.C Year 5 will continue our strong focus on working with end user scientists. These will include (but are not limited to) our existing collaborations with high energy physics, astronomy, geoscience, and bioinformatics. We will expand our identification of large users of the network via the tools available from NetSage and others we will build for ourselves. We are also planning to develop a more targeted approach to our overall engagement efforts that will allow us to devote more time to individual science domains. This approach will help focus our engagements and help us answer specific questions about how different science communities are using research and education networking resources. We are considering offering additional support for scientific data movement in Year 5 through the use of Data Archive and Data Transfer Nodes located at our points of presence in Seattle, Tokyo, Hong Kong, and Guam. If this moves forward this would include partnering with the PRP consortium to have these nodes as part of their “Data Movement SuperHighway”, as well as additional monitoring for performance. We would advertise this service broadly for use by large scale scientific collaborations in the region.

32

9.E Network Analysis WBS 2.3.2, 2.6.5, 4.2.1, 4.3.3 Section 5.C We will continue to expand our work in analyzing the use of the network, correcting anomalous paths, and working to improve end-‐to-‐end performance through the discovery of high-‐connectivity source-‐destination pairs. This will include working not only directly with NetSage and their tools, but in developing extensions for our own uses cases.

9.F Software, Experimental, and Security Support WBS 2.3.1, 3.2.1, 3.2.2, 4.1.1 Sections 7, 8 We will continue to fully support the circuits through their lifespan. We will stay involved in the AutoGOLE/NSI support that Pacific Wave is offering, and evaluate the extension of this framework to include Guam. As Faucet and Nozzle reach stability, we will explore their deployment on the TransPAC circuits for use in DDoS detection. We will also maintain current security practices, and continue to update our cyber risk mitigation processes.

10. Milestones and Progress

1. Planning / Coordination 1.2.1 Evaluate circuit capacity and community needs. Negotiate with vendors and partners for new circuits as capacity demands grow. Phase 2 planning. • ONGOING -‐ Two 10G circuits between Guam and Hong Kong became operational in September 2019 based on feedback from the community. 1.2.2 Finish partner MOU process -‐ Contact partners and start the process of signing Memorandum of Understandings with each. • ONGOING -‐ This year we signed an MOUs with JUCC (Hong Kong), SingAREN (Singapore), and CERNET (China). The only remaining MOU we currently expect to work on is with Taiwan’s Academia Sinica Grid Computing Centre (ASGC). 1.3.1 Evolve network architecture -‐ New network designs over the evolution of the 5 year award. This will include 100G circuit speeds, software defined networking/ exchanges, possible new peering points, and greater than 10G flows. • ONGOING -‐ We added two 10G circuits between Guam and Hong Kong, and will continue discussions with partners to make sure community need is being met. We continue to expand our peering partners. 1.3.2 Coordinate with IRNC:NOC winner -‐ Coordinate with the IRNC:NOC awardee to ensure they have a sufficient and appropriate level of access to all of the TransPAC4 equipment supporting international activities. This includes appropriate logs, SNMP access, portal or login access to obtain data not available via SNMP, etc.

33

• ONGOING -‐ The TransPAC project continues close coordination with the IRNC NOC. 1.3.3 Coordinate with IRNC:AMI winner -‐ Coordinate with the IRNC:AMI awardee for the appropriate distribution of flow data, per our own security and data policies, SNMP and other access as appropriate. • ONGOING -‐ TransPAC shares measurement data, specifically SNMP, perfSONAR, and flow data, with NetSage. TransPAC continues to work closely with NetSage in areas such as Tstat analysis and the Science Registry. We also are coordinating on the use of the dashboards. 1.3.4 Overall Management of the project • ONGOING -‐ Meetings continue more than quarterly with project partners at conferences including APAN, TNC, SC, and Internet2’s Global Summit and TechX. 1.3.5 Project Reporting -‐ Report generation for the life of the project • ONGOING -‐ Reporting infrastructure is in place for up to date reporting; WBS update as part of this report. 1.3.6 Documentation and dissemination • ONGOING -‐ Both private and public facing documentation continues to be updated.

2. Outreach 2.2.1 Analyze usage data to identify geoscience/bioinformatics researchers. Leverage our TransPAC4 partners to provide support and if possible connectivity for these researchers. • ONGOING -‐ See Section 4.B 2.3.1 Coordinate with network partners to extend SDN/SDX to 100G circuits • COMPLETED -‐ Y4Q2, see Section 5.B 2.3.2 Analyze current network traffic and reach out to possible new network users • ONGOING -‐ See Section 4.B 2.3.3 Evangelize Path Hinting • CANCELED, see Section 7. 2.6.1 Attend domestic and international conferences for application identification and relationship maintenance • ONGOING -‐ See Section 3 2.6.2 Coordinate connectivity with existing and new TransPAC Partners • ONGOING – We will continue to hold meetings at APAN, TNC, and Internet2

34

Conferences with our partners. 2.6.3 Ensure connectivity in support of the Large Hadron Collider • ONGOING -‐ We continue our support of the Large Hadron Collider through our efforts in the LHCONE community. See Section 4.B. 2.6.4 Ensure connectivity in support of Belle-‐II • ONGOING -‐ See Section 4.B 2.6.5 Coordinate with network partners and researchers to support large flows • ONGOING – We will continue to develop new flow analysis tools that will assist us in identifying appropriate researchers. See Section 5.C 2.6.6 Explore additional application communities • ONGOING – We continue to look through flow data and discuss with our partners what application communities would most benefit from more intentional engagement. See Section 4.B and 5.C. 2.6.7 Identify and contact US branch campuses in Asia-‐Pacific region • CANCELED, see Section 4.B.

3. Operations 3.2.1 Integrate TransPAC3 SDN Controller -‐ Work with systems engineers to transition the TransPAC3 SDN controller into the TransPAC4 network. • COMPLETED Y4Q2, see Section 5.B. 3.2.2 Deploy SDN DDOS Solution Deploy the SDN based DDOS mitigation solution. • DELAYED -‐ This work will continue in Year 5, see Section 7. 3.2.3 Evaluate and update existing POPs and equipment. Evaluate and install new points of presence and equipment as community demands expands and changes. • ONGOING -‐ See discussions in Section 5 for additional circuits and OXP. 3.2.4 Deploy Path Hinting service into the TransPAC4 routers and work with partners, connectors, and peers to adopt the service. • CANCELED, see Section 7. 3.3.1 Evaluate and deploy new circuits • COMPLETED -‐ See Section 5.A 3.5.1 Refine network measurement and monitoring data. Refine and make network telemetry useful to researchers and the IRNC:NOC. This will include creating public web pages and repositories that provide easy access to data. • ONGOING -‐ We continue to work with IRNC NOC.

35

3.5.2 Tune and support large flows Monitor large flows across the network and work with researchers to fine tune the end points and entire path. Work with researchers to ensure performance is as expected. • ONGOING -‐ We continue to work closely with network researchers to support both large scale demos and day to day activities to ensure effective network performance. See Section 5.C. 3.5.3 Deploy support and telemetry for large flows. Work with partners to configure and allow for large flows across the TransPAC4 network. Work with systems to deploy monitoring solutions for large flows. • ONGOING -‐ The tools we have developed support collection of data for large flows and we will continue to improve them as well as work with our partners to ensure effective network performance. See Section 5.C. 3.5.4 Operate Infrastructure; Pay for circuit, port, maintenance, and hardware costs. • ONGOING

4. Research / Experimentation 4.1.1 SDN for DDOS mitigation -‐ Research the feasibility of using SDN technologies for detection and mitigation of DDOS attacks on the TransPAC network. • DELAYED -‐ See Section 7. 4.2.1 Test larger than 10G flows Test network equipment, configuration, and support for greater than 10G flows. • DELAYED -‐ Delayed until network experimenters express a concerted interest in such activity. 4.2.2 Path Hinting deployment for testing, experimentation, and running community demonstrations. • CANCELED -‐ See Section 7. 4.3.1 SDN at 100G • COMPLETED Y4Q2, see Section 5.B. 4.3.2 Evaluate SDN in an Internet Exchange environment • COMPLETED Y4Q2 -‐ See Section 5.B. 4.3.3 Evaluate routing issues using Flow data • ONGOING -‐ See Section 5.C.

36

11. Financial Reporting Details Year 4 The financial projections at this time have us maintaining the current level of staffing through the end of Year 5 (February 2020). At that point, we have enough funding to continue the full project for an additional 7 months through September 2020, which is the current contract period for the Guam-‐Hong Kong circuits. Table 7 shows the expenditures for Project Year 4. Please note that the circuit expenses in Year 4 appear reduced, but that is simply because in prior years the invoice for the TransPAC-‐PacWave 100G Circuit was paid in November, whereas this year the invoice arrived late and it was paid in December 2018. Also, charges for the two Guam-‐Hong Kong circuit weren’t invoiced and paid until very late in the quarter, so those expenses trail the actual time period as well. Table 8 gives a summary of expenditures to date, and predicted expenditures for Year 5 and a partial Year 6, including full expenses for all current circuits. Table 7: Year 4 expenditures.

Description Dec 2018

Jan 2018

Feb 2018

Mar 2018

Apr 2018

May 2018

Jun 2018

Jul 2018

Aug 2018

Sep 2018

Oct 2018

Nov 2018 Total

Schopf 1,753 1,753 1,753 1,753 1,753 1,753 1,753 1,864 1,864 1,864 1,864 1,864 21,591 Lee 7,457 7,457 7,457 7,457 7,457 7,457 7,457 5,966 5,966 5,966 5,966 5,966 82,027 Addle man 9,403 9,403 9,403 9,403 9,403 9,403 9,403 7,555 7,555 7,555 7,555 7,555 103,596 Southworth 1,208 1,208 2,416 Chevalier 1,835 1,835 1,835 1,835 1,835 1,835 1,835 1,106 1,106 1,106 1,106 1,106 18,375 Moynihan 4,674 4,674 4,674 4,674 4,674 4,674 4,674 1,174 1,174 1,174 1,174 1,174 38,588 Hubbard 3,277 1,457 1,457 1,458 1,531 1,676 1,531 1,115 1,194 976 865 1,367 17,904 Support-‐ Systems 2,227 2,227 2,227 2,227 2,227 2,227 2,227 2,227 2,227 2,227 2,227 2,227 26,724 Support-‐ Network Eng 2,333 2,333 2,333 2,333 2,333 2,333 2,333 2,333 2,333 2,333 2,333 2,333 27,996 F&A 32% 10,547 9,964 9,964 9,965 9,988 10,035 9,988 7,469 7,494 7,424 7,775 7,936 108,549 Subtotal Compensation 43,506 41,103 41,103 41,105 41,201 41,393 41,201 30,808 30,913 30,625 32,073 32,735 447,766 Travel-‐ Lee-‐SC Denver Nov’17 1,322 1,322 Travel-‐Lee-‐ Tokyo/Daejeon Oct’17 3,484 3,484 Travel-‐Lee-‐ APR Singapore Nov’17 1,745 1,745 Travel-‐Addleman Fiona Dec’17 1,187 469 1,656 Travel-‐Schopf PTC Honolulu Jan’18 1,350 1,723 50 222 22 3,367 Travel-‐Lee PTC Honolulu Jan’18 1,133 2,134 3,267 Travel-‐Schopf Cenic SD Mar’18 1,334 -‐100 1,604 494 -‐884 22 2,470 Travel-‐Lee APAN Sing. Mar’18 1,596 1,360 2,956 Travel-‐Tierney APAN Sing. Mar’18 1,818 2,676 4,494 Travel-‐Lee I2 SD May’18 750 520 2,332 1,270

37

Dec 2018

Jan 2018

Feb 2018

Mar 2018

Apr 2018

May 2018

Jun 2018

Jul 2018

Aug 2018

Sep 2018

Oct 2018

Nov 2018 Total

Training Equip APAN 949 407 1,356 Travel-‐Moynihan LHCONE Oxford Mar’18 252 820 1,072 Travel-‐Lee TNC18 Norway Jun’18 956 1,936 1,715 4,607 Travel-‐Addleman APAN NZ Aug’18 369 1,471 1,080 2,920 Travel-‐Lee APAN NZ Aug’18 372 2,330 2,214 4,916 Travel -‐ Addleman -‐NSF Cybersecurity Aug’18 200 667 867 Travel-‐Terzino Install Hong Kong Aug’18 304 2,357 2,661 Travel-‐Schlemmer Install Hong Kong Aug’18 304 2,099 2,403 Travel -‐ Lee SC18 Nov’18 100 100 Travel -‐ Konishi 20th aniv b'ton Aug’18 1,000 1,000 Travel -‐ Myint -‐ LMI Sept’18 387 387 Travel -‐ Moynihan -‐ LMI Sept’18 2,706 86 2,792 Travel -‐ Grimshaw -‐ LMI Sept’18 3,444 3,444 Travel -‐ Moynihan -‐ LHCONE Oct’18 526 526 FedEx 56 11 27 94 F&A 32% 803 3,348 2,081 2,216 158 1,481 1,124 636 7 3,722 1,557 1,270 18,403 Subtotal Other Expenses 3,312 13,812 8,583 9,140 652 6,109 4,636 2,622 29 15,352 6,424 5,240 75,911 Seattle-‐Tokyo co-‐lo (PNWGP) 1500 1500 1500 1500 1500 1500 1500 1500 0 1500 1500 1500 16,500 Guam Circuit-‐ Telstra 19,823 13,359 13,359 46,541 Guam Circuit -‐ ATT 10,400 10,400 Hong Kong Co-‐lo iAdvantage 9750 3250 3801 16,801 CDW -‐ Guam Equip 424 424 807 44 1,699 Dell -‐ Guam Equip 22,921 22,921 Napatech -‐ Guam Equip 10931 10,931 Pier Group splitter-‐ Guam 1,390 1,390 Arista -‐ Guam switch 26719 26719 53,438 OSI -‐ Guam Equip 3739 7711 380 11,830 Apple laptop refresh 3398 19 3,417 Dell -‐ Seattle DTN 11457 11,457

38

Dec 2018

Jan 2018

Feb 2018

Mar 2018

Apr 2018

May 2018

Jun 2018

Jul 2018

Aug 2018

Sep 2018

Oct 2018

Nov 2018 Total

VMWare Software 160 160 Wire Transfer Fees 20 40 40 100 Subtotal Circuit 1,500 1,500 1,500 1,500 0 0 43,904 35,930 14,355 5,656 55,084 25,259 186,188 Grand Total 48,318 56,416 51,186 51,744 41,853 47,502 89,741 69,360 45,297 51,632 93,581 63,235 709,865

Table 8: Summarized expenditures and forecasted expenses.

TransPAC Year 1 Year 2 Year 3 Year 4 Year 5 estimate Year 6 Est TOTAL Dates Mar'15-‐Nov'15 Dec15-‐Nov16 Dec16-‐Nov17 Dec17-‐Nov18 Nov18-‐Feb20 Mar20-‐ Sept 20 #months 9 12 12 12 15 6 Salaries 58,418 224,945 449,976 447,766 613,305 293,561 2,087,971 Travel 7,918 114,384 133,653 75,911 192,649 92,400 616,915 Circuits 0 38,393 509,070 186,188 868,549 475,679 2,077,879 TOTAL 66,336 377,722 1,092,699 709,865 1,674,503 861,640 4,782,765

TransPAC4 Award #1450904 Year 4 Annual and Quarter 4 1 … Y4...1 TransPAC4 Award #1450904 Year 4...

Documents

Transcript of TransPAC4 Award #1450904 Year 4 Annual and Quarter 4 1 … Y4...1 TransPAC4 Award #1450904 Year 4...