perfSONAR: The Road to 100k Nodes

perfSONAR: The Road to 100k Nodes

Eric Boyd, Internet2, [email protected] Luke Fowler, Indiana University, [email protected]

Brian Tierney, ESnet, [email protected]

What is perfSONAR? •  perfSONAR is a tool to:

–  Set (hopefully raise) network performance expectaIons –  Find network problems (“soQ failures”) –  Help fix these problems

•  All in mulI-‐domain environments •  These problems are all harder when mulIple networks are involved

•  perfSONAR is provides a standard way to publish acIve and passive monitoring data –  This data is interesIng to network researchers as well as network operators

2

Current perfSONAR components •  Measurement tools

–  iperf3, bwctl, owamp, traceroute, etc. •  Measurement archive •  Central test mesh management tools •  Host management tools

–  Configure tests, configure NTP, etc. •  Data analysis tools

–  Plot data from the archive –  Dashboard tools

•  Lookup Service

October 15, 2015 3

perfSONAR History •  perfSONAR can trace its origin to the Internet2 “End 2 End performance IniIaIve”

from the year 2000. •  What has changed since 2000?

–  The Good News: •  TCP is much less fragile; Cubic is the default CC alg, autotuning is and larger TCP buffers are

everywhere •  Reliable parallel transfers via tools like Globus Online •  High-‐performance UDP-‐based commercial tools like Aspera •  more good news in latest Linux kernel, but it will take 3-‐4 years before this is widely deployed

–  The Bad News: •  The wizard gap is sIll large •  Under-‐buffered and switches and routers are sIll common •  Under-‐powered/misconfigured firewalls are common •  SoQ failures sIll go undetected for months •  User performance expectaIons are sIll too low

4

The perfSONAR collaboraIon •  The perfSONAR collaboraIon is a Open Source project lead by ESnet, Internet2,

Indiana University, and GEANT. –  Each organizaIon has commiced 1.5 FTE effort to the project –  Plus addiIonal help from many others in the community (OSG, RNP, SLAC, and more)

•  The perfSONAR Roadmap is influence by –  requests on the project issue tracker –  annual user surveys sent to everyone on the user list –  regular meeIngs with VO using perfSONAR such as the WLCG and OSG –  discussions at various perfSONAR related workshops

•  Based on the above, every 6-‐12 months the perfSONAR governance group meets to prioriIze features based on: –  impact to the community –  level of effort required to implement and support –  availability of someone with the right skill set for the task

5

perfSONAR Vision •  Grow perfSONAR to be the ubiquitous, indispensible performance monitoring framework for R&E network customers that meets their needs in the long-‐term, supported in a self-‐sustaining way.

October 15, 2015 6

perfSONAR Challenge •  Get to 100k nodes deployed worldwide – An increase of 2 orders of magnitude

•  Get to 100+ nodes per campus

October 15, 2015 7

perfSONAR Challenge ImplicaIons •  Cut the price per node by an order of magnitude – $4k (perfSONAR 3.3) – $2k (perfSONAR 3.4) – $100 (perfSONAR 3.5)

October 15, 2015 8

perfSONAR Hardware •  These days you can get a good 1U host capable of

pushing 10Gbps TCP for around $500 (+10G NIC cost). –  See perfSONAR user list

•  And you can get a host capable of 1G for around $100! –  Intel Celeron-‐based (ARM is not fast enough) –  e.g.: hcp://www.newegg.com/Product/Product.aspx?Item=N82E16856501007

•  VMs are not recommended –  Tools work becer if can guarantee NIC isolaIon

6/2/15 9

perfSONAR Challenge ImplicaIons •  Scale the infrastructure to support 100k nodes – Does the Lookup Service scale? – Do the various UIs scale? – Does the support model scale? – Grow the number of operaIng systems supported? –  Re-‐evaluate accessibility of nodes?

October 15, 2015 10

perfSONAR Challenge ImplicaIons •  What is the total cost of ownership per node? – O(n2)? O(n)? O(k)?

•  How auto-‐configurable are the nodes? –  If you drop a node in a wiring closet, how much work is required to integrate it into a campus mesh?

October 15, 2015 11

perfSONAR 3.5: What’s new?

perfSONAR 3.5 Themes •  Impact: Increase the number of nodes, reduce total cost of ownership, improve security

•  Usability: Transformed user experience, reduced cost for NOC integraIon

•  Efficiency: Reduced development costs

October 15, 2015 13

perfSONAR 3.5 ImplementaIon •  Modernize the GUIs •  Support for central host management and node auto-‐configuraIon

•  Support for Debian, VMs, and other installaIon opIons

•  Support for low cost, 1Gbps nodes

October 15, 2015 14

Expanded perfSONAR Use Cases •  Current Use Case –  perfSONAR Toolkit

•  Includes CentOS 6 and all perfSONAR components

•  New Use Cases –  perfSONAR tools only

•  Support for both RHEL-‐based and Debian-‐based hosts –  perfSONAR hosts that are centrally managed –  perfSONAR hosts that ‘self-‐configure’

October 15, 2015 15

Improved Support for Central Management

•  Goals: – Make it easy to incorporate perfSONAR hosts into exisIng host management systems (puppet, chef, SaltStack, cfengine, etc.) •  Include sample puppet config files

– Make it easy to manage many perfSONAR hosts at a single insItuIon

– New rpm and debian bundles to support this

October 15, 2015 16

perfSONAR and Puppet

•  Puppet is a configuraIon management system that allows you to configure and maintain many nodes easily

•  IniIal perfSONAR puppet modules have been created –  Alpha/beta level –  Only CentOS6 supported –  We need your help to test/contribute improvements! –  Hoping to expand in perfSONAR 3.6

October 15, 2015 17

Puppet Support Features •  Node classes can be defined, i.e. endpoint or measurement archive •  Puppet will install soQware/dependencies, configure it, keep

packages up-‐to-‐date •  Includes support for Internet2 repos, web100 kernel, Esmond,

bwctl, owamp, rsyslog client or server, apache, meshconfig –  Most services have configurable port ranges –  Puppet also maintains iptables setngs for the configured services

October 15, 2015 18

New perfSONAR InstallaIon opIons •  In addiIon to the tradiIonal “Toolkit” install, you now have the these addiIonal opIons:

–  perfSONAR-‐Tools: •  iperf3, bwctl, owamp, nuccp, etc •  Install this on DTNs, etc to help with troubleshooIng •  Does not support scheduled tesIng •  CentOS and Debian support

–  perfSONAR-‐TestPoint: •  tools plus Lookup Service registraIon and ‘mesh agent’ •  For use in environments with a central measurement archive •  For use on low end/older hardware (e.g.: $100 nodes) •  Supports scheduled tesIng •  CentOS and Debian support

•  See: hcp://docs.perfsonar.net/install_opIons.html

October 15, 2015 19

New perfSONAR InstallaIon opIons (cont.) •  perfSONAR-‐Core:

–  Includes everything except the web interface –  Use this in environments where your site sysadmins want to fully manage the

host configuraIon, but don’t want to setup a central measurement archive –  CentOS only

•  perfSONAR-‐CentralManagement: –  Includes measurement archive, test mesh manager, dashboard –  Use this to manage a collecIon of perfSONAR hosts at your site/campus –  CentOS only

October 15, 2015 20

New perfSONAR InstallaIon opIons (cont.) •  perfSONAR-‐Complete

–  All perfSONAR packages –  Use this environments where your sysadmins want to manage the install, but sIll use

the toolkit web interface, system setng, etc •  the toolkit install will override certain changes every update.

–  CentOS only

•  Other packages to note: –  Separate rpms/debs for iptables config, sysctl config, and ntp packages so you can add

them on top of perfSONAR-‐Core as desired.

October 15, 2015 21

perfSONAR for Network Researchers •  AcIve measurement interesIng for network researchers

–  Traceroute data automaIcally collected along with bwctl/owamp results

•  Data easy to download for analysis –  esmond-‐ps-‐get-‐bulk

•  Output CSV or JSON •  See: hcps://pypi.python.org/pypi/esmond_client

•  AddiIonal InformaIon at: –  hcp://docs.perfsonar.net/client_apis.html

October 15, 2015 22

User Interface Refresh •  Easy to use web interface •  Targeted at users with variety of levels of technical experIse.

October 15, 2015 23

Our goals for perSONAR 3.5: •  improve easy of use and

visual appeal •  integraIon of state of the

art UI components •  to extend UI lifespan.

Current Interface

New Host Management Interface

October 15, 2015 © 2014, hcp://www.perfsonar.net 24

perfSONAR on Low Cost Hardware •  MoIvaIon: make perfSONAR affordable enough to deploy on all subnets

•  AssumpIons: –  1Gbps test nodes –  Centralized measurement archive –  Centralized configuraIon management – Debian Linux

October 15, 2015 25

Auto-‐Updates Rock!

October 15, 2015 26

psUI: the perfSONAR web UI

TechEX 2015

What is psUI? •  Perfect tool for

–  On demand measurements –  Accessing mulIple archived measurements

•  Access MP and MA from –  Global Lookup Service (~ 9000 services) –  Locally maintained list

•  Calls bwctl through OPPD (calls 2 remote bwctl instances) •  Java applicaIon running in tomcat

–  Java 7 and Tomcat 7 –  CentOS and Debian packages

•  To be deployed on separate host (VM is ok) •  Federated login with eduGAIN credenIals

October 15, 2015 28

October 15, 2015 29

1

2

3

October 15, 2015 30

1 2 3

Other resources •  Install your own psUI for your NOC –  InstallaIon instrucIons: hcp://docs.perfsonar.net/install_psui

– Usage instrucIons: hcp://docs.perfsonar.net/using_psui

•  Public demo psUI instance (eduGAIN enabled): hcps://psui.geant.net

October 15, 2015 31

Low Cost Node Project

perfSONAR on Low Cost Hardware •  MoIvaIon: make perfSONAR affordable enough to deploy on all subnets

•  AssumpIons: –  1Gbps test nodes –  Centralized measurement archive –  Centralized configuraIon management – Debian Linux?

October 15, 2015 33

Many Possible SoluIons

August 10, 2015 34

•  Small, Low-‐cost Nodes Examples include the above; NUC, Gigabyte, Pi, cubox, …

Started with ARM-‐based soluIon •  Cheap ARM-‐based

perfSONAR nodes •  e.g.: Cubox

(“perfCUBE”) $120

•  TCP max only 350Mbps

•  owamp works well

35

Evaluated these Celeron-‐based devices

10/15/15 36

ASUS CHROMEBOX-‐M004U ($160) GIGABYTE GB-‐BXBT-‐2807 ($115) Intel NUC Kit DCCP847DYE ($170)

LIVA by ECS •  $100-‐125 USD •  Home:

hcp://www.ecs.com.tw/ECSWebSite/Product/Product_LIVA.aspx?DetailID=1560&LanID=0#Overview

•  hcp://www.newegg.com/Product/Product.aspx?Item=N82E16856501007 –  $105 in Sept, 2015

© 2015, hcp://www.perfsonar.net 37


SpecificaIons

Assembly Required •  Contents of box:

–  Case top and bocom –  Motherboard with onboard memory,

RAM, and 1GE network adapter –  Wireless adapter card –  Antennae for wireless card (1 long,

black; 1 short, white) –  Power Supply (US) with adapter plate

(EU included)


Assembly Completed


Strengths •  Complete package with limited assembly required

–  32Gig of flash onboard –  2Gig of RAM –  True 1GE (tesIng over clean paths 920-‐980M –  Only 15W power required

•  POE PossibiliIes

•  Small and lightweight •  ComparaIvely Low cost to other more robust

hardware/server opIons


eMMC Flash (Onboard) •  perfSONAR’s recommended CentOS 6 does not include driver

support for the onboard storage •  Using instead as a recommended build Ubuntu 12.04.5

Desktop –  Server doesn’t have proper EMMC driver

•  Debian (Ubuntu) perfSONAR packages: hcp://docs.perfsonar.net/install_debian#configuraIon


Power Bucon / USB Issues (minor) •  Power bucon instability has been found on about

1 in 10 boxes. •  Easily overcome by

–  mulIple boot acempts or; –  Long-‐pressing the power bucon for a firm connecIon or;

•  Assume this is either a quirk of the manufacturing process and hope it is worked out on future models

•  USB instability has been found on about 3 in 20 boxes; independent of the power bucon issue


Passive Cooling •  No fan in the box for cooling •  No issues with this were found during

tesIng, but worth considering when placing in warmer spaces.


TesIng Next GeneraIon Liva…


•  “Liva X” – new run of hardware, expected to resolve minor issues.

•  Nearly idenIcal hardware, very similar cost profile •  Arrives pre-‐assembled •  perfSONAR project has acquired (arrived on Monday!),

beginning tesIng/validaIon now. Stay tuned! •  hcp://www.newegg.com/Product/Product.aspx?

Item=N82E16856501011

More InformaIon… hcps://www.perfsonar.net/deploy/hardware-‐selecIon/low-‐cost-‐hardware/ hcp://docs.perfsonar.net/low_cost_nodes.html hcp://github.com/perfsonar/project/wiki/perfSONAR-‐Endpoint-‐Node-‐Project


plans for perfSONAR 3.5.1 and 3.6

perfSONAR Plans •  We are in the early stages of refining the perfSONAR roadmap

•  ExpecIng a minor release in the February Imeframe and a major release in late 2016 (Imeframe TBD)

October 15, 2015 48

perfSONAR Community •  Planning a survey of the perfSONAR user community in October, 2016

•  Planning to publish roadmap in November, 2016

October 15, 2015 49

DRAFT: v3.5.1 (February 2016) •  New Test configuraIon GUI •  New plots based on ESnet Tools team react-‐based plotng tools

•  Node ‘autoconfig’ •  minimize configuraIon required to bring up a new centrally managed

perfSONAR node •  More automated tesIng •  Bug fixes

October 15, 2015 50

DRAFT: v3.6 (Summer 2016) •  Theme for v3.6 will be “Control and Scalability” •  perfSONAR is successful because of the ‘default open’ model.

•  BUT, as the number of perfSONAR hosts worldwide grows, we need a way to control – Who is running tests – How oQen are they allowed to run tests

October 15, 2015 51

Roadmap for v3.6 (cont.) •  CentOS 7 / Debian 8 support •  Pre-‐packaged perfSONAR VM images •  GUI to configure test meshes •  Maddash 2.0 – See next slide

October 15, 2015 52

MaDDash 2.0 •  AlerIng

–  Working in conjuncIon with MaDAlert project to analyze dashboard pacerns and create useful alerts

•  Scalability –  Dashboards keep growing and want to increase frequency and number of checks that can be

run. –  Exploring alternaIve back-‐ends and new modes for distribuIng check execuIon

•  Usability –  Redesign of the web interface. –  Major goals include:

•  Making dashboards easier to understand •  Making it easier to search for and find checks you care about

Test Node Auto-‐ConfiguraIon •  Goal: minimize configuraIon required to bring up a new centrally managed perfSONAR node

•  Site will register in the lookup service: – What hosts to run tests to – Where to send the test results

•  Eventual Goal is a ‘zero config’ mode

October 15, 2015 54

Email Lists and Reference Materials

AcIve and Growing perfSONAR Community

•  AcIve email lists and forums provide: –  Instant access to advice and experIse

from the community. –  Ability to share metrics, experience

and findings with others to help debug issues on a global scale.

•  Joining the community automaIcally increases the reach and power of perfSONAR –  The more endpoints means

exponenIally more ways to test and discover issues, compare metrics

56

•  The perfSONAR collaboraIon is working to build a strong user community to support the use and development of the soQware.

•  perfSONAR Mailing Lists –  Announcement Lists:

•  hcps://mail.internet2.edu/wws/subrequest/perfsonar-‐announce

–  Users List: •  hcps://mail.internet2.edu/wws/subrequest/perfsonar-‐users

perfSONAR Community

57

Useful URLs •  hcp://docs.perfsonar.net/ •  hcps://www.perfsonar.net/ •  hcp://fasterdata.es.net/ –  hcp://fasterdata.es.net/performance-‐tesIng/network-‐troubleshooIng-‐tools/

•  hcps://github.com/perfsonar –  hcps://github.com/perfsonar/project/wiki

58

perfSONAR: The Road to 100k Nodes

Documents

Transcript of perfSONAR: The Road to 100k Nodes