Switch Flow-mod Performance Plots 1. OverviewWewanttoknowhowfastaswitchcaninstall(ormodify)flowtableruleswhentheOpenFlowcontrollertellsittodoso.Theperformanceisexpectedtodependonvariousfactors,includingbutnotlimitedto:thenumberofflowrulestoinstall(ormodify),andtherateofcontrolmessagessentfromthecontroller.Tothisend,wehaveevaluatedvariousswitches,leveragingthemethodologyandtooldescribedinapaperbyKeqiangHe,et.al.,“MeasuringControlPlaneLatencyinSDN-enabledSwitches”,SOSR2015.Theplotsinthisdocumentshowtheflowrulesetupdelayofeachswitchmodel,whichisthetimeswitchusedtocompleteaninstallation(ormodification)ofaflowruleaftertheOpenFlowcontrollerhassentanOpenFlowflow-modcontrolmessage.Forallplots,theX-axisshowsdifferentflowruleinstallationrateswhiletheY-axisshowstheflowsetupdelayinmilliseconds.Section4showsthecomparisonoftheaverageflowrulesetupdelayinallswitchmodelswithahistogrambargraphwithvaryingflowruleinstallationrateandnumberofflowrules.Section5showsthestatisticaldistributionofflowrulesetupdelaymeasurementsforeachswitchmodelwithaboxplot.

2. Factors• NumberofFlowrules(nFlow)

o Thenumberofflowrulesthatwillbeinstalled(ormodified)intheswitch.• Ruleinstallationrate(rate)

o Therate(numberofflow-mods/second)offlow-modmessagesthatwillbepushedtotheswitchbytheOpenFlowcontroller.

• Operation:“Add”or“Mod”o Addinganewflowrulevs.modifyingaflowrule

3. ImportantQuestionso Doestheruleinstallationrateaffectperformance?

§ Yes, especially in low-end switches. In high-end switches, there is not much difference. Better performance can be achieved when the rate is actually close to the switch’s manageable-max. However, how to find that point if it is not stated in its specification is unclear for now.

o Does the number of rules affect the performance? § Yes, especially in low-end switches. Not much in high-end switches.

o Is there a difference between “Add” and “Mod” operation? • Overall:

o “Add” seems to have better performance compared to “Mod” in Arista and HP. However, Cisco shows a reverse trend, where “Mod” has better performance. The cause is possibly by the difference in architecture and handling algorithm, but exact reasons are currently unknown.

• Vendor specific: o Arista

§ “Add” has a consistent performance of 1-3ms. “Mod” shows inconsistent and generally worse performance (3x or more delay), especially starting around rate 40-50 flow-mods/second.

o Cisco 3650 § “Mod” has a shorter delay with 50 flow rules. However, result gets

confusing after 100 and more flow rules, sometimes reporting negative delay values, which is erroneous. Negative values are reported due to the switch flooding packets to all switch-ports, which is possibly caused by the switch being overloaded by too many number of flow rules and flow-mod messages.

• Cisco 3850-inband & 3850 o “Mod” has shorter delay, around 2-3x shorter than “Add”.

• HP o “Add” has shorter delay, around 2-3x shorter than “Mod”. Exception is

when nflows=200: shows similar performance, slighter faster at mod. Reason is unknown.

o Does using the dedicated Management interface port for the control channel connection, instead of any normal switch-port, make a difference?

• We only tested on Cisco devices. “Cisco3850-inband” uses a normal switch-port while “Cisco3850” uses a dedicated management port. No noticeable difference is shown in terms of flow setup delay performance.

o What's the difference between Cisco 3650 and 3850? • There is a significant difference; Cisco 3850 has superior performance while

3650 seems almost unusable. Cisco 3650 reports a delay of 1-10 seconds or more with 100 flow rules, which is probably unacceptable in production networks.

• Cisco 3850 is shows less than 1 second delay (generally around 500ms) for 100 and 200 number of flow rules.

o Overall impression • Arista 7050TX shows exceptional performance, followed by HP, then by

Cisco 3850 and lastly, Cisco 3650. o Notes

• Arista 7050TX-48: CPU: Quad-core x86. System memory: 4GB. • Cisco 3650: CPU: Quad-core x86. System memory: 4GB. • Cisco 3850: Unknown • HP: Freescale PowerPC 8540 @ 666 MHz, 4 MB flash, 1 GB compact flash,

256 MB DDR SDRAM; packet buffer size: 72 MB QDR SDRAM total (36 MB for 1 GbE/10 GbE ports)

4. AverageComparisonBetweenSwitchesThefollowingsetofplotsshowtheaverageflowrulesetupdelayofdifferentswitchmodelswithvaryingflowinstallationrateandnumberofflowstoinstall.TheY-axisisinlog-scale.Notethatifabarismissing,thatmeanstheaveragedelayvalueisanegativenumber,whichindicatesanerroneousmeasurement,possiblyduetooverwhelmingload(stress)ontheswitchduringthetest.

(Fig.4.1)Adding50flows.

(Fig.4.2)Modifying50existingflows.

(Fig.4.3)Adding100flows.

(Fig.4.4)Modifying100existingflows.

(Fig.4.5)Adding200flows.

(Fig.4.6)Modifying200existingflows.

5. BoxplotsforEachSwitchThefollowingplotsshowtheboxplotofmeasuredflowrulesetupdelayvaluesforeachswitchwithvaryingflowinstallationrateandnumberofflows.Boxplotisusedtoshowthedistributionofmeasurementvalues.Theupperwhiskerofaboxdepictsthemaximumvaluewhilethelowerwhiskeroftheboxdenotestheminimumvalue.Theupperlineoftheboxdenotesthethirdquartile,whichisthe75%percentile.Theredlinewithintheboxisthesecondquartile,whichisthemedian.Thelowerlineoftheboxdenotesthefirstquartile,whichisthe25%percentile.Pleasereference(https://en.wikipedia.org/wiki/Box_plot)and(https://en.wikipedia.org/wiki/Quartile)tobetterunderstandboxplots.Notes:

• Weranfiverunsoftestsforeachuniquecombinationof:flowruleinstallationrate,operation,numberofflowrules,andswitchmodel.

• Arista7050TX_nFlows50-modshowsabnormallylowperformancecomparedtoothermeasurementsfromthesavedevice.ThisisalsoobservedinHP-J9307A-v1_nflows50-mod.Weconsiderthemasoutlierresultsfornow;thecauseiscurrentlyunknown.

• AsstatedinSection4,negativedelayvaluesarepossiblyduetooverwhelmingloadontheswitch.

Arista7050TX

Cisco3650

Cisco3850-inband

Cisco3850

HPJ9307A(v1module)

HPJ9538A(v2module)