Voting System Qualification Test...

Voting System Qualification Test Report

Dominion Voting Systems, Inc.

Sequoia WinEDS Release 4.0.175, Version 1 ______________________________________________

May 2012

Florida Department of State Division of Elections

R.A. Gray Building, Rm 316 500 S. Bronough Street Tallahassee, FL 32399

Florida Department of State of 16

Contents

Introduction ............................................................................................................................................ 3 Background ............................................................................................................................................. 3 Conduct of Tests ..................................................................................................................................... 3

Phase I ................................................................................................................................................ 3 Baseline of Operating System ........................................................................................................ 4 Voting System Setup ...................................................................................................................... 4 Installation/Verification of Equipment Firmware .......................................................................... 4 Physical Audit ................................................................................................................................. 4 Functional Audit ............................................................................................................................. 4 Creation/Importing of Election Definitions ................................................................................... 5

Phase II ............................................................................................................................................... 7 Election Simulations ...................................................................................................................... 8 Audit Trail....................................................................................................................................... 8 Security .......................................................................................................................................... 9 Recovery Techniques ..................................................................................................................... 9

Modem capability .............................................................................................................................. 9 Mass Ballot Counts ............................................................................................................................. 9

Insight Precinct Scanner ................................................................................................................ 9 Edge DRE Tabulator ....................................................................................................................... 9 400-C Central Count Scanner ......................................................................................................... 9

Mock Contest Recount ..................................................................................................................... 10 Simulated Catastrophic Failure ........................................................................................................ 10 Ballot Sensitivity ............................................................................................................................... 10 Sound Pressure Level ....................................................................................................................... 13 Edge Audio Language Loop .............................................................................................................. 13 Force ................................................................................................................................................. 13 Daylight Saving Time Change ........................................................................................................... 13 Source Code Review ......................................................................................................................... 14

Insight Memory Pack Engineering Change Order 781, 02/09/2010 .................................................... 14 Test Results and Findings of Compliance ............................................................................................. 14

Exceptions ........................................................................................................................................ 15 WinEDS Release 4.0.175, Version 1 ...................................................................................................... 15


Introduction

On November 11, 2011, Dominion Voting Systems (DVS) submitted the final documentation and source

code files required for their requested certification of the “WinEDS Release 4.0.175, Version 1” voting

system. This system includes a new version of the WinEDS election management system, new firmware

for Optech Insight and Insight Plus, new firmware of AVC EDGE I and II, new software for Optech 400-

C, new software for upgrading uploads using Insight modem capability, and new versions for other

support items. Public testing occurred from March 5 through May 22, 2012, in Tallahassee, Florida. This

report is filed in summary of testing and certification activities.

Background

The “WinEDS Release 4.0.175, Version 1” is a new voting system based on an existing certified system in

the State of Florida. This new voting system is initiated by Palm Beach County’s requirement for

modeming with two or more precinct tabulators in a single precinct. That capability does not exist with

the existing certified system (Release 3.1.077). DVS provided this capability with their new Insight

Listener software and required the new election management software (WinEDS 4.0) and Optech Insight

firmware.

The “WinEDS Release 4.0.175, Version 1” system includes a new version of the WinEDS election

management system, new firmware for Optech Insight and Insight Plus, new firmware of AVC EDGE I

and II, new software for Optech 400-C, new software for upgrading uploads using Insight modem

capability, and new versions for other support items. Thus, this voting system required a full qualification

test, modem test, and mass ballot count tests.

The vendor’s application for Florida certification included the Sequoia WinEDS, Optech Insight and

Insight Plus precinct count tabulators, AVC EDGE I and II direct recording equipment (DRE) as needed

to meet Americans with Disabilities Act (ADA) specifications, and the Optech 400-C central count

tabulator. This is a paper-based voting system with only one element for compliance with HAVA and

ADA provisions for precinct voting: the ACV EDGE, a DRE.1

Conduct of Tests

The test objective was to verify that this voting system met the applicable requirements of Florida Statutes

and Rules and Florida Voting Systems Standards (FVSS).

The certification effort was conducted in two phases: Phase I consisted of establishing a baseline of the

operating system environment, installation and verification of voting system applications and firmware,

and creation/importing of election definitions meeting election criteria outlined below.

Phase II consisted of all activities surrounding election simulation (pre-election activities, early voting

and election day activities, post-election and reporting), modem testing, mass ballot counts, audit and

security examination, recovery techniques, and additional tests.

Phase I

Phase I testing consisted of establishing a baseline of the operating system environment, installation and

verification of voting system applications and firmware, and creation/importing of election definitions

with previously identified parameters designed to assess the system’s compliance with Florida laws, rules,

and standards.

1 By 2016, ADA polling equipment must comply with Section 101.56075(3), Florida Statutes that in

effect requires removal of direct recording electronic (DRE) tabulator.


Baseline of Operating System

The operating system was installed on verified clean machines. The COTS software was installed on the

client and server machines. The server and client machines were then imaged to get a baseline of the

operating system.

Voting System Setup

The election management system [WinEDS, Election Reporting Program (ERP), Ballot Printing System

(BPS), WinEDS Bridge Tool (WBT), Extended Services] was installed on two client machines. The

Insight Listener was installed on a separate client machine. The machines were then imaged.

Installation/Verification of Equipment Firmware

Firmware was upgraded, validated and verified in the following equipment:

Memory Pack Reader / Writer (MPR) – The MPR was upgraded with firmware version

3.01.080422.0552

Optech Insight – The Insight units were upgraded with firmware versions:

o HPX L1.46.100205.1100

o The modem-capable units were upgraded with firmware version CPX J-

L1.16.111003.1633

o Insight Memory Packs were upgraded with APX L2.18.100205.1359

AVC Edge – The Edge machines were upgraded by DVS staff with firmware version 5.2.16

Optech 400-C – The 400-C was installed with WinETP software version 1.16.15 and MS

Windows XP Pro SP2.

Users and workspace was configured and baseline images were created for all the election management

system machines and server.

Physical Audit

The voting system configurations were validated by comparing the system/software identification with

the application for certification descriptions and then for:

System software (WinEDS, COTS)

System hardware (Insight, Edge, 400-C)

Hardware configuration

Functional Audit

A functional audit was conducted on the following:

Insight – validated the voting system configurations by using test scripts

Edge – calibrated and validated the voting system configurations by using test scripts

400-C – validated the voting system configurations by using test scripts

ERP

Extended Services

WinEDS Bridge Tool

SQL Server and Election Reporting Program (ERP) – In the test environment, the name of the

SQL Server included a special character, a hyphen (-), which interfered with the ERP’s ability to

communicate with the server. The problem was solved by changing the host file that is local to

each machine, which restored communication. BVSC recommends that any servers used in


conjunction with the WinEDS election management system follow a naming convention that

excludes special characters.

Creation/Importing of Election Definitions

The test included four simulated elections: Presidential Preference Primary (PPP), Municipal, Primary,

and General. The test plan called for using a primary and general “test” election based on a primary

election from Indian River or Palm Beach County’s 2010 database, and a general election based on Palm

Beach County’s 2010 database. These two elections were modified from restored election definitions,

and the municipal and PPP test elections were created new.

Ballots can be created using one of two methods in the WinEDS suite: ballots can be created within the

WinEDS application, or they can be substantially created in the Ballot Printing System (BPS) application

and ported, using the WinEDS Bridge Tool (WBT), into WinEDS for final editing. The Edge ballots are

created in WinEDS, and the printed ballots can be created in either WinEDS or the BPS. This certification

effort included tests utilizing both methods of ballot production. BVSC found, during the creation

process, that the documentation was confusing and labyrinthine, and considerable consultation with the

helpful DVS staff was necessary in order to achieve the test plan objectives.

BVSC examined the version currently under test, WinEDS, Version 4.0.175, for the added functionality

that automates the previously manual process of verifying contest order, that was in version 3.1.

The four elections used in testing were defined (or modified) to meet the criteria described below.

Municipal Election

Election definition created

Languages: English, Spanish, Haitian Creole

Single sided ballot, 14" long

Three or more precincts with ballots by ballot style

One or more "Vote for x" contests

One or more split precincts

Statistical contests

Precinct scanners set to: o Reject ballot for one or more overvoted contests o Reject a blank ballot o Accept a ballot that contains one or more undervoted contests

Central count scanners set to: o Outstack a ballot for one or more overvoted contests o Outstack a blank ballot o Accept a ballot that contains one or more undervoted contests o Use poll header mode

Tally type for early voting, absentee, provisional, and election day reported by precinct ID and Congressional district

Print at least 10 ballots for each precinct

Ballot definitions produced and unmarked ballots printed

Minimum 4 precincts.

General Election

Election definition based on a general election from Palm Beach County’s 2010 database.

One or more two sided, three column 18” ballots for Optech Insight and 400-C

Three languages (English, Spanish, and Haitian Creole)

At least the maximum precincts as available in Palm Beach County

Ballots by precinct ID

One or more precinct only contests


One or more split precincts

One or more "Vote for x" contests with a write-in

At least one overvoted ballot will include a write-in



Central count scanners set to: o Outstack a ballot for one or more overvoted contests o Outstack a blank ballot o Accept a ballot that contains one or more undervoted contests o Use mixed mode

Tally type (group) by early voting (EV), absentee (except for absentee on ED), absentee on ED, provisional, provisional after 7pm, overseas, and election day (ED)

Summary election results for use with election night reporting

Precinct-level results report as required under section 98.0981(2)(a), Florida Statutes



Setup for early voting for an election definition similar in size to Miami-Dade or Palm Beach County

Minimum 20 precincts, however, EV tabulator(s) election definition must have the capability as Palm Beach precincts.

Presidential Preference Primary (PPP) Election

Election definition created

Languages: English, Spanish, Haitian Creole

Single sided ballot, 14" long

Ten or more precincts with ballots by ballot style

Two or more split precincts

Presidential contest with the Republican and Democrat parties

Three or more congressional districts

At least one contest and candidate name set at the WinEDS maximum field length


Edge configured for audio


Central count scanners set to: o Outstack a ballot for one or more overvoted contests o Outstack a blank ballot o Accept a ballot that contains one or more undervoted contests o Use either poll header mode or precinct header mode

Tally types for early voting, absentee, provisional, and election day reported by precinct and Congressional District


Scan and tabulate single fold absentee ballots




Primary Election

Closed Primary Election (based on an election from Indian River’s or Palm Beach County’s 2010 database) to include the following:

o Republican, Democrat, and Nonpartisan ballots o Three languages (English, Spanish, and Haitian Creole) o At least the maximum precincts as available in Palm Beach County o One or more two sided, three column 18” ballots for Optech Insight and 400-C o One or more precinct only contests o One or more county-wide universal primary contests o One or more district-wide universal primary contests o One or more split precincts o One or more polling place with multiple precincts o Statistical contests o At least one contest and candidate name set at the WinEDS maximum field length

Precinct count scanners set to: o Reject ballot for one or more overvoted contests o Reject a blank ballot o Accept a ballot that contains one or more undervoted contests

Central count scanners set to: o Outstack a ballot for one or more overvoted contests o Outstack a blank ballot o Accept a ballot that contains one or more undervoted contests o Use mixed mode o Scan ballots batched by precinct with precinct header cards

Early voting, absentee, provisional, and election day reported by precinct


Scan and tabulate tri-fold absentee ballots



Additional Criteria

Election officials must be able to program the AVC Edge (I or II) screen fonts from 14 points to 24 points as viewed by the voter (ref. section 101.56062(f) F.S.)

At least one contest and candidate name set at the WinEDS maximum field length

Layout ballot and identify the WAV files needed to support the elections

Setup five or more groups at least on the General election for early voting, election day, absentee to Monday, absentee on election day, provisional, provisional after 7pm, and overseas.

Add/remove provisional ballot (Edge)

Mark and cast paper ballots and use marked folded absentee ballots

Mark ballots with approved marking devices and with the Edge touchscreen o Exercise the voting system’s audio ballots and language selections

Perform deck processing of absentee ballots into WinEDS

Phase II

Phase II consisted of all activities surrounding election simulation (pre-election activities, early voting

and election day activities, post-election and reporting), audit and security examination, recovery

techniques, modem testing, mass ballot counts, and additional tests.


Election Simulations

Election simulations were comprised of pre-election activities, early voting and Election Day activities,

post-election and reporting. Logic and Accuracy testing (L&A) were considered pre-election activities.

Voting was simulated for four elections (municipal, PPP, primary, and general).

Testing activities for L&A and elections included simulating precinct opening, casting ballots, obtaining

zero reports, closing polls, and obtaining results reports.

Pre-Election

Pre-election activities consisted of preparing the following media for election testing: the Insight memory

packs, and Edge cartridges (results, auxiliary, and voter cards). Election definitions were burned on the

appropriate cards in preparation for election testing. The AVC Edge Cartridge Reader/Programmer was

used to program voter access cards for early voting. (Testing of the Edge, however, was accomplished by

both use of the voter card as well as the manual activation method.)

Pre-audited test decks were prepared for elections including patterned machine marked and hand marked

ballots with over voted, under voted, and blank ballots.

L&A testing was conducted on all machines.

Early Voting

Early voting was simulated for all four election definitions. During early voting, each machine (Insight,

Edge, 400-C) was used for ballot casting and/or tabulation. Machines were powered down during “poll

open” state according to procedure, to simulate early voting conditions. Early voting results were

tabulated, compared to test deck audits, and reported in the election management software.

Election Day

Election Day activities were simulated for all four election definitions. Polls were opened (with zero

tapes), ballots cast, polls closed (and results tapes run), and results were reported to the central tabulator

software (via modem and manual upload). Central count results were compared to precinct scanner and

400-C results, and test deck audit results.

Post-Election

Post-election activities included tracking all the precinct scanners and tally groups to ensure that all

precincts reported in. Also, provisional reconciliation was conducted.

Backup activities were conducted using the Extended Services module. Election databases were backed

up to the server.

Reporting

Reporting activities included election night reporting and precinct-level results reporting. The WinEDS

and Election Reporting Program (ERP) were used to generate the required election-night results reports

and the precinct-level reports. The congressional district report was generated, for the PPP election.

Machines were then imaged as part of post-testing activities.

Audit Trail

Audit logs were examined, as well as administrative reports. Some examples of reports that were printed:

Log data – The log data captured by WinEDS 4.0.175 is a considerable improvement over that of

version 3.1. Log data can now be filtered by various specifications.

Users by activity

Users logged in


Users by time/date

OS event logs

Voter registration

Precinct listing

Polling place listing

Ballot style listing

Security

BVSC exercised the voting system’s security by simulating attempts at unauthorized access to all relevant

systems and sub-elements. Each component of the system is password-protected, and functions are

assigned at the user/group access level. Furthermore, staff simulated using a memory card from a

previous election in both the Insight and the Edge. There were no unexpected outcomes.

Recovery Techniques

An Insight failure / replacement on Election Day was simulated. A test of “swapping” the Edge cartridge

during election mode was conducted. A simulation of extracting ballot images after catastrophic failure

was conducted, and a review of the manual recount utilizing stored ballot images was conducted.

Modem capability

The modem-enabled Insights were tested with the Insight Listener modem results-reporting software.

Tests included single-precinct/single-Insight modeming, single-precinct/multiple-Insight modeming,

multiple-precinct/single-Insight modeming, multiple precincts sharing ballot styles (must be reported

manually, per design), catastrophic failure during modeming, and multiple Insights/simultaneous

transmission. The system passed all modem-related tests.

BVSC staff requested and were provided updated keypad codes to finalize modem testing which were

done without incident. BVSC recommends that the vendor provide counties with the most up-to-date

documentation for the Insight keypad codes, and counties train their poll workers in the use of these

codes, as familiarity with the codes is necessary to operate the modem functions of the Insights. Trouble-

shooting instructions would be an improvement. Furthermore, due to changes in the memory pack (see

section below titled “Insight Memory Pack Engineering Change Order 781, 02/09/2010”), any new

installation of modem-capable Insights should be vigorously tested in pre-election activities.

Mass Ballot Counts

Volume testing, or mass ballot counts, was conducted on the precinct scanner (Insight), the central count

scanner (400-C), and the DRE (Edge).

Insight Precinct Scanner

This voting device underwent a mass ballot count test, on a single unit, with the casting of 9,984 ballots

(128 decks cast at 78 ballots per deck). The General Election was used for this effort. There were

504,064 vote targets.

Edge DRE Tabulator

This voting device underwent a mass ballot count, on a single unit, with the casting of 9,900 ballots. The

voting simulator software and the Primary Election were used for this effort.

400-C Central Count Scanner

A full mass ballot count was conducted on this scanner using the General election. One machine was used

in Tallahassee, and two machines were used in Palm Beach County. A total of 193,440 ballots were cast,


with 620 vendor-produced decks cast four times, at 78 ballots per deck. The total number of vote targets

was 9,766,240, with an error rate not exceeding 1 in 1,000,000 targets.

The 400-C returned a Registered Voter/Turnout Check error when it reached around 65,000 ballots. This

error is triggered when the number of ballots exceeds the number of registered voters, or if the number of

ballots in a precinct exceeds 65,536 (216

). According to DVS, the limit is a precinct limit (maximum

number of ballots in a precinct). Because the ballots used in the mass ballot count were all printed in one

precinct, which is highly unlikely to occur outside of test conditions, this error message was not counted

as a memory register overflow.

The 400-C demonstrated sensitivity to static electricity during this test. BVSC recommends that any

installation of the 400-C include steps for static electricity mitigation, including but not limited to: use of

the vendor-provided “ground strap” to ground the machine and the external ballot bin, a rubber or anti-

static mat, and anti-static spray.

Mock Contest Recount

Recount activities were conducted on the 400-C. The General election definition was used for this

activity. The 400-C demonstrated ability to recount a precinct-level race and a district-wide race on a

multi-card election.

Simulated Catastrophic Failure

A complete failure was simulated during scanning of marked ballots to determine the system’s capability

to retain ballot images so that they could be extracted in the event of catastrophic failure, caused by an

event such as a sudden power outage. All machines (Edge, Insight, 400-C) retained ballot information as

designed after power outage.

Ballot Sensitivity

The test decks used to evaluate the Insight and 400-C scanners’ ballot sensitivity were created by marking

blank ballots from the General and Primary Elections that were previously used in this initiative. The test

was performed with ballots marked in such a way as to determine the capability of the scanner to read

marks made by various marking devices. The table below illustrates the specific instruments utilized to

mark the 48 ballots scanned for this activity. The test included a baseline scan of the ballots with selected

broken arrows (“targets”) fully marked (that is, the broken arrow is fully “completed”). A test deck was

then run, with the same targets marked, but with a 1 millimeter line through the center of the target, rather

than fully marked. Also included in the table, in the column labeled ‘Finding,’ are the results obtained for

each scanned ballot. The test was conducted on one Insight and one 400-C, with a separate test deck for

each machine.


Ballot Sensitivity – Optech Insight

Marking Device Device Type Color

Finding A=ballot accepted; UB=unvoted ballot

Fully Marked Target [Baseline]

1 mm Horizontal Line thru Center of Target

1 Office Depot Pencil #2 Gray A A

2 Steadler Pencil #2B Gray A A

3 Steadler Pencil #B Gray A A

4 Steadler Pencil #HB Gray A A

5 Steadler Pencil #F Gray A A

6 Steadler Pencil #H Gray A A

7 Steadler Pencil #2H Gray A A



10 K-Dent

Ball Point (med. pt.)

Blue A A

11 “Hotel” Ink Pen [Renaissance Hotel]


Light Blue A A

12 Pilot G2 Ink Pen


Red A A

13 Papermate


Green A A

14 EF Felt Pen

Felt Tip (med. pt.)

Black A A

15 Sharpie Highlighter Orange A UB

16 Vendor (Micron 08 Pen) Felt Pen (fine pt.)

Black A A

17 Vendor Pen- “X” mark

Felt Tip (fine pt.)

Black N/A A

18 Vendor Pen- “” mark

Felt Tip (fine pt.)

Black N/A A

19 Vendor Pen- “\” mark

Felt Tip (fine pt.)

Black N/A A

20 Vendor Pen- “” mark on oval line

Felt Tip (fine pt.)

Black N/A A






Ballot Sensitivity – 400-C

Marking Device Device Type Color

Finding A=ballot accepted; UB=unvoted ballot

Fully Marked Target [Baseline]

1 mm Horizontal Line thru Center of Target

1 Office Depot Pencil #2 Gray A A


3 Steadler Pencil #B Gray A A

4 Steadler Pencil #HB Gray A A

5 Steadler Pencil #F Gray A A

6 Steadler Pencil #H Gray A A




10 K-Dent


Blue A A

11 “Hotel” Ink Pen [Renaissance Hotel]


Light Blue A A

12 Pilot G2 Ink Pen


Red A A

13 Papermate


Green A A

14 EF Felt Pen

Felt Tip (med. pt.)

Black A A

15 Sharpie Highlighter Orange A A

16 Vendor (Micron 08 Pen) Felt Pen (fine pt.)

Black A A

17 Vendor Pen- “X” mark

Felt Tip (fine pt.)

Black N/A A

18 Vendor Pen- “” mark

Felt Tip (fine pt.)

Black N/A A

19 Vendor Pen- “\” mark

Felt Tip (fine pt.)

Black N/A A

20 Vendor Pen- “” mark on oval line

Felt Tip (fine pt.)

Black N/A A





NOTE: For items with “N/A” results, the test mark was a special mark (such as “” or “X”) using the

vendor’s pen. These were compared against the baseline mark of a fully marked target using the vendor’s pen.


This test shows that the Insight and 400-C scanners are able to detect a wide variety of marks made by

several different marking devices, including ink pens, markers, highlighters, and pencils. The only tested

marking device that was not accepted by the Insight was the orange Sharpie highlighter; it was, however,

accepted by the 400-C. Because of the discrepancy between the two machines’ sensitivity to this marking

device, the use of highlighters is discouraged.

Sound Pressure Level

This test was conducted to test conformance to section 101.56062(1)(g), (h), and (i), F.S., describing the

sound pressure level (SPL) standards for a voting system’s audio voting features. The Edge, which

incorporates the audio voting accessory (hand-held input device) and headphones, was found to conform

to the standard. Voting was conducted using the touch screen and the audio voting accessory using

vendor-supplied standard headphones. The test results are listed below.

Sound Pressure Level Results

Maximum Volume (dB) Default Volume (dB) Intermediate level above 12 dB

Right 97.20 63.18 78.20

Left 100.40 66.50 81.30

Edge Audio Language Loop

At the request of Palm Beach County, the Edge was tested for a possible software bug in which selecting

“help” during audio voting triggers a “loop” of language selection prompts. The Edge was activated for

voting, and then the “Help” button (on the audio voting accessory) was selected at various points in the

voting menus. No language prompt loop was observed, and voting was conducted as usual.

During the Sound Pressure Level test (see above), however, the Edge screen presentations prior to the

ballot screen were confusing. Separate testing was conducted to verify these observations, utilizing both

the touch screen and the audio voting accessory. When the user selected a language at the “language

selection” screen, the audio script proceeded to the next audio instruction, while presenting a redundant

language selection screen. The audio instruction acknowledged the user’s language selection, but the

presented screen did not appear to acknowledge the user’s selection. Upon completion of the audio

instruction and without user input, the program proceeded to the ballot screen. If the user selected a

language at the redundant language selection screen, the screen immediately advanced to the ballot

(without waiting for the recording to end), but did not change the language selection (that was

accomplished at the initial language selection screen).

Although a “loop” was not observed, this duplicate language selection presentation can be a possible

source of voter confusion, leading voters to infer that they must select a language a second time. BVSC

recommends that DVS devise a more user-friendly screen that visually verifies the user’s language

selection while the audio instruction is playing. Poll workers, meanwhile, should be advised that, while

language selection a second time is unnecessary, it will immediately advance the screen to the ballot.

Force

This test was conducted to test conformance to section 101.56062(1)(l), F.S., in which the standard is no

more than 5 pounds of force required to operate a voting system’s controls. The force test was conducted

on both the audio voting accessory and the Edge touch screen. A force gauge was used to measure the

force necessary to activate the buttons on the audio voting accessory, and various regions of the Edge

screen. Both the accessory and the Edge were found to conform to the standard.

Daylight Saving Time Change

This test was conducted to verify the precinct scanners’ ability to automatically display the correct time

during a Daylight Saving time change. Neither the Edge nor the Insight was designed with this capability.


Once the time is set as part of the election definition on the Edge and Insight cartridges, it cannot be

changed during the course of the election. Although time change capability is not an FVSS requirement,

and the absence of this capability does not affect the other functionality of the scanners, counties using

this voting system should make note when they deploy this system for an election that spans a time

change event, as it will affect the date/time stamps on the zero tapes and results tapes.

Source Code Review

The core source code was reviewed by the voting system testing laboratory iBeta Quality Assurance (see

report, “Dominion Voting Systems Sequoia WinEDS 4.0 VSTL Certification Test Report” Version 2.0

(V)2010-30Nov-001(B), November 30, 2010). The recommendation of iBeta was to certify the WinEDS

voting system.

Insight Memory Pack Engineering Change Order 781, 02/09/2010

DVS issued an Engineering Change Order (ECO) in February, 2010 regarding a problem with the RAM

addressing inside the Insight memory pack. This ECO describes a change to the memory pack’s circuit

board configuration, in which a “jumper” is to be soldered on a chip on the circuit board. The memory

packs currently in use in the Florida counties do not have the ECO configuration. The ECO was not

submitted as part of this certification’s technical data package.

The memory packs used during this certification testing effort were of two configurations: the memory

packs with the older configuration, supplied by BVSC, did not have the solder jumper and the current

configuration. Unbeknownst to BVSC, the memory packs supplied by DVS2 had the new configuration

with the soldered jumper for use with the voting system tests. Both types were upgraded with the

firmware as specified in the vendor’s certification application.

During the certification testing, BVSC did not observe any behaviors in the memory packs that might

indicate differences in memory pack configurations, or any unit-specific problems. However, BVSC did

not test the memory packs for this particular variable, as we were not made aware of the ECO until after

testing was complete. According to documentation recently submitted by DVS, the current memory

addressing “does not cause problems in most cases,” but some memory packs without the ECO change

might exhibit problems, “most likely to happen in larger election definitions”. DVS has stated that “the

size of, and successful past history with the Insights in Indian River show that they do not need this

update.”

The ECO has been in effect since February, 2010. Since then, no election issues have become known to

BVSC surrounding the memory packs. BVSC believes the risk of implementing this ECO in the memory

packs used by Florida counties is minimal. Therefore, it is with reservation that BVSC allows this ECO in

the certification for WinEDS 4.0.175, version 1. Furthermore, BVSC recommends that counties

implementing WinEDS 4.0.175, version 1 should vigorously test memory packs during pre-election

activities, especially in election definitions that might be as large as, or larger than, definitions

successfully run in the past. BVSC requires that all memory packs that have the ECO 781 change must

have an external identification on the memory pack.

Test Results and Findings of Compliance

With the exceptions listed below, test results yielded no unexpected outcomes. However, there were items

identified during testing that have prompted BVSC to make recommendations to the vendor and/or the

Florida counties. None of the items below would preclude the certification of this voting system; in fact,

several of the exceptions have to do with the documentation for the voting system, not the system itself.

2 Because some of the vendor-supplied memory packs went back to the vendor at the conclusion of testing

and prior to this issue coming to light, it is unknown if all, or only some, of the vendor-supplied memory packs contained the modification.


Exceptions

1. SQL Server and ERP communication error. Due to the naming convention used for the SQL

Server name, the ERP could not communicate with it. This was resolved by changing the host

files. BVSC recommends that any servers used in conjunction with the WinEDS election

management system follow a naming convention that excludes special characters.

2. During the coding of elections, BVSC found the documentation to be confusing and labyrinthine.

3. The modem functions of the Insight are dependent upon staff having the most updated code

documentation. The vendor should provide counties with the most up-to-date documentation,

with troubleshooting instructions, for the Insight keypad codes, and counties train their poll

workers in the use of these codes. Furthermore, due to changes in the memory pack (see section

titled “Insight Memory Pack Engineering Change Order 781, 02/09/2010”), any new installation

of modem-capable Insights should be vigorously tested in pre-election activities.

4. The Sensitivity test revealed a variance in sensitivity with regards to highlighters as marking

devices. BVSC discourages the use of highlighters as marking devices with this voting system.

5. The 400-C demonstrated sensitivity to static electricity during the Mass Ballot Count. BVSC

recommends that any installation of the 400-C include steps for static electricity mitigation,

including but not limited to: use of the vendor-provided “ground strap” to ground the machine

and the ballot bin, a rubber or anti-static mat, and anti-static spray.

6. Memory packs with the new configuration described by ECO 781 should be vigorously tested

during pre-election activities, especially in election definitions that might be as large as, or larger

than, definitions successfully run in the past. BVSC requires that all memory packs that have the

ECO 781 change must have an external identification on the memory pack.

A copy of the firmware and software provided for this certification effort has been duplicated and has

been provided to Dominion Voting Systems so that they can make certified copies [referred to by the

Division as the ‘Bronze Disks’]. As usual, DVS will create copies of the Bronze Disks and return those

to the Division. The Division will verify that the contents of the media matches the firmware and

software which were certified and will then forward same to a county as needed. The voting system

under test is listed below.

WinEDS Release 4.0.175, Version 1

Election Management System:

WinEDS application, version 4.0.175

o Extended Services 1.0.81

o Election Reporting, version 4.0.77

Optional software

o Ballot Printing System II (BPS) version 3.41 N.05

o WinEDS Bridge Tool (WBT), version 1.0.44 o Insight Listener version 1.0.3

AVC EDGE Cartridge Reader I Programmer (Part No. 460-40070-00) o with COTS device driver as indicated on product label o Optional Sycard PCCextend 70A (replacement/or the EDGE Cartridge Reader /

Programmer)

Memory Pack Reader (MPR) o with firmware version 3.01.080422.0552

COTS software

o SQL Server 2005 SP2


o MS Office 2003 Standard SP2

o MS Visio 2003 Standard SP2

o MS Visio 2002 Standard SP2 (Only for BPS)

o .NET framework 2.0 o Windows SteadyState 2.5 o Optional Adobe Acrobat Standard version 8 o MS Windows XP Pro SP2 (for workstations) o MS Windows Server 2003 R2 Standard (for servers)

COTS hardware

o Server and workstation(s) specifications per Sequoia WinEDS System Operations Prodecures Election Media

o Sequoia Edge ATA/PCMCIA Flash Memory Cartridges for use with EDGE I and II

128MB, 256MB, 512MB,or 1,024MB o Memory pack with ECO 781 for Insight and Insight Plus

APX L2.18.100205.1359

Precinct Count:

Optech Insight and/or Insight Plus o with firmware version:

HPX L1.46.100205.1100 o Optional MODEM

with firmware version CPX v J-L1.16.111003.1633 o Insight Plastic Ballot Box Sequoia Insight Plastic Ballot Box or Election Works, Inc.

folding ballot box ADA polling equipment3:

o AVC EDGE I and/or EDGE II

with firmware version 5.2.16 o Auxiliary equipment/or EDGE I or II:

Optional AVC EDGE Card Activator, version D

with firmware version 5.2.6

AVC EDGE Audio Component, Revision D

with firmware version 8.7.7

Optional AVC EDGE Auxiliary Power Unit

Sip and Puff Interface

Optional Seiko DPU-414 printer

Central / Absentee Count:

Optech 400-C Ballot Counter, Model 3.00P o with WinETP software version 1.16.15 o COTS PC computer System Unit per Sequoia Voting Systems Optech 400-C System Overview

with MS Windows XP Pro SP2

3 Note: By 2016, ADA polling equipment must comply with Section 101.56075(3), Florida Statutes. Therefore,

AVC EDGE I and EDGE II will not be acceptable for any election in 2016 and thereafter.

Voting System Qualification Test...

Documents

Transcript of Voting System Qualification Test...