STATEWIDE HIO Vendor Selection Yolande Calhoun, Chris Davis, Kyle Johnson, Joanne Rimac & Natalie Schwartz 2/22/2011

Table of Contents

1. Introduction: ..................................................................................................................... 3

2. Project Objectives .............................................................................................................. 4

3. Political Environment ......................................................................................................... 4

4. Key Design Decisions and Assumptions .............................................................................. 6

5. Selection Criteria .............................................................................................................. 9

Methodology and Reasoning behind Selection Criteria- HIE Organizational landscape is too new,

No proven sustainable models ......................................................................................................................................... 9

Selection Criteria ................................................................................................................................................................... 12

6. Analysis of Vendor Responses .......................................................................................... 13

Customer Testimonials: ..................................................................................................................................................... 14

7. Scores ............................................................................................................................. 16

8. Cost Proposals ................................................................................................................. 16

9. Acceptance Testing ......................................................................................................... 18

Describe the role of acceptance testing .................................................................................................................... 19

The role of acceptance testing play in the decision ............................................................................................ 20

Acceptance testing performed on the product ...................................................................................................... 21

Subcommittee List of Objectives for the testing process ..................................................................................................... 21

Steering Committee List of Objectives for the testing process .......................................................................................... 21

When will acceptance testing be performed? ........................................................................................................ 22

10. Change Management ..................................................................................................... 22

Statewide HIE Change Request Form ........................................................................................................................ 24

11. Discussion ...................................................................................................................... 25

Attachments ....................................................................................................................... 26

References .......................................................................................................................... 26

3

1. Introduction:

Statewide HIO has been engaged in the process of collecting responses to our Request for Proposals

(RFPs) to assist in the vendor selection process to provide HIE services to state X. After approval by the

Statewide HIO Board of Directors, the RFP was released for responses on February 11, 2011. After the

RFP due date, the Statewide HIO RFP Selection Committee carefully reviewed and compared each

vendors responses. We respectfully submit this document to the Board of Directors with our final

vendor recommendation for approval.

Members of the HIO RFP Selection Committee include:

Evaluator HIO Subcommittee Membership Title/Organization

1.Thomas Brady Clinical Advisory M.D., State X University Hospital

2.Maria D’ Angelo Privacy & Security Privacy Officer, St. Vincent Medical Center

3.Jeffrey Fox Technical CIO, Southern General Hospital

4.Rajendra Kothari Technical Systems Analyst, Federated Bank

5.Carol Manning Clinical Advisory Clinical Nurse Specialist, County General

6.James Prince Clinical Advisory M.D., Erie Community Hospital

7.Sara Rowe Clinical Advisory M.D., State Department of Health

8.Randall Smith Finance J.D., C.P.A., The Smith Group

HIO Steering Committee Members Title

Yolande Calhoun Management Analyst

Chris Davis M.D.

Kyle Johnson CIO

Joanne Rimac R.N.

Natalie Schwartz M.D.

2. Project Objectives

The Statewide HIO Selection Committee considered the following core objectives in evaluating each

vendor’s RFP responses:

Exchange of clinical information in a standard format, such as Continuity of Care Document (CCD).

Exchange of clinical messaging and referrals between participants of the HIE.

The ability to connect to the following entities through virtual or HL7 gateways: o Hospitals o Physician offices o State Public Health agencies (for immunization, syndromic surveillance and

reportable lab registries) o ePrescribing entities o Payers

Provide security of clinical information in a HIPAA compliant fashion.

Conform to national standard vocabularies and code sets being established by the HIT Standards Committee Vocabulary Task Force, the HIT Standards Implementation Workgroup, and the ONC Standards and Interoperability Framework Contractors.

Provide technology which delivers high availability of the HIE services.

Provide the ability to connect to HIEs in other regions, states, and with a National Health Information Network (NHIN).

Support the information technology needs of the HIO (internet, email, web services).

Additional objectives needed to support Phase 2 and 3 of Meaningful Use which were considered in the

RFP process include:

Provide an electronic consent repository.

Provide directory services for providers, payers, and public health entities.

Provide Public Key Infrastructure (PKI) to allow for more sophisticated secondary authentication.

Provide the ability to electronically submit quality reporting to state and federal agencies.

3. Political Environment

Since the implementation of the SOW which includes this RFP, the following changes have occurred:

Acquisition of HIE Vendors by Payers

5

Prior to the release of this RFP, Axolotl and Medicity were acquired by Ingenix and Aetna respectively. These acquisitions are examples of a growing trend for larger established health insurers who are heavily invested in IT to acquire HIE vendors. It is predicted that within the next five years, there will no longer be small, independent HIE vendors. The independent HIE vendors that survive will be a division of a far larger company. These insurers who have the infrastructure to collect and analyze lots of patient data and to

manage risk exposure want to make their technology and data more easily available to

doctors. Procuring HIEs will allow the payers to be able to easily share and access patient

data and electronic health records. With these acquisitions, a closer relationship between

ACOs and payers will be forced. This change does not affect the RFP, but is a change in the

political environment worth noting.

Privacy concerns

The acquisition of HIE vendors will give health insurers the ability to better share administrative patient data with doctors. However, this may also require that doctors also share clinical patient data with insurers. The risk with this is that it could lead to insurers denying care to the riskiest patients. For the statewide HIO, supporting this infrastructure could put the entity at risk for involvement in lawsuits by the patient if such a scenario should occur. With this in mind, the negotiated contract with the selected vendor will have to be written in a language where the statewide HIO is protected from any lawsuits such as these.

The potential risk that HIEs may pose in complying with Health Insurance Portability and Accountability Act (HIPAA) regulations has been expressed by the provider community. Because of these concerns, there may be a reduction in the quantity of data the providers are willing to share which reduces the quantity of data available.

Changes in leadership – Governor, representatives, etc.

With new leadership, come new ideas and new changes. A new governor was elected for

this state in January and has appointed new leadership to sit on the steering committee.

Also, there have been changes in the leadership of the statewide agency with the

appointment of a new commissioner of the healthcare agency sponsoring the statewide

HIO. Whereas, this initiative is still supported, there may be different ideas or changes in

direction from the steering committee on what the statewide HIO may look like than was

originally agreed upon to support this RFP.

Also, with the most current election, a Republican majority U.S. House of Representatives

was elected. Currently, there is a battle over the budget and some of the dollars allotted for

healthcare reform are on the chopping block. If the budget is not passed by March, the

funding that was originally targeted to support this initiative may not be available and may

delay the project’s ability to award a vendor according to the schedule originally defined in

the statement of work. If the budget is passed and there is a reduction in the allotted

dollars for the ONC to work with for this fiscal year, there may be a delay in the trickling

down of money to the steering committee. With this possible reduction or delay in funding,

the state and steering committee will have to seek other sources of funding which will

change the schedule for implementing the statewide HIO. This lack of funding or trickling

affect may also affect the scope of the HIO and may require that a scaled down version of

the statewide HIO be initially implemented.

4. Key Design Decisions and Assumptions

Statewide HIO outlined several key design decisions and assumptions that would be required to meet

the needs of an HIE application. The HIE landscape is immature and in a high state of change and

growth as an industry. Industry experts predict that 50% of all HIEs today will fail and not become

sustainable. The most important consideration for most HIEs is cost, scalability, and technology. The

primary areas that are most important to the Statewide HIO are:

Data Exchange – Statewide HIO has determined the approach to data interoperability that will best meet its overall business needs. If you compare an HIE to music available through the Apple iTunes store, customers log in and download their music from a repository and take the music – or in the case of an HIE – the data away. In a federated model, the data is collected in edge servers where it is indexed or retained in its original repository and accessed by record locator pointers that contain minimal patient identifying data to locate data related to that patient. The Hybrid HIE model stores some data in a central repository within the HIE application for registry or population reporting. Most other data is stored at the edge server level and accessed via the record locator service. The Hybrid model does a better job of managing security and privacy concerns of the HIO and this model generally is lower cost than the central repository.1

Statewide HIO has made a decision to begin with the Federated approach in order to stand up

the HIE quickly. This will allow the HIO to meet basic data exchanged for providers. The long-

term vision of the HIO is to move to the Hybrid approach. The HIO believes the Hybrid approach

best matches their overall business and security needs. Starting with the Federated model will

allow the HIO to spend less dollars up front before moving to the Hybrid model. Both Axolotl and

Medicity offer both approaches, although this is part of the base offering of Axolotl and is an

add-on feature for Medicity.

7

Open Systems Architecture – Statewide HIO seeks to find a vendor who has adopted an Open Systems Architecture approach to their HIE application. Open Systems Architecture can take advantage of all of the healthcare message & transaction standards –

Both Axolotl and Medicity can accommodate the message and transaction standards required

by Statewide HIO.

HITECH Security Standards - Security policies to be considered revolve around access, authentication, audit, and authorization – the administrative safeguards of HITECH. Statewide HIO requires its HIE application to provide all of these appropriate safeguards. Security becomes an important issue for provider willingness to exchange their data.

Both Axolotl and Medicity, in either a Federated or Hybrid data exchange approach, will assist

Statewide HIO in meeting its security goals.

System Reliability, Availability, and Scalability - Consistent system reliability and guaranteed availability of the system are critical for seamless patient data exchange. The data exchange services will become as mission critical as any provider EHR. Statewide HIO is looking for a vendor who has a proven track record for stabile system operations and 99.999% uptime availability. HIE solutions also need to be scalable for future growth of the HIE. HIE’s begin exchanging a few data types – laboratory, radiology, and transcribed document results. The HIE will grow in other data types and services such as referral, consent, e-prescribing, registry, and population reporting services. As the HIE expands in its services, the HIE application will need to scale up to accommodate the growth.

Axolotl offers both reliability guarantees and 99.999% uptime availability and has a proven track

record for meeting HIE scalability based on the HIO needs. Medicity offers the reliable systems

Message Standards and Transport Protocols

CCD

CCR

DICOM

EDI X12 – Administrative transactions

HL7 V2.x

Hl7 V3

NCPDP

SOAP (Web Services)

SSL/TLS

XML

guarantee, and also offers a 99.99% uptime guarantee. Medicity has had some challenges with

scalability, and also requires more planned downtime during the year.

Remote Hosted vs. Self-hosted Technology Models – Statewide HIO decided to use the RFP HIE search and selection process to help inform the HIO about which hardware hosting approach that would be best for the business. Statewide HIO has determined that its core business competency is not hardware hosting and infrastructure and has decided to find a vendor that can provide these services for the HIO.

Both Axolotl and Medicity have cloud computing offerings which will allow the HIO to remote

host its hardware operation for the HIE application.

HIO Privacy Requirements – Statewide HIO’s position on patient consent is “opt-out”. The HIO requires an HIE application that will support an opt-out approach. It is important to the HIO that the HIE application can also support opt-in and no-opt approaches in the future in the event that Statewide HIO changes its consent policy.

Another concern from a privacy perspective is the issue of malpractice. The HIO could find itself

in a malpractice lawsuit if a provider either chose not or did not use information available to

him/her in the delivery of patient care. Statewide HIO will address this as part of the privacy

terms and conditions in the vendor contract.

Both Axolotl and Medicity can provide the selected option for patient consent to the HIO.

Vendor Stability and Reliability – In the HIE application RFP, Statewide HIO outlines stability and reliability its vendors must meet:

o The vendor must have experience within the last 2 years in HIE business planning and development, technical planning and design, implementation and operations support. The response must provide the details of each area of expertise and the exact years of experience.

o The vendor must be the prime vendor in the operation of at least three HIE installations.

Axolotl and Medicity both meet the requirements of vendor stability and reliability.

9

Cost – Cost is an important consideration in choosing an HIE, but not the only consideration. Other important considerations are listed in this section – features and functions; open architecture; security & privacy; system reliability, scalability, and availability; technology; and vendor reliability and stability. All of these aspects of the HIE application will be thoughtfully weighed and the vendor solution providing the closest match to the overall business needs will be chosen.

Medicity is the lowest cost option of our vendors, with Axolotl coming in as a close second.

5. Selection Criteria

Methodology and Reasoning behind Selection Criteria- HIE Organizational landscape is too new, No proven sustainable models

The selection criteria, which follows relied on the Delphi process to decide an appropriate scoring

technique for selecting the vendor. The criteria are a culmination of objective and subjective data

collection, tangible and intangible asset evaluation and evaluation of the vendor’s current customers.

The vendors completed the worksheets identified in the RFP and the responses were analyzed and

applied to the appropriate selection criteria. Due to the known lack of HIE sustainability, the categories

receiving higher scoring priority are as follows: Cost, Vendor Experience, Scalability, Ability to provide

infrastructure, design and host services and Customer site visits.

Cost Comments

A. Implementation costs Costs spread out over 5 years vs. Higher upfront

costs. Fear over how the ARRA money will trickle down

to the HIE level and what the future funding will be after

incentives disappear.

B. Software pricing

C. Professional Services

D. Hardware

Vendor Respondent Experience

A. Two years experience in HIE business

planning

Few vendors can boast significant live customer

experience, but looking for an experienced vendor with

solid, proven solution

B.Prime vendor for 3 HIE’s

Vendor Minimum Requirements

A. Provide basic HIE functions-clinical

messaging, referrals, inquiry, e-prescribing,

public health reporting, EMPI, record locator

services

Depositing patient data directly into the EHR so

that it becomes actionable should not be the immediate

goal. Most successful HIEs to date are those with the

least complex approaches. More focused on passing

packets of data around without taking ownership of the

data---some manual intervention will be required with

almost every transaction.

What will be critical in the future is analytics and

business intelligence and you cannot run reports in a

federated model---but, it is easy to deploy. Centralized

HIE models are fading due to logistical issues, security

liabilities and upfront expenses. Can we afford data

warehouses?

Hybrid HIE with a central server to hold onto data for a

finite period and for compiling reports will become the

industry standard. In the future, PaaS will allow the HIE

vendor to host the service in the cloud and health care

providers will be able to customize it.

B.Conform to national standards Even vendors that are CCHIT certified have a

rudimentary ability to send out results in a standardized

way. Public HIEs- direct and temporary government

backing. Demand strong standards-based technology

C.Comply with privacy & security requirements

for an HIE

Very important factor, since the potential risks that HIEs

may pose in complying with HIPAA may be a barrier to

stakeholder participation

Response to Request for Additional Information Comments

A. Timely Response<1 week

11

B. Answered as directed

Scalability Public HIEs- direct and temporary government backing.

Demand strong potential scalability

A. Consent Repository Vendors that can best manage patient consent

management are most successful

B.Quality Reporting Will require a centralized or hybrid solution

C.Vocabulary Services

D.Security tokens

Infrastructure

A. Software components Vendors with the most flexible, customizable

application programming interfaces and software

development kits will succeed in the PaaS model

B.Hardware requirements

C. Reliability of the System

Design Comments

A.Integration

B.Adapter Framework

C.Disaster Recovery

D.Programming Language

Host Services Advantage for SaaS- costs? Implementation? Scalability?

Maintenance?

A. Independently

B. Relationship with Host Provider

Usability

A. Workflow

B. Warranty

C. Training Tools

D. Support 24/7

Vendor References

A. Listed 3 HIE’s involved with

B. Provided 3 customers who verified project plan

involvement

Vendor Demonstration

Customer Site Visits

Selection Criteria Category Score

Price of the system 15

Vendor respondent experience 15

Vendor qualifications meet the Minimum Requirements stated in the RFP

10

Response to requests for additional information from vendor respondent

10

Scalability of the proposed system to meet 15

13

future growth of the HIE

Vendor respondent ability to provide all aspects of the RFP: Infrastructure, design and host services

15

Usability of the system for end-users and IT staff maintenance within the HIE

10

Vendor References 10

Vendor Demonstration 5

Customer Site Visits 15

Total Score 120

6. Analysis of Vendor Responses

There are very few experienced vendors with live HIE’s who have met their objectives. Four of the

leading HIE vendors within the industry responded to our Statewide HIO RFP by the deadline, Axolotl,

Medicity, RelayHealth, and Epic. We used the Delphi process to support a side-by-side vendor

evaluation of these contenders. We selected eight leading members from our panel of steering

committees, to serve as key evaluators of the respective products. The evaluators represented the

clinical, technical, privacy and security, finance, and legal constituencies of the HIO, as well as the broad

interests of the private and public health care sectors across State X. These evaluators were responsible

for scoring each of the vendors based upon:

Their responses to the Features Functions Worksheet of our RFP

The HIO Vendor Selection Criteria

Submitted Client References

Site Visits

Vendor Demonstrations

Client Testimonials

The eight key evaluators used a Vendor Evaluation Scorecard (Appendix A) to score each of the four

vendors on their individual responses to the Features Functions Worksheet of our RFP. The average

vendor scores were 7.61 for Axolotl, 7.31 for Medicity, and 5.01 for RelayHealth. Epic only connects Epic

software clients and was not fully scored.

Based upon the vendor evaluation scoring, we eliminated RelayHealth and Epic as viable partners in our

Statewide HIE initiative. We continued to vet the remaining vendors, Axolotl and Medicity, which are

industry leaders in HIE innovation and implementation. In fact, according to the 2010 KLAS report,

Medicity and Axolotl were considered in 23 percent and 22 percent, respectively, of the HIE buying

decisions.2

We requested three current HIE client references from both Axolotl and Medicity, and performed full

day site visits at two live HIE operations of each vendor. We also invited our Board of Directors, Steering

Committee members, Key Evaluators, and all interested stakeholders to a number of scheduled vendor

product demonstrations.

Customer Testimonials:

While cost, technology, scalability, integration, and privacy concerns are major determinants in our

selection process of an HIE vendor, there are few proven models of success in this evolving and

precarious industry, and finding the ‘perfect’ vendor has proven to be uniquely challenging. Aside from

the objective criteria, we need to align with a vendor that shares Statewide HIO’s vision for the future of

health information exchange, is committed to transparency and a spirit of partnership in all negotiations

and implementations, and fits the culture of our organization. Some of these softer criteria can be

assessed by focusing in on the marketing strategies and attitudes of the vendors’ representatives during

site visits, vendor demonstrations, and, even, informal conversations.

We have gained a deeper understanding of the organizations behind the RFP responses and marketing

tools of our final two competing vendors, by referencing KLAS, a company that evaluates health care

vendors, products, and services. KLAS spent one year investigating the solutions of the HIE industry

leaders. As part of their initiative, they elicited client satisfaction responses regarding vendor products

and services. 2

15

The KLAS report indicated that Axolotl was one of the “largest commercial vendors in the HIE space,

with a fast-growing client base evenly distributed between acute to acute and acute to ambulatory HIE

organizations. It has the most live acute to acute HIE clients". Clients spoke enthusiastically about a

“straightforward product…that was easy to use”, “well-done implementations”, and “customizable

workflows and flexible report distribution features”. Many customers cited Axolotl’s ‘lite EMR solution’

(for sites that do not have a full-blown EMR) as an impetus for widespread health information exchange.

Most customers claimed measurable improvements in their operational efficiency. Customers were

satisfied viewing data via Web portals and taking advantage of remote access. Interviewed live sites

were content with manual search models and receipt buckets that collected requested information, but

looked forward to Axolotl’s next phase in which information would populate the resident EMR without

interfacing, or, perhaps, have the EMR proactively search for needed information and display it for

physicians. Clients were positive about Axolotl’s account management and executive involvement, and

acknowledged that “the organization contributed to smooth contracting and implementations.” Other

customers commented that the “contracting process was fair with key executives involved throughout”.

One client stated, “Part of my frustration is that Axolotl knows a lot, but we don’t know what to ask. So

we trip over something and Axolotl will have the answer but will never have told us.” In addition, many

customers asked for “better training and faster development cycle for upgrades”. 2

Medicity customers applauded the company’s strength and efficiency in acute to ambulatory exchange

of healthcare information. They described the Novo Grid product as “simple and easy to use…very

successful for pushing information to doctors’ offices.” It was “outpacing the competition by a wide

margin as a facilitator for simple results distribution...funneling lab results from hospital labs and

reference labs to ambulatory sites.” Success of the ProAccess solution was more difficult to measure,

as it was only in use at a few acute to acute HIE organizations. Other positive customer comments

included, “Once properly configured, the operation is fluid and stable to use”, “Novo staff are talented

and intent on implementing client feedback in application upgrades”, “Medicity is really good at getting

disparate software systems to talk to each other”, and “Medicity was the only vendor that came with a

business proposal and not just a technology proposal”. In contrast, some other customers made note

that, “Initial phases of Novo Grid’s implementation can be labor intensive and time consuming while

configuring the HIE platform”, “System documentations for new releases have room for improvement”,

and “Since Medicity acquired Novo, there are longer turnaround times”. Customers also wanted “more

regular feedback on in-progress interfaces”. 2

7. Scores

Selection Criteria Category Score

Axolotl Medicity

Price of the system 15 11.16

12.16

Vendor respondent experience 15 15 15

Vendor qualifications meet the Minimum Requirements stated in the RFP

10 10 10

Response to requests for additional information from vendor respondent

10 10 10

Scalability of the proposed system to meet future growth of the HIE

15 14.50

14.50

Vendor respondent ability to provide all aspects of the RFP: Infrastructure, design and host services

15 12.72

12.74

Usability of the system for end-users and IT staff maintenance within the HIE

10 9.91

9.91

Vendor References 10 9.81

9.44

Vendor Demonstration 5 4.875

4.625

Customer Site Visits 15 12.875

11.125

Total Score 120 110.85 109.50

8. Cost Proposals

17

In our request for Proposal, Statewide HIO was seeking pricing information it could use to do side-by-

side comparisons amongst its vendor respondents. Statewide HIO covers a population of 10 million,

25,000 physicians, and 175 acute care hospitals.

Four vendors were reviewed – Axolotl, EPIC, Medicity, and Relay Health. As a result of our features and

functions review and scoring criteria, only Axolotl and Medicity remain appropriate HIE options for

Statewide HIO.

The table below outlines side-by-side cost comparison of IT costs over a 5-year period of time. Axolotl

took a more traditional approach to their cost proposal. They have included a one-time enterprise

license quote for the entire Statewide HIO service area, professional services to cover phase I and phase

II implementation activities, and annual ongoing costs for remote hosting services that include

hardware, software maintenance, storage, high availability & disaster recovery, 24x7x365 support,

network & telecom fees.

Medicity took a different approach for their proposal. Their quote includes one-time enterprise license

fees for software and implementation services spread over a 4-year period of time. Medicity is Software

as a Service (SaaS) provider. Their annual support costs begin in year 5 and those fees include mostly

software maintenance costs and a small fee for on going SaaS costs. Medicity is not a repository, so the

hardware, storage, and hosting requirements are a completely different model from the Axolotl

proposal.

Axolotl & Medicity Cost Proposals2

One- Time Costs: Year 1 Year 2 Year 3 Year 4 Year 5 Totals

Software License Fees:

-Axolotl $725,000 $0 $0 $0 $0 $725,000

-Medicity $400,000 $800,000 $1,200,000 $2,000,000 $0 $4,400,000

Implementation Fees:

-Axolotl3 $942,000 $942,000 $491,000 $2,375,000

-Medicity $0 $0 $0 $0 $0 $0

Total One-Time Costs:

-Axolotl $3,100,000

-Medicity $4,400,000

Annual Recurring Costs:

Hosting Fees: -Axolotl $400,000 $400,000 $400,000 $400,000 $400,000 $2,000,000

-hardware

-storage

-high availability & disaster recovery

-24x7x365 support

-network & telecom fees

-software maintenance

Subscription Fees: -Medicity $0 $0 $0 $0 $450,000 $450,000

includes SaaS fees

Total Annual Recurring Costs:

-Axolotl $2,000,000

-Medicity $450,000

5-Year Total Cost of Ownership:1

Axolotl $2,067,000 $1,342,000 $891,000 $400,000 $400,000 $5,100,000

Medicity $400,000 $800,000 $1,200,000 $2,000,000 $450,000 $4,850,000

1. TCO does not include HIO labor costs or other professional fees not provided by vendors.

2. These costs represent non-negotiated vendor quotes.

3. Axolotl will charge implementation fees as incurred by year.

Both vendor quotes are within original budget projection. Statewide HIO plans to negotiate aggressively with both vendors to reduce quotes by 20% Medicity has a more flexible payment proposal, less out-of –pocket in the early years. Axolotl and Medicity offered enterprise licensing (no additional licenses would need to be purchased in the future for

current state size projections. Axolotl & Medicity include software maintenance in its annual hosting fee. Overall, Axolotl matches our selection criteria most closely.

9. Acceptance Testing

19

The Statewide HIO Steering Committee Technical experts will evaluate vendor testing on the completed

system. The expected test plan will include verification/validation for the following:

System requirements System design Interoperability Integration and Adapter Data Standards Data Aggregation, De-Identification, Re-Identification and Anonymization Analytics and Reporting Privacy and Security authentication Disaster Recovery

A team of experts (identify the committee members) will develop and initiate user acceptance test case

scenarios.

Describe the role of acceptance testing

The Statewide HIO Steering Committee Technical experts will evaluate vendor testing on the completed

system. The expected test plan will include verification/validation for the following:

System requirements

System design

Functional requirements

Interoperability

Integration and Adapter

Data Standards

Data Aggregation, De-Identification, Re-Identification and Anonymization

Analytics and Reporting

Privacy and Security authentication

Disaster Recovery

A team of experts comprised of the Sub Committee members who are representative of stakeholders

throughout State X and the Steering Committee will have the responsibility of managing the user

acceptance testing. The subcommittee members include Thomas Brady, M.D., Maria D’Angelo, Privacy

Officer, Jeffrey Fox, CIO, Rajendra Kothari, Systems Analyst, Carol Manning, Clinical Nurse Specialist,

James Prince, M.D., Sara Rowe M.D., Randall Smith’s J.D., C.P.A. The team of experts will develop and

initiate user acceptance test case scenarios applicable for the technical requirements identified within

the RFP. The chosen panel of experts from the various disciplines ensures a balanced approach when

testing the system and verifies the vendor meets the agreement set forth by the Statewide HIO.

The role of acceptance testing play in the decision

The Statewide HIO is committed to providing a quality product for its users and the citizens of State X.

The Statewide HIO Steering Committee expects the vendor to follow a rigorous testing plan conforming

to ISO/IEC 29119 Software Testing Standard. The testing process provides a method to ensure

reliability, accountability, and quality assurance of the system.

http://softwaretestingstandard.org/part2.php

The vendor should demonstrate a test plan, which includes the elements listed below prior to any User

Acceptance Test Plan:

Unit Testing

21

Integration Testing

System Testing

System Integration Testing

Software Performance Testing

Software Load Testing

Stability Testing

Usability Testing

Security Testing

Destruction Testing (http://en.wikipedia.org/wiki/Software_testing)

Acceptance testing performed on the product

Subcommittee List of Objectives for the testing process

Provide well-defined, easily understood tests, which are applicable to the functionality of the

system

Enter relevant information in the system and guarantee the expected result of data entry

Validate that all test cases are documented

Document the problems identified and work with the vendor to resolve

Report necessary changes to the Steering Committee

Steering Committee List of Objectives for the testing process

Track and manage test problems

Communicate change recommendations to the vendor

When will acceptance testing be performed?

The user acceptance testing timeline is contingent upon the final contract agreement. The expert team

members have identified high-level objectives and the required tasks applicable to user acceptance

testing in the table below.

Objectives Tasks Associated with Objectives

Plan Acceptance Testing for State HIE Acceptance test schedule

Identify Testing Requirements Test matrix

Establish Test Environment Pilot site agreements

Conduct Acceptance Test Readiness

Review

Draft of test plan

Execute Tests Complete testing checklists

Complete Acceptance Testing Test summary report to Statewide HIO Steering Committee

Document Acceptance Testing Final acceptance test report to Statewide HIO Steering

Committee

Testing Case Scenarios – Please refer to Attachment (Excel Document)

Results Unacceptable during Acceptance Testing

There are several results, which would be unacceptable throughout the user acceptance-testing period,

and it is the expectation a fix occurs within five business days. It is imperative the user, role and patient

authentication is without incident due to federal and state government rules and regulations, which

address patient privacy and organizational policies pertaining to information security. The inability to

transmit what is considered a small amount of aggregated data from Hospital A to Hospital and data not

deidentified is not satisfactory. Lastly, prolonged periods of unplanned downtime is not suitable for the

timely exchange of patient information, which is a primary objective of the Statewide HIO.

10. Change Management

In the SOW, a change management process was established which was to submit a completed change

request form to the statewide HIO steering committee. The change request form contains a description

23

of the change, and a justification for the change. This form is reviewed by the steering committee which

will either approve or reject the change request. The same form presented in the SOW is used to

manage changes to the RFP.

During the review of the RFP, it became apparent that not all vendors were able to offer a complete

product as requested in the RFP. It is understood that the goal of the statewide HIO is to get an HIE up

and it is recommended that a step by step approach be taken so that the selection will allow for the

Statewide HIO to deal with the HIEs as they are. In the future, as analytics are developed, they can work

on developing centralized server at a later time.

The following change request form was submitted and approved by the statewide HIO steering

committee during the vendor evaluation process.

Statewide HIE Change Request Form

Sponsor Name Statewide RFP Vendor Change # RFP – 001

Project Name Statewide HIE Date Submitted 2/22/2011

Requested By Robert F. Proposal Vendor Date Reply Due 2/28/2011

Change Request Type: __New Requirement X Change Requirement __Remove Requirement

Description of Change:

None of the responding vendors were able to offer a complete product as requested in the RFP. To meet the objectives of getting the HIE

established in the desired timeline, this request is for approval to allow the adoption of a step by step approach in the implementation of the

Statewide HIO. The first step will be to begin with a Federated approach in order to stand up the HIE quickly. It will allow the HIO to meet basic

data exchanged for providers. This approach also allows the organization to select an HIE with their current offerings and to enhance their

systems over a defined time period of time to reach the ultimate goal of a hybrid model system.

Justification:

In reviewing the RFP responses, none of the HIE vendors offered a complete offering of a complete product as requested in the RFP. Also,

because of possible funding shortages, approval of this change will allow the HIE to stand up quickly, and meet basic data exchanges for

providers. It will also allow the HIO to spend less dollars upfront before moving to the long-term vision of a hybrid model.

Effect on Schedule, Deliverables, and Project Cost

No. Deliverable / Item Revised End Date Net Change, Hours

Increase or (Decrease)

Net Change, Cost

Increase or (Decrease)

This item affects the overall project

which cannot be outlined. Therefore,

this item is a TBD item to be revisited and completed after the final contract is

awarded and a workplan defined.

Totals

Revised Project End Date

Steering Committee Member Approval:

Signature : _________________________ Title: __________________________ Date: _________

Comments:

25

11. Discussion

While Axolotl and Medicity were very close finalists in this RFP process, Axolotl is recommended by the

HIO RFP Selection Committee for several reasons. First, the pricing structure of Axolotl whereby the

majority of costs are incurred during the beginning of the HIE implementation is more attractive for

Statewide HIO. Given that federal grants are only available to help offset startup costs of HIEs by means

of the HITECH Act, long term sustainability is a valid concern. The pricing model offered by Medicity

escalates costs each year during the 5 year total cost of ownership. If federal funds are no longer

available as health care reform evolves, the transition to HIE sustainability with Medicity could be more

challenging if the majority of the costs of the HIE implementation are still being paid.

Statewide HIO would like to rapidly implement basic functionality of the HIE utilizing a federated model.

As additional functionality is needed to meet stage 2 and 3 meaningful use requirements, transition to a

more hybrid model will be necessary to support features such as a consent repository. Axolotl offers the

tools needed for both a federated and a hybrid model with their base product. In contrast, adding

repository functionality with a hybrid model is not included in the base Medicity product, resulting in

additional costs in the future.

Comparing the experience of these two vendors, Axolotl has more customers live with their HIE

products. They began their clinical networking company in 1995. Axolotl also has experience with other

customers like Statewide HIOs powering both RHIOs and other statewide HIEs.

The references and site visits validated the comments documented in the latest KLAS report. The

customers we spoke with were very satisfied with Axolotl as a vendor. When problems did arise, Axolotl

promptly engaged with the customer in troubleshooting and resolution of the issue to the customer’s

satisfaction.

Two vendors were excluded during the RFP process. While Epic is a KLAS leader in the world of

electronic health records, it’s HIE product, Care Everywhere, is designed to connect Epic customers with

each other. There is a variation of Care Everywhere (formerly called Care Elsewhere) that allows a

similar connection between an Epic customer and other EHRs, but this product is contingent on use of

Epic’s EHR. Therefore, Epic was not considered as an option for the Statewide HIO due to the fact that it

is not a stand-alone HIE software solution and requires the use of Epic’s EHR.

Relay Health was also excluded from the list of HIE vendors during this RFP process. This HIE solution

currently only involves a unidirectional interface between hospitals and ambulatory physicians. While

their ability to connect physician offices with the patient’s personal health record makes them stand out

from a patient centric philosophy, their experience in acute to acute facility exchanges is lacking.

Additionally, Relay Health’s parent company, McKesson, recently reported layoffs of more than 400

employees, bringing their stability as an organization into question.

Attachments

Selection Criteria Final Evaluation.xls

User Acceptance Testing.xlsx

vendor selection form-2.xlsx

References

1. “Whats-on-tap-for-health-information-exchange-at-HIMSS-2011.htm”, SearchHealthIT.com, February 11, 2011.

2. KLAS. Health Information Exchanges. The Reality of HIE Adoption. February 2010. 3. http://chilmarkresearch.com/2010/06/30/where-is-the-consumer-in-hies/ 4. http://infosite.medicity.com/theHIEblog/

5. http://www.ehow.com/how_5224184_perform-user-acceptance-test-_uat_.html 6. http://www.healthcareitnews.com/news/axolotl-initiate-partnership-creates-market-

opportunities

7. www.medicity.com

8. www.relayhealth.com