Tender Evaluation Process Notes

Tender Evaluation Process Notes

May 2009


of 18

Contents

1. INTRODUCTION, PURPOSE AND SCOPE...........................................................3

2. TENDER RESPONSE EVALUATION AND SELECTION..........................................4

2.1 OVERVIEW ...................................................................................................... 4 2.2 EVALUATION PROCESS ........................................................................................ 4 2.3 EVALUATION TEAM .......................................................................................... 13 2.4 EVALUATION CRITERIA...................................................................................... 16 2.5 EVALUATION AND SCORING PROCEDURE ................................................................. 17 2.6 COMMUNICATION WITH RESPONDENTS .................................................................. 18


of 18

1. Introduction, Purpose and Scope

This document contains some notes on a possible approach to tender evaluation.


of 18

2. Tender Response Evaluation and Selection

2.1 Overview

The objectives of the evaluation process are twofold:

• To select the most appropriate proposal to meet requirements

• To create an evaluation audit trail and set of supporting documentation that

will support the evaluation process and final decision.

The output from the process will be a Vendor and Solution Evaluation and

Selection (VSES) report.

The document describes the evaluation criteria that were applied to responses to

the tender.

Any evaluation process is liable to subjectivity. The evaluation process described

below can be subjective in:

• The selection of the evaluation factors and the relative weights assigned

• The scores assigned to each vendor

The purposes of a formal evaluation process are:

• To eliminate subjectivity as much as possible from the evaluation process

• To provide transparency in the evaluation process

• To ensure that all proposals are evaluated in the same way

2.2 Evaluation Process

The generalised solution evaluation process can be customised to meet the exact

needs.

The VSES report will have the following general structure:

Part A – Overview

1. Executive Summary

Part B – Short List Selection

2. List of Vendors Contacted

3. Product Decision Model

4. RFP Findings

5. Summary of Short List Recommendation

Part C – Finalist Selection

6. Reference Checks and Site Visits

7. Functional Demonstration Findings


of 18

8. Technical Review Findings

9. Solution Demonstration Planning Lab Findings

10. Summary of Finalist Recommendation

Part D – Appendices Containing Supporting Material (optional)

Schematically, the high-level view of the evaluation process that is proposed to

be used during the evaluation and selection of a reporting solution is:


of 18

In summary the evaluation process consists of the following elements:


of 18

• Evaluation Team - the evaluation team consists of individuals with

appropriate skills to be able to evaluate responses received from vendors

• Evaluation Process - the evaluation process is the set of steps to be

followed by the evaluation team to conduct an evaluation of the responses

• Evaluation Criteria - this is a set of criteria by which the responses will be

assessed

• Evaluation Score - the score is an objective assessment of the merits of a

response based on the application of the evaluation criteria

• Evaluation Report - this is a document describing the evaluation process

and its results

In more detail, the steps in the process are:

Step Description

Start VSES

Process

The amount of effort involved in starting the Vendor and Solution

Evaluation and Selection (VSES) step depends on:

• How recently you made any VSES scope and direction decisions

• Whether the team has already been selected

• How familiar each team member is with the functional area

within which the application is to be deployed, and the systems

currently in use in BoI to support Monthly Financial Performance

Reporting

• How experienced the team is with vendor and solution

evaluation and selection.

Confirm Scope

and Approach

Review previous decisions made about the scope of the project.

Discuss any scope changes and adjust the work plan to reflect

approved changes.

Consider the urgency of need, and determine how it impacts the

selection strategy.

Determine the timing and general strategy for contract negotiation.

Determine what parties to involve in defining procurement

procedures, handling legal questions, and negotiating the contract

so that the right people can be involved at appropriate points.

Establish and

Train the

Evaluation

Team

Assemble and organise a solution evaluation and selection team.

Observe the following guidelines:

• Include technical personnel who understand the systems

management and support processes that are being implemented

• Include business personnel who understand the operational

objectives, business processes and needs of Executive


of 18

Step Description

stakeholders and sponsors of the project.

• Include subject matter experts in Best Business practice in

Business Performance reporting as needed.

• Include experts who understand the market context within

which the vendors operate so as to ensure an understanding of:

o The capability and capacity of service providers on the

local market where resources will be required

o The extent to which the vendors solutions will need to

be customised to meet the requirements identified in

the RFP

o The market for services based activities

(consulting/training/application development etc)

• Provide for a workshop facilitator with a solid background in the

subject area if you plan to use facilitated workshops as part of

the evaluation and selection process.

• Keep the core group small but representative of the functional

areas the product most affects.

• Include as support members anyone involved in defining

procurement procedures, handling legal questions, and

negotiating the contract.

• Include management personnel responsible for approving the

selection

Once you have established the evaluation and selection team,

designate team members to be the primary and secondary vendor

contacts. The secondary vendor contact works with the vendor

when the primary contact is unavailable, such as during vacation or

illness. These individuals represent the interests of the client and

the systems integrator, answer questions about the solution

evaluation and selection process, strategise with the vendor,

arrange vendor contact with other members of the team, and are

empowered to make informal commitments.

Next, designate a team librarian to administer the vendor

scheduling, mail all vendor requests, record all vendor contact,

obtain and distribute vendor materials, and maintain a filing system

for vendor materials.

Walk through the evaluation and selection activities with team

members to familiarise them with the general process. If the team

is relatively inexperienced with solution evaluation and selection,

consider using a team-building exercise to preview the process.

Scale down and simulate evaluation and selection activities. Save

detailed discussions for just-in-time training sessions.

Discuss the objectives and principles of the evaluation and selection

process with the team to ensure that everyone has a consistent

understanding.


of 18

Step Description

Compile

Function

al

Criteria

Compiling criteria when selecting a solution involves:

• Developing or gathering criteria appropriate to the evaluation

method

• Avoiding excess precision

• Keeping the focus on business priorities

• Designing a score sheet that facilitates getting and analysing

vendor responses

• Incorporating the criteria into the score sheet.

These criteria will be taken from the tender.

In this activity, define the functional criteria - the requirements as

seen from the business user’s point of view. These requirements

represent the business processes the solution needs to

accommodate. For more information on developing application

selection criteria and designing a score sheet, see the Product

Evaluation technique.

Design functional criteria and group them as follows:

• Business processes

• Data

• External system interfaces

• Business performance measurement.

Assemble the functional criteria into one document, the Functional

Criteria List.

Compile

Technical

Criteria

Define technical criteria to represent the solution design and

technical infrastructure aspects of the solution. The technical

requirements concern the solutions operation and infrastructure.

To enhance readability and minimise oversights, develop categories

of technical criteria in the areas of:

• Systems management and support process

• Operational interface

• Technical infrastructure

• Volume and technical performance.

Assemble the technical criteria into one document, the Technical

Criteria List.

These criteria will be taken from the tender.

Compile

General

Criteria

The buyer of a reporting solution is buying more than a service. In

most cases he or she is also developing a long-term relationship

with the vendor, expecting some, if not all, of the following:


of 18

Step Description

• Initial transition, configuration and implementation services

• Custom services

• Consulting services

• Future changes

• Customer support services, such as a hot line and a user group.

General criteria include “vendor” and miscellaneous criteria. These

criteria are factors that influence the solution purchase decision but

are not functional or technical in nature. General criteria require

that you answer them through an iterative process, which requires

different tactics:

Concentrate on criteria that would eliminate vendors from the short

list. Consult with legal and purchasing personnel to extract general

criteria that can quickly narrow the solution selection.

Add criteria to ensure the team understands the vendor’s decision-

making and delivery processes, schedules, and organisation before

making a final solution selection and developing an implementation

plan.

With functional and technical criteria, you are usually looking for a

specific response. With general criteria questions, you are looking

for potential risks and information needed for deployment. Collect

responses to these criteria by asking the vendor directly, by asking

references who use the software, and by doing your own research.

Develop categories to enhance readability and minimise oversights.

Assemble the categories of general criteria into one document, the

General Criteria List.

These criteria will be taken from the tender as well as other more

specific requirements that extend the material in the tender.

Determine

Scoring

Approach

To narrow solutions options to a reasonable number, design an

overall package selection framework to communicate and

summarise findings. Use weighting to reflect the relative

importance of criteria; use scoring to indicate how well the vendor’s

solution meets the criteria.

Weighting and scoring are often represented numerically, giving the

illusion of objectivity. Keep in mind that these numbers are really

just a quick means to communicate subjective judgments. When

making the package decision, check that the bottom-line numbers

are consistent with your overall judgment.

Conduct RFP

Response

Evaluation

The objective of this activity is to obtain and evaluate

comprehensive vendor proposals and presentations that explain

how their products meet the specific criteria, and at what estimated

cost. The RFP has information instructing vendors to respond

formally, in writing, and possibly in a presentation. Vendors are


of 18

Step Description

compared uniformly and strictly as to how well they meet the

criteria, comply with the instructions, and present their solution.

The responses to the RFP will be evaluated with respect to the

criteria developed above.

Develop Script

Cases

During the solution meetings with vendors, the team needs to

compare solutions fairly and evenly, even though different solutions

are being proposed and discussed. This is hard for the team to do if

each vendor is allowed to determine the meeting’s sequence,

contents, and emphasis. By prescribing that all vendors follow a

planned sequence of requirements, or scripts, you:

• Ensure consistency of content coverage across vendors

• Ensure the vendor is presenting those features most important

to you, not just those the vendor puts forward as strong points

• Can more easily spot features and functions that do not quite

match expectations

• Can more easily distinguish how the features and functions

differ among solutions.

Select Vendor

Short List

Conduct a workshop or a work session to obtain team consensus on

whether or not to proceed with solution evaluation and to reduce

the list of vendors under consideration to three or fewer. In

general, use a workshop for large solutions or solutions with a

major impact such as the service being requested.

Check Vendor

References

Check vendor references as another way to confirm that the

vendors’ solution and services could meet the requirements.

Develop a questionnaire to record reference responses. Include

questions covering relevant functional, technical, and general

criteria. Involve members of the team in reference checking. This

distributes the work, counteracts individual bias, and provides

experience useful in subsequent VSES activities.

Conduct

Functional and

Operational

Review

Meetings

Have the short list vendors conduct functional and operation walk-

throughs using the script cases provided, so you can:

• Confirm that the solution, can process operate to your

satisfaction

• Determine, based on your own observations, which solution

provides the best answer to needs

Perform

Technical

Review

Conduct a technical review of the solution with the vendor’s

technical staff. The vendor’s technical staff members are usually

involved in the provision of the solution and its underlying technical

components and are not as oriented as the sales staff toward

selling the solution. They usually paint a more realistic picture of

the solution and the vendor’s ability to support it. They can also

confirm technical infrastructure details about operating the solution.

The technical review typically has several components:


of 18

Step Description

• Demonstration of technical facilities and elements

• Review of remaining questions on technical and general criteria

responses

• Support and operational model comparison

• Performance investigation

• Review of solution architecture

• Conceptual technical infrastructure design

• Review of services delivery and quality

• Software, support and Operations costing

• Review the software release and patching strategy the vendor

employs

The importance and, hence, the amount of effort spent preparing

for and conducting the various components of the technical review

vary based on several factors.

During the technical review, each team member updates the

relevant criteria scores on a copy of the current Technical and

General Criteria Lists. The team members take notes justifying the

scores and identify other risk areas, strengths, and weaknesses. At

the end of the technical review, the team meets to reach consensus

on the technical review findings.

Visit

Reference

Sites

Decide whether to visit vendor reference sites. Plan the approach,

visit the sites, then consolidate the findings.

Perform

Financial

Analysis

This activity is a critical juncture for confirming the vendor’s

financial estimates. Help make sure the vendor’s cost estimates are

as correct as possible by confirming the preliminary cost estimates.

Select Finalist The team is now prepared to select a finalist. Review the decision

to confirm that the reporting approach should still be pursued.

Determine contract negotiation timing and approach. Conduct a

workshop or a work session to select the solution finalist. In

general, use a workshop for a large solution such as that being

requested.

Complete

Evaluation

Complete the evaluation process.

Complete the project records.

Finalise the evaluation report.

Notify the losing vendors. Prepare for any vendor loss meetings.

Conduct any vendor loss meetings

Assess satisfaction with the evaluation project.

Summarise the project results and the lessons learned.


of 18

Step Description

Review and recognise the team performance.

Close out the project records

This is a high-level view of the evaluation process that will be used.

The following additional collateral will be created and maintained during the

evaluation and selection process

• Evaluation Matrix. Create an Evaluation Matrix to record evaluations of

vendor responses to an RFS or RFP.

• Solution Decision Model. Create a Solution Decision Model reflecting how

you measure and evaluate the different evaluation components, such as RFP

responses, demonstrations, and site visits.

• Vendor Demonstration Support Materials. Create a Script Case Plan,

Process Scenario Worksheet, Script Case Summary Score sheets, and EBP

Summary Score sheets to plan vendor demonstrations, record observations,

and summarise team evaluations of the demonstrations.

• Solution Comparison Matrix. Update the Solution Comparison Matrix to

keep track of significant vendor and solution evaluation factors.

• Vendor Reference Questionnaire. Create a Vendor Reference

Questionnaire to record questions and answers for checking a vendor’s

references.

• Negotiated Vendor Agreements. Support the creation of Negotiated

Vendor Agreements. Create the agreements to bind the vendor and company

in a clearly defined and enforceable way regarding a solution the vendor will

supply.

• Strengths and Weaknesses List. Create Strengths and Weaknesses Lists

to record individual and team assessments of vendor and solution strengths

and weaknesses.

2.3 Evaluation Team

The evaluation team may be structured as follows:


of 18

The team will be lead by a team manager.

The team may be divided into a number of groups:

• Functional - this group will be responsible for evaluating the functional

aspects of the responses

• Technical - this group will be responsible for evaluating the technical aspects

of the responses

• General - this group will be responsible for evaluating general factors such as

the responding companies and the finances

No members of the evaluation team will be connected with any of the

respondents.

The team manager may be assisted by a number of specialists in particular

aspects of the development and operation of a reporting system.

The responsibilities of the Evaluation Team Manager will be:

• Managing the evaluation process


of 18

• Consulting with specialists as required

• Documenting the evaluation process

• Assign weights to the evaluation factors in conjunction with the Group Leaders

• Conducting meetings of the evaluation group leaders and of the whole team

• Preparing the final report on the evaluation

• Reviewing and understanding responses from respondents

• Acting as the single point of contact between respondents and other team

members

• Allocating and scheduling work for the evaluation groups

• Reviewing the outputs from the evaluation groups and seeking clarification

The responsibilities of the evaluation group leaders will be:

• Conducting meetings of the evaluation group

• Preparing evaluation task lists and other material to assist evaluation


• Co-ordinating any questions to and responses from respondents from group

members

• Documenting the work of the group

• Reporting to the team manager as required

• Allocating and scheduling work between the members of the group

• Reviewing and discussing the work of the group as it takes place

• Summarising the results of the evaluation group

The responsibilities of the members of the individual group members will be:


• Understanding the evaluation process

• Documenting their work

• Performing their assigned work in a timely and efficient manner


of 18

• Seeking clarification where required

• Responding to queries and requests for updates from the group leader and

team manager

2.4 Evaluation Criteria

The evaluation criteria will be taken from the RFP. They will be grouped. Each

criterion will be assigned a weight to reflect its relative importance in the required

solution.

The following table contains an illustrative list of evaluation criteria:

Category Factor Weight % Total

Category

Weight %

Response

Document

Response document 3.00% 3.00%

Vendor Respondent stability 4.00%

Respondent references 4.00%

Respondent knowledge and

experience

4.00%

Respondent personnel 3.00%

Respondent certifications 1.00%

Project delivery ability 4.00% 20.00%

Methodology Methodology 2.00%

Risk Management 2.00%

Standards 1.00% 5.00%

System Design System design 3.00%

System performance and

throughput

2.00%

System usability 3.00%

Growth and expansion 2.00% 10.00%

Implementation Timescale 4.00%

Team 3.00% 7.00%

Software Tools Development language(s) 2.00%

Database management system 2.00%

Other software components 2.00% 6.00%

Security Connection to End User Company

user directory

2.00%

Flexibilility 2.00% 4.00%

Hardware Server infrastructure 1.50%

Client infrastructure 1.50%

Network infrastructure 1.00% 4.00%

System Operation User reporting 4.00%


of 18

User analysis 4.00%

System administration and

management

3.00%

Office interface 2.00%

Mail interface 2.00% 15.00%

Financial System development 3.00%

Software licenses 3.00%

Support and maintenance 2.00%

Hardware costs 1.00%

Upgrade costs 1.00%

Overall costs 10.00% 20.00%

Future Growth in data volumes 2.00%

Growth in users 2.00%

Additional reports 2.00% 6.00%

TOTAL 100.00% 100.00%

2.5 Evaluation and Scoring Procedure

The evaluation process involves the following steps:

• Weighting or classifying criteria based on importance to the overall project

• Scoring vendors ability to match criteria

• Multiplying score by the weight to derive final notional score

Importa

nce

Weightin

g

(1-5)

Total

Possibl

e

Score

Responde

nt 1

Score

(Unweigh

ted)

.. Responde

nt M

Score

(Unweigh

ted)

Respond

ent 1

Score

(Weighte

d)

.. Respond

ent M

Score

(Weighte

d)

Category 1

Factor 1

...

Factor N

...

Category P

Factor 1

...

Factor N

TOTAL

Deriving a scoring for the cost of the response could involve some formal

comparison of all the costs.


of 18

Respondents could be either assigned a simple score based on their rating in a

table of scores. However, this does not take into account large variations in

costs.

The cheapest response could be assigned the maximum number of points.

The differences between each response and the lowest response could be

calculated. The amount of the total difference each respondent is responsible for

can be used to calculate the number of points to be assigned to the vendor.

In the following example, the cheapest response is 75 and the largest in 100. The

total of all the differences from the cheapest is 110. So a respondent who

proposed a cost of 100 is responsible for nearly 24% of the total variation. The

respondent's score is the maximum score reduced by this percentage.

Cost Difference From

Lowest

Percentage of

Total Variation

Points Assigned

75 0 0.00% 100

80 5 4.55% 95

85 10 9.09% 91

85 10 9.09% 91

90 15 13.64% 86

95 20 18.18% 82

100 25 22.73% 77

100 25 22.73% 77

110

2.6 Communication With Respondents

During the evaluation process, it may be necessary to communicate with

respondents.

All communication will be in writing, by letter, FAX or e-mail. Responses from

respondents must be in writing. These responses will be made available to

members of the Evaluation Team. The communications will take the form of a list

of issues requiring clarification. The communications will not disclose any

information on the perceived strength or weakness of the respondent’s proposal.

All respondent communications will be via a single member of the Evaluation

Team.

Tender Evaluation Process Notes

Business

Transcript of Tender Evaluation Process Notes