Post on 22-Feb-2016
description
NCHRP 20-65 (47) - MTAP Survey Tool Used to Assess FTA Contractor Performance of State DOT Triennial and Other FTA Reviews - An UpdateBiennial FTA State Programs Meeting / State Public Transit Partnerships Conference August 7, 2103
KFH Group, Inc.Beth HambySue Knapp
Background•To assist FTA with improving State Review
processes, MTAP collects input from State DOTs on their experience following each FTA review via an online survey tool.
•Survey was created in Vovici (used by AASHTO for other surveys).
•Value of survey has been somewhat limited by: ▫open-ended answer formats▫challenges in using Vovici
Research Objective:•Update MTAP’s online survey to:
▫improve efficiency / ease of completion▫increase the value added nature of the
process (improve results for ease of use, benefits)
▫incorporate technology improvements▫provide for more detailed responses to 2-3
specific questions of interest to MTAP, SCOPT, FTA - not just a "comments" field
Project Tasks•Task 1 – Assess Survey Tool Improvement
Needs•Task 2 – Revise Survey Content & Structure•Task 3 – Assess Available Survey Technology
Options•Task 5 – Set Up & Test Revised Survey
Using Selected Technology•Task 6 – Prepare Final Report
Project Status
•Task 1 – Assess Survey Tool Improvement Needs – completed
▫Reviewed current survey tool - http://vovici.com/wsb.dll/s/54c9g50982
▫Reviewed reports generated from current tool
▫Gathered feedback/suggestions from States/MTAP Committee FTA Office of Oversight and Program
Guidance MTAP Coordinator
Project Status•Task 2 – Revise Survey Content &
Structure – nearing completion▫Modified existing questions –
standardized responses where possible to make analysis easier
▫Added new questions▫Prepared revised survey for review
by MTAP/ Steering Committee; sent late July
Draft Revised Survey• Numerous new questions seeking specific feedback
▫Basic experience of reviewer▫Consultant performance on specific compliance
topics▫For each scaled rating (Excellent/ Good/ Fair/ Poor),
asks why (open-ended)▫Extent of Regional participation▫Effectiveness of entrance/exit meetings▫SMR workshop effectiveness▫Interest in follow-up discussion with MTAP or FTA
• Standardized responses; limited open-ended questions to explanation and “other”
• Reworded some questions to align with responses
Draft Revised Survey1. Contact and Background Info
▫ Added: Basic experience of respondent2. Consultant Performance
▫ Added: Fields for explanation3. General Experience with Review
▫ Added: Regional office participation▫ Added: Entrance/Exit meetings▫ Added: Overall organization and clarity
4. SMR-Specific Feedback▫ Added: Respondent’s participation▫ Workshop/workbook effectiveness▫ Additional training/tech assistance needs
5. Additional Feedback
Project Status•Task 3 – Assess Available Survey
Technology Options – under way ▫Criteria include:
User-friendliness to create & administer survey
User-friendliness to complete & submit surveys
Functionality/customization options Reporting capabilities Cost
Survey ApproachesGeneral options for getting feedback from
states:
▫Online survey tools – basic to advanced Focus of our assessment
▫Pen & paper Too labor intensive
▫Telephone interviews May be appropriate for follow-up
Basic Online Survey Tools•Examples include:
▫SurveyMonkey Basic / Select / Gold ▫SurveyMoz Free / Premium▫SurveyGizmo Basic▫QuestionPro Professional
Basic Online Survey Tools•Advantages:
▫Easy to learn and use▫Provides some skip logic functionality▫Provides some basic cross-tab analysis▫Can export results to Excel for additional
analysis▫Inexpensive ($0 - $25/month)
Basic Online Survey Tools• Disadvantages:
▫Limited collaboration capabilities▫Limited survey customization options▫Limited reporting / analysis functions – need to
export to third party application
• Basic tools are becoming increasingly sophisticated (e.g., SurveyMoz)
• Some offer higher levels of customization for a higher cost (e.g., SurveyMonkey Platinum - $65/month)
Advanced Online Survey Tools•What MTAP is currently using (Vovici)
•Known as Enterprise Feedback Management (EFM) Systems
•Examples include:▫Key Survey▫Qualtrics▫QuestionPro Corporate▫SurveyGizmo Enterprise▫Vovici
Advanced Online Survey Tools•Advantages:
▫Allows for greater collaboration in survey administration (behind the scenes) Multiple user accounts possible Can share surveys with other survey
administrators Private results through secure dashboard accounts
▫Greater flexibility in branding and customizing ▫Can build sophisticated logic into survey▫Allows for complex analysis▫Custom reports can be developed within the tool
Advanced Online Survey Tools•Disadvantages:
▫Complex interface▫Requires training / steeper learning curve▫May require frequent use to maintain skills▫May require experience in Cascading Style
Sheets (CSS) to customize look and feel of survey
▫Expensive ($100 or more/month – few disclose pricing online) However, no out-of-pocket cost to MTAP for
Vovici; AASHTO already pays for it for other uses
Next Steps•Fine-tune survey questions with input
from MTAP/ Committee•Finish technology assessment•Set up & test revised survey using
selected technology•Prepare final report
For more information or feedbackBeth HambyKFH Groupbhamby@kfhgroup.com206-448-6749